Sourcelibrary V2

Source Library

A digital library of historical primary sources with AI-aided OCR, translation, and scholarly curation.

🌐 Visit Library • 📖 Docs • 🗺️ System Map • 🤝 Contribute

🎬 Experience Source Library

Explore thousands of digitized historical texts with AI-enhanced translations and scholarly curation.

🎯 About Source Library

Source Library is an open digital library dedicated to making early printed books and primary sources readable and citable. We specialize in alchemy, Hermetica, Kabbalah, Rosicrucianism, and early modern science—texts that bridge historical scholarship with contemporary exploration.

🌟 Why Source Library?

• ✨ Originals First — Read the original language text with AI-enhanced translations alongside
• 🎓 Citable Scholarship — Every book gets a DOI and scholarly metadata (USTC alignment, edition tracking)
• 🏛️ Partner Subdomains — Institutions like the Bibliotheca Philosophica Hermetica curate reading rooms on their own domains
• 🔍 Discovery — Collections, galleries of illustrations, and semantic search surface overlooked texts
• ✅ Rigorous QA — Manual verification, image quality scoring, and OCR validation before publication

The platform ingests ~15K pages monthly from Internet Archive, Gallica, Bodleian, Wellcome, and other digital heritage partners.

🚀 Quick Start

💻 Development Environment

# 🍴 Clone and install
git clone https://github.com/Embassy-of-the-Free-Mind/sourcelibrary-v2.git
cd sourcelibrary-v2
npm install

# ⚙️ Configure environment (see .env.example for required variables)
# Must include: MongoDB Atlas connection, Google Gemini API key, Vercel Blob token

# ▶️ Start dev server
npm run dev

Open http://localhost:3000

📋 Tech Stack

📦 Key Dependencies:

• sharp — Image resizing and cropping
• @google/generative-ai — Gemini API integration
• xml2js — USTC metadata parsing
• @modelcontextprotocol/sdk — MCP server for agent integration
• stripe — Donation and subscription handling

📚 Core Features

📖 Reading & Navigation

• ⚡ Page pagination — Instant navigation between 100+ pages
• 🔍 Full-text search — Query across OCR'd text and translations
• 📌 Quote generation — Copy and cite passages with DOI links

🔄 Processing Pipeline

• 🧩 Smart split detection — Automatic gutter detection for two-page spreads (Gemini AI or ML-based)
• ✍️ High-accuracy OCR — Gemini Vision API with language-specific models (Latin, German, Greek, Arabic, etc.)
• 🗣️ Context-aware translation — Maintains continuity across pages for scholarly accuracy
• 🖼️ Gallery extraction — AI-powered detection and cataloging of illustrations

🎨 Curation & Discovery

• 📑 Themed collections — Editorial collections (e.g., "Alchemy & Transmutation," "Kabbalah & Mysticism")
• 🏛️ Gallery browsing — Curated images from all books (museum-quality metadata)
• 📚 Related editions — Link across translations, reprints, and derivative works
• 🔗 Authority linking — Connect to USTC, VIAF, and other scholarly databases

📤 Scholarly Export

• 📱 EPUB generation — Multi-format ebook export
• 📄 PDF with annotations — Preserve layout, add scholarly notes
• 🆔 DOI minting — Version books via Zenodo integration for long-term citation

🏢 Tenant Subdomains

• 🏛️ Isolated reading rooms — Partners host curated subsets on custom domains (e.g., bph.sourcelibrary.org)
• 🎨 Branding & navigation — Full UI customization per tenant
• 🔒 Access control — Public or members-only collections

🏗️ Architecture Overview

📊 Data Model

📚 Books contain structured metadata:

• 🏛️ Bibliographic — Title, author, language, publication date, USTC ID
• 🖼️ Images — Links to source (Internet Archive, Gallica, etc.), archival status
• ⚙️ Processing — OCR status, translation language, extraction metadata
• 🎨 Curation — Collections, tier (featured/standard), visibility flags

📄 Pages store individual page data:

• 📸 Original image — Source photo or PDF page
• ✂️ Split coordinates — Crop boundaries (0-1000 scale) for two-page spreads
• ✍️ OCR output — Raw Gemini extraction + language metadata
• 🗣️ Translation — English translation with scholarly notes
• 🖼️ Illustrations — Detected images with quality scores and descriptions

🖼️ Gallery images are extracted illustrations:

• 🏷️ Metadata — Subject, figures, symbols, style, techniques, period
• ⭐ Quality score — 0–1.0 rating (filters below 0.5)
• 🔗 Provenance — Source book and page, linked back

🎨 Image Tier System

All page images are resized on-demand via /api/image:

Split pages are cropped non-destructively via coordinates; original images are always preserved.

🔄 Processing Pipeline

📥 Import → ✂️ Split Detection → ✍️ OCR → 🗣️ Translation → 🎨 Enrichment → 🌍 Publishing

1. 📥 Import — Upload images, import from IA/Gallica via IIIF, or paste URLs
2. ✂️ Split Detection — Detect two-page spreads; mark crop boundaries
3. ✍️ OCR — Gemini Vision extracts text per page language
4. 🗣️ Translation — Gemini translates to English with prior-page context for continuity
5. 🎨 Enrichment — Extract illustrations, generate summaries, assign collections
6. 🌍 Publishing — Set visible: true, mint DOI, push to search index

Batch endpoints process up to 5 pages/request using Gemini Batch API (50% cheaper).

🔌 API Routes (Key)

Base URL: https://sourcelibrary.org (production) or http://localhost:3000 (local dev with .env.local configured).

Common 404 mistake: paths like /api/bph/books or /api/bph/books/[id] do not exist. BPH catalogue APIs live under /api/embed/bph/.... There is also no top-level /api/[tenant]/books route — tenant book listings use /api/books/library or the embed routes below.

Public read APIs (no auth required today)

Tenant-scoped listing (not /api/bph/...)

Use one of these patterns to filter by partner tenant (e.g. BPH):

The /api/[tenant]/books/[id]/... paths that exist in the codebase are editor/processing routes (batch OCR, index rebuild, etc.) — not public catalogue listings.

Authenticated / internal APIs

These require a signed-in session cookie, editor role, or (for some dataset endpoints) a Bearer API key. Calling them without auth returns 401 or 403.

Full narrative API walkthrough: docs/blog-source-library-api.md. MCP tools (search, quote, read): mcp-server/README.md.

🧪 Trying the API (curl, Postman, browser)

All examples below hit production and need no API key. Replace the base URL with http://localhost:3000 when running locally (MongoDB + env vars required).

curl

# Search translated text
curl -s "https://sourcelibrary.org/api/search?q=quintessence&limit=5" | jq .

# List books (global catalogue)
curl -s "https://sourcelibrary.org/api/books?limit=5" | jq .

# Browse with filters and sort
curl -s "https://sourcelibrary.org/api/books/library?limit=5&sort=recent-translation&has_translation=true" | jq .

# BPH catalogue — note /api/embed/bph/, NOT /api/bph/
curl -s "https://sourcelibrary.org/api/embed/bph/books?limit=5&translated=true" | jq .

# BPH via 

…