# BoM Research Pipeline — Four Facets > **How to go from "I want to research X" to a fully deployed, image-backed primary source page on the VPS.** This document covers the complete lifecycle: **Identification → Acquisition → Organization → Display**. Each facet has its own tools, checklists, and pitfalls. The pipeline is designed for slow, deliberate, verifiable research — not bulk scraping. --- ## Facet 1: Identification **Goal:** Discover primary sources relevant to a research question. Know where to look, what search terms to use, and how to track what you've already searched. ### 1A. Starting Points | Resource | What it has | How to search | |----------|-------------|---------------| | **Mormonr.org** | Curated Q&A pages with 200-300 primary sources each, full transcriptions, 35-tag hearsay taxonomy | Navigate `mormonr.org` → "HARD QUESTIONS" → find topic → append `/research` to URL | | **Joseph Smith Papers** | All extant JS documents, manuscripts, revelations | `josephsmithpapers.org` — search by date range, person, or document type | | **BYU ContentDM** | 49+ collections including BoM papers, early Mormon publications | `contentdm.lib.byu.edu` — browse collections or search | | **Internet Archive** | Out-of-copyright books, periodicals, newspapers | `archive.org/details/` — search by title, author, keyword | | **Chronicling America** | Free digitized US newspapers 1770-1963 | `chroniclingamerica.loc.gov` — search by keyword, date, state | | **Church History Catalog** | LDS Church archives, manuscripts, photographs | `catalog.churchofjesuschrist.org` — search by topic, person, date | | **Newspapers.com** | Subscription newspaper archive | Requires owner credentials | | **GenealogyBank** | Subscription newspaper + obituary archive | Requires owner credentials | ### 1B. Search Strategy Checklist For any new research topic, execute these searches in order: 1. **Mormonr first** — check if there's already a curated page. Browse "Hard Questions" listing. Record the 5-char page ID. 2. **JSP by person** — search for each key adjacent person (see `adjacent-persons.md`) 3. **JSP by date** — search for all documents in the relevant date range 4. **BYU ContentDM** — search by topic keyword + person names 5. **Internet Archive** — search for out-of-copyright books mentioning key figures 6. **Newspapers** — Chronicling America (free) first, then Newspapers.com/GenealogyBank if credentials available 7. **Citation chain** — for each found source, mine its footnotes/ bibliography for additional leads 8. **Record negative searches** — log databases searched with zero results to prevent duplicate work ### 1C. Discovery Log Every search session gets logged. Create `data//discovery_log.md`: ```markdown # Discovery Log — | Date | Database | Search Terms | Results | Follow-ups | |------|----------|-------------|---------|------------| | 2026-06-18 | Mormonr | BoM witnesses | Found page 0Eiiyt (254 sources) | Extract relevant subset | | 2026-06-18 | JSP | "Martin Harris" 1828-1830 | 3 relevant documents | Download images | | 2026-06-18 | Chronicling America | "gold bible" 1829-1830 | 0 results | — | ``` ### 1D. Lead Tracking When a source cites another source you haven't retrieved yet, log it as a **lead** — do NOT attempt retrieval mid-session unless instructed. ```json { "lead_id": "lead-001", "source": "Vogel, Early Mormon Documents, vol. 2, p. 145", "cited_source": "Palmyra Reflector, March 19, 1831", "relevance": "First mention of 'spectacles' in a non-Mormon newspaper", "repository": "Possibly NYS Historic Newspapers or GenealogyBank", "status": "not retrieved" } ``` --- ## Facet 2: Acquisition **Goal:** Get the source data — metadata, transcription, and images — into the local data store. ### 2A. Mormonr Bulk Extraction When a Mormonr Q&A page exists for your topic: 1. **Navigate** to `https://mormonr.org/qnas///research` 2. **Expand all** — click the "EXPAND ALL" button 3. **Extract with browser_console** — see `references/mormonr-extraction-technique.md` for the JS snippet 4. **Parse with Python** — split `detailText` on field labels, clean category suffixes 5. **Filter aggressively** — keep only directly relevant sources. 238 sources → keep maybe 30-50. Exclude late secondary, generic commentary, tangential biblical precedents. 6. **Batch import** — `python3 scripts/add_source.py --data .json --batch filtered_sources.json` 7. **Classify images** — `python3 scripts/seed_images.py --data .json` 8. **Download images** — `python3 scripts/download_images.py --data .json --limit 5 --pages 1` (start small, verify) ### 2B. Manual Source Addition For sources discovered individually (not from Mormonr): ```bash # Interactive — prompts for each field python3 scripts/add_source.py --data .json # From a prepared JSON file python3 scripts/add_source.py --data .json --file source.json ``` ### 2C. Image Acquisition ```bash # Download first page for all undownloaded IA sources python3 scripts/download_images.py --data .json --type internet_archive --pages 1 # Download a specific source python3 scripts/download_images.py --data .json --source src-014 # Download all pages of a specific source python3 scripts/download_images.py --data .json --source src-009 --pages 10 # Check what's downloaded python3 scripts/download_images.py --data .json --list ``` **Domain-specific notes:** - **Internet Archive** (38 sources) — `/page/n{num}` redirects to JPEG. ✅ Reliable. - **Church Catalog** (29 sources) — Direct asset URL download. ✅ Reliable. - **contentDM** (49 sources) — IIIF endpoint at `/digital/iiif-info/{collection}/{id}`. May need manual intervention. - **JSP** (29 sources) — Images at `cdn.churchofjesuschrist.org/ch/jsp/image/`. May need browser-based extraction. - **Newspapers** (13 sources) — Varies by site. Chronicling America has direct JPEG URLs. - **PDF links** (11 sources) — Downloads the PDF. `pdf_to_images.py` (TODO) will extract page images. - **Google Books** (5 sources) — Behind auth. Manual download needed. - **Web archives** (19 sources) — Snapshots of other pages. Follow redirect to original. - **Scripture references** (10 sources) — No manuscript images. ### 2D. Citation Standards Every source must have a complete Chicago Manual of Style citation: - **Manuscript:** Creator, "Title," date, collection, box/folder, repository, location - **Published primary:** Composition date separate from edition publication date - **Newspaper:** "Article," *Paper Name* (City, State), date, page - **Court:** Jurisdiction, court, case name, date, file/case number, repository - **Digital:** Print citation + URL/access date; prefer DOIs and permalinks --- ## Facet 3: Organization **Goal:** Ensure data quality — no duplicates, no dead links, consistent classification, complete metadata. ### 3A. Data Schema Every source in a `data/.json` file: ```json { "id": "src-001", "date": "Aug 11, 1829", "date_sort": 1829.61, "date_source": "Aug 11, 1829", "date_described": "Aug 11, 1829", "date_sort_source": 1829.61, "summary": "One-line description", "source": "Person or entity", "hearsay": "Direct Journalism", "type": "Newspaper", "reference": "Full Chicago citation", "source_link": "https://archive.org/...", "scribe_publisher": "Who recorded/published it", "people": ["Person A", "Person B"], "audience": "Reading Public", "transcription": "Full text of relevant passage", "notes": "Researcher context notes", "tags": ["first-mention", "spectacles"], "images": { "downloaded": true, "source_type": "internet_archive", "local_dir": "src-001", "pages": ["src-001/page_001.jpg"], "page_count": 1, "thumbnail": "src-001/page_001.jpg", "notes": "Bookreader with page images" } } ``` ### 3B. Classification Rules **Hearsay** — use the 35-tag Mormonr taxonomy: | Category | Example Tags | |----------|-------------| | Standard Works | Standard Works | | Direct | Direct, Direct Late, Holograph Direct, Direct Journalism | | Scribed | Scribed Verbatim, Scribed Verbatim Direct, Scribed Summary | | 2nd/3rd Hand | 2nd Hand, 2nd Hand Late, 3rd Hand Late | | Journalism | Journalism, Reprint Journalism, Unsourced Journalism | | Secondary | Secondary | **Document Type** — use the controlled vocabulary: Scripture, Book, Letter, Newspaper, Newspaper Interview, Interview, Periodical, History, Journal, Diary, Institutional Essay, Manual, Legal Document, Court Record, Speech / Sermon, Genealogical Record, Photograph, Map, Other **Tags** — researcher-assigned keywords for filtering. Keep consistent across sources: `first-mention`, `early-account`, `late-recollection`, `firsthand`, `secondhand`, `spectacles`, `seer-stone`, `urim-and-thummim`, `nephite-interpreters`, `two-stones`, `hat`, `newspaper`, `sworn-testimony`, `new-york`, `nauvoo-period`, `cowdery`, `phelps`, `coray`, `silver-bows` ### 3C. Quality Audit Run before deploying any page: ```bash python3 scripts/audit.py --data .json ``` Checks: - [ ] Every source has all required fields (date, summary, source, hearsay, type, reference) - [ ] No duplicate IDs - [ ] All source_links are valid (HEAD request, flag 404/500) - [ ] All hearsay values are in the recognized taxonomy - [ ] All type values are in the controlled vocabulary - [ ] date_sort values are consistent with date strings - [ ] Images with `downloaded: true` have actual files on disk - [ ] No transcription fields contain the word "undefined" or "null" - [ ] Source count matches expected (header metadata) - [ ] All people arrays are non-empty for non-ancient sources ### 3D. Deduplication Two sources are duplicates if they share the same `reference` string or the same `source_link`. When importing, check against existing sources before adding. The `add_source.py` script does NOT check this — verify manually or via audit. --- ## Facet 4: Display **Goal:** Deploy a working, image-backed research page on the VPS that anyone can browse. ### 4A. Local Development ```bash cd ~/Downloads/Hermes/bom-research-viewer python3 -m http.server 8080 # Open http://localhost:8080?data=data/.json ``` The viewer supports any JSON file in `data/` via the `?data=` parameter. ### 4B. Creating a New Page ```bash # Create empty dataset python3 scripts/new_page.py "Book of Mormon Witnesses" # Creates: data/book-of-mormon-witnesses.json # Add sources (interactive or batch) python3 scripts/add_source.py --data book-of-mormon-witnesses.json # Classify and download images python3 scripts/seed_images.py --data book-of-mormon-witnesses.json python3 scripts/download_images.py --data book-of-mormon-witnesses.json --type internet_archive --pages 1 # Audit python3 scripts/audit.py --data book-of-mormon-witnesses.json # Deploy python3 scripts/deploy.py --topic "book-of-mormon-witnesses" ``` ### 4C. Deploying to VPS The VPS serves files from `/var/www/research/` at `http://bomresearcher.fit`. **What to upload:** 1. The data file: `data/.json` → `/var/www/research/data/.json` 2. Downloaded images: `data/images//` → `/var/www/research/data/images//` 3. The viewer itself: `index.html` — only if it changed (shared across all pages) **How:** ```bash python3 scripts/deploy.py --data .json ``` This script: - Creates remote directories - `scp`s the data file, images, and index.html if changed - Verifies the HTTP endpoint returns 200 - Reports the public URL ### 4D. Viewer Features - Sortable chronological table (Date, Summary, Source, Hearsay) - Expandable rows with full metadata + transcription + image gallery - Image gallery with page navigation, thumbnails, full-size view - Full-text search across all fields - Hearsay + Tag filters - URL hash state persistence (sharable links) - Print-friendly CSS - Expand All / Collapse All ### 4E. Post-Deploy Checklist - [ ] Page loads at `http://bomresearcher.fit?data=data/.json` - [ ] Click 3 random rows — detail cards render without JS errors - [ ] At least one image gallery shows correctly - [ ] Search works (type a known term, verify results) - [ ] Sort works (click column headers) - [ ] Filters work (select hearsay type, verify filtering) - [ ] Permalink copies correctly - [ ] Print preview looks clean (Cmd+P) --- ## Complete Pipeline: Example Here's the full end-to-end flow for a new topic, "Book of Mormon Witnesses": ### Phase 1: Identification ```bash # 1. Check Mormonr # Navigate: mormonr.org → HARD QUESTIONS → "Book of Mormon Witnesses" # Found: ID=0Eiiyt, 254 primary sources # URL: https://mormonr.org/qnas/0Eiiyt/book_of_mormon_witnesses/research # 2. Log discovery echo "| 2026-06-18 | Mormonr | BoM witnesses | Found 0Eiiyt (254 sources) | — |" >> data/book-of-mormon-witnesses/discovery_log.md ``` ### Phase 2: Acquisition ```bash # 3. Create the page python3 scripts/new_page.py "Book of Mormon Witnesses" --dir book-of-mormon-witnesses # 4. Extract from Mormonr (browser console — see extraction technique) # ... browser extraction yields filtered_sources.json with ~40 relevant sources ... # 5. Import sources python3 scripts/add_source.py --data book-of-mormon-witnesses.json --batch filtered_sources.json # 6. Classify and download images python3 scripts/seed_images.py --data book-of-mormon-witnesses.json python3 scripts/download_images.py --data book-of-mormon-witnesses.json --type internet_archive --pages 1 ``` ### Phase 3: Organization ```bash # 7. Audit python3 scripts/audit.py --data book-of-mormon-witnesses.json # → Fix any issues, re-audit # 8. Verify locally # python3 -m http.server 8080 # → http://localhost:8080?data=data/book-of-mormon-witnesses.json # → Click through, verify display ``` ### Phase 4: Display ```bash # 9. Deploy python3 scripts/deploy.py --data book-of-mormon-witnesses.json # 10. Verify live # → http://bomresearcher.fit?data=data/book-of-mormon-witnesses.json ``` --- ## File Map ``` bom-research-viewer/ ├── index.html # Universal viewer (shared by all pages) ├── README.md # User-facing quick start ├── PIPELINE.md # ← THIS FILE ├── data/ │ ├── sources_full.json # BoM Translation Methods (238 sources) │ ├── witness_accounts.json # Key Witness Accounts (subset) │ ├── hearsay-taxonomy.md # 35-tag definitions │ └── images/ │ ├── src-009/ # Downloaded page images per source │ ├── src-052/ │ └── src-021/ └── scripts/ ├── add_source.py # Interactive & batch source import ├── new_page.py # Create new topic JSON file ├── seed_images.py # Classify sources & seed image fields ├── download_images.py # Download page images per domain ├── audit.py # Quality check & link validation └── deploy.py # Upload to VPS ``` --- ## Pitfalls 1. **Filtering is critical** — Mormonr pages have 200-300 sources. Do NOT bulk-copy all of them. Filter for directly relevant, first-hand, and contemporary sources. The user values curated, targeted datasets over exhaustive mirrors. 2. **Web archive vs. Internet Archive** — `web.archive.org` is a snapshot service, not a bookreader. These sources (19 in the BoM Translation dataset) redirect to the original page. Classify them as `web_archive`, not `internet_archive`. 3. **Image download pacing** — Default to 1 page per source. The user said "slowly and deliberately." Don't download all pages of a 500-page book unless specifically asked. Use `--pages N` to increase. 4. **contentDM and JSP images** — These may require browser-based extraction or IIIF API exploration. The downloader has handlers, but they may need manual intervention for specific sources. Test one source before downloading a whole batch. 5. **Deploy images with the data** — The `deploy.py` script must upload both the JSON data file AND any downloaded images. A page with `downloaded: true` but missing image files on the VPS will show broken images. 6. **VPS memory** — 238 sources with full-size images could be gigabytes. Start with 1 page per source (typically 200-800KB each). Full-page downloads can wait until specifically needed. 7. **Source Link field parsing** — The `Source Link` label contains `Source` as a substring. When regex-splitting Mormonr detail text, always match `Source Link` BEFORE `Source`. 8. **Citation completeness** — Every source must have a `reference` (Chicago citation) and a `source_link` (or a repository note if not digitized). Missing links should be documented, not silently omitted.