# Datasheet: qipu data San Francisco v0.7.1 Following *Datasheets for Datasets* (Gebru et al., 2021). Every number here comes from `NUMBERS.json` of this release (generated from the published tables, each with its SQL); nothing is typed by hand. Written by `python -m qipu_data.kit v0.7.1`. ## Motivation - **Why it exists.** To make the Internet Archive's scanned San Francisco public records citable as facts: every claim leads back to the archive.org page it was read from. The Archive holds the scans; qipu is a finding aid over them. - **Who made it.** qipu, with Claude models reading the pages under measured, published runs (`runs`). - **Funding.** Self-funded. Model costs are recorded per run in `runs`. ## Composition - **What a record is.** A **claim** says what a record states, not that it is true: subject, property, value, place, time, the page (`:`) and, for records before 1951, a short quote. Hubs: places, persons (officials), decisions, instruments, sources, pages. - **How many.** 8,367,105 claims (4,724,389 of them votes from 171,123 printed roll calls); 539,192 decision rows (539,192 distinct decisions; 344,936 from the v2 read, one per item); 470,596 places with stable ids; 183,888 cited archive.org pages across 6,377 sources (698 in Democracy's Library); 1,428 media items. 86.3% of claims cite an archive.org item. - **Reliability.** Every claim has a tier in `claim_reliability`: 889 verified, 2,962,695 quoted, 5,403,521 unverified. `field_scores` (36 rows) says against what each run was measured (a person, a model-written key, or a model re-reading the page image). - **What a claim means.** **A claim says what a record states, not that it is true.** "The Board adopted resolution 1234 on 15 August 1921" means the Journal page says so; truth needs corroboration. Each claim carries how firmly it is tied to its record (`claim_reliability`): - **verified**: a person checked it against the scan, or two or more independent documents state the same thing; - **quoted**: its quote is printed on the cited page, and the run that read it measured at 0.9 or better (the table says whether against a hand check or a model-written key); - **unverified**: everything else, with the reason (no quote, no page text yet, an unmeasured run, a failed consistency check, a correction). Two columns sound alike and are different things. `claim_reliability.tier` (above) is the claim's reliability. `claims.reliability` (`measured` / `unmeasured`) only says whether the RUN that read the claim has scores in `field_scores`; it says nothing about the claim itself (to be renamed `run_measured` in v0.7). `claim_reliability` is in the Parquet and, from v0.6.2, in the DuckDB too. Found a claim that looks wrong? Flag it (the explorer's "flag this fact"; reviewed by a person before anything changes). - **Personal data.** Private people's names are withheld for 75 years after the record (SF; 92 years in Vancouver); officials are named in their role; companies and organisations are always shown. Item text of Board records after 1951 is not published. ## Coverage and bias These figures were counted on v0.6.1 and will move with each release. - **What the corpus is.** It is what the San Francisco Public Library and the Internet Archive scanned and put on archive.org: 1,045 items, 183,888 cited pages. It is not all of San Francisco's records. Volumes that were never scanned, or that came out as poor OCR, are missing or thin. - **Read in depth.** The Board of Supervisors' Journals of Proceedings and minutes, roughly 1905 to 2015, are read item by item, with decisions, outcomes, money, places and roll-call votes. Most claims come from them: about 0.5 to 1.1 M claims per decade from the 1910s to the 2000s. - **Thinner.** Before 1900 there are under 12,000 claims per decade, mostly maps, directories and catalogue records. After 2015 the claims are mostly DataSF registers and census figures, and there are almost no Board votes. Democracy's Library and other archive.org documents are mostly catalogue metadata (title, date, subject), not read page by page. - **What is missing or weak.** - About 8% of the v2 decision readings (27,643 of 344,936) have no meeting date. - Votes are placed citywide, not at the decision's place. - Only about 1.55 M of 8.37 M claims sit on a door, parcel or building. 3.86 M have no place finer than "unknown", and 2.18 M are citywide. - Private people's names after the embargo line (75 years in SF) are withheld, so the records of ordinary people after about 1951 are deliberately thin. - Person matching has not yet been measured by a person: votes are linked to a supervisor by surname within their years. - A few claims carry impossible years (for example 1000 or 3900; check T3 lists them). - **How to read "central".** When a page or a graph calls a place, person or organisation "central" or "most connected", it means it is **mentioned most in these records**, not that it was most important in the city. A body that kept and filed more paper, or a period that was scanned more fully, looks bigger. - **Who is over- and under-represented.** The Board's own business is over-represented: officials, city departments, utilities, large property owners and firms that dealt with the city. People and neighbourhoods that seldom came before the Board are under-represented, and so is anything decided elsewhere (state, federal, courts, private deals). ## Collection process - **Sources.** Scans and OCR on the Internet Archive (the San Francisco Public Library's partner scans, Democracy's Library and others), plus a few structured public datasets (DataSF, US Census), each marked `on_archive_org = false`. - **How it was read.** Rules for structured text (roll calls, page numbers), language models for the rest (decisions v2: claude-haiku-5-5, one call per page with the previous page as context). Every run is listed in `runs` with its model. - **Politeness.** archive.org metadata and OCR are fetched about once a second with a contact address, and cached. ## Preprocessing, cleaning, labelling - Pages are keyed on IA's leaf number; links use the BookReader page index from the item's leaf map. - Places are resolved to today's EAS addresses, assessor parcels and blocks, streets and neighbourhoods; a claim keeps the text as printed (`place_label`) next to the place it resolved to, with a precision. - Money is kept in the record's dollars, with 2026 dollars beside it (US CPI-U). - The raw extraction tables (`x_*`) are published next to the claims, so every claim can be traced to the row that made it (`claim_support`). ## Uses - Finding what the Board decided at a place, in a year, about a subject; following a supervisor's votes; tracing money. - Citing: cite the archive.org page (each record's `cite`), not qipu as the source of the fact. - **Not for:** treating a single claim as established fact without the page; inferring anything about private people. ## Distribution - data.qipu.org: Parquet, SQLite, CSV (`csv/`), JSON records per source, page and entity, and IIIF annotations on IA's canvases. - Licence: CC0-1.0 for the claims; media keep their holders' rights; Wikipedia paraphrases are CC BY-SA 4.0 and say so. - Releases are immutable, versioned, with a manifest of every file's sha256; a fix is a new version. ## Maintenance - Releases are built by a DAG from registered sources (`sources/registry.yaml`); new items of approved sources flow in under budget caps; new sources need a benchmark and an approval. - Corrections go in the `corrections` layer (0 today), reviewed by a person; takedown requests through www.qipu.org. ## Known limits - Counts: decision rows come from several extraction passes, and one printed item can have up to five rows. Count distinct decisions with `decision_group` (one id per printed item); its duplicate matching is about 0.92 precise and 0.70 complete. - Roll calls are re-read from the page text since v0.6.1, but some across page breaks are still incomplete (1959 hand check: 98 of about 102 roll calls with a No are found). - Street links: on the audit sample, 39 of 68 street links and 20 of 46 corner links are still seriously wrong. - About one numbered decision in five before 1998 carries the number of a neighbouring item. - About 8% of decisions v2 rows have no date (no dated heading on their page). - Many claims are citywide (votes); the table above gives the door-level count. Votes sit at the city, not at what was voted on. - Feature geometry (creek, shipyard, freeway, landfill, redevelopment area) is a circle around the Wikidata point, marked approximate. - The accuracy numbers are agreement with Opus-written keys, not hand-checked samples (except the appraisals); no held-out set, no intervals yet. - Vote confidence is 0.87 (the measured roll-call count rate on the SF key, n = 46), 0.6 where the printed counts disagree with the names; person matching is unmeasured (a 300-name hand check is ready: build/handcheck/vote_names_300_HOWTO.txt). - Date ranges: until v0.6.2, 1.68 M vote claims of the v0.6.1 re-read had `t_from` but no `t_lo`/`t_hi`; v0.6.2 fills them and check T6 stops a build that misses any. - The name hider is a pattern (two or three capitalised words), not a full name recogniser: it over-hides some places and can miss a lone surname. Raw quotes of embargoed records are not published at all. - OpenSFHistory photos are link-only in our products until WNP agrees to inline display.