← qipu data

qipu data: San Francisco

An open, sourced record of what happened at places in San Francisco, read from the Internet Archive's scans of public records.

qipu reads the scans (Board of Supervisors journals, property files, reports) and turns them into claims: small statements, each tied to a place, a time and a thing, and each pointing to the archive.org page it came from. The design is in spec/ (ARCHITECTURE_v1.md, written by qipu's data architect); this folder is the build.

What exists today (v0.3, counted from the release) and what is only specified

today (v0.3) specified, not built yet
claims 2,232,486; 95.6% cite an archive.org item ~50 M for SF read in depth
votes 1,492,054 vote claims from 123,611 printed roll calls (1906-1999) and decisions v2: 1,259,855 yes, 56,798 no, 175,683 absent, 697 excused; 99.4% of printed names matched to 294 supervisors (122 of them have a Wikidata id) the 172 missing supervisor items on Wikidata
decisions 350,014 (161,015 corpus items, 112,079 re-swept, 55,692 roll-call items without a summary, 717 decisions v2 not matched to a v1 item, 13,496 map-only, 6,286 site-only, 729 hand-written) decisions v2 for the whole corpus (running)
places 466,238 with stable ids; 459,606 dated geometries (parcels, blocks, addresses, streets, neighbourhoods, the 1906 burn zone, 7 approximate features); 1.28 M relations from geometry historic addresses and street renames (the historic place index)
place precision 73% of claims are citywide (mostly votes); 97,318 claims sit at a door (address, parcel or building), 279,002 on a street door-level sources (directories, block books)
sources and pages 6,673 sources; 567 are in Democracy's Library and carry 1,381 claims (0.06%); the backbone is the SF Public Library's partner scans (not in DL). 100,974 pages, 97,309 with IA's leaf number OCR version pinned per page (text_sha1)
accuracy field_scores (26 rows): decisions v2 against an Opus-written key (SF, 115 decisions, no record of a hand check: all-three 0.95); Vancouver 0.98 (Opus key, 21 decisions hand-checked); SFRA appraisals read by hand (18 pages); roll-call counts 40/46 on the SF key a hand-checked, held-out random sample with intervals
provenance runs (26), claim_support (793,758 rows: which extraction rows support each claim), id_redirects (1,031) prompts with hashes, costs, an inputs item on archive.org
checks the spec's CI (spec/tests) runs in every build and stops the publish on an error: v0.3 passes 40, warns on 5, skips 2
corrections the corrections table exists, empty review and takedown workflow

The SF proof: six places, every kind of record

On the explorer ("SF proof"): Islais Creek and its watershed, Hunters Point Shipyard, the Embarcadero Freeway, Haight Street, 1805 Geary (the Fillmore, in the Western Addition A-2 redevelopment area) and Mission Bay. Each gathers what the Archive holds across collections (Democracy's Library reports, SFPL scans, Board decisions, Redevelopment appraisals, films, concert and radio recordings, TV news, Wayback captures, Open Library books, maps), found by metadata search only (no model): each item has a link, a year, a rights note and a confidence that says how the name matched. "Click anywhere" returns every place containing a point, most specific first.

Stable ids

qipu:sf/place/address/eas-289463        680 Haight St (DataSF EAS id)
qipu:sf/place/parcel/0847-019           assessor block 0847, lot 019
qipu:sf/place/block/0847                assessor block
qipu:sf/place/street/haight-st          a street;  qipu:sf/place/street/geary = "Geary" printed without its type
qipu:sf/place/hood/western-addition     neighbourhood (DataSF Analysis Neighborhoods)
qipu:sf/place/creek/islais-creek        a feature (approximate geometry)
qipu:sf/place/burn-zone/1906            the area burned in 1906
qipu:sf/person/dianne-feinstein         a person
qipu:sf/instrument/ord/1975-338         an ordinance (res|ord|bill / <year>-<number>)
qipu:sf/decision/<volume>|<item>        a decision (an item in a Journal)
<identifier>:<leafNum>                  a page: IA's leaf number (the jp2 / IIIF number), not the BookReader display index
c:<32 hex>                              a claim: 128 bits, hashed from its slot (subject, property, object, page, role), not from the reading

How it's built

scripts/night_build.sh v0.4 runs everything, one step at a time (DuckDB capped at 2 GB and 2 threads), checks, then publishes:

step script what
1 01_supervisors_roster.py the supervisor roster (names, years served, Wikidata ids) from Wikipedia's list
2 20_rollcalls.py every printed roll call in the Board text -> roll calls + one vote per supervisor
3 10_stage.py, 11_stage_extra.py, 25_v2.py the map's files, the extraction runs, decisions v2, the media registry, the appraisals and the DL tags -> claim-shaped rows
4 80_showcase_fetch.py, 81_showcase_stage.py the SF proof: metadata search across the Archive, Open Library and the Wayback Machine
5 15_ia_meta.py archive.org metadata of every cited item; Democracy's Library membership by collection
6 30_build.py the DuckDB build (spec v1.0 tables, vocabularies, claim ids, support, redirects) -> Parquet
7 35_geometry.py the place graph: geometries, relations, places_at()
8 spec/tests/run_checks.py the spec's checks (stops on an error)
9 40_export.py, 41_public_scrub.py the public, privacy-filtered SQLite and Parquet (the full tables stay local); every free-text column of embargoed rows filtered, and the build stops if a name pattern is left
10 45_docs.py, 46_pages.py, 50_entities.py, 70_products.py dictionary, release pages, one JSON per entity, the map's block files regenerated from claims and diffed
11 60_publish.sh upload to R2

Inputs (scripts/config.json) are local today: the Board corpus, the re-sweep and decisions-v2 run folders, the map's files and the leaf maps. Publishing them as a qipu-data-inputs archive.org item is on the list (review 1, M7).

Use it

Known limits