# qipu data: San Francisco

An open, sourced record of what happened at places in San Francisco, read from the Internet Archive's scans of public records.

qipu reads the scans (Board of Supervisors journals, property files, reports) and turns them into **claims**: small statements, each tied to a **place**, a **time** and a **thing**, and each pointing to the archive.org page it came from. The design is in `spec/` (ARCHITECTURE_v1.md, written by qipu's data architect); this folder is the build.

- Release: **v0.7.1** (20/10/2026; the table below is generated from the newest built release's NUMBERS.json). Changes: [CHANGELOG](CHANGELOG.html).
- Staging site: https://tiles.qipu.org/data/ (data.qipu.org is not live yet: it needs a DNS step).
- Licence: see LICENSE. Facts read from public records are CC0; sentences paraphrased from Wikipedia are CC BY-SA 4.0 and say so; media keep their holders' rights.
- **Privacy:** names of private people are withheld for 75 years (SF; 92 in Vancouver) in everything published, including names inside model-written summaries. Corrections and takedowns: through www.qipu.org (a dedicated address is being set up).
- **Model-written text:** one-line summaries (`plain`) are written by a model (claude-haiku-5-5, or an earlier pipeline model); every one links to its scan. Check the scan before quoting.

<!-- NUMBERS:README BEGIN -->
## What exists today (v0.7.1, counted from the release) and what is only specified

Generated from `NUMBERS.json` (`python -m qipu_data.numbers render`); every number has its SQL there.

| | today (v0.7.1) | specified, not built yet |
|---|---|---|
| claims | 8,367,105; 86.3% cite an archive.org item | ~50 M for SF read in depth |
| votes | 4,724,389 vote claims from 171,123 printed roll calls and decisions v2: **4,014,743 yes, 148,174 no, 560,569 absent**; 294 officials (122 with a Wikidata id) | rosters beyond supervisors |
| decisions | 539,192 decisions (printed items), read 718,093 times by several extraction passes (an item can have up to five readings; one is preferred), 344,936 readings from the v2 read | meetings and day dates for v2 decisions |
| places | 470,596 with stable ids; 459,992 dated geometries | historic addresses and street renames |
| place precision | 26.1% of claims are placed citywide; **1,543,612 claims sit at a door** (address, parcel or building) | more door-level sources |
| sources and pages | 6,377 sources (698 in Democracy's Library); 183,888 cited pages, 182,845 with IA's leaf number | |
| reliability | tiers: 889 verified, 2,962,695 quoted, 5,403,521 unverified (`claim_reliability`); `field_scores` has 36 rows | a hand-checked, held-out random sample with intervals |
| provenance | `runs` (43), `id_redirects` (135,661), `corrections` (0) | prompts with hashes and costs |
<!-- NUMBERS:README END -->

## The SF proof: six places, every kind of record

On the explorer ("SF proof"): Islais Creek and its watershed, Hunters Point Shipyard, the Embarcadero Freeway, Haight Street, 1805 Geary (the Fillmore, in the Western Addition A-2 redevelopment area) and Mission Bay. Each gathers what the Archive holds across collections (Democracy's Library reports, SFPL scans, Board decisions, Redevelopment appraisals, films, concert and radio recordings, TV news, Wayback captures, Open Library books, maps), found by **metadata search only** (no model): each item has a link, a year, a rights note and a confidence that says how the name matched. "Click anywhere" returns every place containing a point, most specific first.

## Stable ids

```
qipu:sf/place/address/eas-289463        680 Haight St (DataSF EAS id)
qipu:sf/place/parcel/0847-019           assessor block 0847, lot 019
qipu:sf/place/block/0847                assessor block
qipu:sf/place/street/haight-st          a street;  qipu:sf/place/street/geary = "Geary" printed without its type
qipu:sf/place/hood/western-addition     neighbourhood (DataSF Analysis Neighborhoods)
qipu:sf/place/creek/islais-creek        a feature (approximate geometry)
qipu:sf/place/burn-zone/1906            the area burned in 1906
qipu:sf/person/dianne-feinstein         a person
qipu:sf/instrument/ord/1975-338         an ordinance (res|ord|bill / <year>-<number>)
qipu:sf/decision/<volume>|<item>        a decision (an item in a Journal)
<identifier>:<leafNum>                  a page: IA's leaf number (the jp2 / IIIF number), not the BookReader display index
c:<32 hex>                              a claim: 128 bits, hashed from its slot (subject, property, object, page, role), not from the reading
```

- **Ids are never reused and never deleted.** A merged or retired id gets a row in `id_redirects` and keeps resolving.
- **A place id is the place, not its state** (Open Library's work vs edition): names, numbers, buildings and occupants change; they are claims about the place, each dated and sourced.
- Claim ids changed between v0.2 and v0.3 (to slot hashes); no corrections pointed at them yet. From v0.3 on they are meant to stay.
- Every place, person and instrument id resolves to a JSON document: `qipu:sf/person/dianne-feinstein` → `https://data.qipu.org/sf/person/dianne-feinstein` (staging: `https://tiles.qipu.org/data/sf/...`).

## How it's built

`scripts/night_build.sh v0.4` runs everything, one step at a time (DuckDB capped at 2 GB and 2 threads), checks, then publishes:

| step | script | what |
|---|---|---|
| 1 | `01_supervisors_roster.py` | the supervisor roster (names, years served, Wikidata ids) from Wikipedia's list |
| 2 | `20_rollcalls.py` | every printed roll call in the Board text -> roll calls + one vote per supervisor |
| 3 | `10_stage.py`, `11_stage_extra.py`, `25_v2.py` | the map's files, the extraction runs, decisions v2, the media registry, the appraisals and the DL tags -> claim-shaped rows |
| 4 | `80_showcase_fetch.py`, `81_showcase_stage.py` | the SF proof: metadata search across the Archive, Open Library and the Wayback Machine |
| 5 | `15_ia_meta.py` | archive.org metadata of every cited item; Democracy's Library membership by collection |
| 6 | `30_build.py` | the DuckDB build (spec v1.0 tables, vocabularies, claim ids, support, redirects) -> Parquet |
| 7 | `35_geometry.py` | the place graph: geometries, relations, `places_at()` |
| 8 | `spec/tests/run_checks.py` | the spec's checks (stops on an error) |
| 9 | `40_export.py`, `41_public_scrub.py` | the public, privacy-filtered SQLite and Parquet (the full tables stay local); every free-text column of embargoed rows filtered, and the build stops if a name pattern is left |
| 10 | `45_docs.py`, `46_pages.py`, `50_entities.py`, `70_products.py` | dictionary, release pages, one JSON per entity, the map's block files regenerated from claims and diffed |
| 11 | `60_publish.sh` | upload to R2 |

Inputs (`scripts/config.json`) are local today: the Board corpus, the re-sweep and decisions-v2 run folders, the map's files and the leaf maps. Publishing them as a `qipu-data-inputs` archive.org item is on the list (review 1, M7).

## What a claim means

**A claim says what a record states, not that it is true.** "The Board adopted resolution 1234 on 15 August 1921" means the
Journal page says so; truth needs corroboration. Each claim carries how firmly it is tied to its record
(`claim_reliability`):

- **verified**: a person checked it against the scan, or two or more independent documents state the same thing;
- **quoted**: its quote is printed on the cited page, and the run that read it measured at 0.9 or better (the table says
  whether against a hand check or a model-written key);
- **unverified**: everything else, with the reason (no quote, no page text yet, an unmeasured run, a failed consistency check,
  a correction).

Two columns sound alike and are different things. `claim_reliability.tier` (above) is the claim's reliability. `claims.reliability`
(`measured` / `unmeasured`) only says whether the RUN that read the claim has scores in `field_scores`; it says nothing about the
claim itself (to be renamed `run_measured` in v0.7). `claim_reliability` is in the Parquet and, from v0.6.2, in the DuckDB too.

Found a claim that looks wrong? Flag it (the explorer's "flag this fact"; reviewed by a person before anything changes).

## Use it

- **Explorer:** https://tiles.qipu.org/data/ (questions, the SF proof, click anywhere, SQL over SQLite or over the Parquet).
- **Parquet:** `releases/v0.7.1/parquet/<table>.parquet` (the public, privacy-filtered tables).
- **Datasette:** `pip install datasette && datasette qipu_sf.db -m metadata.json`.
- **AI (MCP):** `mcp/qipu_data_mcp.py`, six tools: `place_history(address)`, `person_votes(name, years)`, `search_decisions(text, years)`, `places_at(lat, lon, date)`, `showcase(place)`, and `verify_claim(claim_id)` (a model re-reads the claim's scan page; rate-limited and spend-capped; it checks the record, not the world). Every answer carries its archive.org links; cite them as "SF Board of Supervisors, Journal of Proceedings, <date>, archive.org/details/<id>/page/n<n>". Run: `QIPU_DB=/path/to/qipu_sf.db python mcp/qipu_data_mcp.py`, or `claude mcp add qipu-data -- python /path/to/mcp/qipu_data_mcp.py`.
- **Benchmark:** `bench/` (golds, prompt, scorer) to test a model on public records.

## Known limits

- Counts: decision rows come from several extraction passes, and one printed item can have up to five rows. Count
  distinct decisions with `decision_group` (one id per printed item); its duplicate matching is about 0.92 precise and 0.70 complete.
- Roll calls are re-read from the page text since v0.6.1, but some across page breaks are still incomplete (1959 hand check:
  98 of about 102 roll calls with a No are found).
- Street links: on the audit sample, 39 of 68 street links and 20 of 46 corner links are still seriously wrong.
- About one numbered decision in five before 1998 carries the number of a neighbouring item.
- About 8% of decisions v2 rows have no date (no dated heading on their page).
- Many claims are citywide (votes); the table above gives the door-level count. Votes sit at the city, not at what was voted on.
- Feature geometry (creek, shipyard, freeway, landfill, redevelopment area) is a circle around the Wikidata point, marked approximate.
- The accuracy numbers are agreement with Opus-written keys, not hand-checked samples (except the appraisals); no held-out set, no intervals yet.
- Vote confidence is 0.87 (the measured roll-call count rate on the SF key, n = 46), 0.6 where the printed counts disagree with the names; person matching is unmeasured (a 300-name hand check is ready: build/handcheck/vote_names_300_HOWTO.txt).
- Date ranges: until v0.6.2, 1.68 M vote claims of the v0.6.1 re-read had `t_from` but no `t_lo`/`t_hi`; v0.6.2 fills them and check T6 stops a build that misses any.
- The name hider is a pattern (two or three capitalised words), not a full name recogniser: it over-hides some places and can miss a lone surname. Raw quotes of embargoed records are not published at all.
- OpenSFHistory photos are link-only in our products until WNP agrees to inline display.