← qipu data

qipu data: changelog

Releases are numbered v0.x until the schema settles. Each release is a folder (releases/<version>/) and, when published, an archive.org item. Entity ids (places, persons, decisions, instruments) never disappear between releases: a retired or merged id gets a redirect. Claim ids changed twice before any correction pointed at them (v0.3: slot hashes) and are meant to stay from v0.3 on.

v0.4 (14/10/2026, early morning): Vancouver, provenance, exports, privacy everywhere

What changed - Vancouver (qipu:van/...): the decisions v2 run (claude-haiku-5-5, $23.95, Wayback council minutes 1995-2026 and IA scans of the 1970s): 76,652 decisions read, 14,231 items split across two pages merged with the next page's heading (their second halves redirect to the first), 62,421 decisions and 252,551 claims (outcome, concerns_place, party, moved, money, voted, refers_to, rezoned). Vancouver minutes mostly print only who opposed ("CARRIED (Ald. X opposed)"), so its votes are 12,775 no, 4,920 absent, 448 yes. Councillors are public officials by role; their names stay names as printed until a roster is loaded. Canada's 92-year rule applies to everything else (all of it is embargoed). - Privacy, every column (coordinator spot-check): a second pass (41_public_scrub.py) filters every free-text column of every public table with a year (vote rows, roll-call quotes, decision titles, x_ tables, quotes) and the build stops if a person-name pattern is left in an embargoed row. The name finder now needs a title, an initial or a Census first name, so phrases like "Finally Passed" are no longer hidden. On the published v0.3 copy: 36,393 name patterns in the public SQLite and 90,024 in the public Parquet before; 0 and 0 after (republished 14/10 ~05:00). - OCR provenance (review M5): ocr_files (9,416 OCR files of the cited items with md5, sha1 and mtime) and pages.ocr_file_sha1 on 115,800 of 118,891 pages. - Exports (review M10, m8): Linked Places Format (2,148 places for the World Historical Gazetteer), 1,000 W3C Web Annotations on IIIF canvases (pre-1940 decisions), and a reviewed-but-not-uploaded QuickStatements batch adding "member of the San Francisco Board of Supervisors" (Q116005093) to 115 supervisor terms with a Journal-page reference. - Reach: REACH.html (Democracy's Library by place: 91% placeable from metadata, but 92% of USGovernmentDocuments are court dockets placed only by district; of the other ~0.9M items ~15% name a city or county); MINUTES_INVENTORY.html (10,612 council and board minutes items on archive.org, ~1.68M pages, ~$2,100 to read with decisions v2); new-city plans for Vancouver and Oakland (95_new_city.py). - Benchmark kit (bench/): golds, prompt, a dependency-free scorer, honest labels. - Hand-check sheets (build/handcheck/, local): 100 held-out decisions and 300 vote names, stratified by decade, for a person to check against the scans. - MCP server: places_at and showcase tools; citations documented.

Checks: 41 pass, 5 warn, 1 skipped (S9: the runner skips it when the previous release is attached), 0 fail; the public privacy check passes.

Still weak - SF decisions v2 is still the 1,747 decisions of the run's first parquet (3 of 52 batches collected; the runner writes its parquet at the end). - Vancouver has no places beyond the city (no address register loaded), and no council roster. - The name finder is a pattern with a first-name list: it still misses a lone surname ("Smith gets a permit") and over-hides some phrases that start with a first name. - The public explorer's privacy filter runs a Python function over millions of cells: the export takes ~30 minutes.

v0.3 (14/10/2026, night): the spec v1.0 fixes, the place graph, the SF proof

What changed - Adopted the architect's spec v1.0 P0 list (spec/HANDOFF.md 1-6): content-hash claim ids (no run id, no resolved place) with claim_support (one row per supporting extraction row); decisions v2 readings attach to their v1 decision (1,030 v0.2 v2 ids redirect to v1 ids in id_redirects); search scores moved out of confidence into qualifiers.score; place precision on the closed list plus place_role (at / near / along / about); licence on claims is a short code, the prose lives in media.rights_note; vocabulary tables (properties, place_types, cities, privacy_rules, events); places.type_id; the spec checks run in every build and stop the publish on an error. - Also from the spec: EDTF time with t_lo / t_hi; Sanborn lines are building observations (not built); the 1906 fire is affected_by the event; nearby photos and camps are place_role = near with distance_m; rent carries its period where the row prints one; private names carry privacy_class and public_after_year (sf 75 years, van 92); runs.gold_id, field_scores from the benchmarks, claims.reliability measured / unmeasured; media get media_kind, created_edtf (from the item, not the beat) and use_policy. - The place graph: place_geometries (459,606 dated geometries: DataSF parcels and assessor blocks as polygons, EAS address points, street centerlines per street, the 41 neighbourhoods, the 1906 burn zone, the showcase features) and place_relations (1.28 M: within / crosses / overlaps from geometry, on_street / within parcel / within zoning district declared), computed every build; no fixed hierarchy. places_at(lon, lat, date) returns every place containing a point at a date, most specific first (680 Haight today: address, parcel 0847-019, block 0847, Haight Ashbury; Market and 4th on 20 April 1906: block, parcels, South of Market, the burn zone). - Today's zoning districts as places (from the DataSF parcel attribute), parcels declared within them. - The SF proof: six places (Islais Creek and its watershed, Hunters Point Shipyard, the Embarcadero Freeway, Haight Street, 1805 Geary / the Fillmore with the Western Addition A-2 area, Mission Bay) with what the Archive holds about each across collections, found by metadata only (archive.org search, Open Library, Wayback CDX; no model): 3,382 hits become sources and concerns_place / depicted_in / recorded_at claims with a confidence by where the name matched and a rights note by collection. A timeline page per place on the explorer, and "click anywhere" (a point to its block, neighbourhood and features, with their records). - Explorer: a DuckDB-WASM SQL console over the Parquet release (sql.html).

Checks (spec/tests/run_checks.py on v0.3, previous release v0.2): 40 pass, 5 warn (T3 219 claims dated outside their volume's catalogue years; T4 7 claims before their place existed; V4 4 roll calls where one printed name stayed unmatched; V6 903 rents without a printed period; I6 claim ids of v0.2 changed with the move to content hashes), 0 fail; 2 skipped (S9: the runner skips it when a previous release is attached, it passes when run alone; R11: no corrections yet).

Still weak - Feature geometry is a circle around the Wikidata point (marked approximate); real boundaries (sewersheds, the shipyard, the freeway alignment, the made land) are the next layers. - The metadata search catches namesakes: "The Fillmore" brings concerts at other Fillmores, "Hunters Point" brings the wider neighbourhood; those have lower confidence but are not removed. - Democracy's Library hits are few for these places by metadata alone (Hunters Point 22, Mission Bay 3, Islais Creek 0): DL's IGS items rarely name them in title or subject; their pages do. - Claim ids changed from v0.1/v0.2 (content hashes from now on): nothing pointed at them yet (no corrections).

v0.2 (14/10/2026, night)

What changed - Decisions v2 (claude-haiku-5-5, schema v2.1, one call per page) ingested as its own run next to v1, never over it: 1,747 SF decisions so far (the run is still landing; Vancouver's first batches had not landed). x_decisions now holds v1 and v2 rows (v2 columns: names in the roll call, parties with roles, money, places with cross streets, zoning, related numbers, dates, interest). - New claims from v2: 11,789 votes (100% resolved to supervisors by the roll-call matcher), 790 parties, 570 amounts, 763 places, 307 movers, 157 references, 24 rezonings. 1,030 v2 decisions are recognised as the same decision as a v1 item (same_as); 114 v1 decisions took their outcome from that v2 twin. - Vancouver has its own namespace (qipu:van/...); its votes stay names as printed until a council roster is loaded. - Media registry: 1,405 photos, films, recordings, clips and documents the map shows (timelines, covers, hood stories, block items, films, clips), each with credit, licence, holder and a rights note (OpenSFHistory/WNP: ask before public use; Live Music Archive: non-commercial), and 5,088 depicted_in / recorded_at claims tying them to their place and year. - The Redevelopment Agency appraisals (Western Addition A-2): 3,189 rows as x_appraisals, 14,823 claims (appraised_at, assessed_at, rent, sold_for, building, use, units, owned_by). - Democracy's Library items read for their places (x_dl_tags, 1,131 items): 1,289 concerns_place claims. - Products: the map's block decision files regenerated from the claims, compared with the map's files (build/products/v0.2/DIFF.md; the map is not touched).

Why: the map's whole content becomes traceable (every photo and film has a holder and rights), and the richer v2 extraction joins the same structure without replacing v1.

Still weak - v2 coverage is what landed by 02:07 (2 of 52 SF batches); rebuild to take more. - The v2 runner stamps CAD on SF amounts and links pages by OCR leaf; both are corrected in the release, not in the run files. - v2 money has no direction (paid or received), so it is money, not paid/received. - The map's two layers disagree with each other on the one-line summary for 3,989 of 37,758 block rows (the block index line vs the curated site line); the dataset keeps the curated one. - DuckDB is capped at 2 GB, but the build still peaks at about 3.5 GB of memory (Python + DuckDB).

v0.1 (14/10/2026)

First release, built only from data qipu already had (the map's files and the decision extraction runs).