Synced from the repo — do not edit here

Canonical source: docs/claude/data-source-of-truth-audit.md. This page is generated by docs/scripts/sync-handbook.mjs. Edit the source file in the repo; changes appear here on the next build.

Data Source-of-Truth Audit — Provider ↔ Client ↔ Atlas

Filed: 2026-07-14 · Trigger: George — "Atlas needs to see the same data source that all surfaces see. We can't have Atlas reference one value, basisflow-web another, basishybrid/basisweb another for the same metrics (e.g. zone 2 minutes last week)."

Scope: two-way data matching across the four surfaces that read/write the same health data — basisflow-web (provider layer), basisweb + basishybrid (client layers), and Atlas (the AI, which must quote the identical number the humans see). Three data domains audited: protocol assignment/scheduling/notes, wearable metrics (zone/exercise minutes), and labs (values/units/verdicts).

This is the master synthesis of four parallel deep-dive audits. Companion docs: plan-metrics-data-contract.md (the wearable/lab read contract) and plan-tab-fixes.md (running divergence list). Related open issue: #272 (benchmark/verdict consolidation) — the RANGE half of the labs problem; this doc covers the DATA half.

---

0. The one pattern behind every divergence

Every mismatch in all three domains is the same architectural failure:

There is no single computed value. Each surface independently re-reads raw-ish data and re-derives the "same" metric with its own code, thresholds, aggregation, timezone, and unit handling. They agree only by coincidence — and drift the moment any one of the four reimplementations changes.

Concretely, the "single source of truth" that exists in intent is defeated in execution in three recurring ways:

1. The canonical doc isn't authoritatively written. dailyDigest (wearables) is written only by the mobile app, never server-side; the protocol recurringEvents doc is canonical but no surface except the app reads it. So the nominal source is stale/absent exactly when it matters, and everyone falls back to raw + improvises.
2. N schedule/metric generators for one value. Protocols have three independent schedule generators; zones have four independent max-HR/threshold bases; labs have per-surface benchmark tables and unit conversions.
3. Atlas is the most divergent reader, because it was wired to be permissive (reads a superset of sources) but does the least normalization (no unit conversion, no benchmark fallback, and — for zones — no source field at all).

The fix is the same in all three domains: compute the value once, server-side, into a canonical doc, and make every surface and Atlas read that doc verbatim — deleting the downstream re-derivations. Below, per domain: the source map, the ranked root causes (with file:line), and the collapse-to-one plan.

---

1. WEARABLE METRICS (zone / exercise / active minutes)

Source map — "zone 2 minutes last week"

Surface	Doc → field	Computation
basishybrid (app UI)	Local SQLite (Rust FFI), not Firestore	Recomputes live. maxHR = measured ?? 220−age (default age 35 → 185). Scalar zoneMinutes is weighted z1×1..z5×5; per-zone series raw. Week = 7 device-local days. analyzer_hr.dart:97-101,171,185-236
dailyDigest (what the app wrote)	dailyDigest.zone1Minutes..zone5Minutes	From hrDaily z1..z5. maxHR = DOB 220−age clamp(100,210) fb 185, never measured. Raw per-zone. service_health_firestore_writer.dart:2255-2259,2836-2863
basisflow-web (provider)	dailyDigest.zoneNMinutes + recompute	Reads digest zones then ADDITIONALLY recomputes from non-Apple workout HR samples (own thresholds: 220−age, else absolute 120/140/160/180 bpm) and SUMS onto digest zones. Card = 7-day sum, provider browser tz. MetricsTab.tsx:131-181,1136-1212,1549-1580
basisweb (client portal)	—	Does not show wearables at all (gap).
Atlas	NONE	No zone entry in _DIGEST_METRIC_FIELD (functions_ai_agent.py:1978-1995), type_mapping (:2149-2176), or snapshot METRIC_DEFS (functions_orchestrator.py:359-375). Atlas cannot read zone minutes at all.

Exercise minutes: app = local sum; digest = exerciseMinutes keep-max/day; web = latest-day-max; Atlas get_health_metrics = avg/day (functions_ai_agent.py:1988,2073); Atlas snapshot = avg/day (functions_orchestrator.py:370,395). Same digest doc, four aggregations. ("Active minutes" on web is a dead interface, effectively = Exercise.)

The structural fact

dailyDigest is app-written only (service_health_firestore_writer.dart:2480-2493 + add_manual_metric mirror). The Terra webhook path writes healthSummaries only, never dailyDigest (mirror_firestore.py:28-42). So Terra data reaches the digest only when the app runs, computes locally, and pushes. Web-only client / backgrounded / uninstalled → stale/absent digest → web + Atlas fall back to healthSummaries and improvise.

Root causes (ranked)

• W1 (CRITICAL): dailyDigest written client-side only; Terra never populates it server-side → not authoritative when the app isn't running. mirror_firestore.py:41, writer :2493.
• W2 (CRITICAL): basisflow-web double-derives zones (digest zones + its own recompute, summed) → double-counts for any Terra user. MetricsTab.tsx:1136-1212.
• W3 (HIGH): four max-HR/threshold bases (app UI, digest writer, web recompute, Terra per-workout startPercentage) → same HR stream, different zone attribution everywhere.
• W4 (HIGH): weighted vs raw zone minutes (mobile scalar weighted ×1..5; digest+web raw).
• W5 (HIGH): different aggregation for the same metric (exercise: web latest-max / Atlas avg / app sum).
• W6 (MEDIUM): different week-window timezones (app device-local / web provider-browser / Atlas UTC).
• W7 (= the founder's example): Atlas has no zone-minute source at all → fabricates or returns nothing.

---

2. LABS (values / units / verdicts)

Source map — a single lab value

Surface	Reads (path)	Value/unit	Unit conversion on read?	Verdict source
basisflow-web (provider)	clinicsv2/{c}/clinic_users/{u}/labs/{id} LabsTab.tsx:1921	d.value/d.unit	display-only, opt-in preferSI, NOT applied before verdict lib/lab-unit-conversions.ts:240	clinic benchmarks/{key} → per-doc range → local DEFAULT_LAB_BENCHMARKS
basisweb (client)	users/{u}/labs/{id} portal/health/page.tsx:333 — different collection	d.value/d.unit	display-only	its own DEFAULT_LAB_BENCHMARKS page.tsx:181
basishybrid (mobile)	clinic and user labs via parseLabDoc lib/services/lab_doc.dart:98	value/unit	no (glucose-only in chart)	TrendType/InsightType benchmark or per-doc flags analyzer_biomarkers.dart:401
Atlas	user + both clinic labs + labResults functions_ai_agent.py:2586	value|result|numericalValue	NO — raw {value} {unit} :2662	NONE of its own — only echoes stored isAbove/isBelowMaxRange :2569

Write side is consistent: users/{u}/labs and the clinic mirror are written from one dict in one batch (functions_ai_documents.py:2708/2712; Junction functions_junction.py:1890/1891). Divergence is across ingestion routes and across readers, not within one write.

Root causes (ranked)

• L1 (CRITICAL): no enforced canonical unit at rest. import_lab_values normalizes only the ~28% of analytes in lab_unit_conversions.json with an exact source-unit match; Junction stores fully raw (functions_junction.py:1860-1861). Same analyte, different unit depending on ingestion route.
• L2 (CRITICAL, = #272 mechanism): surfaces compare the raw stored value against a benchmark in a different unit, no reconciliation — on every surface independently (LabsTab.tsx:1944, analyzer_biomarkers.dart:402). Aliasing amplifier: ratios share the HDL family type → "HDL 3.9 / ratio".
• L3 (HIGH): 3–4 disagreeing unit authorities — lab_analytes.json.primary_unit (display only), runtime lab_unit_conversions.json (canonical_unit), Dart BasisLabType hardcoded units. The one used at write is not the registry.
• L4 (HIGH): TS analyte mirrors (lib/lab-analytes-generated.ts) carry no conversions/ranges and their generator (generate_lab_analytes_ts.py) is missing from the repo → hand-drifted.
• L5 (MEDIUM): analyte-id key namespaces differ — Atlas keys snake_case analyteKey (ANALYTE_KEY_NAMES), web/mobile key camelCase labXxx → dropped names/flags.
• L6 (MEDIUM): registry itself is bimodal (483 rich / 301 legacy) → ~38% of analytes have no machine-usable unit or range at source.
• Atlas-specific: reads the widest source set, applies zero conversion, has no benchmark fallback → reports a different number AND an absent verdict vs the portal for the same lab.

---

3. PROTOCOL ASSIGNMENT / SCHEDULING / NOTES

Data flow — write → store → read

Provider writes through one callable (clinic_service assign_protocol / update_protocol) — good — but the backend then quadruple-writes: (1) users/{u}.recurringEvents (full event JSON), (2) clinicsv2/{c}/clinic_users/{u}/protocols/{id} (minimal _habit_min shape), (3) users/{u}/protocols/{id} (same minimal mirror), (4) clinic_users/{u}/events/expected-* (materialized fan-out). Each surface reads a different subset + a different schedule generator:

Surface	Protocol content	Schedule instances
Provider (PlanTab)	clinic mirror (2)	clinic events (4) + user events + client-side materializeProtocolActivities PlanTab.tsx:2916-3135
basisweb	clinic mirror (2)	clinic events (4); dates via new Date(string)
basishybrid	recurringEvents (1) + both mirrors (2)+(3) merged	local SQLite from the Rust recurrence engine over (1) service_recurring_events.dart:1394

Root causes (ranked)

• P1 (CRITICAL): three independent schedule generators for one protocol (Python fan-out, mobile Rust engine, provider materializeProtocolActivities) — different horizons/weekday-math/time-defaults → different instances per surface.
• P2 (CRITICAL): assign vs update fan-out inconsistent — horizon 90d vs 14→180d; eventType stored raw on assign but resolved on update; date-only TZ special-case on update only (functions_clinic.py:1333,2037,1371,2205,2239).
• P3 (HIGH): recurringEvents (1) and mirror (2/3) are different shapes — (1) has defaultSupplements/normalized preset/description; mirror drops them → supplements/descriptions differ by surface.
• P4 (HIGH): notes stored in ≥4 places and UpdateHabitInput has no notes field (:1528-1535) → the habits-patch path cannot edit notes; QuickActivityModal writes notes directly to the event doc bypassing update_protocol (PlanTab.tsx:428-438). ("Adding notes isn't foolproof.")
• P5 (HIGH): basishybrid FFI round-trip strips recurringEvents/protocolId (service_firebase.dart:759-763); no scheduledFrom on BasisRecurringEventV1.
• P6 (HIGH): fan-out expected-* events leak into user events — sync_clinic_event_to_user mirrors ALL clinic events with no source=='expected' filter (functions_event_mirror.py:330-387) → duplicate instances.
• P7–P10 (MED/LOW): non-atomic dual-mirror writes; date-only UTC off-by-one for US clinics (#302); provider activities-wins vs backend habits-only; timesPerWeek habits never fanned out.

---

4. Atlas alignment scorecard

Domain	Atlas reads the canonical doc?	Same number as the surfaces?	Gap
Wearable scalars (exercise)	✅ dailyDigest (intent correct, functions_ai_agent.py:2102)	⚠️ No — avg/day vs web latest-day-max; UTC vs browser tz	aggregation + tz contract
Wearable zones	❌ no source field	❌ impossible	add zone1..5 to digest map + snapshot
Labs	⚠️ widest source set, but	❌ no unit conversion, no verdict fallback	normalize + read one path + persist verdict
Protocol	❌ reads mirrors/fan-out, not recurringEvents	⚠️ depends which of 3 generators	one canonical protocol + one generator

Atlas is ~70% aligned on the happy path and the single most divergent reader on the edges. It was built permissive-read / thin-normalize; the correct end state is thin-read / normalize-upstream — i.e. Atlas reads the same canonical doc every surface reads, with the computation already done.

---

4b. Write-side door maps (ingestion audit, added 2026-07-14)

George's follow-up: metrics/labs arrive through MANY doors (Apple Health, manual mobile, manual staff, Terra; provider manual, PDF scan, Junction JSON, mobile scan). Two dedicated audits mapped every door. Findings:

Labs — 6 doors collapse to 3 code paths; ONE is already correct

Path	Doors	Normalizes units?	Dedup?
import_lab_values (functions_ai_documents.py:2336)	provider manual (web), PDF scan, general manual	✅ _canonicalize() (200+ rules + registry synonyms) + _normalize_unit() → canonical unit, keeps originalValue/Unit	✅ deterministic IDs
Junction webhook (functions_junction.py:1794)	Junction JSON sync	❌ 11 slug maps only; stores raw units (:1860-1861) — but DOES persist ranges/interpretation	✅ deterministic
Mobile scan direct Dart write (modal_activity_details.dart:2031-2137)	basishybrid lab-report scan	❌ raw; Dart enum matching	❌ auto-IDs → duplicates possible; user-path only (clinic mirror via separate share call)

basisweb has NO lab-entry door (read-only). The labs write fix is therefore convergence, not construction: route the 2 bypassing doors through the existing good gate.

Wearables — NO source precedence exists; last-write-wins by sync timing

All digest writers (app _digestAppend service_health_firestore_writer.dart:2102-2110, add_manual_metric functions_clinic.py:13880) hit the same dailyDigest/{date} doc field-by-field with set(merge:true): no read-before-write, no max/union, no ranking — whoever syncs last wins. (Apple 8,000 steps → Terra overwrites 7,500 → manual overwrites 8,200 → digest shows 8,200 because it synced last.) Two half-built pieces exist: _sources[field]={device,at,writer} provenance IS stamped on every field, and users configure heartDataPreferences.devicePreferences — but that policy is consulted on the READ side only. Fix = apply the existing preference policy at WRITE time using the existing provenance stamp (#393). Also confirmed: the Terra path derives NO zones at all (only hr_avg_bpm scalars) — #381 adds server-side zone derivation.

The two-layer stack (values vs verdicts)

A benchmark/verdict ("is this HDL optimal?") is a second derived layer on top of the value layer and obeys the same one-computation principle. #272 (already specced: wire the 784-analyte registry into verdicts, unit-normalize at eval, retire benchmark-defaults.ts + trend_type.dart) is that layer for BOTH labs and wearable-metric benchmarks, now a child of #377 with Atlas as a third required reader (Atlas has no benchmark engine; it echoes persisted flags only — #392 persists the computed verdict at write so Atlas quotes what the UIs show).

---

5. Target architecture — one value, computed once

The unifying principle across all three domains:

Derive each metric once, server-side, into a canonical document. Every surface (including Atlas) reads that document verbatim. Delete all downstream re-derivation.

Three shared building blocks to create:

1. A server-side dailyDigest writer in the Terra ingestion pipeline (terra_adapter.py / mirror_firestore.py) so wearable scalars + zones are computed once on the backend from healthSummaries/DuckDB, independent of whether the app runs. App keeps writing the same doc opportunistically; server becomes the guaranteed writer.
2. A single canonical-unit normalizer for labs at write, applied on all ingestion routes (PDF import and Junction), storing canonical {value, unit} + originalValue/originalUnit, failing loudly on misses. One generated registry (lab_analytes.json → conversions + TS mirrors + Dart units, CI drift check).
3. One canonical protocol serializer + one schedule generator — pick recurringEvents (or the mirror) as authoritative, derive the other from one serializer with identical fields; make the backend fan-out the single materialized-events source all surfaces read (retire the provider client-side materialization and, for protocol-assigned habits, the mobile Rust duplicate).

Then a shared "resolver" seam per domain that Atlas calls too:

• wearable metric resolver (canonical field map + zone thresholds + aggregation + tz) — today reimplemented in TS biomarker-utils.ts and Python _read_metric_from_daily_digest;
• lab verdict resolver (normalize value → benchmark unit → verdict) shared by web evaluateBenchmark, mobile analyzer_biomarkers, and Atlas — the join with #272;
• protocol read resolver (one doc shape, one instance list).

Definition of done: ask any surface and Atlas for "zone 2 minutes last week" / a lab verdict / a protocol's schedule for the same patient/window and get the byte-identical answer.

---

6. Epic & children — FILED (2026-07-14)

Epic #377 with 16 sub-issues, dependencies declared via Blocked by: + blocked labels. The phased plan is posted as a comment on #377.

Phase 0 — measure: #384 parity harness (cross-surface diff tool; RED baseline first; the acceptance test every child cites).

Phase 1 — make canonical sources authoritative (write side): #381 server-side dailyDigest writer in Terra pipeline + one shared max-HR/zone module (wearables keystone) · #383 Junction → import_lab_values normalizer (bug,P2) · #385 mobile lab-scan → backend import path (bug,P2) · #387 one canonical protocol habit serializer, atomic mirrors · #388 skip expected-* in sync_clinic_event_to_user (bug,P2, small early win) · #389 date-only scheduledFrom/Until in clinic tz (bug,P2, relates #302) · #390 single notes path (UpdateHabitInput.notes + QuickActivityModal via update_protocol) (bug,P2)

Phase 2 — converge readers: #393 write-time source precedence for the digest (kill last-write-wins) (blocked by #381) · #394 basisflow-web reads digest zones verbatim, delete recompute (blocked by #381) · #391 basisweb wearables from digest (blocked by #381) · #382 Atlas zones + one aggregation/tz contract (unblocked — fields already exist) · #395 one schedule generator + unify assign/update fan-out (blocked by #387)

Phase 3 — verdict layer: #272 benchmark consolidation (labs + metric benchmarks; Atlas added as third required reader; needs-human design decisions) · #392 persist computed verdict at write → Atlas verdict parity (blocked by #383 + #272) · #386 one generated analyte registry + CI drift check (supports #272)

Recommended first moves: #384 → #381 + #388 in parallel → #382/#394 (completes the founder's "zone 2 minutes" case end-to-end).

Cross-cutting: each child extends plan-metrics-data-contract.md with its metric's canonical doc/field/computation/tz — the contract every surface + Atlas cite.