Skip to content

Observability and runbooks

Purpose

The PRD's reliability bar is that cadences never silently skip and a missed run is itself escalated. The Harness therefore has to know, within minutes, when it is late, wrong, loud, expensive, or blind, and the on-call engineer needs a written path from symptom to fix. This doc defines the SLOs, golden signals per component, the four dashboards the Sea Suite team lives in (tenant health, trust budget, outreach volume, cost per managed person), alerting and on-call, and runbooks for fifteen failure modes, each a closed loop with containment first, because containment is what protects trust.

Scope / Non-scope

In scope

  • SLIs/SLOs and error-budget policy for brief delivery, reply latency, cadence adherence, connector freshness, harness and Org Model availability, audit write success, approval routing.
  • Golden signals (latency, traffic, errors, saturation) per component, plus the custom signals each needs.
  • Dashboard specifications and the trust-budget accounting model.
  • Alert catalogue, severities, paging policy, on-call rotations.
  • Runbooks RB-01–RB-15.
  • Edge cases (EC-OBS-*) and risk-control loops for the observability system itself.

Out of scope

  • The security incident process (13-security-privacy-compliance.md §12); several runbooks hand off to it.
  • Graduation and regression rules (16-rollout-and-operations.md); the trust budget feeds them.
  • Output-quality metric definitions (14-evaluation-and-testing.md §7); this doc consumes them.
  • Cloud-provider infrastructure runbooks (platform wiki).

1. SLOs

All SLOs are per tenant and rolled up fleet-wide. Windows are 28 days rolling. Error budgets follow the standard policy: when a budget is more than 50% burned, feature promotion for that component pauses; when exhausted, only reliability work ships.

SLI SLO Measurement Budget policy notes
Brief delivery 99.5% of scheduled briefs delivered inside the person's window (default 30 min) message.delivered for kind=brief inside window / briefs scheduled Same-day-but-late counts as failed; briefs suppressed by a correct rule (PTO) are excluded.
IC reply latency p50 ≤ 5 s, p95 ≤ 20 s from inbound message to first visible reply or acknowledgment Surface receipt minus inbound ts An acknowledgment counts only if the full reply lands within 90 s.
Cadence adherence 100% of cadence runs either complete or raise cadence.missed within 15 min of the slot; 99.5% complete on time Scheduler ledger The 100% is a hard invariant; a silent skip is SEV2 (RB-11).
Connector freshness 99% of connector-minutes inside the freshness SLO (comms 15 min, work 1 h, people/finance 24 h, atoms 15 min) Health sampler Vendor outages count; the customer experiences them.
Harness availability 99.9% Scheduler ticks + API success
Org Model availability 99.9% reads, 99.5% writes API success
Audit write success 100% (fail-closed) audit.write_failed = 0 Any failure halts execution (RB-09), so audit availability bounds harness availability.
Approval routing 99% of over-threshold items reach the inbox within 10 min of detection decision.proposed → inbox render
Escalation delivery 100% of ic.escalate delivered to the manager of record within 60 s escalation.ic_escalation_to_human → receipt Hard invariant (P2).

2. Golden signals per component

Component Latency Traffic Errors Saturation Custom signals
Harness scheduler tick-to-dispatch p95 runs dispatched/min dispatch failures, late dispatch queue depth by priority, workers busy % cadence.missed, run.stuck, runs per tenant vs baseline
Manager run (per Chonk) duration p50/p95 by run type runs/min by type failures by cause (model, policy, connector, budget) concurrent runs, context tokens p95 vs limit actions and messages per run, grounding rate, injection flags
Policy engine decision latency p99 decisions/min policy.error (must be ~0), rule timeouts rule-pack cache misses humanonly_blocked, hardline_hit, class-distribution drift
Function services solve time p95 by solver solves/min infeasible / timeout CPU, memory, queue constraint-violation attempts
Org Model store read/write p95 ops/s conflicts, RLS denials connections, replication lag version growth, dangling edges
Evidence store ingest lag records/min by connector classifier failures, drop-field violations index lag unknown-field rate, sensitive-lexicon hits
Memory store retrieval p95 retrievals/min integrity failures namespace size purge backlog, orphans
Connectors vendor call p95 calls/min by priority 4xx/5xx/429 by vendor budget utilization, breaker state health score, freshness age, dedupe rate, scope drift
Communication engine compose-to-send p95 messages/min by surface send failures, undelivered per-surface queue cap and quiet-hour suppressions, duplicates caught
Content engine render p95 renders/day render failures GPU queue grounding on rendered scripts
Audit writer append p99 entries/s write failures (page immediately) segment size chain verification, anchor lag
Surfaces end-to-end reply latency inbound/outbound per surface delivery errors webhook receiver backlog thread reply rate, brief open rate
Model gateway TTFT, total p95 by provider requests/min, tokens/min provider 5xx, 429, timeouts, refusals provider concurrency cost/min, cache hit rate, fallback usage

Metrics carry labels tenant_id, chonk, function, run_type, surface where applicable. Per-person labels are never emitted; per-person facts live in the audit log and the person's page.

3. Dashboards

3.1 Tenant health

One page per tenant, the first thing support opens.

Panel Content
Status strip Mode, graduation state per function (SHADOW/PROPOSE/AUTONOMOUS), kill switch state, manager of record, listening-period countdown if applicable
SLO tiles Brief delivery, reply latency, cadence adherence, connector freshness, with 28-day budget remaining
Connector health Score per connector, state, freshness age, last error, scope conformance
Cadences Next 24 h of scheduled runs; last 7 days run/late/missed
Escalations Open items in the approvals inbox with age; IC escalations to human (count, ack latency)
Trust budget Trust score per function (§3.2) with distance to the regression line and 7-day net delta
Outreach Messages/person/day vs cap; suppressions; one-tap feedback counts
Cost Spend per managed person MTD vs ceiling
Recent audit Last 50 decision.*, config.*, graduation.*, killswitch.set entries

3.2 Trust budget

The trust budget is defined in 07-closed-loop-risk-mitigation.md §6: a per-(tenant, function) trust score from 0 to 100, credited by verified decisions, unedited approvals, shadow agreement and "useful" taps, and debited by overrides (BAD_TIMING/TONE −1, WRONG_FACTS/WRONG_JUDGMENT −3), verification mismatches (−5) and attributed incidents (SEV3 −10, SEV2 −25, SEV1 → 0 and SHADOW). Advancement needs trust ≥ 20 (SHADOW → PROPOSE) and ≥ 60 (PROPOSE → AUTONOMOUS); regression fires below 40 and below 10. This dashboard is the operational view of that ledger; it does not define new deltas.

Signals raised by this doc's alerts are recorded as doc 07 events:

Operational signal Recorded as Δ
"didn't need this" tap on outreach override reject, reason BAD_TIMING −1
"this wasn't stuck" tap on a chase override, reason WRONG_FACTS −3
Wrong-person chase confirmed (RB-07) WRONG_FACTS plus verification mismatch −3, −5
Quiet-hours or cap violation (RB-14) verification mismatch −5
Duplicate outreach (RB-06) BAD_TIMING per duplicate −1
Inbox item marked noise BAD_TIMING −1
Incident attributed at review SEV3 / SEV2 / SEV1 −10 / −25 / → 0

Panels: trust score per function with the 10/40 regression lines and the 20/60 advancement lines; 7-day net delta; top debit sources; projected line-crossing date at the current net rate; days at the current level (the 14-day hysteresis countdown); fleet distribution by function, which is the product's behavior-improvement backlog. Alerts: projected to cross a regression line within 7 days → warning to trust on-call; within 48 h, or a single-day net debit ≥ 15 → page.

3.3 Outreach volume

Panel Content
Messages per person per day Histogram against caps; anyone at cap is listed
By manager and reason Brief, check-in, chase, feedback, ritual, quiet HR, escalation
Suppressions Cap, quiet hours, degraded surface, duplicate caught, TTL expired
Storm detector Messages/min per tenant vs 7-day baseline (RB-03)
Unnecessary-outreach rate 14-day rolling with one-tap breakdown
Reply and open rates Per surface
Quiet-hour heat map Send times in recipient-local time; anything inside quiet hours is red and should be empty

3.4 Cost per managed person

Panel Content
Spend per managed person per month Model, function services, content rendering, connectors; vs the 10%-of-price ceiling
By Chonk, sub-agent and run type planner / writer / chaser / analyst; brief, check-in, chase, plan, owner update, ritual, ask-anything, packet
Token profile Context tokens p50/p95 per run type; cache hit rate
Anomalies Tenants > 2× fleet median; run types > 2× their 7-day baseline (RB-04)
Provider mix Primary vs fallback usage; price per 1k tokens

4. Alerting and on-call

4.1 Severities

Severity Definition Response Examples
SEV1 Trust or safety breach, or the product is down for many tenants Page primary + secondary + trust on-call; incident commander within 15 min; exec notified HUMAN_ONLY executed; cross-tenant exposure; audit writer down fleet-wide; message storm in progress
SEV2 A tenant is materially wrong or silent Page primary; contain within 1 h Cadence skipped without escalation; wrong-person chase confirmed; tenant kill switch pulled; connector paused for a tenant's primary surface
SEV3 Degraded, contained, not urgent Ticket, business hours, fix within 5 days Connector degraded with fallback working; cost anomaly under 2×; stuck run auto-recovered
SEV4 Hygiene Backlog Dashboard gaps, noisy alert

4.2 Rotations

  • Harness on-call (primary/secondary, weekly, follow-the-sun across two regions): everything in this doc.
  • Trust on-call (weekly; product + eng pair): behavior incidents (RB-03, RB-06, RB-07, RB-14), trust pages, owner-misuse reviews.
  • Security on-call: 13-security-privacy-compliance.md §12.
  • Handoff is a daily 10-minute written note: open incidents, degraded tenants, budgets under 50%.

4.3 Alert catalogue

Alert Condition Sev Runbook
ConnectorFleetDown Median health for a connector across tenants < 40 for 10 min SEV2 RB-01
ConnectorTenantPaused A tenant's connector enters PAUSED SEV2 if primary surface, else SEV3 RB-01
ModelProviderErrors Provider 5xx/429/timeout rate > 20% for 5 min SEV2 (SEV1 if fallback also failing) RB-02
MessageStorm Tenant outbound > 5× 7-day baseline for 5 min, or any person receives > cap SEV1 RB-03
CostSpike Tenant spend rate > 3× baseline for 1 h, or fleet > 1.5× SEV2 RB-04
RunStuck Run age > 3× expected for its type, or > 15 min absolute SEV3 (SEV2 if brief/cadence) RB-05
DuplicateOutreach Two messages with same dedupeKey to same recipient within 24 h SEV2 RB-06
WrongPersonChase One-tap "this wasn't stuck" + "not my item" on a chase, or attribution mismatch detected post-send SEV2 RB-07
PolicyEngineError Any policy.error (not a deny) SEV2 (SEV1 if > 10/min) RB-08
AuditWriteFailure Any audit.write_failed SEV1 RB-09
KillSwitchEngaged killswitch.set to L1 or above for any tenant SEV2 (SEV1 at L3) RB-10
CadenceMissed A cadence slot passes with no run and no cadence.missed escalation within 15 min SEV2 RB-11
MemoryIntegrity Memory checksum job failure, orphan rate > 1%, or retrieval returns cross-person entries SEV1 RB-12
WebhookSigFailures Signature failure rate > 5% for 10 min on a receiver SEV3 (SEV2 if sustained 1 h) RB-13
QuietHoursViolation Any manager-initiated message inside recipient quiet hours, or over cap SEV2 RB-14
ReadBoundaryMismatch Nightly INV-READ-1 diff non-empty SEV2 RB-15
SLOBurn Any SLO burning at > 4× for 1 h SEV2 per SLI
TrustBudgetBurn Function trust projected to cross a regression line within 48 h, or single-day net debit ≥ 15 SEV2 (trust on-call) §3.2, 16-rollout-and-operations.md §3.4

Hygiene rules: every alert links a runbook; an alert that fires three times in a week without action is either fixed or deleted at the weekly incident review; alerts page on symptoms (a person did not get a brief), not causes (CPU high).

5. Runbooks

Template for every runbook: Symptoms → Detect → Contain → Diagnose → Correct → Verify → Learn → Comms. Containment comes before diagnosis. Every runbook ends with a learning step that changes a rule, test, alert, or persona; a runbook without one is incomplete.

flowchart LR
    S[Symptom-based alert] --> C[Contain: freeze, mute, degrade, or fail closed]
    C --> D[Diagnose from run trace and audit]
    D --> F[Correct: fix, roll back, reconcile]
    F --> V[Verify: SLI recovers, scenario replays green]
    V --> L[Learn: rule, test, alert, or persona changed]
    L --> T[Trust delta confirmed at incident review]
    T -. feeds .-> G[Graduation and regression, doc 16]

RB-01 Connector outage (vendor-side or tenant-side)

  • Symptoms: stale data on health views; briefs missing work items; connector state DEGRADED/PAUSED; fleet alert if vendor-wide.
  • Detect: ConnectorFleetDown, ConnectorTenantPaused, freshness SLO burn.
  • Contain: Confirm the degradation profile engaged (11-integrations-spec.md §6): stall clocks paused, stale-sensitive actions downgraded, briefs rerouted to fallback surfaces. If not, engage it manually: harness connector degrade --tenant T --connector C. Never resume chasing on stale data.
  • Diagnose: Fleet-wide → vendor status, our app-level quota, key rotation. Single tenant → token revoked (401), scope drift, webhook secret rotated, admin uninstalled. Read connector.state_changed.reason.
  • Correct: Vendor outage: wait, keep P3 paused, reconcile on recovery. Token/scope: send the admin the templated re-consent link. Our bug: hotfix through the rings (16-rollout-and-operations.md §5); connector fixes skip opt-in since they do not change manager behavior.
  • Verify: Health ≥ 90 for 15 min; reconcile complete; expired queued messages dropped, not sent; stall clocks resumed without backfilling.
  • Learn: If the profile did not engage automatically, add the signature to the health sampler; add the outage shape to the sim (FakeX.outage).
  • Comms: Tenant admin within 30 min for PAUSED; manager of record if > 50% of ICs lost their primary surface; status page if fleet-wide.

RB-02 Model provider outage

  • Symptoms: run failures with cause=model; reply latency SLO burn; briefs late.
  • Detect: ModelProviderErrors.
  • Contain: The gateway fails over automatically for run types whose prompts have passed live-model replay on the secondary provider; run types without a qualified fallback queue (briefs up to their window, chases up to 4 h) and the health strip shows "manager is slow today". Interactive threads send one honest acknowledgment ("I'm having trouble thinking right now; I'll reply within the hour"), not repeated ones.
  • Diagnose: Provider status; our rate-limit tier; a bad gateway deploy; token-limit changes.
  • Correct: Wait or fail over. Never raise retry aggressiveness (it worsens 429 storms). If the fallback produces behavioral diffs on canaries, restrict it to non-outbound run types.
  • Verify: Error rate < 2% for 15 min; queued briefs delivered in window or explicitly skipped with a note; cost dashboard shows the fallback premium.
  • Learn: Every run type must have a qualified fallback or an explicit queue policy; add the missing one. Record the outage shape in sim W9.
  • Comms: Status page; no tenant email unless briefs were skipped.

RB-03 Message storm

  • Symptoms: a person or a channel receives many messages in minutes; Slack rate limits; angry thread replies.
  • Detect: MessageStorm (the communication engine's storm breaker should trip first; the alert means it did or it failed).
  • Contain: Engage the tenant's outbound freeze: harness comms freeze --tenant T (keeps P2 IC-escalation deliveries and the kill notice only). Then engage per-person mute for affected recipients for 24 h. If fleet-wide, freeze fleet outbound.
  • Diagnose: Almost always a trigger loop (connector event → run → write → connector event), a cadence misconfigured to a minute schedule, a debounce regression, a sim persona script leaking into a real tenant, or a rule change that made everything look stalled. Follow run.trigger chains in the run graph.
  • Correct: Break the loop (idempotency key or debounce fix), revert the rule or config, delete the unsent queue. Then send one human-approved apology per affected person in the Chonk's voice with the count and the why; it counts as outreach and against the trust budget deliberately.
  • Verify: Outbound rate at baseline; no queued duplicates; storm scenario replays green.
  • Learn: Add the loop shape to the storm breaker; add a property test if it was a debounce/cap regression.
  • Comms: Tenant owner within 30 min with the count and the fix; affected people via the apology; incident review within 5 days.

RB-04 Cost spike

  • Symptoms: spend per managed person trending toward the ceiling; provider bill alert.
  • Detect: CostSpike; nightly cost gate in CI (G8).
  • Contain: Per-tenant cost breaker: above 3× baseline, non-interactive run types drop to the cheaper model tier and ask-anything gets a daily budget; interactive replies are never degraded silently.
  • Diagnose: Context bloat (a memory namespace without compaction), a retry loop, a new content render type, a tenant that tripled headcount, a provider price change, a prompt change that broke caching.
  • Correct: Fix the bloat or loop; restore caching; adjust the run-type budget; if it is legitimate growth, update the baseline.
  • Verify: Spend within 1.5× baseline for 24 h; cost per run type inside budget.
  • Learn: Add the pattern to the anomaly detector; if a prompt change caused it, G8 becomes blocking for that run type.
  • Comms: Internal unless the breaker degraded a tenant's service.

RB-05 Stuck run

  • Symptoms: a brief or cadence has not completed; a thread reply never came; run age alarm.
  • Detect: RunStuck.
  • Contain: The watchdog cancels runs at 3× expected duration and re-dispatches once with resume_from_checkpoint. If the second attempt sticks, the run is failed and, for a cadence, cadence.missed goes to the manager of record (the "never silently skip" invariant). Interactive threads get the honest acknowledgment.
  • Diagnose: A connector call with no deadline, an infeasible solve that does not time out, a looping rule, a hanging model call, a lock on the Org Model version. The run trace shows the last checkpoint.
  • Correct: Deadlines on every external call; solver timeouts with an infeasibility explanation; fix the rule.
  • Verify: Re-run from checkpoint succeeds; no duplicate side effects (idempotency keys held).
  • Learn: Any hang class without a deadline gets one; add a fault-injection scenario.
  • Comms: The affected person gets the acknowledgment; the manager of record gets the cadence.missed if applicable.

RB-06 Duplicate outreach

  • Symptoms: a person receives the same brief or chase twice.
  • Detect: DuplicateOutreach (dedupe key collision post-send, or one-tap "I got this twice").
  • Contain: Per-recipient mute for the message kind for 24 h; freeze the run type for the tenant if more than three recipients are affected.
  • Diagnose: A retry after a send that succeeded without an ack (vendor timeout), two scheduler instances dispatching the same slot, a debounce miss on a burst of connector events, or two managers generating the same chase (Chief Build Officer and Chief Operating Orca on the same blocker).
  • Correct: Idempotency on dedupeKey at the surface adapter, not only the engine; scheduler slot leases; cross-manager dedupe on (recipient, subjectEvidenceId, kind) within 24 h.
  • Verify: Replay the recorded run; assert one send. Sim W9 duplicate scenario green.
  • Learn: Add the path to the property "no two messages share a dedupe key". Trust: BAD_TIMING −1 per duplicate.
  • Comms: No apology for a single duplicate (that would be a third message); apology only at three or more.

RB-07 Wrong-person chase

  • Symptoms: someone was chased about work that is not theirs; "not my item" tap; a thread reply saying so.
  • Detect: WrongPersonChase; trust on-call reviews all "this wasn't stuck" taps daily.
  • Contain: Immediately reply in-thread (templated, human-approved within the hour): acknowledge, say what the manager got wrong, and that it has been corrected. Freeze chases for the tenant for the affected entity type until diagnosed.
  • Diagnose: Identity resolution error (EC-INT-003/004), attribution lag after an issue move (EC-INT-007), reviewer request to a bot or a former employee, a stale snapshot during degradation (RC-INT-2), or an assignee field that the work tool uses differently from our mapping.
  • Correct: Fix the mapping or resolution rule; re-run attribution; apologize once; if the person is a former employee, purge and check retention.
  • Verify: Attribution diff between the work tool and the Org Model is empty for the tenant; chase precision back above 0.9.
  • Learn: Scenario added; the mapping table in 11-integrations-spec.md §3 updated; trust: WRONG_FACTS −3 plus mismatch −5.
  • Comms: The person (in-thread); the manager of record if the person escalated.

RB-08 Policy engine returns an error

  • Symptoms: runs failing at the policy step; actions neither allowed nor denied.
  • Detect: PolicyEngineError. Note: a deny is not an error; an error is an exception, timeout, or malformed decision.
  • Contain: Fail closed. The Harness treats an error as deny for AUTONOMOUS and PROPOSE_CONFIRM (the action does not happen; the run records policy.error), and treats it as no packet for HUMAN_ONLY. IC escalation delivery does not depend on the policy engine and continues. If errors exceed 10/min, freeze outbound for the tenant and page.
  • Diagnose: A rule-pack deploy with a syntax or type error, a rule referencing a config field that a tenant lacks, a rule timeout on a large Org Model, a schema migration mismatch between the Harness and the engine.
  • Correct: Roll back the rule pack (16-rollout-and-operations.md §5 rollback is one command and takes effect on the next run); fix and re-promote through the rings.
  • Verify: policy.error = 0 for 30 min; replay strict on the affected runs shows correct decisions; the actions that were denied by error are re-evaluated and executed if still valid (each with a fresh receipt) or dropped if superseded.
  • Learn: Rule packs get a compile step and a schema-conformance test against every tenant config before ring 1; P9 determinism test extended with the failing input.
  • Comms: Internal unless briefs were missed.

RB-09 Audit log write failure

  • Symptoms: execution halts tenant-wide; runs complete with halted: audit_unavailable; dashboards show a red audit tile.
  • Detect: AuditWriteFailure (any single failure pages).
  • Contain: Already failed closed. Confirm reads, the "what my manager knows" page, and IC-escalation delivery still work (escalations go to a separate minimal ledger reconciled into the chain afterward). Confirm decision.executed is flat since the first failure.
  • Diagnose: Object-lock bucket permissions, sequencer leader loss, index disk full, clock skew making ts non-monotonic (the writer refuses), a hash mismatch from a corrupted prior segment.
  • Correct: Restore the writer; verify the chain from the last daily anchor; if there is a gap, write a signed audit.gap entry with range and reason (the chain never pretends). Replay halted runs from checkpoint with fresh policy evaluation.
  • Verify: Chain verifies end to end; executed count equals audit entries; halted runs recovered or dropped with reason.
  • Learn: Writer redundancy (two independent stores, both must succeed); monthly chaos test.
  • Comms: Tenant owners told the manager paused for N minutes and why, with the audit range; 13-security-privacy-compliance.md §12 if integrity was in question.

RB-10 Tenant kill switch pulled

  • Symptoms: KS-GLOBAL (07-closed-loop-risk-mitigation.md §7) raised to L1 (outreach freeze), L2 (write freeze) or L3 (full stop) for a tenant by an owner, manager of record, ops, or a breaker. The IC escalation path, audit log, "what my manager knows" page and read-only console stay up at every level.
  • Detect: KillSwitchEngaged (killswitch.set).
  • Contain: Already contained by definition. Confirm the level's guarantees hold (P12): at L1 no outbound beyond the allowed replies and safety alerts; at L2 no connector writes and no new inbox items; at L3 no runs. At L2+ confirm the plain-words notice reached every IC thread with the manager of record named.
  • Diagnose: Call whoever set it within 15 min; the reason is the incident. Classify: manager behavior (trust), data or integrity concern (security; L3 is automatic on audit-chain break, tenant-isolation failure, or HUMAN_ONLY execution), tenant business reason, or accidental.
  • Correct: Address the cause. Lowering follows doc 07: L1 clears after the cause is cleared plus a human ack (queued messages re-observed, anything older than 48 h withdrawn); L2 only by owner or ops with a recorded reason; L3 only by ops after the integrity check passes. No automated process lowers a level. Graduation after release reflects the trust ledger: an attributed SEV2 regresses the function to PROPOSE, a SEV1 to SHADOW.
  • Verify: killswitch.set to L0 audited with reason; withdrawn messages logged, not sent; first post-release cadence runs cleanly.
  • Learn: Every raise, including accidental, gets an incident review; the cause becomes a scenario; if a breaker should have tripped first, its threshold is tuned.
  • Comms: Owner and manager of record continuously; ICs via the level notice and a release notice.

RB-11 Cadence skipped

  • Symptoms: a standup recap, weekly review, owner update or huddle did not happen and nobody was told.
  • Detect: CadenceMissed (a scheduler slot passed with neither a run nor a cadence.missed escalation). This should be impossible; the alert exists because impossible things happen.
  • Contain: Raise cadence.missed to the manager of record manually with the slot and reason; if people wait on the output (a brief, a schedule), run it now with a "late, here's why" preface if still useful, otherwise skip explicitly.
  • Diagnose: Scheduler failover losing a slot lease, a tz/DST bug in slot computation, a cadence edited to an invalid schedule, a degradation profile that said "skip" without escalating (a bug), a stuck run that failed without the watchdog raising.
  • Correct: Fix; backfill missing slot records so adherence metrics are honest.
  • Verify: The next three slots run on time; sim DST scenarios green.
  • Learn: Slot computation gets a property test over tz and DST; degradation profiles are audited for any skip without escalate.
  • Comms: Manager of record; participants of the cadence with an honest one-liner.

RB-12 Memory store corruption

  • Symptoms: a manager "remembers" something wrong or about the wrong person; retrieval returns entries from another person's namespace; checksum job fails.
  • Detect: MemoryIntegrity; a person disputes a memory entry on their page; one-tap flags.
  • Contain: Freeze retrieval for the affected namespaces (runs proceed on evidence only, receipts marked memory_unavailable); if cross-person leakage is suspected, freeze tenant outbound and treat as SEV1 under 13-security-privacy-compliance.md §12.
  • Diagnose: A namespace key bug in a writer, an index rebuild with stale mappings, a partial purge, a compaction that merged entries, a backup restore that skipped tombstones (R-DEL-4).
  • Correct: Rebuild the namespaces from the evidence store and decision log (memory is derived state and must always be rebuildable); re-run purges; replay tombstones; verify checksums.
  • Verify: Integrity job green; sampled pages match evidence; INV-TEN-5 canary retrievals return nothing across namespaces.
  • Learn: Namespace assertion at the type level on every memory write; automated tombstone-replay check in the restore procedure; scenario added.
  • Comms: If memory leaked between people, both people and the owner are told per §12; otherwise internal plus the disputing person.

RB-13 Webhook signature failures / spoofing

  • Symptoms: receiver rejects a rising share of events; connector switches to polling; freshness dips.
  • Detect: WebhookSigFailures.
  • Contain: Nothing enters on a failed signature (by design); polling fallback engaged. Capture samples for security if payloads look like probes.
  • Diagnose: Vendor rotated the signing secret; ours rotated without redeploy; the signature scheme changed; someone is probing.
  • Correct: Rotate/refresh the secret via the vendor app config; redeploy; keep polling until push is clean for 30 min.
  • Verify: Failure rate < 0.1%; push resumed; no rejected event was processed (audit shows none).
  • Learn: Secret rotation becomes a scripted runbook per vendor; probing patterns feed the WAF rules; a scenario with rotated secrets in the fake.
  • Comms: Internal; security if probing.

RB-14 Quiet-hours or cap violation

  • Symptoms: someone got a message at 11pm or a sixth message in a day.
  • Detect: QuietHoursViolation from the policy audit (post-send check).
  • Contain: Per-person mute until their next working window; freeze the run type for the tenant if more than one person is affected.
  • Diagnose: tz missing or wrong on the person (defaulted to UTC), DST transition, a location tz applied instead of the person's for atoms, an urgent tag applied by a rule that should not have, a cap counter reset bug at midnight UTC instead of local, or a message sent by a fallback surface path that bypassed the engine.
  • Correct: Fix the tz source of record (11-integrations-spec.md §4.1: the person sets it; the connector's is a hint); all surfaces must go through the engine (no side paths).
  • Verify: P4 and P3 green with the failing input as a regression case; the quiet-hour heat map is empty inside quiet hours for 7 days.
  • Learn: Trust: verification mismatch −5; the specific path becomes a property-test generator case.
  • Comms: One apology in the person's next working window, not immediately (that would be another quiet-hours message).

RB-15 Read-boundary invariant mismatch (INV-READ-1)

  • Symptoms: nightly diff finds a run that read a source not listed on the person's page, or a page listing a source no run reads.
  • Detect: ReadBoundaryMismatch.
  • Contain: Freeze runs about the affected persons (EC-SEC-011); if the extra source is one the manager may never read (a DM, a private channel without invite), escalate to SEV1 under 13-security-privacy-compliance.md §12.
  • Diagnose: A connector entity not registered with the page renderer, a missing provenance stamp, a channel invite not propagated to member pages, or (worst) a scope regression.
  • Correct: Fix provenance or registration; purge out-of-boundary records and derived memory; regenerate any outputs that went out.
  • Verify: Diff empty for 3 nights; sim SC-SEC-011 green.
  • Learn: Page registration becomes part of the connector checklist gate (11-integrations-spec.md §10, already listed; add the CI check); RC-SEC-2 metric updated.
  • Comms: Per §12 if any never-read source was involved; otherwise internal.

6. Edge cases

ID Scenario Signal / detection Rule Action (autonomy class) Escalation Test
EC-OBS-001 Metrics pipeline is down; dashboards are blank but the product works. Metrics ingest lag > 10 min; heartbeat missing. Absence of metrics is an alert (dead-man switch). Page harness on-call; the Harness keeps a local ring buffer of the last hour of SLI events for reconstruction (AUTONOMOUS). SEV2. Kill the metrics sink in staging; assert page within 10 min.
EC-OBS-002 A tenant with 3 people trips the storm detector on a normal Monday (baseline too small). Storm alert with baselineSamples < 50. Minimum absolute thresholds for small tenants (10 msgs/min or cap breach), not multiples of baseline. Alert suppressed; absolute rule applies (AUTONOMOUS). None. Sim with a 3-person tenant; assert no false storm.
EC-OBS-003 DST transition doubles or skips an hour; cadence adherence shows a missed slot. Slot computed in UTC differs from local expectation. Slots are computed in the cadence's tz with explicit DST rules; the adherence job uses the same function. No false cadence.missed (AUTONOMOUS). None. Property test over tz/DST; W7 Airport scenario.
EC-OBS-004 A trust debit is applied twice (one-tap plus incident review for the same event). Same eventId debited twice. Debits are idempotent on eventId; incident review replaces, not adds, one-tap debits for the same event. Dedupe (AUTONOMOUS). None. Unit test on the ledger.
EC-OBS-006 Fleet-wide alert fatigue: 200 tenants each page on the same vendor outage. Correlated ConnectorTenantPaused within 5 min. Alerts group by connector when > 10 tenants fire within 5 min; one page. Grouping (AUTONOMOUS). None. Simulate 50 tenant pauses; assert one page.
EC-OBS-008 Per-person metric labels sneak into the metrics store via a new counter. Cardinality lint finds person_id label. No per-person labels in metrics (§2). CI fails (AUTONOMOUS). None. Lint test.

7. Risk controls

RC-OBS-1 — Silent failure (the system is wrong and no alert fires). Owner: Chief Build Officer. Metric: incidents detected by a human before an alert (target: < 10% of incidents). - Detect: every incident review records "who noticed first"; dead-man switches on metrics, scheduler, audit anchors. - Contain: add a temporary alert on the symptom the human saw. - Correct: permanent symptom-based alert; SLI added if missing. - Learn: the "human noticed first" rate is reviewed monthly; alerts that never fired in 90 days are reviewed for relevance.

RC-OBS-2 — Alert fatigue. Owner: Chief Build Officer. Metric: pages per on-call week (target ≤ 5); actionable-page rate ≥ 80%. - Detect: weekly page count and "was this actionable" tag on every page. - Contain: group correlated alerts; raise thresholds temporarily with an expiry. - Correct: fix noisy conditions; delete alerts that fire without action three times. - Learn: alert catalogue review at the weekly incident review.

RC-OBS-3 — Runbook rot. Owner: Chief Build Officer. Metric: runbooks exercised in the last quarter (target 100% via chaos or real incidents). - Detect: runbook last-exercised date; chaos schedule. - Contain: on-call pairs with the runbook author when a stale runbook is invoked. - Correct: monthly chaos day exercises three runbooks in staging with the sim org. - Learn: every exercise produces a diff to the runbook or a note that it held.

Open questions

  1. Should the trust budget be visible to tenant owners in the Owner Console? Showing it builds trust but invites gaming of the one-tap signals.
  2. Fleet-level vs per-tenant SLOs: a 99.5% fleet brief SLO can hide one tenant at 80%. Proposal: page on any tenant below 95% regardless of fleet.
  3. The IC-escalation ledger used during audit outages (RB-09) is a second write path; it must be as trustworthy as the chain. Is that acceptable, or does escalation delivery need its own anchor?
  4. Should provider failover for outbound run types require live-model replay parity on the fallback, or is strict parity on structured outputs enough?