> ## Documentation Index
> Fetch the complete documentation index at: https://docs.payfi.global/llms.txt
> Use this file to discover all available pages before exploring further.

# Production evaluation observability

> Inspect redacted pipeline signals, latency, retention, and the two-hour release observation window.

This runbook applies only to Railway `production`
(`b7270b25-8992-4b89-b34d-a88fe40d1a9e`). Use structured API and worker logs
together with the organization-scoped `operation_processing_events` timeline.

## Safe operational events

The worker emits `payfi.pipeline.stage_completed` after a terminal stage event
is persisted. Safe fields include scoped IDs, stage, status, duration, workflow
and execution IDs, contract version, and version identifiers from immutable
provenance.

Trust-boundary failures and abstentions use stage-specific signals:

| Condition                     | Events                                                                                                                                                                                |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Schema or non-JSON response   | `payfi.requirements.schema_failure`, `payfi.document.schema_failure`, `payfi.decision.schema_failure`                                                                                 |
| Other stage failure           | `payfi.requirements.stage_error`, `payfi.document.stage_error`, `payfi.decision.stage_error`                                                                                          |
| Partial or untrusted result   | `payfi.requirements.degraded`, `payfi.document.degraded`, `payfi.decision.degraded`                                                                                                   |
| Low-confidence critical field | `payfi.document.low_confidence_critical`                                                                                                                                              |
| Activation gate blocked       | `payfi.analysis_v2.gate_blocked`                                                                                                                                                      |
| Raw-object cleanup failed     | `payfi.raw_document.deletion_failed` for a document deletion, or `payfi.raw_document.cleanup_failed` when the expiry reconciler itself fails; timeline `raw_document.deletion_failed` |

Logs and new timeline rows must not contain document bytes or text, prompts,
model responses, original file names, document hashes, signed URLs, secrets,
signatures, authorization headers, account data, beneficiary names, reviewer
notes, or arbitrary raw input/output/error payloads.

## Railway queries

Use the Railway Log Explorer for the production services. Example filters:

```text theme={null}
@event:payfi.pipeline.stage_completed
```

```text theme={null}
(@event:payfi.requirements.schema_failure OR @event:payfi.document.schema_failure OR @event:payfi.decision.schema_failure)
```

```text theme={null}
(@event:payfi.requirements.degraded OR @event:payfi.document.degraded OR @event:payfi.decision.degraded OR @event:payfi.document.low_confidence_critical)
```

```text theme={null}
@event:payfi.pipeline.stage_completed AND @durationMs:>90000
```

```text theme={null}
@event:payfi.raw_document.deletion_failed

@event:payfi.raw_document.cleanup_failed
```

Do not put customer names, file hashes, document contents, account data, or
other personal data into log searches or release reports.

## P95 from the durable timeline

```sql theme={null}
SELECT
  "stageKey",
  count(*) AS samples,
  round(
    percentile_cont(0.95) WITHIN GROUP (ORDER BY "durationMs")::numeric,
    0
  )::bigint AS p95_ms
FROM operation_processing_events
WHERE "createdAt" >= now() - interval '2 hours'
  AND "durationMs" IS NOT NULL
  AND status NOT IN ('PENDING', 'RUNNING')
GROUP BY "stageKey"
ORDER BY p95_ms DESC;
```

Scope drill-down queries by both organization and operation:

```sql theme={null}
SELECT
  "createdAt",
  source,
  "stageKey",
  status,
  "durationMs",
  "documentId",
  "analysisJobId",
  "n8nWorkflowId",
  "n8nExecutionId",
  "versionMetadata"
FROM operation_processing_events
WHERE "organizationId" = 'org_example'
  AND "operationId" = 'op_example'
ORDER BY sequence, "createdAt";
```

Record the environment ID, UTC window, activation set, application commits,
workflow active version IDs and hashes, sample counts, query result, and
reviewer approvals. A percentile without its sample size and activation set is
not release evidence.

## Retention evidence

n8n production workflows persist no successful, failed, or manual execution
payloads. Correlation metadata and terminal status live in the redacted PayFi
timeline.

Raw operation objects are deleted immediately after a trusted success or a
terminal failure. During an eligible retry or a pending human review they may
remain for at most 24 hours. For each release run, verify both the absence of
expired objects and the audit outcome of the reconciler. A cleanup error keeps
the release blocked without affecting commercial risk scoring.

## Two-hour observation

After the 20-operation gate, observe at least:

* Schema, provenance, policy, corpus, and activation-set failures.
* Partial, uncertain, conflicted, and human-review dispositions.
* Dependency timeouts and retry exhaustion.
* Duplicate/replay/callback/nonce concurrency.
* Raw-object cleanup failures.
* Cross-tenant denials and unexpected authorization outcomes.
* Stage sample counts, error rate, and P95.

The sanitized release manifest remains `PENDING_OBSERVATION` until the full
window ends and the alerting backend has been verified.

Grafana authentication is a release prerequisite. If the connector returns
`401`, record the observation as unavailable and keep the release blocked; do
not substitute the presence of Railway logs for verified alerts.