Skip to main content
This runbook applies only to Railway production (b7270b25-8992-4b89-b34d-a88fe40d1a9e). Use structured API and worker logs together with the organization-scoped operation_processing_events timeline.

Safe operational events

The worker emits payfi.pipeline.stage_completed after a terminal stage event is persisted. Safe fields include scoped IDs, stage, status, duration, workflow and execution IDs, contract version, and version identifiers from immutable provenance. Trust-boundary failures and abstentions use stage-specific signals: Logs and new timeline rows must not contain document bytes or text, prompts, model responses, original file names, document hashes, signed URLs, secrets, signatures, authorization headers, account data, beneficiary names, reviewer notes, or arbitrary raw input/output/error payloads.

Railway queries

Use the Railway Log Explorer for the production services. Example filters:
Do not put customer names, file hashes, document contents, account data, or other personal data into log searches or release reports.

P95 from the durable timeline

Scope drill-down queries by both organization and operation:
Record the environment ID, UTC window, activation set, application commits, workflow active version IDs and hashes, sample counts, query result, and reviewer approvals. A percentile without its sample size and activation set is not release evidence.

Retention evidence

n8n production workflows persist no successful, failed, or manual execution payloads. Correlation metadata and terminal status live in the redacted PayFi timeline. Raw operation objects are deleted immediately after a trusted success or a terminal failure. During an eligible retry or a pending human review they may remain for at most 24 hours. For each release run, verify both the absence of expired objects and the audit outcome of the reconciler. A cleanup error keeps the release blocked without affecting commercial risk scoring.

Two-hour observation

After the 20-operation gate, observe at least:
  • Schema, provenance, policy, corpus, and activation-set failures.
  • Partial, uncertain, conflicted, and human-review dispositions.
  • Dependency timeouts and retry exhaustion.
  • Duplicate/replay/callback/nonce concurrency.
  • Raw-object cleanup failures.
  • Cross-tenant denials and unexpected authorization outcomes.
  • Stage sample counts, error rate, and P95.
The sanitized release manifest remains PENDING_OBSERVATION until the full window ends and the alerting backend has been verified. Grafana authentication is a release prerequisite. If the connector returns 401, record the observation as unavailable and keep the release blocked; do not substitute the presence of Railway logs for verified alerts.