Skip to content

Production checklist

Use this list as a release gate, adapting it to the application’s threat model and availability requirements.

Protocol and application

  • [ ] Route IDs are unique, stable, and >= 100.
  • [ ] Protobuf changes are backward compatible across the rollout window.
  • [ ] Client retries reuse request IDs and have bounded exponential backoff.
  • [ ] Handler timeouts, idempotency, and transaction behavior are tested.
  • [ ] Load tests cover expected concurrency, payloads, reconnects, and pushes.

Secrets and network

  • [ ] Ticket key, internal token, and admin token are unique values of at least 32 bytes.
  • [ ] Secret rotation and rollback procedures are documented.
  • [ ] Only the Gateway client listener is public.
  • [ ] World and admin listeners are isolated by firewall/NetworkPolicy.
  • [ ] Public TLS and internal TLS/mTLS match the threat model.
  • [ ] Proxy Protocol trusts only known load-balancer CIDRs.

Distributed correctness

  • [ ] Multiple Gateways share a replay store.
  • [ ] Duplicate-login policy has the required online and session-control transports.
  • [ ] Online dashboards distinguish authenticated Sessions from users deduplicated by user_id.
  • [ ] Durable presence projections reconcile against lease expiry instead of relying only on SessionObserver.
  • [ ] Cross-Gateway push is configured when business logic requires it.
  • [ ] Discovery removes unready/terminated Worlds promptly.
  • [ ] Stateful Worlds use an ownership and epoch strategy.
  • [ ] Redis Cluster key prefixes/hash tags are compatible with required operations.
  • [ ] Outbox dispatchers run, drain, retry, and expose health independently.

Capacity and resilience

  • [ ] Gateway connection, per-IP, queue, rate, payload, and timeout limits are tuned.
  • [ ] World connection and in-flight limits match Gateway pool capacity.
  • [ ] Circuit-breaker thresholds are exercised in failure tests.
  • [ ] Resource requests/limits and autoscaling signals are based on load tests.
  • [ ] Termination grace periods exceed drain/shutdown timeouts.
  • [ ] PodDisruptionBudgets match the actual replica count.

Realtime simulation

  • [ ] Input target-Tick, redundancy, reorder, and replay windows are bounded and tested.
  • [ ] Scene ownership prevents the same logical match from simulating on two Worlds.
  • [ ] Fixed-step catch-up limits and overload behavior are explicit.
  • [ ] AOI and replication streams are partitioned by observer and scene rather than globally.
  • [ ] Prediction, interpolation, replication, and rewind capacities are sized from measured RTT and jitter.
  • [ ] Prediction and lag-compensation snapshots contain only simulation or collision data.
  • [ ] Weak-network tests cover loss, jitter, reordering, duplication, queue pressure, and constrained bandwidth.
  • [ ] Historical hit validation cannot mutate live state before the result is accepted.

Operations

  • [ ] Health/readiness/version endpoints are monitored.
  • [ ] Prometheus metrics are scraped with admin authentication where required.
  • [ ] Alerts cover readiness, failure rates, saturation, and open circuits.
  • [ ] Logs include trace, request, route, session, and instance identifiers without secrets.
  • [ ] Admin mutations are available only through an audited operator path.
  • [ ] Dashboards and incident runbooks have been validated in a game-day exercise.

Released under the MIT License.