Skip to main content

Documentation Index

Fetch the complete documentation index at: https://internal.september.wtf/llms.txt

Use this file to discover all available pages before exploring further.

bap-engine handles two kinds of secrets: platform API keys (used by products to authenticate to the orchestrator) and engine API keys (generated for each provisioned engine, used by the product to authenticate to that engine). Plus the master key that encrypts engine keys at rest. This page covers what each secret is, where it lives, and how to rotate it.

​
The keys

​
Master key (ORCH_MASTER_KEY)

The most critical secret in the whole system. A Fernet key (base64-encoded 32 bytes) used to encrypt engine API keys at rest. Without it, no engine API key can be decrypted; products lose access to every engine they admit.
  • Format: Fernet.generate_key().decode() (43 chars, URL-safe).
  • Where it lives: ORCH_MASTER_KEY env var on the orchestrator. Should be injected at runtime from your secret manager.
  • Where it doesn’t: Postgres, logs, the orchestrator’s filesystem.
  • Rotation: Heavy operation. Requires re-encrypting every engine’s engine_key_enc. There’s no built-in rotation script today; doing it requires a custom migration.
If ORCH_MASTER_KEY ever leaks, every plaintext engine API key the orchestrator could decrypt is now exposed. Treat it like a database master password — store in a secret manager, rotate on schedule, audit access.

​
Admin key (ORCH_ADMIN_KEY)

Used by POST /products/register and PUT /products/{product_id}/policy. Without it, you can’t onboard new products.
  • Format: any opaque string (we use secrets.token_urlsafe(48)).
  • Where it lives: ORCH_ADMIN_KEY env var on the orchestrator.
  • How it’s checked: plain string comparison against the X-Admin-Key header.
  • Rotation: trivial. Generate a new value, swap the env var, restart the orchestrator. Tooling that uses the old value gets INVALID_KEY; update them.

​
Platform API key (per product)

Generated by the orchestrator at /products/register. The product uses it on every request as X-Platform-Key.
  • Format: pk-sept-<URL-safe random> (42+ chars).
  • Where it lives: products.api_key_hash (SHA-256). The plaintext is returned once at registration and never stored.
  • Rotation: today, no built-in rotation endpoint. To rotate, generate a new key (re-register the product) and update the consumer. Future versions will add POST /products/{id}/rotate-key.

​
Engine API key (per engine)

Generated by the orchestrator at provision time. The product uses it to authenticate to the specific engine via X-Engine-Key.
  • Format: sk-sept-<URL-safe random> (42+ chars).
  • Where it lives:
    • engines.engine_key_hash — SHA-256, stored in Postgres.
    • engines.engine_key_enc — Fernet-encrypted plaintext, stored in Postgres. The orchestrator decrypts when admitting the same user later.
    • The engine’s ENGINE_KEY_HASH env var — the engine validates incoming X-Engine-Key against this.
  • Rotation: POST /engines/{user_id}/rotate-key. Generates a fresh key, updates both forms in Postgres, swaps the engine’s hash, returns the new plaintext.

​
What’s never stored

  • Plaintext platform API keys.
  • Plaintext engine API keys (the encrypted form is stored, but plaintext only exists in flight: at provision/admit response, and at the engine’s ENGINE_KEY_HASH validation step).
  • ORCH_MASTER_KEY itself — read from the env at startup.

​
The auth flow end-to-end

​
What the orchestrator does NOT enforce

  • Per-user authentication beyond the platform key. The orchestrator trusts that the product authenticated the user upstream. From the orchestrator’s perspective, user_id is just an identifier.
  • Encryption of in-flight network traffic. TLS belongs at the reverse proxy upstream of the orchestrator. The orchestrator itself serves HTTP.
  • DDoS protection. Rate limiting at the policy layer is per-product, not per-IP. Layer 7 protection should be upstream.
  • MCP credential security. The engine handles per-user OAuth and Fernet-encrypts MCP credentials at the engine layer using a separate AD_ENCRYPTION_KEY. The orchestrator doesn’t see them.

​
Trust boundaries

The boundaries:
  • Public ↔ TLS terminator. TLS handshake. Standard.
  • Product ↔ orchestrator. Internal network. Plaintext platform key on the wire (HTTP), but only inside the trusted network. Run on a VPC or internal subnet.
  • Orchestrator ↔ Postgres. Internal. Use Postgres TLS if you cross VPCs.
  • Orchestrator ↔ engine container. Internal Docker network. Plaintext engine key on the wire. Never expose engine ports beyond the host.

​
Rotation playbook

​
Rotate ORCH_MASTER_KEY

This requires downtime and is heavy. Don’t rotate unless you must.
  1. Schedule maintenance window.
  2. Take a Postgres snapshot.
  3. Stop the orchestrator.
  4. Run a one-off script that:
    • Reads every engines.engine_key_enc row.
    • Decrypts with old ORCH_MASTER_KEY.
    • Re-encrypts with new key.
    • Writes back.
  5. Update ORCH_MASTER_KEY env on the orchestrator.
  6. Start the orchestrator.
  7. Verify by admitting a known user; confirm engine key still works.
If you rotate without re-encrypting, every product that admits an existing user gets back an undecryptable key, and the engine’s ENGINE_KEY_HASH won’t match — every existing engine is effectively locked. Provision new ones; the old ones are dead weight.

​
Rotate ORCH_ADMIN_KEY

  1. Generate new value.
  2. Update env on the orchestrator.
  3. Restart.
  4. Update tooling that calls /products/register or /policy.
Trivial.

​
Rotate a platform API key

Today, register the product again with a new slug or update the plaintext secret out-of-band. Future: a dedicated rotation endpoint.

​
Rotate an engine API key

curl -X POST "$ORCH_URL/engines/{user_id}/rotate-key" \
  -H "X-Platform-Key: $PLATFORM_KEY"
Returns the new plaintext. Old key invalid immediately. The product must update its cached key.

​
What goes wrong

SymptomLikely cause
INVALID_PLATFORM_KEY for known productsPostgres restored from a backup older than the registration. Re-register.
All engine keys reject after restartORCH_MASTER_KEY changed without re-encrypt.
Product gets a fresh key on every admitIgnoring the cached key returned by previous admit. Cache and reuse.
Engine returns INVALID_KEY despite using the orchestrator’s keyThe engine’s ENGINE_KEY_HASH env var is stale (engine restarted before the orchestrator pushed the new hash). Re-rotate.

​
See also