Day 36

Isolation, keys and scale thinking

You ship
A hardened, optionally single-tenant deployment with runbooks, a game day and a completed security questionnaire.

What you'll build today

After Day 35, WorkBot's infrastructure is code. One pipeline can build it from nothing on three targets, release it, test it and roll it back. A regulated customer, such as a dental group whose conversations carry patient data, asks for more than a shared deployment. It wants its data kept apart from other customers and from the internet, keys it can account for, and proof that the system holds up under growth and failure.

Today you harden the deployment for that customer. You split the network into tiers with a single way out, and you keep the encryption keys in a key management service. A customer who needs it gets a stack of its own from one more variable. Then you push a deployment until it breaks, fix what broke, and rehearse an outage. You learn how isolation works layer by layer, how envelope encryption works, and how to find bottlenecks one at a time.

By the end of the day, WorkBot has a hardened deployment, runbooks for its likely failures, and true answers to a customer's security questionnaire.

By the end of the day you can

  • Design network isolation for an agent system that handles customer data, and explain it to a security team
  • Run key management for the infrastructure and for the agent's credentials
  • Plan capacity and failures for 10 times the current load, without rebuilding

Topics

Infrastructure Isolation

  • Tiers: a public tier for the load balancer only, a private tier for services, and an isolated tier for data. Build them from VPCs (virtual private clouds), subnets and security groups
  • Private connectivity: VPC endpoints to cloud services, and private links or VPNs (virtual private networks) to customer systems. No database is reachable from the internet
  • Single-tenant or multi-tenant. A tenant is how WorkBot keeps one customer's data and settings apart. A regulated customer gets its own account or VPC. The rest share infrastructure, kept apart as on Day 30
  • Egress control, over the calls that leave the system: an agent that can call the internet is a path for data to leave. Control it with allowlists and a proxy that logs every request

Key Management

  • A cloud key management service (KMS) for encryption keys, with rotation schedules
  • Envelope encryption for data at rest: each data key is itself encrypted by a key held in KMS
  • A secrets manager for application secrets. The app reads them at runtime through an IAM (identity and access management) role, never from a file in the image
  • Agent credentials (Day 30): encrypted with a tenant key, with every access logged
  • Audit: who read which secret, and when, with alerts on anomalies

Data Separation at the Infrastructure Level

  • Per tenant, by tier: storage prefixes, databases or schemas, with backups and restores per tenant
  • Residency, the region where data is kept: provider endpoints per region, enforced by configuration
  • Logs and traces: redaction (Day 30), retention, and deletion per tenant on request

Scale Thinking

  • Bottlenecks, in order: provider rate limits, queue depth, database connections, worker memory, egress cost
  • Capacity arithmetic: requests per second × tokens per request, against the provider's limits. Then connections per worker, against the database's limits
  • Failure planning: a provider down for an hour, a tenant that imports a million records, a worker in a loop. Write a runbook for each
  • Observability: saturation metrics for each bottleneck, with alerts on the trend as well as on the threshold
  • 10-times tests: double the load until something breaks, fix the cheapest thing, and repeat

Build

Features shipped today

  • WorkBot: The single-tenant option for regulated customers. KMS keys, envelope encryption and audited secrets. Data residency enforced by configuration, and egress control on the agent

Harden the deployment for a regulated customer:

  1. Network: 3 tiers in the VPC, with no public database. Add VPC endpoints for cloud services, and an egress proxy that allowlists and logs the agent's outbound calls
  2. Keys: KMS keys per environment, and envelope encryption for tenant data and agent credentials. Configure rotation, and audit every access to a secret
  3. Single-tenant option: a Terraform variable that deploys an isolated stack for one tenant. Deploy one
  4. Scale: a capacity table for 10x current load. Load test to the first bottleneck, fix it, and repeat twice. Record what broke each time
  5. Runbooks for 3 failures: a provider outage, a runaway tenant import and a looping worker. Run one as a game day, a planned rehearsal of the failure
  6. Security questionnaire: answer the 20 most common questions from a standard vendor questionnaire, for WorkBot

Step by step

Network tiers and one way out

Isolation starts with the network. The data should sit where the internet cannot reach it, and the agent, which fetches pages for people, needs a single way out that you can watch.

Study first

  • Learn the three tiers. The edge takes traffic from the internet, the app tier is reached only from the edge, and the data tier is reached only from the app. The curriculum calls them the public, private and isolated tiers. Private routes connect the app to the cloud's own services, so those calls never cross the internet. AWS's guidance on network layers draws the same three tiers, with a control at each boundary that admits only the traffic the next tier needs.
  • Learn what egress control means: control over the calls that leave the system. An agent that can fetch any page is the easiest path out for a customer's data. A forward proxy closes that path, with an allowlist of hosts and a log of every request. Squid's access control FAQ shows how its rules are read. They run from top to bottom, the first match decides, and the list should end by denying everything else.

Build

  1. Add an egress proxy in infra/egress. It runs Squid with an allowlist of the services WorkBot calls, and nothing else. When a customer wants the agent to read a page, add that page's host to the allowlist for that customer.
  2. Send every outbound call through the proxy. Set its address in HTTPS_PROXY, and Node's fetch honours it once NODE_USE_ENV_PROXY=1 is set, as Node's proxy support describes. Point the agent's browser at the proxy directly as well.

Done when

  • An allowed host answers through the proxy, any other host is refused with 403, and both requests appear in the log.
Isolation
In the reference run, the model's API answers through the proxy and example.com is refused, both logged. A pod in the shared namespace cannot reach Harbor Dental's database, while Harbor's own pod can.
Technical notes
  • Squid drops its privileges before it opens its logs, so it cannot write to standard output and needs a log file of its own.
  • On your machine the proxy is an app setting. In a virtual private cloud (VPC), the private network the deployment runs in, the proxy is the only route out.

Encryption keys and access logs

With the network closed, the next question a security team asks is who holds the keys. A key management service (KMS) keeps the main key where no one can copy it out, encrypts the data and the secrets under it, and records every use.

Study first

  • Read how envelope encryption works. Each piece of data is encrypted with a data key of its own, and each data key is encrypted with a key that never leaves KMS. Then read about customer-managed keys, key rotation, and the audit logs that record every use of a key.

Build

  1. In infra/gcp, create one KMS key per environment, rotated every 90 days, and encrypt Cloud SQL and every Secret Manager secret under it. The agent's credentials from Day 30 are sealed with SECRETS_KEY, which is a secret too, so the same key covers them.
  2. Turn on data-access logs for Secret Manager and KMS, because they are off by default.
  3. Extend the Day 35 terraform test, which plans against a mocked provider. Check that rotation is set, that the database and the secrets are encrypted under the key, and that both logs are on.

Done when

  • The plan shows the database and the secrets encrypted under the environment's key, and every read of either is logged.

A stack for one customer

Row-level security already keeps customers apart inside one database. Some customers need more than that, so the next step is a stack of their own, made from the same code with one more variable.

Study first

  • Compare the ways to keep a tenant apart, where a tenant is one customer's data and settings inside WorkBot. You can use a tenant column with row-level security (Day 30), a namespace and database per tenant, or a project per tenant. Know what each one costs to run.
  • Read Kubernetes' page on multi-tenancy, which weighs a shared cluster against a cluster per tenant. You should be able to say why a namespace alone does not keep tenants apart, since every pod can reach every other until a network policy says otherwise.

Build

  1. On Google Cloud, write a variable file, infra/gcp/harbor-dental.tfvars. It gives Harbor Dental, the second customer from Day 29, its own project, database, key and host.
  2. On the cluster, add a tenant in infra/kubernetes, with a namespace that holds its own secrets and Postgres. Give the namespace a network policy that admits traffic only from the namespace itself, the gateway and KEDA (Kubernetes Event-driven Autoscaling), the autoscaler from Day 35. Add moved blocks so that Terraform carries the existing state over to the new resource addresses, as the moved block reference explains.
  3. Deploy Harbor Dental's stack on its own host, then try to reach its database from the shared namespace.

Done when

  • The shared namespace cannot connect to the tenant's database, and the tenant's own pods can.

Capacity for ten times the load

An isolated deployment still has to carry the customer's growth. So you plan for ten times today's users on paper first, then load a running deployment until something breaks, and fix one thing at a time.

Study first

  • Learn capacity planning from the volumes in the Day 34 cost model, such as requests, steps, connections and rate limits. Then learn load testing to the first bottleneck, with one fix per round so that each round shows what its fix changed.
  • Read the testing for cascading failures section of Google's Site Reliability Engineering book, on load testing past the breaking point. Then read Brendan Gregg's method of checking utilization, saturation and errors for every resource. You should be able to say which resource gave out first in each round, and how you know.

Build

  1. Write a capacity table for ten times today's users in docs/isolation.md.
  2. Add pnpm bench, a load test whose workers create and list requests on a running deployment.
  3. Raise the load until something breaks, fix it, and run again, writing down each round.

Done when

  • Each round names what broke and what fixed it, with the numbers before and after.
Bench
The reference run's bench ran four rounds. An unbounded list broke first, then a scaler blind to CPU, then the schema's data definition language (DDL) and a missing index under scale-out, and last the machine's own limit.
Technical notes
  • Cap every list. In the reference run, the list returned all 10,000 rows, 1 MB a call, and returning the newest 100 multiplied throughput by six.
  • Scale on CPU as well as on the queue. KEDA scaled the web tier only on the queue, so one replica took every request until a CPU trigger beside the queue's trigger fixed that. The CPU trigger reads from metrics-server, so install it where the cluster has none.
  • Apply the schema once per version, not at every start. After the first two fixes, four replicas were slower than one, because each new replica re-ran the schema's alter table statements. Those statements take exclusive locks, and they deadlocked with inserts. The list also sorted the table, because it had no index. The schema now runs once per version, recorded in the database.
  • The last round's limit is the machine itself, since one 4-CPU node runs the bench, Postgres and every replica.

Runbooks and a rehearsed outage

Load tests show where the system bends, and runbooks say what to do when it does. Rehearsing one of them now means the first real outage is not the first time anyone has followed it.

Study first

  • Learn what a runbook holds. It lists the signs, the first checks and the actions, and it says how to know the incident is over. Google's Site Reliability Workbook calls these playbooks and asks for one per alert.
  • Read the test-induced emergency in Google's Site Reliability Engineering book, a planned failure test that went wrong. You should be able to say what a rehearsal finds that a written runbook cannot. In that test, it was a hidden dependency and a rollback nobody had tried.

Build

  1. Write runbooks in docs/isolation.md for a provider outage, a runaway tenant import and a looping worker.
  2. Rehearse the outage as a game day, which is a planned rehearsal of a failure. Mark the small model as down with OUTAGE (Day 24), then run the chat smoke suite from Day 25. The smoke suite is a small set of chat evals, and an eval is a scored test of WorkBot's output.

Done when

  • The rehearsal shows what the fallback does to quality, cost and latency.
Game day
In the reference run, with the small model out, the large model answered. It scored 0.99 against the 0.94 baseline, at about 22 times the cost per answer.

The security questionnaire

Everything built today ends up as answers in a vendor review, the security review a customer runs before it buys. Writing those answers against the code shows where a promise would get ahead of what is built.

Study first

  • Read the Cloud Security Alliance's page on its controls matrix and questionnaire, whose yes-or-no questions for cloud providers cover 17 domains. Map your 20 questions to those domains, such as key management, logging and incident management.
  • Read the technical safeguards that the Health Insurance Portability and Accountability Act (HIPAA) sets for patient data. You should be able to say which of them WorkBot meets today, and where a reviewer can check each one.

Build

  1. Write docs/questionnaire.md, answering the 20 questions a vendor review asks most, each for WorkBot as built. Point each answer at the place where a reviewer can check it.

Done when

  • Every answer is true of the code today, and none promises what is not built.

Resources