Home / Stories / LLM egress & profiles
Story · Data boundary Live

Your hostnames stay yours. The model still does the work.

Some of your infrastructure can go to any model. Some of it may only leave with its identifiers masked. Some of it must never leave your own hardware. The assistant treats "where does this prompt go" as a declared decision per call, not an accident of configuration, and lets you pick a different model for each class of problem.

Starts fromAny LLM call the assistant makes: triage, drafting a change, reviewing a merge request.
Looks atThe assembled prompt, the declared class of the destination, and the strictest egress requirement of your tenants.
You getCalls that go only where you allowed, with identifiers masked where required and restored on the way back, and a record of each decision.
01 · One door out

Every model call leaves through the same engine.

The assistant has many agents but a single path to a language model. Each call carries a label naming the kind of work (review_mr, analyse_manifest, chart_upgrade…), and that label decides which profile, and so which endpoint, it goes to.

A test suite pins that door: a new place in the code that talks to a model provider directly fails the build until it is classified.

02 · Classify what is in the prompt

Redact, pseudonymise, or pass. Decided by rules, not by a model.

Before anything leaves, the prompt is sorted into data classes by a deterministic rule pack. No model classifies it, because the payload would have to leave first.

  • Redact: secret material, by a separate, older secret scrub on the paths that can carry it, backed by a fail-closed check on every write to your forge.
  • Pseudonymise: your company and project names, public hostnames, private IP addresses, email addresses, cluster ids, Vault mount paths.
  • Pass: Kubernetes kinds and label keys, public chart and image names, cluster.local, upstream documentation hosts.
03 · Where is it going?

Destinations are declared, never inferred.

Every profile is declared in config as one of three destination classes. A self-hosted box you run is local controlled: no third party sees the data. A provider you have accepted for masked payloads is approved remote, whether it is a regional provider or a frontier model. Anything not declared is denied.

The endpoint URL is not trusted as evidence. A profile missing from the declaration is refused, not guessed.

04 · What does the tenant require? In development

Each tenant states its own floor.

A tenant's egress requirement is one of any, pseudonymised or local_only. The checks are built. Until every call carries reliable tenant provenance, a shared engine applies the strictest requirement configured to every call, so one cautious tenant raises the floor for all.

pseudonymised lets a call reach an approved remote provider only as a validated masked envelope. local_only keeps it on your own hardware.

05 · Mask the identifiers

Same shape, obviously fake, consistent within the request.

Each identifier becomes an alias that keeps its grammar: a hostname stays a valid hostname under the reserved .invalid domain, a private address becomes a documentation-range address with the same prefix width. The same value always gets the same alias, so the model can still reason about "this host talks to that host".

There is no stored table of real names and aliases. The mapping is rebuilt per request from what is actually in it.

06 · Restore on the way back

The answer comes back in your terms, or not at all.

Aliases in the reply are swapped back to the real values before any other code sees them. An alias that cannot be restored, or one that would end up in a file, an MR or a tool argument, rejects the reply instead of leaking a fake name into your repo.

A model is only allowed on a masked lane after a paired run proves its replies restore cleanly. The evidence is bound to that exact provider, endpoint and model.

07 · When the only allowed endpoint is down In development

It stops and asks. It does not quietly go elsewhere.

Profiles can fall back to another tier when an endpoint fails. Every fallback is checked again against the policy of the tier it lands on. There is no automatic reroute to a different residency.

If a local_only tenant's box is down, the engine enters a named HALT state and waits for a person. Losing a night of work is the designed cost; sending that tenant's data to a cloud model is not an option.

08 · Mix and match

Route each class of problem to the model that suits it.

A profile maps call labels to endpoints. Bulk, high-volume work goes to a small, cheap model. Careful generation of a diff goes to a stronger one. The review of the finished MR goes to a different family again, so it is not marking its own homework.

Unmapped labels fall through to the profile's default. A mapping that names a profile which does not exist is reported at startup.

Agentsupgrades · resources · alerts · review…
LLM enginethe only path to a model
Egress gateclassify · check destination · mask
Endpointchosen by the call's label
call: label=review_mr · profile=review-tier · 1 MR diff + 3 cluster reads
classification · deterministic rule pack
  • password: s3cr… secret · REDACT
  • orders.shop.acme.example public hostname · PSEUDONYMISE
  • 10.40.12.7 private IPv4 · PSEUDONYMISE
  • acme company name · PSEUDONYMISE
  • oncall@shop.acme.example email · PSEUDONYMISE
  • kind: HelmRelease Kubernetes vocabulary · PASS
  • ingress-nginx public chart name · PASS
  • orders-api.shop.svc.cluster.local in-cluster name · PASS
Local controlledyour own GPU box · raw data may go here
Approved remoteregional provider · frontier model · masked where required
Deniedanything not declared · never a payload
config.yaml · egress_policy
egress_policy:
  local-gpu-box:      local_controlled
  regional-provider:  approved_remote
  frontier-model:     approved_remote
  # not listed ⇒ refused once enforcement is on
tenant requirement × destination
  • any → approved remote raw payload allowed
  • pseudonymised → approved remote masked envelope only
  • pseudonymised → local controlled raw allowed, never leaves
  • local_only → local controlled raw allowed
  • local_only → approved remote refused
  • any → undeclared endpoint refused
tenants: shop=any · payments=pseudonymised ⇒ engine floor = pseudonymised
prompt as assembledstays in memory
host: orders.shop.acme.example
upstream: 10.40.12.7:5432
vault path: secret/acme/orders/db
alert contact: oncall@shop.acme.example
what the provider receivesmasked
host: egress-fake-3b9e1c07a4d2f6e815c0.invalid
upstream: 198.51.100.37:5432
vault path: secret/egress-fake-a71f0c92d8e4b3561f2a/orders/db
alert contact: egress-fake-5c2d…@contact.invalid
# note to the model: copy placeholders verbatim
Illustrative values. Aliases are keyed hashes: stable, announced-fake, never mistaken for real.
model reply
The ingress for egress-fake-3b9e…f6e815c0.invalid points
at a Service with no ready endpoints; the database at
198.51.100.37 is unaffected.
after restoration
The ingress for orders.shop.acme.example points
at a Service with no ready endpoints; the database at
10.40.12.7 is unaffected.
  • every alias restored reply accepted
  • an unknown alias in a proposed file reply rejected, nothing written
Calltenant: local_only
local-gpu-boxconnection refused
Fallback?frontier-model · approved remote
HALT · restricted destination unavailable
Fallback to frontier-model refused: tenant requirement local_only. Queued calls wait; an operator restarts or reconfigures to resume.
recorded: HALT · label=analyse_manifest · reason=local-only destination exhausted
config.yaml · profiles
profiles:
  acme-routed:
    # default tier: bulk triage, cheap and fast
    base_url: http://gpu-box.internal:8000/v1
    model: small-local-model
    call_profiles:
      # judgement: drafting and checking changes
      analyse_manifest: regional-provider
      chart_upgrade:    regional-provider
      diff_complex:     regional-provider
      # review: a different model family
      review_mr:        frontier-model
      review_issue:     frontier-model
Triagelocal GPU box
Generateregional provider
Reviewfrontier model

What a masked review looks like to you.

Illustrative. The assistant reviews its own draft MR before it reaches you, through a masked lane. You see real names; the provider did not.

Draft

Add readiness probe to shop/orders-api

from gitops-ai/reliability-orders-api-probe-7a2e into main · 1 file · +8
category:reliabilitysource:repo_scan:acme-deployrisk:lowxrev::LGTM
ReviewWhat the provider sawEgress record
Automated review

The probe targets port http on /healthz, which the live pods at orders.shop.acme.example already answer with 200. The Service selector matches the Deployment's labels. The database dependency on 10.40.12.7 is not part of the probe, so a database blip will not take the pods out of rotation.

Verdict: LGTM.

Excerpt of the outbound prompt
Live pods behind egress-fake-3b9e1c07a4d2f6e815c0.invalid answer
GET /healthz with 200. DB upstream 198.51.100.37:5432.
… the diff, Kubernetes kinds and public chart names pass unchanged …

Identifiers were masked with keyed aliases for this request only. This is pseudonymisation of declared identifier classes: a provider could still infer things from structure and chart choices, and this lane does not claim otherwise.

FieldValue
labelreview_mr
profilefrontier-model
destination classapproved remote
engine floorpseudonymised
aliasedfqdn 2 · rfc1918-ipv4 1 · customer-name 1
round tripall aliases restored · reply accepted
qualificationpaired-run evidence on file for this provider + model

Counts are recorded, never the values.

What keeps it honest.

The boundary is deterministic code with tests that fail loudly, not a promise in a prompt.

Sealed masking codeThe classification and masking module cannot import any network client or model SDK. It can transform a prompt; it can never send one.
Mask the copy, not the sourceOnly the outbound envelope is masked. Your manifests and the proposals built from them stay real, so a fake name can never be committed.
Aliases are a taintAn alias that survives into a file, a tool call or a forge write rejects the reply, and the forge write gate blocks alias-shaped text too.
Refusal is a named stateUnknown destination: refused. Restricted tenant with its endpoint down: HALT. Both are recorded with a reason code; nothing degrades silently.
Qualified before maskedA model joins a masked lane only with round-trip evidence bound to that exact provider, endpoint, model and masking version.
Honest about limitsMasking covers declared identifier classes. It does not stop a provider inferring who you are from structure or style, and we do not say it does.

Where this stands.

Live
  • One engine for every model call, pinned by tests
  • Declared destination classes; undeclared endpoints refused
  • Identifier masking and restoration for MR review calls, with per-model round-trip evidence
  • Profiles routing call labels to different endpoints, with startup checks
  • Secret scrub, with a fail-closed check on every forge write
In development
  • Per-tenant any / pseudonymised / local_only requirements and HALT (built, not yet in customer use)
  • Masking for issue review, then further call labels
  • Self-hosted local endpoints as a routine production tier
  • Observation of review tool servers (results not yet masked)
Roadmap
  • Masking of tool results and of manifest-editing calls
  • Per-call tenant provenance, so each call gets its own tenant's floor instead of the strictest
← Previous storyBeyond GitLab: other forges