ADR-0022: Verify Gateway Scope Enforcement with a Dedicated Measurement Tool, Not the API Test Automation Suite¶
- Status: ✅ Accepted
- Date: 2026-08-10
- Owners: @emre, @enver
-
Context: The PrivateApiGateway declares a required scope per route, and a user holding that scope is supposed to reach the route while a user holding a different one is not. Nothing verified either half. The reported problem class was concrete: "the user has no update permission, but the update call does not return 403".
The surface is large. The deployed gateway declares 1058 route+method pairs, and proving a scope claim about one of them takes two calls — one holding exactly the declared scope, one holding a different scope on the same resource. That is ~2100 requests per full pass, and the number grows with every route anyone adds.
Until now the check was manual, endpoint by endpoint, done by the test team before a release. It was not repeatable, its coverage was unknown, and it cost human time per release. Two further needs shaped the decision: the checks had to be runnable per environment on demand, whether or not a release touches scopes at all, and they had to be runnable locally by the test team rather than only in CI.
Decision¶
Scope enforcement is verified by a dedicated project — ac-scope-tester, its own repository — that
derives its cases from the deployed gateway's own configuration rather than from hand-written test
cases, and probes every route twice against a live environment.
Per route: the owner identity rewrites a custom role to hold exactly the scope the route declares and
the probe user calls the route (expected: anything except 403); then the role is rewritten to hold a
sibling scope on a different resource and the same call is made again (expected: exactly 403). Only
403 counts as a scope denial — a 400, 404 or 500 means the request got past the gateway, which
is all the positive leg needs to prove.
The route corpus is extracted from the container image that is actually deployed, once per run, by
reading GET /versions and pulling that image. It is deliberately not committed: one hand-copied
snapshot had 112 of 1028 route+method pairs declaring a scope the deployed gateway did not, and
three measurement runs were spent before anyone noticed.
The run is a manually triggered Jenkins job parameterised by environment, and its exit code maps to the
build colour: 0 clean, 1 a declared scope is refused, 2 unauthorized access is possible, 3
nothing was measured. The result is posted to #automated-tests with the findings themselves. The same
binary runs locally with the same configuration.
Options Considered¶
- Option A — Add the cases to the existing API test automation suite
- Pros: one place for all API tests; existing reporting and Jenkins integration.
- Cons: 1058 routes × 2 legs as hand-written cases. The suite's case count is already high and its maintenance and refactor cost is the reason we are not extending it. Worse, hand-written cases encode somebody's understanding of the gateway's configuration rather than the configuration itself: a route added without a test is silently uncovered, and the suite would have to be edited every time a scope is renamed.
- Option B — Keep the manual per-endpoint check
- Pros: no engineering cost; a human notices things a tool does not.
- Cons: not repeatable, coverage unknown and unmeasurable, human time per release, and no way to answer "is this environment correct right now".
- Option C — Static analysis: compare the ocelot configuration against the Keycloak scope catalogue
- Pros: cheap, fast, no test cases, no live environment, no credentials.
- Cons: it cannot see the largest failure class. The configuration looked correct; 36 routes' declared scopes were nonetheless inert, because Ocelot compiled a trailing placeholder to a pattern matching any nested path, so a shallower route resolved the request and a different scope was enforced. Only a live request shows that. Of the 51 findings the first run produced, 36 were invisible to any amount of config reading.
- Option D — A dedicated tool deriving its cases from the deployed gateway's configuration —
selected
- Pros: no case list to maintain, because there is no case list — coverage grows with the gateway automatically. The measurement is live, so runtime-only classes such as route shadowing are caught. Runs per environment on demand, and locally.
- Cons: a second project to own; needs privileged credentials and a dedicated organization per environment; its own runtime cost (~12 minutes per full pass).
Consequences¶
Positive
- The first full run found 51 real findings: 36 routes whose declared scope was inert through route shadowing, and 15 declaring a scope Keycloak does not define (7 name drifts, 8 with no scope that can grant them at all). All were fixed in the gateway (PRs #397, #404, #407–#410), and the same run now reports 810 measured, 0 unauthorized access on dev, reproducing identically across runs.
- Nothing to maintain per route. A new route is measured on the next run without anyone being told.
- The tool measures a deployed environment, so it answers "is prep correct right now" independently of whether the release being shipped touches scopes.
- Release health became observable: a run's exit code is a build colour, and the findings arrive in Slack with the route and the scope on each line.
- It is not tied to Ocelot. The tool measures behaviour from the outside — HTTP requests and status codes — and knows nothing about the gateway's implementation. Under ADR-0010 (replacing Ocelot with Apache APISIX) it is not thrown away; it becomes the migration's regression harness, and the only practical answer to "does APISIX enforce the same scopes on all 1058 routes".
Negative
- A run is destructive by design. It rewrites a custom role's scopes twice per route and issues
authorized writes and deletes, including
DELETE. Each environment therefore needs a dedicated, disposable organization and a throwaway user inside it; the tool refuses to start unless the owner identity belongs to the configured organization, and production additionally requires the environment name typed as a confirmation. This is a standing operational obligation, not a one-off setup. - It depends on the gateway reporting its own build. dev reports a commit SHA; prep and prod do not
(prep returns a fragment of its version, prod the literal
master.Sha), so for those the image is resolved from the version tag — the deployed version, but not a commit-pinned artifact. The run says so in its log. - Throughput is capped by the platform being measured: the identity custom-role
PUTis 200/min per user, and each probe spends two of them, so ~100 probes/min regardless of parallelism. A full pass is ~12 minutes and cannot be meaningfully shortened without additional probe identities. - Two runs against one environment must not overlap — they share one role and would interleave their writes, producing wrong verdicts in either direction with nothing in the output saying so. The Jenkins job serialises itself; a local run racing a CI run is not covered by anything but discipline.