Saguaro Cloud Systems
Back to the notes

configuration & platform data

Honest Per-Environment Scoring: When a Glob Eats Your Sandbox

A `**/production/**` glob that quietly swallows a sandbox nested under production, and why you should compute the criteria that don't vary once instead of faking N independent audits.

3 min read
020 criteria
  • Env-scoped; 5 of 20 · re-scored per environment
  • Service-level · source repo; 10 of 20 · computed once
  • Service-level · API & observability; 5 of 20 · computed once
Only 5 of 20 criteria carry per-environment signal; the other 15 are computed once and labeled service-level, not re-scored per environment. Counts are the actual documented split, not illustrative.

Scoring service maturity is straightforward until you have more than one environment. The moment production and a sandbox share a deployment repo, a single score per service quietly becomes a lie: it pools evidence from both tiers and represents neither.

I hit this building a fleet maturity scorer; a tool that grades each service against a roughly 20-criterion SRE rubric by mining its repositories, CI, and observability data. The first version produced one scorecard per service by globbing the whole deployment repo (minus a hardcoded non-production/**) and pooling everything it matched. Splitting that into per-environment scorecards surfaced two problems worth generalizing.

The layout fights the obvious glob

The deployment repos nested the sandbox underneath the production tier. Paths looked like production/<cluster>/production/... and production/<cluster>/sandbox/...; the sandbox literally sits inside a directory named production. So the obvious **/production/** glob for the production environment matches the outer tier directory and slurps every sandbox file beneath it. Production’s score silently absorbs sandbox config. This isn’t exotic: any glob keyed on a path segment that also names an ancestor directory has the same bug.

The fix is to make the environment boundary data, not code; a small validated registry with explicit per-env include and exclude globs:

environments:
  - name: production
    path_globs:    ["**/production/**"]
    exclude_globs: ["**/sandbox/**", "**/non-production/**"]
    display_order: 1
  - name: sandbox
    path_globs:    ["**/sandbox/**"]
    exclude_globs: ["**/non-production/**"]
    display_order: 2

Each deployment-repo query runs, then its matched files are filtered to path_globs AND NOT exclude_globs. If filtering drops every file, the match is discarded and the maturity loop falls through to the next level, no phantom evidence promotes a score. Adding a third environment is one YAML entry; the scorer, storage, API, and UI all read the registry at startup. The loader is validated (unique names, non-empty path globs, strictly increasing display order), so a bad entry fails at load rather than mid-scan, and the very first test written for the migration was the cross-contamination case.

Don’t fake independence

The second problem is subtler. Of the ~20 criteria, only 5 read the deployment repo: statelessness, load balancing, scalability, deployment, IaC. The other 15 are properties of the service’s source code or its shared operational accounts. The same production APM account backs both environments, so re-querying it per environment burns API budget to produce identical numbers. Running all 20 twice would look like two independent audits while being nothing of the sort; a lie.

So classification is derived, not asserted. A criterion is env-scoped only if one of its queries reads the deployment repo; anything API-driven or source-repo-only is service-level. The 15 service-level criteria are computed once and attached to every environment’s scorecard with a service-level scope tag. The UI says so plainly: production and sandbox differ on 5 deployment-repo criteria; the remaining 15 reflect codebase and operational properties common to both. A unit test pins the 5/15 split, so adding or reclassifying a criterion forces a deliberate review instead of silently inflating apparent per-environment precision.

The general lesson

When a rubric spans environments, separate what genuinely differs from what doesn’t. Isolate the differing evidence with explicit, config-driven boundaries; includes and excludes, because directory conventions will betray a single-segment glob. Then refuse to manufacture per-environment precision your data can’t support. A service-level label on 15 criteria is more honest, and cheaper, than two scorecards that pretend to be independent audits of things that never varied.