A reachability test has a property that breaks the obvious design: the answer is only true from the source network. “Can this cluster reach that host on that port?” cannot be answered by a web service dialing the target, because the web service sits elsewhere on the network. A firewall or route that blocks the workload’s namespace may wave the API server through. The probe has to run inside the specific cluster the engineer is asking about; one of many in a fleet, possibly not the one the control app runs in.
The instinctive fix is a resident agent: deploy a small probe service to every cluster, expose an RPC, call it. That gives you a clean typed response. It also gives you a standing thing on every cluster to secure, patch, grant RBAC, and monitor; for a workload that runs a few seconds at a time. Across a fleet that is a real operational tax, and a permanent attack surface for an intermittently-used feature.
The system I looked at took the other trade. Each test is shipped as an ephemeral Kubernetes Job. The control app authenticates to the target cluster (per-cluster: workload identity for remote clusters, a mounted service-account token for its own, kubeconfig for local dev), creates a one-shot Job running a standalone probe binary with the parameters as CLI args, waits, then deletes the Job. BackoffLimit: 0, RestartPolicy: Never; exactly one attempt, under a strict context: runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, capabilities.drop: [ALL]. Between tests nothing runs. Nothing to patch, nothing to reach.
The bill for “no resident agent” comes due at the boundary: with no network API, there is no response to read. The probe prints one JSON result to stdout and exits; the result is recovered by reading the pod’s logs. That is the transport, and it is where the fragility lives, so it gets handled deliberately.
The catch is stream merging. The binary writes its result to stdout and structured logs to stderr. Run it as a subprocess and you capture stdout alone. Run it as a pod and the log API hands you stdout and stderr interleaved; you cannot json.Unmarshal the blob. So the reader anchors on a sentinel and brace-counts to the matching close:
for i, line := range allLines {
if !inJSON && strings.TrimSpace(line) == "{" &&
i+1 < len(allLines) && strings.Contains(allLines[i+1], `"target"`) {
inJSON, jsonLines, braceCount = true, []string{line}, 1
} else if inJSON {
jsonLines = append(jsonLines, line)
braceCount += strings.Count(line, "{") - strings.Count(line, "}")
if braceCount == 0 { break } // balanced: payload ends here
}
}A standalone { followed by a line containing a known field marks the start; balanced braces mark the end. Because indentation is guaranteed by MarshalIndent, the closing brace lands on its own line and the count resolves cleanly. BackoffLimit: 0 matters here too; one attempt means one payload, nothing to disambiguate. And the binary emits valid JSON even on failure (exit 1), so a blocked connection is a parseable result, not a parse error.
The probe does DNS-with-timing, a TCP or UDP dial classified into a human hint (refused → “no service listening”; timeout → “likely firewall filtering”), an ICMP ping, and, opportunistically, if the TCP port answers a handshake, a TLS inspection with InsecureSkipVerify: true to read the certificate chain, validity, cipher and version rather than trust them. One tension is worth naming: the same drop-ALL context that makes the Job cheap also drops CAP_NET_RAW, so in-cluster ICMP hits EPERM. The binary detects that and returns a targeted hint instead of crashing.
The general lesson: sometimes the cheapest correct transport is a platform primitive you already have. A Job plus its logs is not elegant, it is a text stream you scrape, but it needs zero standing infrastructure and puts the probe provably in the right network with the right identity. The design question is not “what’s the nicest API,” it’s “what will I trade for having nothing to run between requests.” Here the trade is a log parser you write defensively. Name that boundary, sentinel it, and count your braces.