Access Logs That Should Be Metrics

Important Signal: Logs

Problem

The service ships one log line per HTTP request, and what those lines are actually used for is aggregate: request rate, error rate, and latency by route. Per-request lines don’t deduplicate — every request is distinct — but they collapse into a handful of metric series that carry the same answers at a fraction of the volume.

Insights flags a service when at least 1,000 access-log records land in each of two consecutive 5-minute windows and the records outnumber the estimated metric series they would collapse into by at least 10×, using a conservative bound on the series estimate. Method, route, and status are resolved from log attributes, from structured bodies, or from a quoted request line in a plaintext body — so collectors that forward the raw access-log line unparsed are caught too.

Impact

• You pay log ingest and storage per request for answers a few metric series already carry.
• Health checks and synthetic probes are often most of the stream: high-volume records nobody will ever read.
• The volume grows linearly with traffic, so the waste compounds exactly when you can least afford it.

Example

An ingress-nginx controller writes one JSON line per request. The collector’s filelog receiver forwards the raw line as an unparsed body, so millions of lines per day reach the backend carrying method, path, and status that would collapse into an http.server.request.duration histogram with a few hundred series.

How to fix

1. If the access logs have forensic, audit, or compliance value, identify and preserve the records that purpose requires before reducing anything; an ERROR-only log path is enough only when that review says so.
2. Check the top routes for health, readiness, status, and synthetic-monitoring endpoints, and filter the successful probe records at the collector — keep failing probes, and keep a probe metric if an alert depends on probe rate or latency, because records filtered here never reach the derived metrics.
3. Convert the stream to the http.server.request.duration histogram by normalized route, method, and status in the Collector. Its count component carries the request count, so no separate request counter is needed — your backend derives the rate from it the same way it does from any counter.
4. Roll out per service: validate route normalization, run metrics alongside logs for a window, and compare before cutting over. Only then reduce or sample the raw lines — never drop records on this finding alone.

Expected gains

The flagged ratio is the ceiling on what the stream collapses into; the realized saving depends on the metric export interval and on how many raw lines you keep, so measure it during the side-by-side window before cutting over. Dashboards and alerts keep the same request, error, and latency answers from the derived metrics.

See the Insights catalog for everything Insights can detect.