Skip to main content
On this page

Duplicate Telemetry Stream

Important Signal: Traces, Logs, Metrics

Problem

A service’s telemetry arrives more than once: the same span, log record, or metric datapoint, byte-identical down to the nanosecond event timestamp, delivered in distinct export requests. That is the fingerprint of a redundant export path, not of an application logging too much — one measurement left the service once and something in the pipeline shipped it twice.

The usual causes:

  • An agent-and-gateway collector topology where both tiers export to the backend.
  • A cluster-scoped receiver (such as k8sclusterreceiver or a Prometheus scrape job) running in multiple collector replicas without leader election, so every replica reports the same targets.
  • A migration dual-write that was never cut over, leaving the old and new export paths both running.

Insights flags a service when its duplication factor (records delivered ÷ distinct records) holds at 1.2 or above for two consecutive 15-minute windows on meaningful volume. Copies repeated inside a single export request don’t count — those are application shape, like multi-line log writes — so a finding always means copies crossed the pipeline in separate requests. The finding reports which identity layer collided: the exact record, or the record minus the collector’s self-identification, which catches the same stream flowing through two different collectors.

Impact

  • Duplication multiplies ingest directly: a factor of 2 means you pay twice for every span, record, and datapoint the service emits, with zero added signal.
  • Storage, processing, and query cost scale with the duplicated volume across every downstream tool.
  • It hides as organic growth. A bill that doubled after a collector rollout looks like traffic until someone compares distinct counts.

Example

A Kubernetes fleet runs a collector agent on every node plus a central gateway. The agents are configured to export both to the gateway and directly to the backend, and the gateway exports everything again. Every span the fleet produces arrives twice, in two different export requests, byte-identical including the timestamp. The duplication factor sits at 2.0 all day, and the ingest bill reads double what the workload justifies.

How to fix

  1. Map every export path that carries this service’s telemetry: SDK exporters, collector agents, gateways, and their destinations.
  2. Remove the redundant path. Point agents at the gateway or the backend, never both; finish the migration cutover; enable leader election or run a single replica for cluster-scoped receivers.
  3. If the duplication is deliberate (a dual-write during a migration, a short-lived comparison), treat the finding as a reminder of the standing cost and a prompt to finish.

Do not add a deduplication or suppression processor. It hides the cost while both paths keep running, and it cannot deduplicate across collector instances anyway.

Expected gains

Ingest for the affected service drops by the duplication factor — often half or a third of the volume — with zero information loss, because every removed record was a byte-identical copy of one you keep.

See the Insights catalog for everything Insights can detect.