Building a Telemetry Pipeline with OpenTelemetry Collector

From Zero to Pipeline: An OpenTelemetry Collector Guide

Your services emit telemetry data, but how does it reach observability backends? The answer lies in a centralized OpenTelemetry Collector pipeline, which replaces vendor-specific agents with a unified system for collecting, processing, and routing data.

This article demonstrates building a pipeline that ingests logs, traces, and metrics, processes them with batching and filtering, and exports to multiple destinations like Jaeger and Dash0.

Why This Matters

Vendor-specific agents create brittle, costly systems that lock you into single vendors. The OpenTelemetry Collector addresses this by acting as a universal receiver and processor. Failure to adopt such a system risks inefficiency, data loss, and increased operational complexity, especially as telemetry volume scales.

Key Insights

• “Debug exporter used for console logging in 2025 example”: The debug exporter prints detailed OTLP data to stderr for verification.
• “Batch processors reduce network overhead”: Batching minimizes individual span/metric transmissions, cutting backend load.
• “Transform processors correct OTLP data format”: Fixes misplaced trace context and redundant attributes in log records.

Working Example

# otelcol.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      exporters: [debug, otlp/dash0]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger, otlp/dash0]

# docker-compose.yml
services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.129.1
    volumes:
      - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml
  otelgen-logs:
    image: ghcr.io/krzko/otelgen:v0.5.2
    command: ["--otel-exporter-otlp-endpoint", "otelcol:4317", "--insecure", "logs", "multi"]

Practical Applications

• Use Case: Dash0 for centralized observability with multi-backend support.
• Pitfall: Overloading the Collector without memory limits leads to crashes and data loss.

References:

• https://dev.to/dash0/from-zero-to-pipeline-an-opentelemetry-collector-guide-58l0

---