Mastering Distributed Tracing with OpenTelemetry for Microservices

Debugging and understanding the flow of requests in a monolithic application can be challenging. When that application evolves into a microservices architecture, the complexity multiplies significantly. A single user request might traverse dozens of services, each with its own logs and metrics. Pinpointing the root cause of latency or errors in such an environment becomes a daunting task. This is where distributed tracing provides a critical solution.

Distributed tracing offers visibility into the entire lifecycle of a request as it moves through various services. It helps developers visualize the path, identify performance bottlenecks, and quickly diagnose issues across a distributed system. OpenTelemetry, an open-source observability framework, has emerged as the standard for implementing distributed tracing, along with metrics and logging. This guide provides a practical, step-by-step tutorial on integrating OpenTelemetry for distributed tracing in a microservices setup.

The challenge of microservices observability

Microservices break down large applications into smaller, independent services. This approach offers benefits like improved scalability, resilience, and development agility. However, it introduces new operational complexities. When a user experiences a slow response or an error, determining which service or interaction caused the problem is difficult. Traditional logging, while useful for individual services, fails to provide a holistic view of a request’s journey across service boundaries. Metrics offer aggregate data but lack the granular, request-specific detail needed for deep troubleshooting.

Distributed tracing bridges this gap by providing a complete, end-to-end view of a request. It allows you to see the sequence of operations, their durations, and any associated metadata, all linked together by a unique identifier.

Core concepts of distributed tracing

Before diving into OpenTelemetry, understanding the fundamental concepts of distributed tracing is essential.

Traces, spans, and context

A trace represents the complete journey of a single request or transaction through a distributed system. It is a directed acyclic graph (DAG) of spans.

A span is a single operation within a trace. It represents a logical unit of work, such as an HTTP request, a database query, or a function call. Each span has a name, a start time, an end time, and attributes (key-value pairs) that provide additional context. Spans can have parent-child relationships, forming the structure of the trace. For example, a span representing an incoming API call might have child spans for database operations and calls to other microservices.

Context refers to the information that links spans together. This includes the trace ID and the parent span ID. Context propagation is the mechanism by which this information is passed between services, ensuring that all operations related to a single request are correctly grouped into a single trace.

Trace ID, span ID, and parent span ID

Trace ID: A unique identifier for an entire trace. All spans belonging to the same trace share the same trace ID.
Span ID: A unique identifier for a specific span within a trace.
Parent Span ID: The span ID of the parent operation. This establishes the hierarchical relationship between spans, allowing a tracing backend to reconstruct the trace graph.

Distributed Tracing Diagram — Photo by Google DeepMind on Unsplash

Understanding OpenTelemetry

OpenTelemetry is a vendor-neutral set of APIs, SDKs, and tools designed to instrument, generate, collect, and export telemetry data (traces, metrics, and logs). It provides a standardized way to instrument applications, freeing developers from vendor lock-in and allowing them to choose their preferred observability backend.

OpenTelemetry architecture

The OpenTelemetry architecture consists of several key components:

API (Application Programming Interface): Defines how telemetry data is created. This includes interfaces for creating traces, spans, and metrics.
SDK (Software Development Kit): Implements the API and provides the logic for processing and exporting telemetry data. This is where you configure things like samplers, processors, and exporters.
Instrumentation Libraries: These are pre-built libraries that automatically instrument popular frameworks, databases, and HTTP clients. They reduce the amount of manual code developers need to write.
Collector: An optional but highly recommended component. The OpenTelemetry Collector is a proxy that receives, processes, and exports telemetry data. It can receive data in various formats, perform transformations, and send it to multiple backends. Using a collector decouples your application from the specifics of the observability backend.
Exporters: Components within the SDK or Collector that send telemetry data to an observability backend (e.g., Jaeger, Zipkin, Prometheus, Datadog).

Instrumentation: automatic versus manual

OpenTelemetry supports two main approaches to instrumentation:

Automatic Instrumentation: This uses pre-built libraries that automatically instrument common frameworks, libraries, and protocols (like HTTP, gRPC, database drivers). It requires minimal code changes and is ideal for getting started quickly.
Manual Instrumentation: This involves explicitly adding OpenTelemetry API calls to your code. It provides fine-grained control over what is traced and allows you to add custom spans and attributes for specific business logic. A combination of both is often used, with automatic instrumentation providing baseline coverage and manual instrumentation adding depth where needed.

Setting up a basic tracing environment

To demonstrate OpenTelemetry in action, we will set up a simple microservices environment. We will use two Python Flask services, service-a and service-b, where service-a makes an HTTP call to service-b. We will use Docker Compose to orchestrate these services along with an OpenTelemetry Collector and Jaeger as our tracing backend.

Prerequisites

Docker and Docker Compose installed.
Python 3.8+ installed.

Project structure

Create a directory named otel-tracing-example and set up the following structure:

otel-tracing-example/
├── docker-compose.yaml
├── service-a/
│   ├── app.py
│   └── requirements.txt
└── service-b/
    ├── app.py
    └── requirements.txt

Docker Compose configuration

Create docker-compose.yaml in the root directory:

version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.55
    ports:
      - "16686:16686" # Jaeger UI
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    networks:
      - otel-network

  otel-collector:
    image: otel/opentelemetry-collector:0.90.1
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "6831:6831/udp" # Jaeger receiver
      - "4317:4317"     # OTLP gRPC receiver
      - "4318:4318"     # OTLP HTTP receiver
    depends_on:
      - jaeger
    networks:
      - otel-network

  service-a:
    build: ./service-a
    ports:
      - "5000:5000"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
      - OTEL_RESOURCE_ATTRIBUTES=service.name=service-a
      - SERVICE_B_URL=http://service-b:5001/hello
    depends_on:
      - otel-collector
    networks:
      - otel-network

  service-b:
    build: ./service-b
    ports:
      - "5001:5001"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
      - OTEL_RESOURCE_ATTRIBUTES=service.name=service-b
    depends_on:
      - otel-collector
    networks:
      - otel-network

networks:
  otel-network:
    driver: bridge

Create otel-collector-config.yaml in the root directory:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

This configuration sets up:

Jaeger: The tracing backend and UI.
OpenTelemetry Collector: Receives OTLP (OpenTelemetry Protocol) data from our services and exports it to Jaeger.
Service A & B: Our Python Flask applications.

Step-by-step implementation

Step 1: Project setup and dependencies

service-a/requirements.txt:

Flask==2.3.3
requests==2.31.0
opentelemetry-api==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-exporter-otlp==1.22.0
opentelemetry-instrumentation-flask==0.43b0
opentelemetry-instrumentation-requests==0.43b0

service-b/requirements.txt:

Flask==2.3.3
opentelemetry-api==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-exporter-otlp==1.22.0
opentelemetry-instrumentation-flask==0.43b0

Step 2: Instrumenting service-a

service-a/app.py:

import os
import requests
from flask import Flask
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

# Configure OpenTelemetry
resource = Resource.create({
    "service.name": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service-a-default"),
    "application": "my-microservice-app"
})

trace.set_tracer_provider(
    TracerProvider(resource=resource)
)
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter to send data to the collector
otlp_exporter = OTLPSpanExporter(
    endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
    insecure=True
)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

app = Flask(__name__)

# Instrument Flask and Requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

SERVICE_B_URL = os.environ.get("SERVICE_B_URL", "http://localhost:5001/hello")

@app.route("/")
def home():
    with tracer.start_as_current_span("home-request"):
        return "Hello from Service A!"

@app.route("/call-service-b")
def call_service_b():
    with tracer.start_as_current_span("call-service-b-endpoint"):
        print(f"Calling Service B at: {SERVICE_B_URL}")
        try:
            response = requests.get(SERVICE_B_URL)
            response.raise_for_status()
            message = response.text
            return f"Service A called Service B: {message}"
        except requests.exceptions.RequestException as e:
            return f"Error calling Service B: {e}", 500

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Explanation for service-a/app.py:

Resource Configuration: We define a Resource to attach metadata like service.name to all telemetry data from this service. This helps identify the source of traces in the backend.
TracerProvider Setup: TracerProvider is the entry point for all tracing operations. We set it globally.
OTLP Exporter: OTLPSpanExporter sends spans to the OpenTelemetry Collector using the OTLP gRPC protocol. The endpoint is configured via an environment variable.
BatchSpanProcessor: This processor batches spans before exporting them, which is more efficient than sending each span individually.
Instrumentation: FlaskInstrumentor().instrument_app(app) automatically instruments incoming Flask requests, creating spans for them. RequestsInstrumentor().instrument() instruments outgoing requests calls, ensuring context propagation to service-b.
Manual Span: The with tracer.start_as_current_span(...) block demonstrates how to create a custom span around specific logic, providing more granular detail within a trace.

Step 3: Instrumenting service-b

service-b/app.py:

import os
from flask import Flask
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Configure OpenTelemetry
resource = Resource.create({
    "service.name": os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "service-b-default"),
    "application": "my-microservice-app"
})

trace.set_tracer_provider(
    TracerProvider(resource=resource)
)
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter to send data to the collector
otlp_exporter = OTLPSpanExporter(
    endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
    insecure=True
)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

app = Flask(__name__)

# Instrument Flask
FlaskInstrumentor().instrument_app(app)

@app.route("/hello")
def hello():
    with tracer.start_as_current_span("hello-from-service-b"):
        return "Hello from Service B!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5001)

Explanation for service-b/app.py: The setup for service-b is similar to service-a, focusing on Flask instrumentation. When service-a calls service-b, the RequestsInstrumentor in service-a automatically injects tracing context into the HTTP headers. The FlaskInstrumentor in service-b then reads this context, allowing service-b to continue the trace started by service-a. This is the essence of context propagation.

Step 4: Running with a collector and backend

Navigate to the otel-tracing-example directory in your terminal and run:

docker compose up --build

This command builds the Docker images for service-a and service-b, and starts all the services defined in docker-compose.yaml.

Step 5: Verifying traces

Once all services are running:

Access Service A: Open your browser or use curl to hit service-a’s endpoint that calls service-b:
```
curl http://localhost:5000/call-service-b
```
You should see a response like: “Service A called Service B: Hello from Service B!”
Access Jaeger UI: Open your browser and navigate to http://localhost:16686.
Find Traces: In the Jaeger UI, select “service-a” from the “Service” dropdown and click “Find Traces”. You should see traces corresponding to your requests.

Click on a trace to view its details. You will see a waterfall diagram showing the sequence of spans:
- An initial span for the incoming request to service-a.
- A child span for the call-service-b-endpoint in service-a.
- A child span for the outgoing HTTP request from service-a to service-b.
- A span for the incoming request to service-b.
- A child span for hello-from-service-b in service-b.
This visualization clearly shows the entire request flow, including the time spent in each service and operation. You can inspect individual spans to see their attributes, which provide valuable context for debugging.

Jaeger UI Trace View — Photo by Matheus Bertelli on Pexels

Advanced considerations

Sampling strategies

Collecting every single trace can generate a massive amount of data, especially in high-traffic systems. OpenTelemetry supports sampling to reduce the volume of telemetry data. Samplers decide whether a trace should be recorded or dropped at the beginning of its lifecycle. Common strategies include:

AlwaysOnSampler: Records all traces. Useful for development and low-traffic environments.
AlwaysOffSampler: Records no traces.
ParentBasedSampler: Respects the sampling decision of the parent span.
TraceIdRatioBasedSampler: Samples a certain percentage of traces based on their trace ID.

You configure samplers in your TracerProvider.

Adding custom attributes

Attributes are key-value pairs that provide additional context to spans. You can add custom attributes to spans to include business-specific information, such as user IDs, order IDs, or specific function parameters. This enriches your traces and makes them more useful for debugging.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.route("/process-order/<order_id>")
def process_order(order_id):
    with tracer.start_as_current_span("process-order-endpoint") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.id", "some_user_123")
        # ... rest of your order processing logic
        return f"Processing order {order_id}"

Error handling and events

OpenTelemetry allows you to record exceptions and events within spans. This is crucial for understanding when and where errors occur in a distributed system.

import sys
from opentelemetry import trace
from opentelemetry.sdk.trace import SpanKind, StatusCode

tracer = trace.get_tracer(__name__)

@app.route("/risky-operation")
def risky_operation():
    with tracer.start_as_current_span("risky-operation", kind=SpanKind.SERVER) as span:
        try:
            # Simulate an error
            result = 1 / 0
            return f"Operation successful: {result}"
        except Exception as e:
            span.set_status(StatusCode.ERROR, description=str(e))
            span.record_exception(e)
            return f"Operation failed: {e}", 500

Setting the span status to ERROR and recording the exception provides clear indicators in your tracing backend that something went wrong.

Integration with other observability signals

While this guide focuses on tracing, OpenTelemetry also supports metrics and logs. A complete observability strategy integrates all three. Traces provide the request-level detail, metrics offer aggregate performance data, and logs give detailed event information. OpenTelemetry aims to provide a unified approach to collecting all these signals, allowing for powerful correlations and comprehensive system understanding.

Benefits and best practices

Implementing distributed tracing with OpenTelemetry offers significant advantages:

Faster root cause analysis: Quickly identify which service or component is causing latency or errors.
Performance optimization: Pinpoint bottlenecks and areas for improvement in your request flows.
Understanding service dependencies: Visualize how services interact and depend on each other.
Improved developer experience: Developers gain better insights into their code’s behavior in a distributed environment.

For best practices, ensure consistent instrumentation across all services. Use automatic instrumentation for common libraries and frameworks, and supplement with manual instrumentation for critical business logic. Leverage the OpenTelemetry Collector to centralize data processing and export, providing flexibility and resilience. Regularly review your traces to understand system behavior and identify potential issues proactively.

Conclusion

Distributed tracing is an indispensable tool for managing the complexity of modern microservices architectures. OpenTelemetry provides a powerful, standardized, and vendor-neutral way to implement this crucial observability practice. By following the steps outlined in this guide, developers can gain deep insights into their distributed systems, leading to faster debugging, improved performance, and a more robust application environment. Embracing OpenTelemetry is a strategic move towards building more resilient and understandable software.

Works Cited

“Mermaid Gantt diagrams for displaying distributed traces in Markdown (2023).” brycemecum.com, https://brycemecum.com/2023/03/31/til-mermaid-tracing/