Connecting the dots using OpenTelemetry (Part - 1)

Getting Started With OpenTelemetry: A Developer's Walkthrough (Part 1)

This article highlights the benefits of OpenTelemetry for measuring and collecting telemetry data, and how the Bureau uses it in its infrastructure.

Author
Mohit Shukla

OpenTelemetry is a set of tools, APIs, and SDKs that allows us to measure and collect telemetry data such as metrics, logs and traces from the software. By utilizing these tools, we can gain insight into the performance and behavior of our system, which can aid in debugging, troubleshooting, and identifying performance bottlenecks.

At Bureau, we have been using OpenTelemetry quite extensively. We rely on multiple data sources, internal and external to generate insights. These data sources are often inconsistent, opening up a lot of blind spots in the infrastructure. OpenTelemetry has helped us in understanding how well these sources are performing.

Why OpenTelemetry?

As of 2023, OpenTelemetry continues to be the second most active CNCF project after Kubernetes. It gives a couple of advantages over the others.

Vendor agnostic: This means that you do not have to change the instrumentation in your applications if you decide to move away from one backend to another. Apart from it, you should also be able to design your observability strategy by mixing and matching independently to the backend.
Speed: Being one of the most active projects of CNCF and having a good number of contributors helps it move way faster than any proprietary tool ever can.
Data Governance: It gives capabilities to govern the data by offering features such as controlling the data retention time, data masking/redacting, data filtering, and data sampling.
Community: OpenTelemetry is an industry standard for gathering telemetry data from the system. It is backed by strong and supportive community members.

Architecture

How do we use OpenTelemetry at Bureau?

We use OpenTelemetry to increase observability by collecting data from all layers of an application stack — including distributed traces, metrics, and logs — providing a comprehensive view that helps us make informed decisions quickly while responding faster when issues arise within production environments and customer-facing services.

We have an OTEL Collector running as an HTTP service that all other services use to export their telemetry data. Later, This data is exported to their respective backends.

What is an OpenTelemetry Collector?

The OpenTelemetry Collector is a vendor-agnostic way of receiving, processing and exporting telemetry data from different systems. It removes the need to spawn multiple agents to get any signal data. It supports majorly all open-source observability data formats (e.g. Jaeger, Prometheus, Fluent Bit, etc.) to receive and eventually send to one or more open-source or commercial back-ends like honeycomb.io, newrelic.com, lightstep.com, etc.

OTEL Collector supports multiple components to receive, process, and export data.

Receivers
Processors
Exporters
Extensions

We use the filter, tail_sampling, and batch processors to filter health checks routes, samples, and batch traces, respectively, before exporting them to the backend. This helps us to have governance over the data, control the cost of the backend and focus on the interesting traces from the system. There are various processors available as part of the collector or collector repository.

Below is the typical otel collector configuration.

receivers:
  otlp:
    protocols:
      http:
      grpc:
  fluentforward:
    endpoint: 0.0.0.0:24224
  prometheus/infra:
    config:
      scrape_configs:
        - job_name: '${SRV_ENV_NAME}-kafka'
          scrape_interval: 60s
          scrape_timeout: 60s
          static_configs:
            - targets: ['kafka-1.${SRV_ENV_NAME}.internal:11002', 'kafka-2.${SRV_ENV_NAME}.internal:11002', 'kafka-3.${SRV_ENV_NAME}.internal:11002']
              labels:
                resource: 'amazon-msk'
                env: '${SRV_ENV_NAME}'
                type: 'node-exporter'
            - targets: ['kafka-1.${SRV_ENV_NAME}.internal:11001', 'kafka-2.${SRV_ENV_NAME}.internal:11001', 'kafka-3.${SRV_ENV_NAME}.internal:11001']
              labels:
                resource: 'amazon-msk'
                env: '${SRV_ENV_NAME}'
                type: 'jmx-exporter'
        - job_name: '${SRV_ENV_NAME}-otel-collector-service'
          scrape_interval: 30s
          static_configs:
            - targets: ['0.0.0.0:8888']
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: '.*grpc_io.*'
              action: drop
processors:
  filter:
    spans:
      exclude:
        match_type: regexp
        attributes:
          - key: http.target
            value: (^/health|^/metrics)
  batch:
  tail_sampling:
    decision_wait: 60s
    num_traces: 10000
    expected_new_traces_per_sec: 1000
    policies:
      [
        {name: errors-policy, type: status_code, status_code: {status_codes: [ERROR]}},
        {name: randomized-policy, type: probabilistic, probabilistic: {sampling_percentage: 25}},
      ]
exporters:
  logging:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
  otlp:
    endpoint: $NEW_RELIC_ENDPOINT
    headers:
      api-key: $NEW_RELIC_API_KEY
  prometheusremotewrite/1:
    endpoint: $ENDPOINT
    auth:
      authenticator: basicauth/prw
    external_labels:
      server: otel-collector
extensions:
  basicauth/prw:
    client_auth:
      username: $TOKEN_USERNAME
      password: $TOKEN_PASSWORD
  health_check:
    endpoint: :13133
  zpages:
    endpoint: :55679
service:
  telemetry:
    logs:
      level: "$LOG_LEVEL"
    metrics:
      address: :8888
  extensions: [health_check,zpages,basicauth/prw]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter, tail_sampling, batch]
      exporters: [otlp]
    metrics:
      receivers: [prometheus/infra]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [fluentforward, otlp]
      processors: [batch]
      exporters: [otlp]

Below is the example of a distributed trace where one or more spans have attribute otel.status_code as “ERROR” where an HTTP POST request failed. We can get more information, like the HTTP endpoint, HTTP status code, etc., from the attributes for the analysis and alerts.

A log message is linked with the trace using a trace.id attribute that will help you understand what happened before and after the failed span. You can also use it to carry the complete payload.

Limitations of using OpenTelemetry

There are a couple of limitations as well when using OpenTelemetry collector.

It is still a work in progress. Many of its components are still in the alpha or beta stage.
Scalability is still a concern for the collector. When some components support running multiple instances of the collector, many don’t. For example, tail based sampling processor does not support running multiple instances of the collector without load balancing exporter.

Special Thanks to Abhinav, Nandeesh, Shekh from the Bureau Team.

At Bureau, top tech minds in fraud prevention are sharing valuable insights and uncovering the trade’s best-kept secrets, all with one goal: empowering fraud fighters everywhere to work smarter and more effectively.