The Gateway ships distributed traces and structured logs to SigNoz over OTLP/HTTP using the OpenTelemetry SDK. The entire integration lives in a single function — initialize_opentelemetry(app) in src/main.py:80 — and is opt-in: it is a complete no-op unless OTEL_EXPORTER_OTLP_ENDPOINT is set. There is no SigNoz stack in this repo (the signoz/ directory is empty); the collector is deployed and operated externally, and the Gateway is only a client that POSTs to it.
Two things surprise almost everyone the first time: (1) only auto-instrumentation produces spans — no code in this service opens a manual span, so if a request path isn’t Flask/requests/SQLAlchemy it produces no trace; and (2) OTEL_RESOURCE_ATTRIBUTES is defined in settings but never applied. Both are covered in detail below.

​
How export is wired

initialize_opentelemetry(app) is called from the composition root at src/main.py:179, immediately after DBConnector(app) (it needs a live db.engine to instrument SQLAlchemy) and before Swagger, CORS, JWT, and blueprint registration. See Startup, Validation & Composition Root for the full boot order. The function does six things, in order (src/main.py:98-134):
1

Gate on the endpoint

if settings.OTEL_EXPORTER_OTLP_ENDPOINT: — if the endpoint is unset, the function returns without touching anything. There is no console exporter fallback, so a service with OTEL disabled emits nothing to any OTel backend.
2

Build the Resource

A Resource is constructed with exactly two attributes: service.name (from OTEL_SERVICE_NAME) and deployment.environment (from DEPLOYMENT_ENVIRONMENT). It uses Resource(attributes=...) directly — not Resource.create(...) — which is the reason OTEL_RESOURCE_ATTRIBUTES is ignored (see gotchas).
3

Set up the tracer provider

TracerProvider(resource=resource) is registered globally and a tracer is fetched — but that tracer object is never used. All spans come from auto-instrumentation.
4

Parse export headers

OTEL_EXPORTER_OTLP_HEADERS is split on , into key=value pairs (each split with maxsplit=1, so a base64 token’s trailing = is preserved) and passed to both exporters. This is how the signoz-access-token reaches the collector.
5

Wire trace + log exporters

OTLPSpanExporter(headers=headers) behind a BatchSpanProcessor, and OTLPLogExporter(headers=headers) behind a BatchLogRecordProcessor. Neither exporter is given an endpoint argument — the endpoint is read from the environment by the SDK itself (this matters for the path footgun below). A LoggingHandler at LOG_LEVEL is attached to the Python root logger so every log line is also exported.
6

Auto-instrument

FlaskInstrumentor().instrument_app(app), RequestsInstrumentor().instrument(), and SQLAlchemyInstrumentor().instrument(engine=db.engine) (the last inside an app_context). These three libraries are the only sources of spans.
src/main.py:98
if settings.OTEL_EXPORTER_OTLP_ENDPOINT:
    resource = Resource(
        attributes={
            "service.name": settings.OTEL_SERVICE_NAME,
            "deployment.environment": settings.DEPLOYMENT_ENVIRONMENT,
        }
    )
    tracer_provider = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer_provider)
    tracer = trace.get_tracer(__name__)  # created but never used

    headers = {}
    if settings.OTEL_EXPORTER_OTLP_HEADERS:
        for header in settings.OTEL_EXPORTER_OTLP_HEADERS.split(","):
            key, value = header.split("=", 1)
            headers[key.strip()] = value.strip()

    # Endpoint is auto-read from OTEL_EXPORTER_OTLP_ENDPOINT; only headers passed.
    otlp_trace_exporter = OTLPSpanExporter(headers=headers)
    tracer_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))

    logger_provider = LoggerProvider(resource=resource)
    otlp_log_exporter = OTLPLogExporter(headers=headers)
    logger_provider.add_log_record_processor(BatchLogRecordProcessor(otlp_log_exporter))
    handler = LoggingHandler(level=settings.LOG_LEVEL, logger_provider=logger_provider)
    logging.getLogger().addHandler(handler)

    FlaskInstrumentor().instrument_app(app)
    RequestsInstrumentor().instrument()
    with app.app_context():
        SQLAlchemyInstrumentor().instrument(engine=db.engine)

​
What actually gets instrumented

SignalSourceProduces
TracesFlaskInstrumentorOne server span per HTTP request to any /api/v1/* route
TracesRequestsInstrumentorClient spans for outbound requests calls (VPN status probes, Janus/WHIP REST, ECR/S3 via boto3 are not requests-based so are not covered)
TracesSQLAlchemyInstrumentorSpans for DB queries issued through db.engine
Logsroot LoggingHandlerEvery Python log record at or above LOG_LEVEL, exported via OTLP
Socket.IO telemetry and rosbridge WebSocket traffic are not traced. Auto-instrumentation only covers the WSGI request lifecycle, requests, and SQLAlchemy. The high-volume real-time paths — Socket.IO events (src/routes/socket_routes.py) and the per-drone rosbridge connection pool (src/rosbridge/connection.py) — run outside the Flask request context and emit no spans. See Socket.IO Telemetry Streaming and Rosbridge Connection & Reconnect.

​
Environment variables

Only three OTEL_* vars are read by the code, plus LOG_LEVEL and DEPLOYMENT_ENVIRONMENT which feed the log-export level and the deployment.environment resource attribute. Full config lives in Gateway Environment Variables.
VariableDefaultEffect
OTEL_EXPORTER_OTLP_ENDPOINT(unset)Master switch. If unset, OpenTelemetry is entirely skipped. When set, must be the base OTLP/HTTP URL (e.g. http://host:4318); the SDK appends /v1/traces and /v1/logs.
OTEL_EXPORTER_OTLP_HEADERS(unset)Comma-separated key=value headers attached to both exporters (e.g. signoz-access-token=<redacted>).
OTEL_SERVICE_NAMEskyhub_gateway_serviceservice.name resource attribute.
OTEL_RESOURCE_ATTRIBUTESdeployment.environment={DEPLOYMENT_ENVIRONMENT}Defined but ignored — see below.
DEPLOYMENT_ENVIRONMENTserverSupplies the deployment.environment resource attribute (not OTEL_RESOURCE_ATTRIBUTES).
LOG_LEVEL20 (INFO); prod uses 10 (DEBUG)Level of the OTLP LoggingHandler. Records below this level are neither logged nor exported.
The OpenTelemetry SDK packages are declared unpinned in requirements.txt:76-83. The HTTP protobuf exporter (opentelemetry-exporter-otlp-proto-http) is the one imported — this is why the endpoint must be an HTTP OTLP endpoint. When you talk directly to a collector that means the HTTP port 4318, not the gRPC port (4317); the prod endpoint uses :4317, but that hits an nginx reverse proxy that forwards to the 4318 HTTP receiver (see below). 
OTEL_EXPORTER_OTLP_ENDPOINT=http://<office-docker-host>:4318/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=signoz-access-token=<redacted>
OTEL_SERVICE_NAME=skyhub_gateway_service

OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://<office-docker-host>:4318}
OTEL_EXPORTER_OTLP_HEADERS: ${OTEL_EXPORTER_OTLP_HEADERS:-signoz-access-token=<redacted>}
OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME:-skyhub_gateway_service}

OTEL_EXPORTER_OTLP_ENDPOINT=http://jumphost-private.skyhub-prod.internal:4317
OTEL_SERVICE_NAME=skyhub_gateway_service
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod
LOG_LEVEL=10

​
Footguns & gotchas

settings.py:139 defines OTEL_RESOURCE_ATTRIBUTES (default deployment.environment={DEPLOYMENT_ENVIRONMENT}), but initialize_opentelemetry builds the Resource with a hardcoded attribute dict using Resource(attributes=...) instead of Resource.create(...). Resource.create() is what merges the OTEL_RESOURCE_ATTRIBUTES env var; Resource(attributes=...) does not. Any extra resource attribute you set via that env var is dropped.Concrete consequence: docs/aws_prod_env_vars.env sets DEPLOYMENT_ENVIRONMENT=server and OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod. Because the code reads DEPLOYMENT_ENVIRONMENT, spans/logs are actually tagged deployment.environment=server — the prod value never takes effect. To change the environment tag today you must change DEPLOYMENT_ENVIRONMENT (which also flips CORS, JWT, and VPN-middleware behavior), not OTEL_RESOURCE_ATTRIBUTES.
The exporters are created with no endpoint argument, so the SDK reads OTEL_EXPORTER_OTLP_ENDPOINT itself and appends the signal path (/v1/traces, /v1/logs). That means the env var must be the base URL:
  • Correct: OTEL_EXPORTER_OTLP_ENDPOINT=http://<office-docker-host>:4318 → traces POST to .../v1/traces, logs to .../v1/logs. (This is the docker-compose.yml default.)
  • Broken: OTEL_EXPORTER_OTLP_ENDPOINT=http://<office-docker-host>:4318/v1/traces (the .env.example value) → the SDK appends again, so traces go to .../v1/traces/v1/traces and logs go to .../v1/traces/v1/logs. Both 404 at the collector, and log export is broken even if you only meant to configure traces.
If you ever need per-signal endpoints, set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (used verbatim, no path appended) — but the code doesn’t read those explicitly; they’re honored only via the SDK’s own env lookup.
The code imports the HTTP exporter (opentelemetry.exporter.otlp.proto.http), which speaks OTLP over HTTP. When you point it directly at a collector, use the HTTP port 4318 — 4317 is generically the OTLP gRPC port, and an HTTP exporter talking to a raw gRPC listener will not deliver spans. This is why the dev .env.example targets <office-docker-host>:4318 directly.The prod endpoint http://jumphost-private.skyhub-prod.internal:4317 (docs/aws_prod_env_vars.env:111) is not a bug despite the :4317. The WireGuard jumphost nginx (skyhub_terraform/nginx/default:174-181) has a server { listen 4317; location / { proxy_pass http://<office-docker-host>:4318; } } block that accepts the plain-HTTP OTLP POST on 4317 and reverse-proxies it to the OTEL HTTP receiver on 4318 — so the HTTP exporter posting to :4317 works in prod. Don’t “correct” it to :4318; that bypasses the jumphost ingress and aims the exporter directly at <office-docker-host>, breaking telemetry.
No manual span is opened anywhere in the codebase. The tracer = trace.get_tracer(__name__) line exists but is unused. Traces therefore exist only for Flask requests, outbound requests calls, and SQLAlchemy queries. Business logic in services (drone control, SITL orchestration, video rooms, billing) shows up only insofar as it runs inside an HTTP request span or issues DB queries — there are no custom child spans describing what it’s doing.
Each header must contain a literal =. A malformed OTEL_EXPORTER_OTLP_HEADERS entry without = raises ValueError at startup inside initialize_opentelemetry, failing the boot. Commas separate headers, so a value that itself contains a comma cannot be expressed. The base64 SigNoz token is safe because the split uses maxsplit=1, preserving trailing = padding.

​
Adding custom spans

Because the global TracerProvider is already registered by initialize_opentelemetry, any module can obtain a tracer and open spans without further setup — the challenge is purely that nobody does it yet. To instrument a code path (e.g. a rosbridge command or a SITL lifecycle step): 
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def push_mission(self, user_id, drone_id, mission_id):
    with tracer.start_as_current_span("push_mission") as span:
        span.set_attribute("drone.id", drone_id)
        span.set_attribute("mission.id", mission_id)
        ...  # existing logic; nested requests/SQLAlchemy spans attach automatically
Guard for the OTEL-disabled case: when OTEL_EXPORTER_OTLP_ENDPOINT is unset, no provider is configured and trace.get_tracer(...) returns a no-op tracer — start_as_current_span is safe to call and simply does nothing. You do not need to branch on whether OTEL is enabled.

​
Troubleshooting: traces/logs not showing in SigNoz

1

Confirm the endpoint is set at all

If OTEL_EXPORTER_OTLP_ENDPOINT is empty, initialization is skipped entirely — check the startup log for Initializing OpenTelemetry / OpenTelemetry initialized and instrumented (src/main.py:99, 134). No line means OTEL never ran.
2

Check for a doubled signal path

Ensure the endpoint is the base URL (http://host:4318), not .../v1/traces. A baked-in signal path produces /v1/traces/v1/traces and /v1/traces/v1/logs — the classic reason traces “kind of work in dev but logs never arrive.”
3

Check the port/protocol

HTTP exporter → HTTP port 4318 when talking directly to a collector; a direct endpoint on 4317 is aimed at the gRPC receiver and will silently fail to deliver. The prod :4317 endpoint is the exception — it goes through the jumphost nginx proxy that forwards to the 4318 HTTP receiver (see the port footgun), so leave it as-is.
4

Verify the access token header

OTEL_EXPORTER_OTLP_HEADERS=signoz-access-token=<redacted> must match a valid ingestion key on the collector; a rejected token drops data server-side.
5

Expect a batch delay

BatchSpanProcessor / BatchLogRecordProcessor buffer and flush asynchronously, so data appears after a short delay rather than instantly. Under the single gunicorn worker (Gateway Build, Docker & Runtime) there is one exporter per process; batches flush from a background thread.
6

Logs missing but traces present?

The log LoggingHandler fires at LOG_LEVEL. If LOG_LEVEL=20 (INFO) you will not see DEBUG records; prod sets LOG_LEVEL=10. Also confirm the endpoint path isn’t a trace-only URL (previous step) — that breaks the log exporter specifically.

Environment Variables

Full OTEL_*, LOG_LEVEL, and DEPLOYMENT_ENVIRONMENT reference plus every other Gateway knob.

Startup & Composition Root

Where initialize_opentelemetry sits in the boot sequence and why order matters.

Gateway Build & Runtime

Single gunicorn/gevent worker model that hosts the exporter.

Auth & Security Model

Why DEPLOYMENT_ENVIRONMENT changes far more than the OTEL tag.