An OpenTelemetry API for PostgreSQL and its extensions

Bringing distributed tracing to the whole of PostgreSQL using a pluggable interface that enables any extension to emit traces
Schedule

Presented by:

Craig Ringer

EnterpriseDB

Craig is a PostgreSQL contributor with a long history in the community. At 2ndQuadrant he was an early developer on the BiDirectional Replication (BDR) project and pglogical - which later formed the foundations of PostgreSQL's built-in logical replication.

Craig's PostgreSQL contributions include replication slot catalog_xmin feedback, the txid_status feature, and the first implementation of libpq's pipelined query mode. He also found and reported the Linux/PostgreSQL fsync() data-loss bug that became known as 'fsyncgate'.

Craig currently works primarily on PostgreSQL observability in cloud-hosted environments. He makes numerous smaller contributions across many open source projects. His recent focus has been on full-stack OpenTelemetry distributed tracing in PostgreSQL and Kubernetes.

Craig lives in Wellington in New Zealand, where he likes to hike in the Tararua ranges or climb into the clouds.

No video of the event yet, sorry!

This talk presents otel_api, a generic OpenTelemetry distributed tracing and metrics API usable by all extensions. It explains why this API is needed - and how extension and application authors can benefit from it. I demonstrate working examples of useful tracing - including synchronous replication timing and tracing through foreign data wrapper (FDW) calls. Because the API is available as an extension for current PostgreSQL releases, I show how it can benefit users now - without having to wait for its inclusion in future PostgreSQL versions.

Distributed tracing helps operators observe and understand the behaviour of complex multi-layered systems in real-world conditions. Tracing can be immensely helpful for understanding how applications interact with databases and how databases execute real-world queries. But tracing support in postgres is limited, so it has not seen wide adoption in the postgres community to date. A few existing extensions can trace aspects of query execution within one postgres instance. They cannot trace queries between PostgreSQL instances, or from postgres into other systems, which is where distributed tracing really shines.

By introducing a new pluggable OpenTelemetry tracing and metrics API for PostgreSQL I hope to greatly expand the ability of users and developers to benefit from tracing. The new API decouples trace and metric collection instrumentation from exporters. It allows tracing to be extended into extensions whose primary purpose is unrelated to observability - such as storage access methods, foreign data wrappers and replication extensions. Unrelated extensions interoperate to produce integrated traces without needing to be aware of each other.

The metrics support means that extensions can also expose health and performance statistics directly via OpenTelemetry metrics without needing to integrate with an external SQL-to-metrics scraper.

I demonstrate a functional prototype API and implementation for unpatched PostgreSQL 18+ that supports distributed tracing across FDWs, synchronous replication and third-party extensions. The prototype adds no external library dependencies to traced components. It adapts the generic OpenTelemetry API to the unique needs of PostgreSQL's runtime environment and extension ecosystem, making it available for other extensions. Optional core patches are available to further improve tracing functionality - adding protocol-level trace context messages and structured key/value logging.

Distributed tracing is an extremely powerful operational and development tool when it is sufficiently well supported by all software in a stack. PostgreSQL query traces become visible in Tempo or Jaeger alongside the application requests that initiated them. Any downstream steps where the query calls upon other systems can be integrated into the same query trace tree. With further work it should even be possible to link together all participants in a deadlock cycle, see how queries waiting on a unique index affect each other, and much more. To make this possible it will be necessary to get some kind of lightweight OpenTelemetry support into PostgreSQL itself. This talk presents one design; feedback from the community will determine whether it or something like it belongs in PostgreSQL core.

Date:
Duration:
40 min
Room:
Conference:
PG Down Under 2026
Language:
Track:
Development
Difficulty:
Medium