Cloudflare Logpush Transformers Enable In-Pipeline SQL Filtering and Redaction

Cloudflare has made Logpush Transformers generally available, integrating SQL-based processing directly into the Logpush export pipeline. This allows observability teams to filter records, redact sensitive data, and reshape output structures before logs reach SIEMs, analytics platforms, or object storage, eliminating the need for separate external processing services.

How Transformers Work

A Transformer is a SQL query applied to a specific Logpush dataset before export. Records failing the WHERE clause are dropped, while the SELECT statement defines the final record structure. The output is exported in NDJSON format. Users can create and manage Transformers via the Cloudflare dashboard or API, with each update creating a new version for auditability.

Architecture and Performance

To maintain high throughput and reliability, Cloudflare embedded the transformation logic within its delivery service, Bufstream, rather than using external compute like Cloudflare Workers. This avoids network latency and transient timeouts in the export hot path. The system uses Apache DataFusion, a Rust query engine built on Apache Arrow, to execute SQL queries in-process.

Key architectural details include:

• Data Flow: Bufstream parses Cap'n Proto records into typed Apache Arrow RecordBatches before execution.
• Go/Rust Integration: Data is exchanged across the Go and Rust boundary using Arrow IPC via a C ABI and cgo, ensuring efficient, batch-oriented serialization without per-record parsing overhead.
• Performance: Benchmarks showed executing a prepared Transformer on a 5,000-row batch took about 3 ms, equivalent to roughly 1.6 million rows per second.
• Reliability: Transformers fail closed; if execution fails, the job does not advance, preventing untransformed or raw data from violating downstream schema assumptions.

Availability

Transformers are available on Free, Pro, Business, and Enterprise plans. Users can access Transformer Studio in the Cloudflare dashboard to write queries, preview results against sample records, and attach Transformers to compatible NDJSON Logpush jobs.