Debugging Oblivious HTTP with Cloudflare’s Open-Source pvcli

Operating privacy-preserving technologies at scale introduces unique debugging challenges. When running protocols like Oblivious HTTP (OHTTP) across millions of requests per second, troubleshooting network issues requires inspecting complex cryptographic handshakes, multi-hop relay configurations, and binary-encoded HTTP payloads. Traditional network testing tools like Standard curl are not natively equipped to encapsulate binary HTTP (BHTTP) payloads or handle multi-party decryption flows natively.

To solve this, Cloudflare open-sourced pvcli (Privacy Client CLI) under the Apache-2.0 license. Designed with a familiar, curl-like command-line interface, pvcli abstracts the underlying cryptographic framing and multi-hop routing required for protocols such as OHTTP, MASQUE, and Privacy Pass, making end-to-end testing accessible and deterministic.

The Debugging Bottleneck in Multi-Party Privacy Protocols

Privacy-preserving architectures rely on distributed trust, where no single entity possesses complete context regarding both the requester’s identity and the requested content. Cloudflare powers privacy features for major implementations—including Apple’s Private Relay, Microsoft’s Edge Secure Network VPN, and Flo Health’s Anonymous Mode—using protocols built on this principle.

However, when a request fails in a multi-hop privacy system, identifying the point of failure is notoriously difficult. A standard request involves four distinct entities and multiple translation steps. Without specialized tooling, engineers must manually craft binary encodings, verify hybrid public key encryption (HPKE) key configurations, and correlate raw hexadecimal streams across isolated server logs.

Decoupling Identity from Intent: How Oblivious HTTP Operates

Oblivious HTTP (RFC 9458) guarantees anonymity by decoupling the IP address of the client from the target origin server. This architecture relies on two non-colluding proxy roles: an OHTTP Relay and an OHTTP Gateway.

The standard OHTTP execution lifecycle consists of eight distinct sequential steps:

1. Key Fetching: The client retrieves the public key configuration of the Gateway (often via a trusted configuration service or direct key endpoint).
2. Client Encryption: The client encapsulates the HTTP request into Binary HTTP (BHTTP) format and encrypts it using HPKE directed at the Gateway’s public key.
3. Relay Forwarding: The client sends the encrypted encapsulated message to the Relay. The Relay strip the client’s IP address and identity headers.
4. Gateway Decryption: The Relay forwards the encrypted payload to the Gateway. The Gateway uses its private key to decrypt the payload, recovering the inner BHTTP request.
5. Origin Request: The Gateway decodes the BHTTP payload into standard HTTP and forwards it to the final target origin server.
6. Origin Response: The target origin processes the request and returns a standard HTTP response to the Gateway.
7. Gateway Response Encryption: The Gateway converts the HTTP response into BHTTP format, encrypts it using the response context derived during request decryption, and sends it back to the Relay.
8. Client Decryption: The Relay returns the encrypted payload to the client, which decrypts it to extract the original HTTP response.

The Manual Debugging Process: Hex Parsing and BHTTP Encoding

Prior to pvcli, verifying an OHTTP deployment required manual construction and validation of each protocol step. To understand why this process was error-prone, consider an end-to-end test targeting an endpoint such as https://target.ohttp.info/anything.

1. Key Configuration Verification

Fetching the Gateway key configuration via standard HTTP returns a raw binary blob represented in hex format:

0029550020b9bb667e2230dc01c6d6cc047f94a1083beb185c63e50ec09f7692a5a0832540000400010001...

According to RFC 9458 Section 3, engineers had to break down the hex sequence manually to ensure algorithm compatibility:

• 0029: Length prefix indicating a 41-byte key configuration entry.
• 55: Key ID (Decimal 85) used to identify which key pair the Gateway should use for decryption.
• 0020: KEM (Key Encapsulation Mechanism) ID specifying DHKEM(X25519, HKDF-SHA256).
• b9bb667e...a0832540: The 32-byte X25519 public key payload.
• 0004: Symmetric algorithm length prefix (4 bytes).
• 0001 0001: Cipher suite identifiers mapping to HKDF-SHA256 (KDF) and AES-128-GCM (AEAD).

2. Binary HTTP Formatter Constraints

Next, the plaintext HTTP request must be converted into Binary HTTP (RFC 9292). A simple JSON POST request translates to the following framed bytes:

0204504f5354056874747073117461726765742e6f687474702e696e666f092f616e797468696e670c636f6e74656e742d74797065106170706c69636174696f6e2f6a736f6e0a757365722d6167656e740c6665727265742f302e312e30000a7b2274657374223a317d00

BHTTP uses strict length-prefixed encoding. A single character byte error invalidates the payload. For instance, in the hex sequence 0a 7b2274657374223a317d, the prefix 0a specifies a 10-byte body ({"test":1}). Adding an unintended trailing space (0x20) shifts the stream offset, causing the OHTTP Gateway to return an uninformative malformed request error without exposing which byte failed validation.

Introducing `pvcli`: A Native Command-Line Tool for Privacy Protocols

The pvcli tool eliminates manual cryptographic framing by integrating key discovery, BHTTP encoding, HPKE encryption, and transport-layer routing into a unified CLI. Built in Rust for performance and memory safety, pvcli can be installed via Cargo:

# Install Rust toolchain if needed
curl https://sh.rustup.rs -sSf | sh

# Install pvcli from official repository
cargo install --git https://github.com/cloudflareresearch/pvcli

Designed around the principle of least surprise, basic GET and HTTP/3 execution syntax matches curl standard patterns:

pvcli https://cloudflare.com/cdn-cgi/trace
pvcli --http3 https://cloudflare.com/cdn-cgi/trace

Executing End-to-End OHTTP Requests with `pvcli`

Instead of manually orchestrating key fetching, string concatenation, and HPKE encryption scripts, an entire OHTTP transaction across a Relay, Gateway, and Origin Target can be executed with a single command using the --ohttp flag:

pvcli -vvv --ohttp 
  --first-hop https://relay-cloudflare.ohttp.info 
  --proxy https://gateway.ohttp.info 
  -X POST 
  --header "content-type: application/json" 
  --data '{"test":1}' 
  https://target.ohttp.info/anything

By supplying the verbosity flag (-vvv), pvcli provides step-by-step trace logs of the internal cryptographic assembly, showing the parsed key configuration, BHTTP structure, and output ciphertext:

TRCE Full decoded client config: ClientConfig { key_configs: [KeyConfig { key_id: 85, key: PublicKey { kem_id: X25519HkdfSha256, bytes: b9bb667e... }, symmetric_algorithms: [(HkdfSha256, AesGcm128)] }] }
DEBG Creating POST request to https://target.ohttp.info/anything
TRCE BHTTP encoded request bytes, hex: 0204504f5354056874747073117461726765742e6f687474702e696e666f...
TRCE Encrypted request bytes, hex: 5500200001000164f03cef18f625f1dcd9fb26aa802081196bd6d7ba225bf6...

Advanced Relay Authentication: Custom Headers and mTLS

In production environments, OHTTP Relays often enforce access controls, such as bearer token authorization or mutual TLS (mTLS), to verify that incoming client traffic originates from authorized software applications. Because standard header arguments apply to the target origin inside the encrypted outer envelope, pvcli provides dedicated flags targeting the Relay (first hop).

1. Passing Relay Headers

To pass authentication tokens directly to the outer HTTP envelope bound for the OHTTP Relay, use --first-hop-header:

pvcli -vvv --ohttp 
  --first-hop https://relay-cloudflare.ohttp.info 
  --first-hop-header "authorization: Bearer relay-token-here" 
  --proxy https://gateway.ohttp.info 
  -X POST 
  --header "content-type: application/json" 
  --data '{"test":1}' 
  https://target.ohttp.info/anything

2. Authenticating via Mutual TLS

When the Relay requires cryptographic client identity verification at the transport layer, pvcli accepts client certificates and key files using --first-hop-client and --first-hop-key:

pvcli -vvv --ohttp 
  --first-hop https://relay-cloudflare.ohttp.info 
  --first-hop-client ./relay-client.pem 
  --first-hop-key ./relay-client.key 
  --proxy https://gateway.ohttp.info 
  -X POST 
  --header "content-type: application/json" 
  --data '{"test":1}' 
  https://target.ohttp.info/anything

Future Protocol Support and Open-Source Roadmap

While existing libraries provide language-specific OHTTP reference implementations, pvcli aims to serve as a unified testing utility across multiple modern privacy protocols. Cloudflare’s roadmap for pvcli includes expanding capabilities beyond basic OHTTP:

• MASQUE Protocols: Support for proxying TCP over HTTP/3, as well as UDP and IP encapsulation over HTTP/2 and HTTP/3.
• Privacy Pass: Native issuance and redemption flow testing for zero-knowledge privacy tokens.
• Post-Quantum Cryptography: Support for post-quantum hybrid KEMs within OHTTP key encapsulation workflows.
• Observability: Latency and timing metrics for individual network hops across multi-party paths.

By consolidating binary framing, HPKE encryption, and transport abstraction into an open-source tool, pvcli lowers the barrier to entry for developing, debugging, and maintaining privacy-preserving network architectures.

Primary reference: Review the original announcement for exact release details. This article is an independent explanation and does not reproduce the source text.