Add ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798
Add ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798aram356 wants to merge 45 commits into
ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798Conversation
- Compare r.from case-insensitively in RuleTable::first_match to enforce the lowercase invariant regardless of how Rule.from was built - Reject trailing-colon inputs (empty port string) as RuleError::Port in Authority::parse; add rejects_empty_or_missing_port test - Assert scheme_is_tls in rewrite_default_preserves_from_host_and_sets_sni_to_to and rewrite_host_uses_to_authority_with_port
Add ConfigError::BasicAuthFile variant with a path-carrying display message so file-not-found and permission errors are no longer reported as the misleading "--basic-auth must be USER:PASS" format error. The read failure now maps to BasicAuthFile; only parse failures map to BasicAuth. Includes a unit test that asserts the correct variant is returned for a missing file.
Replace the write-then-chmod pattern in CertAuthority::persist() with a single OpenOptions::create_new(true).mode(0o600) open, eliminating the window where the private key was briefly world/group-readable on disk.
…NNECT - Thread the raw buffered bytes through `RequestHead` so `blind_forward_http` can write the complete request head to the upstream before piping the rest of the socket bidirectionally (spec §8.4). Previously the head was discarded, sending a truncated/empty request. - Update `blind_forward_http` doc comment to reflect that it now replays the original head rather than falsely claiming it always did. - Add `unmatched_connect_off_loopback_is_refused_with_403` integration test. The proxy listener is bound on `127.0.0.1:0` (real socket) but `cfg.listen` is patched to `0.0.0.0:<port>` before being handed to `serve_on`, so `is_loopback` is computed as false while the test can still connect via loopback. Asserts that an unmatched `CONNECT` receives `403` and no tunnel is established.
…s-dev-proxy-spec # Conflicts: # Cargo.toml
Replace the dead `std::thread::park()` restore thread in `launch_safari`
with a file-based persist-and-recover scheme. Before applying the PAC
URL, write `<ca_dir>/safari-proxy-restore` capturing the network service
name and the prior auto-proxy URL (or an empty second line when
auto-proxy was off). Add `restore_system_proxy_if_pending(ca_dir)`,
which reads and deletes that file then runs the appropriate `networksetup`
command to put things back.
Wire it into `run()` in two places: at startup (crash recovery from a
previous hard-killed run) and in a `tokio::select!` on `ctrl_c()` (clean
exit). Also move the function-local `use std::time::{SystemTime,
UNIX_EPOCH}` out of `make_temp_dir` to the top-level imports per project
convention.
…-spec' into worktree-review-ts-dev-proxy-spec
…h, and service detection
…ell-quote restore URL
… trust revocation
…FROM validation - Abort `ca regenerate` when the old CA's keychain revocation can't be confirmed, so on-disk key material never outlives its OS trust. - Declare the CLI macOS-only via `compile_error!` on non-macOS targets (keychain, Safari, and networksetup are all macOS-specific), and gate the macOS-only helpers (`manual_restore_command`, `restore_auto_proxy`) while ungating `shell_quote` so the shared launch path compiles cleanly. - Shell-quote the keychain, cert, and profile paths in the `ca install` and Firefox `certutil` fallback instructions. - Validate rule FROM as a bare hostname before embedding it in the generated PAC, browser URL, and upstream Host header. - Ignore the workspace-excluded crate's `target/` directory. Spec, plan, and guide updated to match.
The macOS-only `compile_error!` lived inside the proxy module while all native deps stayed unconditional, so a build for the repo-default wasm32-wasip1 target failed first in tokio/ring/aws-lc-sys instead of with the intended "macOS only" error — an easy developer footgun (a plain `cargo check` in the crate inherits the wasm default from .cargo/config.toml). - Move every dependency (and dev-dependency) under `[target.'cfg(target_os = "macos")'.dependencies]` so unsupported targets build none of the native TLS/networking stack. - Lift the platform gate to the crate root: `compile_error!` in lib.rs and `#[cfg(target_os = "macos")]` on the command modules, the CLI types, and the binary entry point. The proxy module's own `compile_error!` is removed. - Gate the e2e test crate to macOS (its deps are now macOS-scoped). Now `cargo check --target wasm32-wasip1` emits exactly one error — the macOS-only message — with no dependency build attempts. Native build/test unchanged (31 unit + 6 e2e pass). Spec and plan updated to match.
Let `--to` target a bare IP and `--rewrite-host` carry the hostname that
endpoint expects. The flag now takes an optional value:
- omitted -> Host = FROM (default; unchanged)
- --rewrite-host -> Host = TO host (the prior bare-flag behavior)
- --rewrite-host <HOST> -> Host = <HOST> and TLS SNI = <HOST>
Connection still dials `--to`, so pointing at an IP works: the proxy presents
the explicit hostname for both SNI and Host while the socket goes to the IP.
Replaces `Rule.preserve_host: bool` with a `HostMode { PreserveFrom, UseTo,
Explicit }` enum threaded through `rewrite_for`; the explicit host is validated
as a hostname (new `ConfigError::InvalidRewriteHost`). Spec, plan, and guide
updated; adds tests for all three forms plus invalid-value rejection.
The `compile_error!` fired on a plain `cargo build` (no `--target`), which the repo's `.cargo/config.toml` resolves to `wasm32-wasip1` — so it tripped even on macOS and blocked the natural build command. Remove the gate. The deps and command modules stay scoped to `target_os = "macos"`, so unsupported targets build an empty shell instead of dragging tokio/ring/aws-lc-sys through an unsupported build. Build natively with an explicit `--target` (proper no-`--target` defaulting is separate, edgezero-aligned work). Also fix a stray `x` that slipped into a config test assertion, which broke the lib-test build. Spec and plan updated to match.
Replace the overloaded `--rewrite-host <HOST>` (which set both Host and SNI to reach an IP upstream) with two orthogonal knobs: - `--resolve HOST:IP` (repeatable, curl-style) pins where a hostname's upstream connection dials, leaving the SNI/Host derivation untouched. This is the self-contained way to reach a server by IP while keeping `--to` a hostname so the TLS SNI and certificate stay valid — no /etc/hosts edit. - `--rewrite-host` is a plain bool again: send `Host: TO` instead of `Host: FROM`. The SNI is always the TO host. Drops the `HostMode` enum in favor of `Rule.rewrite_host: bool` named and passed straight from the flag (no `preserve_host` inversion). Adds an e2e proving a non-resolvable TO host still reaches the upstream via --resolve. Spec and guide updated; the guide now leads with the --resolve flow.
The DNS pin only covered the matched-rule MITM upstream, so a CONNECT to a host that isn't a configured --from (a directly-hit host, a sub-resource) still went through the blind tunnel via real DNS and ignored the pin. Route all three connection paths (MITM, blind tunnel, plain-HTTP forward) through a shared `connect_upstream` helper that honors the pin case-insensitively. Also log each pin at startup (`--resolve pin: HOST -> IP`) so it's visible at the default info level — the per-connection summary shows the hostname (SNI/cert use it) even though the socket dials the pinned IP.
…n host Trusted Server derives request_host from X-Forwarded-Host (then Host) and anchors all first-party URL rewriting to it. The proxy now always sends X-Forwarded-Host: FROM — standard forward-proxy behavior — so TS emits production-host URLs (tsjs, GPT, DataDome, …) even when --rewrite-host sends Host: TO for an upstream that routes/validates on its own hostname. This decouples routing (Host) from the displayed first-party host: use --rewrite-host freely for host-validating upstreams without skewing the rewritten URLs onto TO. Adds an e2e asserting Host=TO + X-Forwarded-Host=FROM under --rewrite-host. Spec and guide updated (the earlier "avoid --rewrite-host for TS" guidance no longer applies).
aram356
changed the title
ts dev proxy: a local MITM dev proxy serving production hostnames from staging
- rewrite: match the port-stripped host with eq_ignore_ascii_case instead of allocating a lowercased copy per rule (FROM is already stored lowercase). - config: warn when a --resolve HOST matches no rule's TO host (a likely typo whose pin would otherwise silently never apply); still succeeds. Add a test. - config: document why is_valid_host rejects underscores. - browser: correct the Linux chrome_command comment (it doesn't fall back to chromium; the arm is unreached on the macOS-only build anyway).
Addresses two review findings on the dev proxy (the trusted-host design for --rewrite-host through the real adapter is tracked separately): - server: strip any inbound `Forwarded` before stamping `X-Forwarded-Host`. Trusted Server resolves the request host from `Forwarded` → `X-Forwarded-Host` → `Host`, so a client-supplied `Forwarded` would otherwise outrank the FROM host the proxy injects. Add a unit test. - docs: reconcile stale `X-Orig-Host`-only guidance in the spec/plan — the functional first-party-host header is `X-Forwarded-Host` (TS reads it for request_host); `X-Orig-Host` is an informational duplicate. Note the inbound-`Forwarded` strip in the spec/plan and the `RewriteOutcome` field doc.
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Automated review: I reviewed the new trusted-server-cli dev proxy implementation, CA/browser orchestration, CI wiring, and the related docs. CI is currently green and I did not find a blocking issue that requires requesting changes, but I found several correctness/security edge cases that should be addressed before relying on this tool broadly.
| Remove the old CA manually (Keychain Access), then retry.", | ||
| )); | ||
| } | ||
| std::fs::remove_file(&cert_path).ok(); |
There was a problem hiding this comment.
Automated review: P1 — ca regenerate can silently reuse the old CA if deletion fails.
These remove_file(...).ok() calls discard failures, and the following load_or_generate() will happily reload the existing cert/key if they are still readable. That means ts dev proxy ca regenerate can print regenerated CA even though the old key material remains in use, which breaks the security promise users rely on after a suspected key compromise. Please propagate deletion failures (or verify the new cert/key differs from the old pair) and only print success after the files were actually replaced.
| headers.insert(HeaderName::from_static(X_FORWARDED_HOST), value.clone()); | ||
| headers.insert(HeaderName::from_static(X_ORIG_HOST), value); | ||
| } | ||
| if let Some(auth) = basic_auth |
There was a problem hiding this comment.
Automated review: P1 — --allow-non-loopback plus --basic-auth exposes the injected credentials to any network client that can reach the proxy.
Matched CONNECTs are MITM'd even on non-loopback binds, and this branch injects the developer's upstream Authorization header for every matched request that lacks one. A LAN client can therefore proxy through https://<FROM>/ and reach the gated staging upstream using the developer's credentials. Please reject --basic-auth/--basic-auth-file with non-loopback listens unless a separate explicit unsafe flag or proxy authentication is configured, or otherwise restrict matched MITM traffic to trusted clients.
| // keeping emitted first-party URLs on the production host even when | ||
| // `--rewrite-host` sends `Host: TO` for routing/validation (spec §8.3). The | ||
| // `insert`s below already overwrite any inbound `X-Forwarded-Host`/`X-Orig-Host`. | ||
| headers.remove("forwarded"); |
There was a problem hiding this comment.
Automated review: P2 — plaintext upstreams lose the original HTTPS scheme signal.
For --upstream-plaintext, the browser still visited https://FROM, but the upstream leg is plain HTTP and the proxy only stamps host headers. Trusted Server derives URL-rewrite scheme from TLS metadata, Forwarded, X-Forwarded-Proto, or Fastly-SSL; with a local plaintext upstream it will default to http (or honor a spoofed inbound scheme header), so generated first-party URLs can be downgraded. Please strip inbound scheme-forwarding headers here and stamp an authoritative X-Forwarded-Proto: https (and/or the signal the local adapter expects) to preserve the browser-facing scheme.
| if cert_path.exists() { | ||
| let cert = cert_path.to_string_lossy(); | ||
| let profile = tmpdir.to_string_lossy(); | ||
| let certutil = Command::new("certutil") |
There was a problem hiding this comment.
Automated review: P2 — Firefox CA import is attempted against an uninitialized NSS database.
tmpdir is a freshly-created profile directory, but certutil -A -d <dir> expects an existing NSS DB (and without sql: it may also target the legacy DB format). On a clean temp profile this commonly fails with SEC_ERROR_BAD_DATABASE, after which Firefox launches without trusting the dev CA and proxied HTTPS fails despite --launch firefox. Please initialize the profile DB first (for example certutil -N --empty-password -d sql:<profile>) and import with -d sql:<profile>, or launch Firefox once to create the DB before importing.
| /// proxy; HTTP and other schemes bypass it (spec §9). | ||
| fn launch_chrome(cfg: &ResolvedConfig) { | ||
| let port = cfg.listen.port(); | ||
| let proxy_arg = format!("https=127.0.0.1:{port}"); |
There was a problem hiding this comment.
Automated review: P2 — browser launch ignores the configured listen address.
The CLI accepts arbitrary loopback/non-loopback --listen addresses, but Chrome (here), Firefox, and Safari hard-code 127.0.0.1:<port> when configuring the proxy/PAC URL. Launching with --listen [::1]:..., 127.0.0.2:..., or a specific non-loopback address can bind successfully while every launched browser points at the wrong endpoint. Please derive a browser-connect address from cfg.listen (normalizing wildcard binds intentionally) and use it consistently in Chrome, Firefox, Safari, and the manual instructions.
2 participants
Closes #824
Summary
Adds
ts dev proxy— a local TLS-terminating (MITM) developer proxy that serves a production publisher hostname from a dev/staging upstream by swapping the TLS SNI. A real browser shows the production domain in the address bar while a Compute/staging service answers, so you can exercise Trusted Server's host-anchored URL rewriting against real first-party behavior without DNS or/etc/hostschanges.What's included
New
tsdeveloper CLI (crates/trusted-server-cli, a native binary excluded from the wasm workspace):Proxy flags
--map FROM=TO(repeatable) or-f/--from+-t/--to HOST[:PORT]— the rewrite rule(s); rules are explicit (notrusted-server.tomlinference).--rewrite-host— sendHost: TOupstream (default keepsHost: FROMso core's URL rewriting stays anchored to the production host).--resolve HOST:IP(repeatable) — curl-style DNS pin; dials the given IP on every upstream path while keeping SNI/Hoston the hostname, so the cert still validates. Makes the proxy self-contained (no hosts-file edits).--basic-auth USER:PASS/--basic-auth-file PATH— injectAuthorizationwhen absent (clears a gated upstream).--listen ADDR(default127.0.0.1:18080),--launch chrome|firefox|safari,--insecure(accept self-signed upstreams),--allow-non-loopback.Example
--resolve staging.example.net:192.0.2.10dials192.0.2.10for the upstreamleg while leaving the TLS SNI and
Hostasstaging.example.net, so theupstream certificate still validates — the curl
--resolvemodel. Repeat theflag to pin multiple hosts. IPv6 targets work too (the value is split on the
first
:), e.g.--resolve staging.example.net:2001:db8::10.Architecture
TO) signed by a per-machine local CA; the decrypted stream is proxied request-by-request over keep-alive. Unmatched hosts are blind-tunnelled on loopback (never MITM'd), and aHostmatching no rule is refused421(no smuggling through the CONNECT-authority rule).ca installadds it to the login keychain;uninstall/regeneraterevoke prior trust before replacing key material)./proxy.pacroute; Chrome/Firefox/Safari launch helpers (Safari persists & restores system-proxy state across hard kills).X-Forwarded-Host/X-Orig-Host = FROMso the upstream can keep emitting first-party URLs on the production host even when--rewrite-hostsendsHost: TO.Safety
403(can't become an open proxy) unless--allow-non-loopback.cfg-scoped to macOS, so non-macOS targets compile to an empty shell with a clear message instead of a hardcompile_error!(which previously brokecargo buildunder the repo's wasm default target).Tests, CI, docs
tests/proxy_e2e.rs): MITM rewrite/forward, blind tunnel cert identity,--resolvepinning,--rewrite-hostForwarded-Host behavior, injected Basic auth, keep-alive multi-request,421/403guards.wasm32-wasip1by default).docs/guide/ts-dev-proxy.md(setup, trust, troubleshooting) + the design spec and implementation plan underdocs/superpowers/.Notes