Introduction

proxybroker is a Rust library and command-line tool that finds, checks, and serves public HTTP(S) and SOCKS4/5 proxies. It scrapes proxy lists from many providers, verifies that each proxy actually works, classifies its anonymity level, and can run a local rotating proxy server in front of the working set.

It is a from-scratch Rust/tokio rewrite of proxybroker2 (Python/asyncio), which is itself the maintained successor to the original ProxyBroker. All three are Apache-2.0; this crate is a derivative work carrying the same licence. See the Data & Licensing chapter for attribution and a statement of changes.

Why it exists

There was no Rust equivalent that shipped a real library API. The project is library-first: everything the CLI does is available as public types, and the proxybroker binary is a thin shell over the library. You can embed the broker, the checker, the pool, and a drop-in rotating connector directly in your own async Rust program.

Highlights

The three core verbs

Everything centres on three commands (mirrored by Broker methods):

A minimal find:

proxybroker find --types HTTP HTTPS --limit 10

The same thing as a library:

#![allow(unused)]
fn main() {
use proxybroker::{Broker, FindQuery, Proto, TypeSpec};
use futures_util::StreamExt;

let broker = Broker::builder().build();
let mut stream = broker.find(
    FindQuery::builder()
        .types(vec![TypeSpec::any(Proto::Http)])
        .limit(10)
        .build(),
).await?;
while let Some(proxy) = stream.next().await {
    println!("{}", proxy.addr());
}
}

Alongside the three verbs, the CLI also offers check (verify a list of proxies you already have) and, behind build features, top (a live terminal dashboard) and mcp (serve the live pool over MCP stdio).

How this book is organized

• Getting Started — install the CLI or library, then run the quick-start commands.
• CLI Reference — every subcommand and flag: overview & global options, grab, find, check, serve, top, mcp, and output formats.
• Library Guide — using the crate from Rust: the broker & queries, the Proxy type, the pool & selection, persistence, the rotating connector, and worked examples.
• Architecture — how it works inside: the module map, providers & scraping, the checking pipeline, geolocation & ASN, and the feature flags.
• Reference — data & licensing and observability.
• Project — roadmap, deferred backlog, the systematic refactor design record, and contributing.

Installation

Zuli ProxyBroker Extended v1.0.0 is published as a public GitHub Release with Linux, macOS, and Windows binaries plus SHA-256 checksums, and GHCR is the canonical container registry. The next release, v1.1.0 (distribution closure), is being prepared: it adds an Android (aarch64-linux-android) release binary, prepares crates.io publication, and adds a non-blocking Docker Hub mirror. None of the v1.1.0 artifacts or submissions exist yet — v1.1.0 is not published to crates.io and no v1.1.0 release or image has been produced.

Build from source

Local source builds require:

• Rust 1.85 or newer;
• Cargo; and
• Git.

In an existing checkout, build the release binary with the locked dependency set:

cargo build --release --locked

After the public Zuli repository is created, clone it with:

git clone https://github.com/zuli2021/zuli-proxybroker-extended.git
cd zuli-proxybroker-extended
cargo build --release --locked

The executable remains proxybroker at target/release/proxybroker. The Cargo package is zuli-proxybroker-extended, while the Rust library crate remains proxybroker. The pinned rust-toolchain.toml selects stable Rust and the package declares Rust 1.85 as its minimum.

Rust dependency

The v1.1.0 crate package is prepared for crates.io publication as zuli-proxybroker-extended, but publication has not occurred (owner-gated, preceded by a collision check immediately before upload). Until it does, depend on the stable tag as a Git dependency:

[dependencies]
proxybroker = { package = "zuli-proxybroker-extended", git = "https://github.com/zuli2021/zuli-proxybroker-extended.git", tag = "v1.0.0" }

Pin a tag or commit rather than relying on an unpinned branch.

Prebuilt static binary (install.sh)

After the first tagged Zuli release, the installer will download the matching release archive for Linux (musl) or macOS, verify its SHA-256 checksum, and install it without a build toolchain or sudo:

curl -fsSL https://raw.githubusercontent.com/zuli2021/zuli-proxybroker-extended/main/install.sh | sh

SHA-256 verification is mandatory: the installer requires either sha256sum or shasum and fails rather than installing an archive with a missing, malformed, or mismatched checksum. These environment variables control the future installation:

The binary is installed under PROXYBROKER_BIN_DIR, while the three legal files are installed under PROXYBROKER_DOC_DIR. Supported installer targets are x86_64/aarch64 Linux musl and x86_64/aarch64 Apple Darwin. Windows is not supported by install.sh.

The Linux release binary is intended to be a fully static musl build, with the geo database and provider list embedded.

Android (aarch64-linux-android)

v1.1.0 adds a canonical Android release binary: proxybroker-<tag>-aarch64-linux-android.tar.gz (executable proxybroker, plus LICENSE, LICENSE-DATA, NOTICE, README.md) and its .sha256 checksum, built with the same default feature set as desktop (cli, server, geo, geo-bundled) in a dedicated Android CI workflow. Install by downloading the matching asset from the release, verifying the SHA-256, and extracting it on an ARM64 Android device (for example inside Termux). Android is not covered by install.sh (Linux/macOS only).

Docker (FROM scratch)

The canonical image path is ghcr.io/zuli2021/zuli-proxybroker-extended. GHCR is the authoritative registry; a Docker Hub mirror (docker.io/zuli2021/zuli-proxybroker-extended) is planned for v1.1.0 as a non-blocking copy and never gates the canonical release path.

After a verified tagged release publishes an image, replace <tag> with the release version:

docker run --rm ghcr.io/zuli2021/zuli-proxybroker-extended:<tag> find --types HTTP --limit 5
docker run --rm -p 8888:8888 ghcr.io/zuli2021/zuli-proxybroker-extended:<tag> \
  serve --host 0.0.0.0:8888

Publishing a port from the container requires serve --host 0.0.0.0:8888. A local checkout can also build the repository Dockerfile, subject to the local Docker environment:

docker build -t zuli-proxybroker-extended .
docker run --rm zuli-proxybroker-extended find --types HTTP --limit 5

Feature flags in one paragraph

The crate is split into Cargo features so library users can pull in only what they need. The defaults — cli, server, geo, geo-bundled — give you the full binary with the local server and the bundled country database. Optional features add a metrics endpoint, a progress bar, SQLite/Redis persistence, a terminal dashboard (tui), an MCP server (mcp), a filesystem watcher (watch), and a drop-in hyper connector (connector). See Feature Flags for the full table and what each one enables.

A geo-free build

Building with --no-default-features gives you the library only, with no geo data, no server, and no CLI dependencies — and therefore no data-attribution obligation:

cargo build --no-default-features

You can also keep geolocation code while dropping the bundled database (turn off geo-bundled but keep geo) and supply your own database at runtime with --geo-db. See Geolocation & ASN and Data & Licensing for details.

Quick Start

This page walks through the most common commands. It assumes proxybroker is on your PATH (see Installation). For every flag, see the full CLI Reference.

Grab proxies (no checking)

The fastest path: scrape the providers and print addresses, without verifying them.

proxybroker grab --limit 10

Output is host:port, one per line. Results are unverified — many will not work. Use grab when you want raw candidates and will check them yourself. See grab.

Find checked HTTP proxies

find scrapes, checks that each proxy actually works, and classifies its anonymity level. --types is required.

proxybroker find --types HTTP HTTPS --limit 10

Only working proxies are emitted. Add --show-stats for an aggregate summary on stderr:

proxybroker find --types HTTP --limit 20 --show-stats

Restrict by country and anonymity level (--lvl applies to HTTP):

proxybroker find --types HTTP --countries US GB --lvl "High Anonymous"

See find for the full flag set (--dnsbl, --judges, --timeout, --max-conn, retry knobs, and more).

Serve a rotating proxy

Run a local proxy server that finds working proxies and rotates through them. Point any HTTP client at it and each request goes out through a pooled upstream.

proxybroker serve --types HTTP --host 127.0.0.1:8888

Then use it like any HTTP proxy:

curl --proxy http://127.0.0.1:8888 https://example.com

The pool tops itself up to --limit working proxies (default 100) and picks an upstream per request according to --strategy (best, round-robin, random, or sticky). See serve for selection strategies, health thresholds, authentication, and country filtering.

Check a list you already have

Verify proxies from a file or stdin instead of scraping. Input is host:port addresses.

# from a file
proxybroker check --types HTTP --infile proxies.txt

# from stdin
cat proxies.txt | proxybroker check --types HTTP SOCKS5

Only working proxies are emitted, exactly as with find. See check.

Machine-readable output

Every command that emits proxies supports --format. For example, NDJSON (one JSON object per line):

proxybroker find --types SOCKS5 --limit 10 --format json

Other formats include json-array, csv, url (scheme://host:port), and a custom --output-format template. See Output Formats.

Save and reload

Append every working proxy to an NDJSON file with --save, then reload it later without re-checking via check --load or serve --load:

proxybroker find --types HTTP --limit 50 --save working.ndjson
proxybroker serve --load working.ndjson

Next steps

• Browse the full CLI Reference for every subcommand and flag.
• Embed the broker in your own program via the Library Guide.
• Understand how checking works in The Checking Pipeline.

CLI Overview & Global Options

The proxybroker binary is a thin shell over the library. Every invocation has the shape:

proxybroker [GLOBAL OPTIONS] <SUBCOMMAND> [SUBCOMMAND OPTIONS]

--help and --version work at every level. The version string also prints the DB-IP attribution required by the bundled geo data’s CC BY 4.0 license (see Data & Licensing).

proxybroker --help
proxybroker find --help

Global options

These are declared global = true, so they may appear before or after the subcommand and apply to whichever command runs. proxybroker --log debug find … and proxybroker find --log debug … are equivalent.

Notes:

• --geo-db and --asn-db are only honored when the binary is built with the geo feature (on by default). See Feature Flags.
• --provider-dir may be passed multiple times; each directory’s configs are appended. Pair with --providers-only to replace the bundled registry entirely. See Providers & Scraping.

Subcommands

serve, top, and mcp are compiled in only when their feature is enabled. A stock build may not list them in --help.

Value syntax

Two argument types recur across the checking subcommands:

• Protocols (--types): case-insensitive wire names — HTTP, HTTPS, SOCKS4, SOCKS5, CONNECT:80, CONNECT:25. Pass several space-separated: --types HTTP HTTPS SOCKS5.
• Anonymity levels (--lvl): Transparent, Anonymous, or High (case-insensitive). Levels apply to HTTP only; other protocols ignore them.

Output formatting (--format, --output-format, --outfile, --save) is shared by grab, find, and check — see Output Formats.

grab

Gather proxy candidates from the providers without checking them. grab scrapes each configured provider and emits every candidate address it finds — none are dialed, so there is no guarantee any of them actually work. Use find when you want only proxies that pass a live check.

proxybroker grab --limit 50

Options

--countries filtering requires geo data, so a candidate’s country is only known when the binary is built with the geo feature (default) or a --geo-db is supplied.

Because grabbed proxies are unchecked, several template/CSV fields are empty or defaulted: there are no confirmed {{protocols}}, no {{anon}} level, and {{scheme}} falls back to http. Timing and error-rate fields read zero. Country and ASN are populated only if geo/ASN databases resolved the address.

Examples

Grab up to 100 US or German candidates as scheme://host:port URLs:

proxybroker grab --limit 100 --only-cc US,DE --format url

Grab everything into a file as NDJSON:

proxybroker grab --format json --outfile candidates.ndjson

Feed grabbed candidates straight into a check:

proxybroker grab --limit 500 | proxybroker check --types HTTP HTTPS

See check for validating a candidate list and find for the combined scrape-and-check path.

find

Scrape candidates from the providers and check that they work, classifying HTTP anonymity as it goes. find streams each proxy to output the moment it passes, so results appear incrementally rather than all at once.

proxybroker find --types HTTP HTTPS --limit 10

--types is required. Everything else has a default.

Selection & filtering

Judges, timing & concurrency

Retry policy

Capability filters

These probe and optionally require specific proxy behaviors. Filtering flags drop proxies that fail the requirement; --relaxed-validity loosens what counts as a passing check.

Output & reporting

The --show-stats summary aggregates every checked proxy (working or not), not just the ones written to output. --save writes only working proxies. --progress, --state, and the store backends are gated behind feature flags; without the backing feature, --state prints a notice and is otherwise inert.

Examples

Find 20 high-anonymity HTTP/HTTPS proxies in the US, as JSON, with a summary:

proxybroker find --types HTTP HTTPS --lvl High --only-cc US \
  --limit 20 --format json --show-stats

Deep check with the transient retry set and a liveness fallback, saving a reloadable pool:

proxybroker find --types HTTP --retry-on transient --max-tries 4 \
  --liveness-url https://httpbin.org/ip --save pool.ndjson

The library equivalent of these queries is shown in Broker & Queries and the checking_depth example.

check

Check a list of proxies you already have. check reads host:port addresses from stdin (or a file), dials each through the same checking pipeline as find, and streams the working ones to output. Unlike find, it does not scrape providers — you supply the candidates.

cat proxies.txt | proxybroker check --types HTTP HTTPS

--types is required unless --load is used (which emits already-checked proxies without re-checking).

Input

With --load, no network activity occurs and --types is ignored; the saved proxies are streamed straight to output. Timing fields are not persisted, so under --show-stats the avg_resp_time/error counters read zero while total/working and the protocol/anonymity/country breakdowns remain meaningful.

Selection & filtering

Judges, timing & concurrency

check uses a plain retry-on-timeout policy (--max-tries attempts); the richer --retry-on/--backoff-ms and capability filters of find are find-only.

Output & reporting

Input parsing is lenient: addresses are extracted line-by-line, and if nothing parses, check prints a notice to stderr and exits cleanly.

Examples

Check a file of addresses for working HTTPS proxies, saving winners:

proxybroker check --infile candidates.txt --types HTTPS \
  --save working.ndjson --show-stats

Re-emit a previously saved pool without touching the network:

proxybroker check --load working.ndjson --format url

Pipe grabbed candidates directly into a check:

proxybroker grab --limit 500 | proxybroker check --types HTTP --limit 50

See grab for producing an unchecked candidate list and find for the scrape-and-check path.

serve

Run a local rotating proxy server. It accepts client connections and relays each one through a pool of checked proxies, transparently retrying on a different proxy when one fails. Point any HTTP client (or a SOCKS5 client) at serve’s address and every request rides an upstream proxy picked from the live pool.

serve is behind the server feature, which is on by default. See feature flags for the optional add-ons (metrics, watch, store-sqlite, store-redis) that unlock some flags below.

# Fill the pool with working HTTP proxies, listen on 127.0.0.1:8888.
proxybroker serve --types HTTP

The server binds immediately and fills its pool in the background from a live find. It prints its listen address to stderr and runs until Ctrl-C.

How it fills the pool

There are two ways to source proxies:

--types and --load are mutually exclusive; one of them is required. The --load file is the NDJSON artifact written by find --save / check --save.

# Serve a previously-saved pool without re-finding.
proxybroker find --types HTTP HTTPS --limit 50 --save pool.ndjson
proxybroker serve --load pool.ndjson

Options

All find-style filters (--types, --lvl, --strict, --post, --dnsbl, --countries) are threaded into the pool-fill query. Protocol and anonymity values are exactly as in find (e.g. HTTP HTTPS SOCKS5 CONNECT:80).

Listener

Pool fill

Selection and eviction

Selection strategies

--strategy accepts:

With --prefer-connect, CONNECT:80 support is a tie-break for best/sticky (health still dominates) and a primary filter for round-robin/random.

Country filter

--countries (alias --only-cc) constrains the pool to specific ISO country codes. The filter is applied on admission — even on the --load path, which never ran find’s country filter — so a warm or bring-your-own pool is screened too. A proxy with no geolocation is rejected whenever a country filter is set.

# US or German exit proxies only. Both spellings are equivalent.
proxybroker serve --types HTTP --countries US DE
proxybroker serve --types HTTP --only-cc US,DE

Persistence and re-checking

These need a persistence build (the persist feature; see feature flags). --state additionally needs a store backend (store-sqlite or store-redis) to durably remember proxies; --recheck alone works on any persist build, keeping its scores in memory when no --state is given.

Metrics

With the metrics feature built in:

Any request to the metrics endpoint returns the current pool metrics in Prometheus text exposition format (version=0.0.4):

proxybroker_pool_size{scheme="http"}            <gauge>
proxybroker_pool_size{scheme="https"}           <gauge>
proxybroker_pool_error_rate_avg                 <gauge>
proxybroker_pool_resp_time_avg_seconds          <gauge>
proxybroker_pool_probe_latency_avg_seconds      <gauge>
proxybroker_evictions_total                     <counter>
proxybroker_rotations_total                     <counter>

Error rate is an aggregate gauge over the pool, not per-address — per-proxy labels would be unbounded cardinality for a rotating pool. Per-proxy detail lives behind the control API below.

proxybroker serve --types HTTP --metrics 127.0.0.1:9090
curl -s http://127.0.0.1:9090/

Live reload

With the watch feature built in:

# Edit pool.ndjson while the server runs; changes are reconciled into the live pool.
proxybroker serve --load pool.ndjson --watch

Client authentication (--auth)

--auth USER:PASS gates clients. An HTTP client without a matching Proxy-Authorization: Basic base64(user:pass) gets 407 Proxy Authentication Required before any pool proxy is touched. The same credential also gates the SOCKS5 front-end via RFC 1929 (username/password). Absent, the server is open.

proxybroker serve --types HTTP --auth alice:s3cret
curl -x http://alice:s3cret@127.0.0.1:8888 http://example.com/

The client’s gate credential is a hop-by-hop secret: it is stripped from the request before it is forwarded upstream, so it never leaks to the (untrusted) upstream proxy.

Upstream proxy authentication

If a pooled proxy carries its own credentials (a paid/authenticated upstream), the server applies them automatically — as SOCKS5 RFC 1929 during negotiation, or as Proxy-Authorization on a CONNECT/forward request. These upstream secrets are never serialized, so they stay out of --format json. Supplying authenticated upstreams is a library-level operation; see the serve_authenticated example.

The X-Proxy-Info header

The server tells the client which upstream served the request via an X-Proxy-Info: <host>:<port> header:

• On an HTTP forward request, it is injected after the response status line.
• On a CONNECT tunnel, it rides the 200 Connection established ack (the one place a CONNECT client can see it — the tunnel body is opaque).
• A SOCKS5 tunnel is opaque and carries no X-Proxy-Info; its success reply uses a stub bound address so the upstream identity is not leaked.

SOCKS5 front-end

The same listener accepts plain HTTP, HTTP CONNECT, and SOCKS5 — the protocol is auto-detected from the client’s first byte (0x05 ⇒ SOCKS5). Only the SOCKS5 CONNECT command is supported (BIND/UDP are rejected); all three address types (IPv4, IPv6, domain) work. With --auth, the SOCKS5 client must authenticate via RFC 1929, symmetric with the HTTP 407 gate.

proxybroker serve --types SOCKS5
curl --socks5 127.0.0.1:8888 http://example.com/

The proxycontrol control API

Introspect and steer a running server — without a restart — by sending it requests, as its own client, addressed to the magic proxycontrol host. These requests are intercepted before proxy selection, so they never consume a pool proxy. On an authenticated server the client-auth gate is checked first, so introspection cannot reveal pool membership.

# With the server as your HTTP proxy:
curl -x http://127.0.0.1:8888 http://proxycontrol/api/remove/203.0.113.5:3128
curl -x http://127.0.0.1:8888 "http://proxycontrol/api/history/url:http://example.com/"

See also

• find — the finding/checking pass whose filters serve reuses.
• top — a live TUI dashboard over the same pool.
• mcp — expose the live pool to agents over MCP.
• feature flags — which features gate which flags.

top

A live terminal dashboard (a top-style TUI) over a working proxy pool: a sortable table of proxies plus a response-time sparkline for the selected row. Built with ratatui.

top is behind the tui feature, which is off by default — it pulls in ratatui and crossterm. Build with it enabled:

cargo build --features tui
proxybroker top --types HTTP HTTPS

The command fills a pool by running find in the background (optionally warm-started from --state) and redraws the dashboard on a fixed interval.

Keep --log at its default warn. Higher log levels write to stderr, over the dashboard.

Options

Protocol values are exactly as in find.

Layout

The screen has three regions:

1. A one-line pool summary header:

   total <N> | http <N> | https <N> | avg err <x.xx> | avg resp <x.xx>s
   

2. A bordered Proxies table:

   Column	Contents
   Addr	host:port
   Protos	Confirmed protocols, comma-joined
   Err%	Rolling error rate (0.00–1.00)
   Resp(s)	Average response time, seconds
   Country	ISO country code (blank if geo absent)

3. A bordered Selected resp time (ms) sparkline of the highlighted row’s recent response-time history (up to 60 samples, one per refresh).

Keybindings

Any other key is a no-op. Sorting re-orders in place without re-fetching the pool.

See also

• serve — the rotating proxy server over the same pool machinery.
• mcp — expose the same live pool to agents.
• feature flags — enabling tui and store backends.

mcp

Serve a live proxy pool over the Model Context Protocol on stdio, so agent tooling can pull healthy proxies and feed failures back into the same eviction machinery. It is a thin veneer over the same pool used by serve.

mcp is behind the mcp feature, which is off by default. Build with it enabled:

cargo build --features mcp
proxybroker mcp --types HTTP HTTPS

stdout is the MCP JSON-RPC channel; all human-facing logging goes to stderr. The pool fills in the background, so get_proxy may return null until the first proxies land.

Options

Protocol values are exactly as in find. There is deliberately no --max-error-rate / --max-resp-time: those pool thresholds only gate re-admission via the server relay, which the MCP handlers never call.

Exposed tools

The server exposes exactly three tools:

get_proxy

Check out the best healthy proxy for a scheme, optionally filtered to a country. The proxy stays in the pool (it is returned immediately, so it keeps rotating by priority).

Arguments:

Result (or null if none is available):

{
  "proxy": "1.2.3.4:8080",
  "types": ["HTTP", "HTTPS"],
  "avg_resp_time": 0.42,
  "error_rate": 0.0
}

pool_status

A snapshot of the live pool. No arguments. Result:

{
  "total": 100,
  "working": 100,
  "by_protocol": { "HTTP": 80, "HTTPS": 40 },
  "by_country": { "US": 30, "DE": 12 },
  "avg_resp_time": 0.71,
  "errors": {}
}

report_dead

Report a proxy as dead so it is removed from the pool and no longer handed out. The failure happened out-of-process, so the honest action is removal — not synthesizing an error into the histogram.

Arguments:

Result:

{ "removed": true }

See also

• serve — the rotating proxy server over the same pool.
• top — a live TUI dashboard over the same pool.
• feature flags — enabling mcp and store backends.

Output formats

The grab, find, and check subcommands share one output layer. It offers a set of built-in formats (--format), a fully custom line template (--output-format), and — for the --show-stats summary on find/check — a choice of text or JSON (--stats-format).

Proxy output goes to stdout (or --outfile <PATH>). Summaries and progress always go to stderr, so they never mix with the proxy stream on stdout.

--format

The default is default.

proxybroker find --types HTTP --limit 5 --format url
# http://1.2.3.4:8080
# socks5://5.6.7.8:1080

The url scheme

The scheme is how a client dials the proxy, which is what tools like curl --proxy need:

• SOCKS5 proxies → socks5://
• SOCKS4 proxies → socks4://
• The whole HTTP family (HTTP, HTTPS, CONNECT:*) → http:// — these are all reached over plain HTTP. An HTTPS/CONNECT capability describes target traffic the proxy can tunnel, not a TLS endpoint on the proxy itself.

CSV columns

--format csv emits a header row followed by one row per proxy:

host,port,protocols,anon,country,resp_time,error_rate
1.2.3.4,8080,HTTP|HTTPS,High,US,0.42,0

Every field is comma-free by construction: protocols are |-joined, country is the ISO code only, and the rest are numeric — so no CSV quoting layer is needed. Unchecked (grabbed) proxies have empty protocols/anon/country columns.

json vs json-array

json is NDJSON — exactly one self-contained JSON object per line, ideal for streaming pipelines (jq -c, log ingestion). json-array wraps the same objects in a single well-formed [...] document (streamed, not buffered), for consumers that want one parseable array. An empty stream under json-array yields [].

--output-format templates

--output-format <TEMPLATE> renders each proxy through a custom line template, overriding --format (output is always plain lines — a template ignores JSON-array wrapping). Tokens are substituted; unknown {{...}} tokens are left literally, so the template needs no escaping.

proxybroker find --types HTTP --asn-db GeoLite2-ASN.mmdb \
  --output-format '{{proxy}} {{country}} AS{{asn}} {{asn_org}}'
# 1.2.3.4:8080 US AS15169 Google LLC

{{asn}} and {{asn_org}} only resolve when a global --asn-db <PATH> (a MaxMind-format ASN database) is supplied; nothing ASN-shaped is bundled, so without the flag both render empty.

--stats-format

find and check accept --show-stats, which prints an aggregate summary (by protocol / anonymity / country) to stderr after the run. --stats-format picks its rendering:

--stats-format is inert without --show-stats.

proxybroker find --types HTTP --limit 20 --show-stats --stats-format json 2> stats.json

See also

• find · grab · check — the subcommands that emit these formats.
• serve — consumes the NDJSON produced by find --save / check --save.

Broker

The Broker is the entry point to the library: it turns a set of providers into a stream of proxies. Two operations sit on top of it:

grab returns immediately (the work runs in a spawned task). find and check do their fail-fast setup up front — discovering the host’s external IPs and verifying at least one judge — so [Error::NoTypes], [Error::ExtIpUnknown], and [Error::NoJudges] surface from the await, not as a silently-empty stream.

The crypto provider note

This crate builds reqwest with rustls-no-provider (to keep aws-lc-rs out of the dependency graph, which is the musl cross-compile blocker). The trade-off: reqwest bakes in no crypto provider, so a bare reqwest::Client::new() panics until one is installed.

BrokerBuilder::build and Resolver::new call install_default_crypto_provider() for you, so the normal paths need nothing. You only call it yourself when you build a custom reqwest::Client to pass to BrokerBuilder::client:

#![allow(unused)]
fn main() {
use proxybroker::install_default_crypto_provider;

install_default_crypto_provider(); // idempotent; call before building your own reqwest::Client
let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(10))
    .build()?;
let broker = proxybroker::Broker::builder().client(client).build();
Ok::<(), Box<dyn std::error::Error>>(())
}

install_default_crypto_provider() is idempotent (a std::sync::Once), so calling it more than once is harmless.

BrokerBuilder

Broker::builder() returns a BrokerBuilder. Every setter is consuming (returns Self); unset fields fall back to defaults. build() is infallible.

With default features (geo-bundled), build() auto-attaches the bundled DB-IP Country-Lite database unless you supplied one or called without_geo(). See feature flags for what each feature pulls in.

#![allow(unused)]
fn main() {
use proxybroker::Broker;

let broker = Broker::builder().build(); // bundled providers + bundled geo
}

FindQuery

FindQuery describes what to find and check. types is required — an empty types makes find return [Error::NoTypes]. Every other field has a default. You can construct it as a struct literal (with ..Default::default()) or through FindQueryBuilder.

FindQueryBuilder

FindQuery::builder() returns a FindQueryBuilder. build() is infallible — the NoTypes guard lives in find/check, not the builder, so the builder stays composable. The builder covers the common fields; the A4/A6 capability flags (require_cookie, require_connect25, trust_check, …) have no builder setter — set those public fields directly on the struct.

GrabQuery

GrabQuery is much smaller — grabbing does not check, so there are no judge/timeout/retry knobs.

GrabQuery derives Default, so GrabQuery::default() grabs everything the providers return.

ProxyStream

Both grab and find return a ProxyStream, which implements futures_util::Stream<Item = Proxy>. Consume it like any stream:

#![allow(unused)]
fn main() {
use futures_util::StreamExt;
async fn f(stream: &mut proxybroker::ProxyStream) {
while let Some(proxy) = stream.next().await {
    println!("{}", proxy.addr());
}
}
}

The stream ends when the source is exhausted, the limit is reached, or the stream is dropped — dropping fires a cancellation token that aborts in-flight checks, so there is no detached-task leak.

For find (not grab), the stream also carries running statistics over every checked proxy, working or not. Read them after the stream is fully drained for a complete picture:

#![allow(unused)]
fn main() {
fn f(stream: &proxybroker::ProxyStream) {
if let Some(stats) = stream.stats() {
    // aggregate over all checked proxies — see the Stats type
    let _ = stats;
}
}
}

stats() returns Some only for find; it is None for grab (nothing is checked).

A real find example

This mirrors examples/find.rs. It finds up to ten working HTTP or HTTPS proxies and prints each as it is confirmed.

use futures_util::StreamExt;
use proxybroker::{Broker, FindQuery, Proto, TypeSpec};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let broker = Broker::builder().build();

    let mut proxies = broker
        .find(FindQuery {
            types: vec![TypeSpec::any(Proto::Http), TypeSpec::any(Proto::Https)],
            limit: Some(10),
            ..Default::default()
        })
        .await?;

    while let Some(proxy) = proxies.next().await {
        // schemes() is HTTP/HTTPS support; types() has the per-protocol anonymity level.
        println!(
            "Found proxy: {:<21} {:?}  {:.2}s",
            proxy.addr(),
            proxy.schemes(),
            proxy.avg_resp_time(),
        );
    }
    Ok(())
}

The same query via the builder, which normalizes limit(0) to unlimited:

#![allow(unused)]
fn main() {
use proxybroker::{FindQuery, Proto, TypeSpec};

let query = FindQuery::builder()
    .types(vec![TypeSpec::any(Proto::Http), TypeSpec::any(Proto::Https)])
    .limit(10)
    .build();
let _ = query;
}

rotating (feature connector)

Broker::rotating is the one-call convenience for the whole discover → pool → rotate pipeline. It runs find, feeds the resulting stream into a background-warming Pool, and wraps that pool in a ready RotatingProxyConnector:

#![allow(unused)]
fn main() {
pub async fn rotating(
    &self,
    query: FindQuery,
    cfg: RotateConfig,
) -> Result<RotatingProxyConnector, Error>
}

The pool fills as find streams; the connector routes each connection through a healthy proxy (dead ones self-eject via the pool’s health thresholds). It hands hyper a raw tunnel to the target and does not terminate TLS — for an https:// target, layer TLS over the connector (e.g. a hyper-rustls HttpsConnector).

#![allow(unused)]
fn main() {
use proxybroker::{Broker, FindQuery, Proto, TypeSpec};
use proxybroker::connector::RotateConfig;

async fn f() -> Result<(), Box<dyn std::error::Error>> {
let broker = Broker::builder().build();
let connector = broker
    .rotating(
        FindQuery {
            types: vec![TypeSpec::any(Proto::Http)],
            ..Default::default()
        },
        RotateConfig::default(),
    )
    .await?;
// Hand `connector` to a hyper_util Client — see the connector page.
let _ = connector;
Ok(())
}
}

For a pre-seeded pool or a custom PoolConfig, compose the pieces yourself: find → Pool::spawn → RotatingProxyConnector::from_pool.

See also

• Proxy — the value type each ProxyStream yields.
• Pool — feed a ProxyStream into a rotating proxy pool.
• Rotating connector — what rotating returns; a drop-in hyper Service.
• feature flags — geo, server, and friends.

Proxy

Proxy is the library’s value type: an address, the protocols it is expected to (and confirmed to) support, timing/error statistics, and geolocation. It is not a connection handle — the socket lives in the checker/negotiator and is passed in. Proxy is plain data plus a few recording methods.

Fields

Construct one with Proxy::new(host, port, expected_types). The address renders with addr(), which brackets IPv6 per RFC 3986:

#![allow(unused)]
fn main() {
use proxybroker::{Proto, Proxy};
use std::collections::BTreeSet;

let mut p = Proxy::new("1.2.3.4".parse().unwrap(), 8080, BTreeSet::from([Proto::Http]));
p.add_type(Proto::Http, None); // mark HTTP confirmed-working
assert_eq!(p.addr(), "1.2.3.4:8080");
}

Key methods

Geo model: Country and Region

geo is an Option<Country>. The bundled DB-IP Country-Lite database resolves the country only, so region/city stay None for it — they populate only from a richer MaxMind City database opened via --geo-db. The JSON shape is fixed either way.

#![allow(unused)]
fn main() {
pub struct Country {
    pub code: String,           // ISO country code, e.g. "US"
    pub name: String,           // e.g. "United States"
    pub region: Option<Region>, // subdivision — City DB only
    pub city: Option<String>,   // City DB only
}

pub struct Region {
    pub code: String, // ISO 3166-2, e.g. "CA"
    pub name: String,
}
}

ASN model: Asn

asn is network-ownership attribution, orthogonal to Country (geolocation and ownership are independent facts about an IP). It is populated only from a user-supplied ASN database (--asn-db); nothing ASN-shaped is bundled.

#![allow(unused)]
fn main() {
pub struct Asn {
    pub number: u32,         // e.g. 15169
    pub org: Option<String>, // e.g. "Google LLC"; None when the DB omits it
}
}

The v1 JSON contract

Proxy implements serde::Serialize and serde::Deserialize by hand. Serialize matches proxybroker2’s as_json — a nested object, not a flat struct — so serde_json::to_string(&proxy) is the replacement for the Python as_json() method. The top-level key set is frozen at v1: host, port, geo, asn, types, avg_resp_time, error_rate.

{
  "host": "1.2.3.4",
  "port": 80,
  "geo": {
    "country": { "code": "US", "name": "United States" },
    "region":  { "code": "",   "name": "" },
    "city": null
  },
  "asn": null,
  "types": [ { "type": "HTTP", "level": "High" } ],
  "avg_resp_time": 0.0,
  "error_rate": 0.0
}

The round-trip is lossy on stats by design. Serialize emits the computed avg_resp_time / error_rate for humans, but Deserialize restores only the persistent identity — host, port, geo, asn, confirmed types — and leaves expected_types / requests / errors / runtimes empty. A loaded proxy’s timing history restarts. asn serializes as null when no --asn-db resolved it; an empty country code maps back to geo: None.

Only additive, always-present, backward-compatible fields may join without a format bump; a breaking change must bump the --format variant (e.g. json2).

Credentials redaction

Credentials (username/password for an authenticated upstream proxy) is carried on Proxy and applied by the negotiator (SOCKS5 RFC 1929) and the server (HTTP Proxy-Authorization). It is never serialized — secrets stay out of --format json — and its Debug impl is redacted, so a Proxy debug print cannot leak the secret. Attach it builder-style:

#![allow(unused)]
fn main() {
use proxybroker::{Credentials, Proto, Proxy};
use std::collections::BTreeSet;

let creds = Credentials { username: "user".into(), password: "pass".into() };
let p = Proxy::new("1.2.3.4".parse().unwrap(), 8080, BTreeSet::from([Proto::Socks5]))
    .with_auth(creds);
assert!(p.auth().is_some());
}

Because auth is never serialized, it is never deserialized either — a loaded proxy always has auth: None.

Caps and trust

Two more profiles ride on a checked proxy, neither serialized:

• Capabilities (from capabilities()): cookie_echo, referer_echo, and connect25 (a confirmed SMTP tunnel, derived from the types). caps() returns the raw Caps — the two header-echo flags — OR-accumulated across protocols. These back the --require-cookie / --require-referer / --require-connect25 filters.
• Trust (from trust()): the honeypot verdict. Empty (trusted) unless --trust-check ran. Backs --require-trusted.

Saving and loading: read_ndjson / write_ndjson

The library ships flat-file persistence as NDJSON — one serde_json object per line, the exact bytes --format json emits. This is the minimal persistence step (no schema/index/migration).

#![allow(unused)]
fn main() {
use proxybroker::{read_ndjson, write_ndjson, Proto, Proxy};
use std::collections::BTreeSet;
use std::io::Cursor;

let mut a = Proxy::new("1.2.3.4".parse().unwrap(), 8080, BTreeSet::new());
a.add_type(Proto::Http, None);

let mut buf = Vec::new();
write_ndjson(&mut buf, &[a])?;                     // W: std::io::Write
let back = read_ndjson(Cursor::new(buf))?;         // R: std::io::BufRead
assert_eq!(back.len(), 1);
Ok::<(), Box<dyn std::error::Error>>(())
}

write_ndjson is generic over std::io::Write; read_ndjson over std::io::BufRead. read_ndjson skips blank lines and aborts on the first malformed line with a std::io::ErrorKind::InvalidData error.

See also

• Broker — produces a stream of Proxy values.
• Pool — a rotating pool of checked proxies.

Pool

Pool is a rotating pool of checked proxies, refilled from a Stream<Item = Proxy> (typically a ProxyStream from find) and drained one proxy at a time to serve client requests. It lives behind the server feature and is re-exported from proxybroker::server.

The pool avoids proxybroker2’s heapq selection (which raises TypeError on tied f64s, since Python compares the Proxy objects and they define no __lt__). Selection here orders ties with f64::total_cmp, so equal response times are deterministic, never fatal.

Building a pool

Both return an Arc<Pool>. On import, each proxy is screened against config.countries — a warm or BYO pool that never went through find’s country filter is still admission-checked.

#![allow(unused)]
fn main() {
use futures_util::stream;
use proxybroker::server::{Pool, PoolConfig};
use proxybroker::{Proto, Proxy};
use std::collections::BTreeSet;

async fn f() {
let mk = |ip: &str| {
    let mut p = Proxy::new(ip.parse().unwrap(), 8080, BTreeSet::from([Proto::Http]));
    p.add_type(Proto::Http, None); // confirmed-working for HTTP
    p
};
let source = stream::iter(vec![mk("203.0.113.1"), mk("203.0.113.2")]);

let pool = Pool::spawn(source, PoolConfig::default());
pool.wait_ready(1).await;                 // block until warm (or source exhausted)
println!("pool warmed: {} proxies", pool.len());
}
}

wait_ready(n) blocks until at least n proxies are pooled or the source is exhausted — so a too-small source can never hang startup forever. wait_ready(0) returns immediately.

PoolConfig

PoolConfig tunes eviction and selection. It implements Default.

Strategy

Strategy chooses which eligible upstream serves each request:

Selection is two-tiered: ready proxies (never benched, or the bench window elapsed) are ranked first; only if none are ready does the pool fall back to benched ones (better than a 502).

ClientKey

Strategy::Sticky keys each session on a ClientKey:

#![allow(unused)]
fn main() {
pub enum ClientKey {
    Ip(IpAddr),      // the client's peer IP (the default)
    Header(String),  // the value of --sticky-header, HTTP requests only
}
}

Checking proxies in and out

Mutating a live pool

remove is exactly what GET http://proxycontrol/api/remove/<ip:port> does to a running server:

#![allow(unused)]
fn main() {
use proxybroker::server::{Pool, PoolConfig};
use proxybroker::{Proto, Proxy};
use std::collections::BTreeSet;
fn f(pool: std::sync::Arc<Pool>) -> Result<(), Box<dyn std::error::Error>> {
let removed = pool.remove("203.0.113.2".parse()?, 8080);
println!("removed → {removed}; pool now has {}", pool.len());
Ok(()) }
}

PoolSnapshot

pool.snapshot() returns a cheap PoolSnapshot — a live view taken under a single lock:

#![allow(unused)]
fn main() {
pub struct PoolSnapshot {
    pub http: usize,          // proxies serving Scheme::Http
    pub https: usize,         // proxies serving Scheme::Https
    pub total: usize,
    pub avg_error_rate: f64,  // mean over the pool
    pub avg_resp_time: f64,   // mean over the pool, seconds
}
}

For a richer aggregate (counts by protocol/anonymity/country, latency percentiles), feed pool.proxies() into Stats::from_proxies:

#![allow(unused)]
fn main() {
use proxybroker::{server::Pool, Stats};
fn f(pool: &Pool) {
let stats = Stats::from_proxies(&pool.proxies());
let _ = stats.total;
}
}

The pool also exposes cumulative counters evictions() and rotations() (both u64).

serve() and ServerHandle

serve starts the local rotating proxy server on addr, relaying every client connection through the pool and retrying on a different proxy when one fails.

#![allow(unused)]
fn main() {
pub async fn serve(
    addr: SocketAddr,
    pool: Arc<Pool>,
    resolver: Arc<Resolver>,
    timeout: Duration,
    min_queue: usize,        // wait for this many pooled proxies before serving (B13)
    backlog: u32,            // TCP listen backlog
    auth: Option<String>,    // "user:pass" gate; Some → 407 without matching credentials
) -> std::io::Result<ServerHandle>
}

It binds immediately (so local_addr() works at once) and runs the accept loop in a background task. ServerHandle controls its lifetime:

#![allow(unused)]
fn main() {
use proxybroker::server::ServerHandle;
fn f(handle: ServerHandle) {
let addr = handle.local_addr(); // useful when bound to port 0
handle.shutdown();              // stop accepting and shut down
let _ = addr; }
}

Dropping the handle also shuts the server down.

See also

• Broker — find produces the ProxyStream that fills a pool.
• Proxy — the value type the pool holds.
• feature flags — server, metrics, and friends.

Persistence (--state)

By default Zuli ProxyBroker Extended keeps a proxy’s history for a single process. A flat find --save/serve --load snapshot restores which proxies to try, but it cannot accumulate a success EWMA across runs or record “last seen 3 days ago”. Durable cross-run state (internally “D2”) escalates to a real store only for that history, behind feature gates so pure-library users stay zero-dependency. The relevant module is proxybroker::persist.

The Store trait

Every backend implements one small, synchronous contract:

#![allow(unused)]
fn main() {
pub trait Store: Send + Sync {
    /// Fold one finished proxy's current-session outcome into its durable
    /// record (upsert on `(host, port)`): accumulate requests/errors, move the
    /// success/latency EWMAs, bump uptime.
    fn upsert(&self, proxy: &Proxy) -> Result<(), Error>;

    /// Reconstruct every remembered proxy for a warm start, seeding
    /// priority-relevant aggregates.
    fn load(&self) -> Result<Vec<Proxy>, Error>;
}
}

The trait itself lives behind the persist feature and pulls in no backend dependency. A pure-library user can implement Store for any backend of their own; the store-sqlite and store-redis features each provide a concrete one.

The observer hook

The write seam is already a plain observer on the broker — a CheckObserver, i.e. Arc<dyn Fn(&Proxy) + Send + Sync> — so any Store plugs in without new machinery. You install it with Broker::builder() / with_observer, and warm-start the pool from Store::load:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use proxybroker::persist::{SqliteStore, Store};
use proxybroker::Broker;

let store = Arc::new(SqliteStore::open("proxies.db")?);

// Warm start: reconstruct remembered proxies before find runs.
let history = store.load()?;

// Fold every checked proxy's outcome back into the store as it finishes.
let upsert = store.clone();
let broker = Broker::builder()
    .with_observer(Some(Arc::new(move |p| {
        let _ = upsert.upsert(p);
    })))
    .build();
}

with_observer takes an Option<CheckObserver>; passing None (the default) disables it. The broker calls the observer once per finished check.

CLI: --state <PATH_OR_URL>

The binary wires all of this up from one flag on find and serve:

# SQLite: a filesystem path.
proxybroker --state proxies.db find --types HTTP --limit 20

# Redis: a redis:// or rediss:// URL.
proxybroker --state redis://127.0.0.1/0 serve --port 8888

--state remembers proxies across runs: it warm-starts the pool from stored history and folds each fresh check back in. The backend is chosen by the spec — a redis:// / rediss:// URL selects Redis, anything else is treated as a SQLite file path. If the matching backend feature is not compiled in, the CLI prints a hint (--state <spec>: a file path needs --features store-sqlite) rather than silently doing nothing.

--state gives adaptive re-checking (--recheck) a durable score to re-check into, so scores survive restarts. But --recheck no longer requires --state: without it, the re-checker folds into an in-memory MemoryStore instead (scores reset on restart). See serve for the --recheck* cadence flags.

Warm start: Proxy::restored

Both backends reconstruct proxies through Proxy::restored:

#![allow(unused)]
fn main() {
pub fn restored(
    host: IpAddr,
    port: u16,
    types: BTreeMap<Proto, Option<AnonLevel>>,
    requests: u32,
    errors_total: u32,
    avg_resp_time: f64,
) -> Proxy
}

The reconstruction is lossy on the error histogram, faithful on the error rate: errors_total is seeded under a single bucket, so per-bucket breakdowns are gone but error_rate() is exact. avg_resp_time is seeded as one runtime sample, so avg_resp_time() returns it. Warm start only needs priority(), never the per-bucket breakdown — so nothing priority-relevant is lost.

SqliteStore (store-sqlite)

The bundled SQLite backend. rusqlite is compiled with its bundled feature — SQLite is statically linked from source, so there is no dependency on a system libsqlite3 (this matches the static-musl goal). One denormalized proxies table keyed on (host, port), no per-attempt rows.

#![allow(unused)]
fn main() {
use proxybroker::persist::SqliteStore;

let store = SqliteStore::open("proxies.db")?; // creates the file if absent
}

open sets journal_mode = WAL and a 5-second busy_timeout so the re-checker (which opens a second connection to the same DB) and the upsert observer can write concurrently without an un-retryable SQLITE_BUSY. The single connection sits behind a Mutex (rusqlite’s Connection is Send but not Sync), which makes Arc<SqliteStore> shareable across tasks.

The atomic EWMA fold is done in SQL with ON CONFLICT(host, port) DO UPDATE — ewma_success = 0.3 * excluded.ewma_success + 0.7 * proxies.ewma_success — so each check is one atomic round-trip. A failing re-check’s empty types are guarded: the prior confirmed types are kept rather than erased.

SCHEMA_VERSION

#![allow(unused)]
fn main() {
pub const SCHEMA_VERSION: i64 = 1;
}

The current on-disk schema version, written to PRAGMA user_version. open runs migrate, which reads the stored version and creates the table if it is below 1. A schema change bumps this constant and adds a migration arm.

MemoryStore (persist)

The in-memory backend: a HashMap<(host, port), Record> behind a Mutex (so Arc<MemoryStore> is shareable across the re-check tasks), gated on the persist feature alone — no backend dependency. It exists so serve --recheck can keep an adaptive re-check’s decay/score bookkeeping without a --state SQLite/Redis backend; the state lives only for the process and nothing survives a restart.

#![allow(unused)]
fn main() {
use proxybroker::persist::MemoryStore;

let store = MemoryStore::new(); // empty; no file, no connection
}

Its upsert fold reproduces SqliteStore’s ON CONFLICT arithmetic byte-for-byte — alpha = 0.3 new + 0.7 prior on the success/latency EWMAs, accumulate requests/errors/uptime, and keep the prior confirmed types on a failing (empty) sample — so a MemoryStore-backed re-check decays identically to the durable path. load reconstructs through Proxy::restored like both durable backends.

This is what the CLI selects when --recheck runs without --state (or on a persist-only build with no backend compiled in); it prints re-checking into memory only (no durable --state); scores reset on restart.

RedisStore (store-redis)

The Redis backend (added in Wave 9). One blocking redis::Connection behind a Mutex, mirroring SqliteStore’s connection handling (redis::Connection is Send but not Sync). It works with local Redis over redis:// and managed Redis (ElastiCache / Upstash / Redis Cloud) over rediss://; TLS uses the crate’s ring-only rustls with bundled webpki roots, so there is no OS cert-store dependency.

#![allow(unused)]
fn main() {
use proxybroker::persist::RedisStore;

let store = RedisStore::open("redis://127.0.0.1/0")?;
}

The atomic upsert that SqliteStore does with SQL ON CONFLICT is a Lua EVAL here: Redis runs scripts single-threaded, so the whole read-modify-write fold is race-free without a WATCH/MULTI transaction, even with two RedisStores on a shared fleet upserting concurrently. It reproduces the SQLite arithmetic byte-for-byte (alpha = 0.3), including the confirmed-types guard.

Each proxy is a Redis hash under proxybroker:proxy:<host>:<port>; a set at proxybroker:proxies enumerates members for load. A string key proxybroker:schema stands in for SQLite’s PRAGMA user_version; a present-but-different value is a hard error (redis store schema mismatch) rather than a silent misread of old-shape hashes.

Feature gates

See feature flags for the full matrix. The Store trait is synchronous, so store-redis uses redis-rs’s blocking Connection — no tokio-comp — matching the trait’s shape.

Errors from either backend surface as Error::Persist(String).

Rotating connector (connector)

RotatingProxyConnector is a drop-in [tower_service::Service<Uri>] that routes every outbound connection through a rotating pooled proxy — so a Rust program gets pooled, self-healing proxy rotation with no local server and no listening port. It is gated behind the connector feature (internally “E1”).

Per connection it checks out a healthy proxy from a Pool, negotiates the tunnel with the shared negotiator, retries a different proxy on failure (dead ones self-eject via the pool’s existing health thresholds), and hands hyper the negotiated byte stream.

RotateConfig

#![allow(unused)]
fn main() {
#[derive(Debug, Clone)]
pub struct RotateConfig {
    /// Proxies to try (each a different checkout) before returning an error.
    pub max_tries: usize,
    /// Per-connection negotiation timeout.
    pub timeout: Duration,
}
}

RotateConfig::default() is max_tries: 3, timeout: 8s.

Plugging into hyper-util

reqwest 0.13 exposes no custom-connector hook, so the intended drop-in target is hyper_util::client::legacy::Client, not reqwest::Client. You build the connector from an already-fed pool with from_pool, then hand it to the client builder:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use std::time::Duration;
use proxybroker::connector::{RotatingProxyConnector, RotateConfig};
use proxybroker::resolver::Resolver;
use proxybroker::server::{Pool, PoolConfig};
use hyper_util::client::legacy::Client;
use hyper_util::rt::TokioExecutor;

// The pool must already be populated — via Pool::spawn(find_stream, ..) or
// Pool::from_proxies(..). The connector wraps an existing pool + resolver.
let resolver = Arc::new(Resolver::new(Duration::from_secs(8))?);
let connector = RotatingProxyConnector::from_pool(
    pool,          // Arc<Pool>
    resolver,
    RotateConfig::default(),
);

let client: Client<_, http_body_util::Empty<bytes::Bytes>> =
    Client::builder(TokioExecutor::new()).build(connector);
// Every request this client makes now dials through a rotating pooled proxy.
}

from_pool is the honest seam: the pool must already be populated (there is no hidden find inside the connector). The service is always ready — checkout happens per call, in Service::call, which runs the retry loop over up to max_tries proxies. Each failed dial records the error so the pool benches or ejects the proxy through its normal thresholds; the response type is ProxyConn, a bare negotiated tunnel wrapped for hyper.

Scope (v1): tunnel-only

The connector gives hyper a transparent byte stream to the target; hyper then speaks origin-form HTTP over it. That is correct for CONNECT/SOCKS tunnels, where the stream really reaches the target. Concretely, tunnel_proto prefers SOCKS5 → SOCKS4 → CONNECT:80, and only falls back to plain-HTTP passthrough when the target scheme is http.

A plain forward-HTTP proxy (which needs absolute-form requests) is not the intended fit — prefer CONNECT/SOCKS proxies. An HTTPS-only proxy for a given connection is skipped and another is tried.

Deferred: TLS-to-target

For an https:// URL the connector returns the tunnel and the caller layers its own end-to-end TLS (e.g. a hyper-rustls HttpsConnector wrapping the connector). Terminate-and-verify TLS to the target is a later feature with its own consumer. See the deferred backlog.

The Broker::rotating() convenience constructor that was deferred here has now shipped — see Broker::rotating for the one-call find → pool → connector pipeline.

Security note

The checker uses a liveness-only AcceptAllVerifier when it probes a proxy’s TLS — that is fine for deciding whether a proxy is alive, but it accepts any certificate. The connector deliberately never reuses it for real client traffic: doing so would be a silent MITM hole. The server’s protocol picker can return Proto::Https (which upgrades TLS to the target with the accept-all verifier) and the SMTP-specific Connect25; both are excluded from the connector’s tunnel_proto, so it only ever hands hyper a plain, un-terminated byte stream. The caller’s own TLS stack — not the checker’s — validates the target certificate.

Feature dependencies

The connector feature pulls in server (for Pool), tower-service, and hyper-util/client-legacy. See feature flags.

Examples

Every runnable example lives in the repository’s examples/ directory. After the public Zuli repository is created, it will be available at the planned examples/ path. Run one with cargo run --example <name>.

Six examples declare required-features = ["server"] in Cargo.toml. Because server is part of the default feature set, the plain cargo run --example <name> command works out of the box — you only need an explicit --features flag if you build with --no-default-features. The other examples depend only on the core library and need no features at all.

Find and grab

Serving (server feature — on by default)

If you build without default features, add the flag explicitly, e.g.:

cargo run --example serve --no-default-features --features server

Related

• Broker — find, grab, and the builder used throughout.
• Pool & server — Pool, PoolConfig, and serve.
• Persistence — warm-starting from stored history.
• feature flags — the full feature matrix.

Architecture overview

Zuli ProxyBroker Extended is an independently maintained derivative of proxybroker-rs, itself a Rust/tokio port of Python’s proxybroker2. Its Rust library crate remains proxybroker and is organized as a set of small modules with exactly one home for each concept, wired together by the Broker. This page is the map: what each module does, how data flows through a find / grab / serve run, and the one ownership rule that shapes the whole design.

Module map

Every pub mod from src/lib.rs. Modules marked (feature) only compile when their cargo feature is enabled.

Data flow

grab — scrape only

providers (ProviderSpec)  ──fetch──►  page body  ──extract──►  Candidate { host, port, protocols }

Broker::grab fetches each provider concurrently and runs ProviderSpec::extract, which scans the whole page for IP:port pairs, canonicalizes the IP, and deduplicates. No proxy is contacted — a grabbed Candidate is unverified.

find — scrape, then check

grab ──► Candidate ──► Proxy (unchecked)
                          │
                          ▼
             Checker::check(&mut proxy)
                          │
   resolver → negotiator → judge/liveness → anonymity + trust
                          │
                          ▼
             ProxyStream yields working Proxy

Broker::find feeds each candidate to the Checker. The checker connects, negotiates the protocol, sends a test request to a judge, and classifies the result. Only proxies that pass are yielded on the ProxyStream, up to the query limit.

serve — check, then route

serve runs a find internally to fill a live Pool, then listens locally and forwards each client connection through a selected upstream proxy (by Strategy). The server feature is required.

Ownership: Proxy is plain data, the socket lives in the checker

A Proxy is a plain value: an IP, a port, its confirmed protocols and anonymity levels, geolocation, and timing stats. It owns no socket, no connection, no task. It is Clone, serializable to NDJSON, and cheap to pass around.

Every live resource — the TcpStream, the negotiated Stream, the judge round-trip — is created, used, and dropped inside Checker::attempt. The checker opens TcpStream::connect((proxy.host, proxy.port)), negotiates, sends the request, reads the response, and records the outcome back onto the &mut Proxy (record_attempt, add_type, record_trust). When attempt returns, the socket is gone; the Proxy carries only the facts observed.

This separation is why the design has no process-global judge state and no asyncio.Event to deadlock on (see checking): the checker owns its judges and its sockets for exactly the lifetime of a check, and the Proxy that survives is inert data. It is also why persistence (the Store trait) can save and reload a Proxy losslessly on identity — there is nothing live to serialize.

Providers

A provider is a page that lists proxies. Zuli ProxyBroker Extended treats providers as data, not code: provider sites rot continuously (measurement on 2026-07-15 found ~10 of proxybroker2’s 38 registry entries already dead), so a dead provider is a config edit, not a recompile-and-republish.

The ProviderSpec model

Each provider is a ProviderSpec, deserializable from YAML or JSON so the bundled registry and user configs share one shape:

#![allow(unused)]
fn main() {
pub struct ProviderSpec {
    pub url: String,            // the page to fetch
    pub protocols: Vec<Proto>,  // claimed protocols; empty = unknown (check all)
    pub pattern: Option<String>,// optional bespoke (host, port) regex
    pub timeout: u64,           // request timeout in seconds (default 20)
    pub kind: Option<String>,   // proxybroker2 `type`; only `simple` is supported
}
}

Extraction produces Candidate { host, port, protocols } values — canonical IP, port, and the protocols the provider claims. A Candidate is not yet a Proxy: it has not been checked.

The whole-text IP:port scanner

By default a provider needs no format-specific parser. Extraction runs parse::find_addrs_global over the entire page body — a whole-text scanner that finds every IPv4 and pairs it with the nearest following port. This one scanner subsumes the three formats a per-site parser zoo would otherwise need:

Extraction then filters exactly as the Python pipeline does: ports that are empty or zero are dropped, IPs are canonicalized (leading-zero and out-of-range matches removed), the unspecified address (0.0.0.0 / ::) is rejected as a non-routable sentinel, and the result is deduplicated.

The scanner uses proxybroker2’s exact IPv4 octet sub-pattern, so its quirks are byte-identical — including that 999.1.1.1 yields the valid substring 99.1.1.1. Rust’s regex crate rejects the lookahead of the original global pattern, so find_addrs_global is a two-pass scanner (regex for IPs, code for pairing) verified against a characterization oracle. See the checking page for the resolver/negotiator pipeline the resulting candidates flow into.

Custom pattern regexes

A provider whose page needs bespoke extraction supplies a pattern: a regex with two capture groups, (host, port). When present it replaces the whole-text scanner for that provider.

url: "https://example.com/odd-format"
protocols: [SOCKS5]
pattern: "IP=(\\d+\\.\\d+\\.\\d+\\.\\d+) PORT=(\\d+)"

Adding your own providers

Write one provider per file into a directory and point the CLI at it. config_template() (also printed by the library) is a ready-to-edit starting point:

# one provider per file; a filename starting with `_` is skipped (rename to disable).
url: "http://example.com/proxy-list.txt"
protocols: [HTTP, HTTPS]
# pattern: "(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"   # omit to use the default scanner
timeout: 20

proxybroker --provider-dir ./my-providers find --types HTTP
proxybroker --provider-dir ./my-providers --providers-only grab

The loader (load_provider_dir) reads files sorted by name, skips names starting with _, and logs and skips any file that fails to parse — one bad config never sinks the rest. Two deliberate deviations from proxybroker2:

• Only type: simple is supported. A config declaring paginated or api is warned about and skipped, rather than loaded as a silently-broken plain GET. A type-less config is treated as simple. Harmless unknown fields (name, format, max_connections) are ignored, so an existing proxybroker2 simple config loads directly.
• No Python execution. proxybroker2 can execute .py provider files; there is no safe Rust equivalent, so only data files are loaded.

The bundled registry

50 sources ship embedded (data/providers.yaml, read via include_str! and exposed as bundled_registry()). Only sources confirmed live and yielding are carried over; proxybroker2’s dead entries are not. The set combines the surviving proxybroker2 sources (proxyscrape HTTP/SOCKS4/SOCKS5, sslproxies, free-proxy-list, socks-proxy, proxydb, and others) with a curated 2026-07-17 expansion of hourly-refreshed GitHub-hosted lists (TheSpeedX, monosans, proxifly, and more).

The bundled proxyscrape SOCKS entries fix a live upstream bug: in proxybroker2 the SOCKS providers’ proto=("SOCKS4") has no trailing comma, so it is a string, not a tuple, and the type filter silently drops both — despite them being among the highest-yield sources still alive. In Rust a Vec<Proto> cannot be a string, so the bug is unrepresentable.

Scheduled liveness audit

Provider liveness is checked by a scheduled GitHub Actions workflow (.github/workflows/ provider-audit.yml), which runs Mondays at 06:00 UTC (plus manual workflow_dispatch). It fetches every bundled provider and fails, listing the dead URLs, if any source yields zero proxies.

The audit is deliberately not run on pull requests or pushes: a source rotting upstream must never block a merge. A red audit is a maintenance signal — “curate data/providers.yaml” — not a broken PR. An offline test guards the registry’s shape; the audit guards its liveness.

Checking

The Checker validates one proxy across the protocols it is expected to support, and classifies how much it reveals about the client. This page walks the pipeline a proxy travels from a raw address to a confirmed, classified result.

The pipeline

Proxy (host, port)
   │
   ├─ DNSBL check ............ rejected early if listed in any --dnsbl zone
   │
   ▼  for each protocol in (expected ∩ requested), in Proto::ALL order
connect TCP ──► negotiate ──► send test request ──► read + validate ──► classify
   │              │                                                        │
resolver       negotiator                                          anonymity + trust

Each protocol is checked independently; a proxy can confirm several. The set actually checked is the intersection of the protocols the provider claimed (expected_types) and the protocols the caller requested — iterated in the fixed Proto::ALL order, never HashMap order (which is randomized and would make check order, and therefore emitted bytes, nondeterministic). An empty expected set means “unknown”, so all requested protocols are checked.

Resolver

Before any judge can be used it must be reachable, and the host’s own external IP must be known — it is the anonymity baseline. The Resolver resolves host names via hickory (IP literals pass straight through, including leading-zero IPv4) and discovers this machine’s external IPs by probing a set of IP-echo endpoints concurrently. external_ips returns a set: on a dual-stack host both the IPv4 and IPv6 addresses, because the anonymity check must trip if either appears in a judge’s echo.

Negotiator

The negotiator turns a fresh TCP connection into a stream tunnelled to the target, dispatching on Proto. The protocol set is closed — six variants — so this is a match, not a trait object (users extend providers, never protocols):

CONNECT:25 has no test request — a granted tunnel plus the 220 banner is the whole check. Everything else proceeds to a request/response round-trip against a judge.

Judges

A judge is an endpoint that echoes the request headers and the client IP back, so the checker can see what the proxy forwarded. The JudgePool is probed eagerly when the checker is built and owned by it: Checker::new returns Error::NoJudges if none verify, so check is simply unconstructible before the baseline exists. There is no process-global judge state and no asyncio.Event — the deadlock trap of a naive port does not exist here.

Judges are grouped by JudgeScheme (Http / Https / Smtp). Each protocol routes to a random working judge of the scheme its judge_scheme() selects; HTTPS uses an HTTPS judge, CONNECT:25 an SMTP judge, everything else an HTTP judge. On probe, an HTTP/HTTPS judge is kept only if it returns 200 and echoes one of the host’s real external IPs and echoes a random marker — and its baseline via/proxy counts are recorded for the anonymity comparison below.

Anonymity levels

Only HTTP carries anonymity information (the judge sees the client’s headers directly; tunnelled protocols hide them). The measured AnonLevel, ordered worst → best:

Because the ordering is worst → best, a filter like level >= AnonLevel::Anonymous is meaningful. A request narrows protocols and, optionally, levels via TypeSpec; --strict requires the measured level to match exactly.

Judge-less liveness mode

If no judge comes up (all unreachable) the checker normally fails with Error::NoJudges. When the caller supplies a liveness URL (CheckerConfig::liveness_url, the CLI --liveness-url), the checker instead falls back to a plain fetch-through-the-proxy: a GET that must return 200. This is graceful degradation — the proxy is confirmed working but its anonymity is unclassifiable without a judge, so it carries level None. Combining --liveness-url with an anonymity-level filter therefore yields nothing.

Honeypot / trust verdict

With --trust-check (CheckerConfig::trust_check), each judge round-trip is assessed for hostility and a TrustReport is recorded. It reports specific signals rather than a bare “untrusted” boolean:

An empty report means trusted. The check is an opt-in heuristic with documented false-positive guards: Via / X-Forwarded-For are the anonymity signal and are allow-listed so they do not double-count as injection, and the injected-header scan only detects against judges that echo raw Name: value request headers (the bundled defaults emit JSON or HTTP_NAME = value, so the scan is inert — safe, but pass a raw-header-echo judge via --judges for real detection).

Geolocation & ASN

When built with the geo feature, checked proxies can be attributed to a country (and, with a richer database, a region and city) and to the Autonomous System that owns their IP. This page covers the GeoDb lookup, the bundled database, and the two user-supplied database hooks.

Geo support is feature-gated: geo compiles the lookup code, and geo-bundled additionally embeds the default country database. Both are on in the default build.

GeoDb

GeoDb is an opened MaxMind-format (MMDB) database. There are three ways to obtain one:

#![allow(unused)]
fn main() {
use proxybroker::GeoDb;

let db = GeoDb::bundled()?;              // the embedded DB-IP Country Lite (needs `geo-bundled`)
let db = GeoDb::open("/path/to.mmdb")?;  // any MMDB you supply
}

It exposes two lookups:

#![allow(unused)]
fn main() {
db.lookup(ip)     // -> Option<Country>
db.lookup_asn(ip) // -> Option<Asn>
}

A single decode path serves both a Country-only and a full City database: geoip2::City’s fields are all optional, so the bundled Country record decodes cleanly with empty region/city, and those fields fill in only when the supplied database actually carries them.

The data model

lookup returns a Country; lookup_asn returns an Asn. Both hang off the Proxy as proxy.geo and proxy.asn.

#![allow(unused)]
fn main() {
pub struct Country {
    pub code: String,           // ISO country code, e.g. "US"
    pub name: String,           // English country name
    pub region: Option<Region>, // subdivision — City DB only
    pub city: Option<String>,   // city name — City DB only
}

pub struct Region {
    pub code: String,           // ISO 3166-2 subdivision code
    pub name: String,
}

pub struct Asn {
    pub number: u32,            // autonomous system number, e.g. 15169
    pub org: Option<String>,    // owning organization, e.g. "Google LLC"
}
}

Country and Asn are orthogonal: geolocation and network ownership are independent facts about an IP, and come from independent databases.

The bundled database — DB-IP Country Lite

The embedded database is DB-IP Country Lite (CC BY 4.0), not MaxMind GeoLite2. It resolves an IP to a country only — region and city are always None from the bundled data, and lookup_asn always returns None (it carries no ASN fields). Shipping only country data is a licensing hygiene constraint, covered in data & licensing.

Using the bundled data triggers the CC BY 4.0 attribution duty, which the crate surfaces in --version, the README, and LICENSE-DATA:

IP Geolocation by DB-IP (https://db-ip.com)

--geo-db — bring a City database

Pass any MaxMind-format country/city database to override the bundled one. A full City database populates region and city, which the bundled Country-Lite cannot:

proxybroker --geo-db /path/to/GeoLite2-City.mmdb find --types HTTP --format json

You may lawfully use your own MaxMind GeoLite2 copy under your own license key — the crate simply does not bundle MaxMind data. Both flags are global, so they precede the subcommand.

--asn-db — attribute the Autonomous System

ASN lives in a separate database from country/city (e.g. GeoLite2-ASN.mmdb). It is opt-in and unbundled — no ASN data ships with the crate — so proxy.asn stays None unless you supply one:

proxybroker --asn-db /path/to/GeoLite2-ASN.mmdb find --types HTTP --format json

Each proxy then carries an asn object ({ "number": 15169, "org": "Google LLC" }, or null when nothing resolved) in --format json, and the {{asn}} / {{asn_org}} tokens work in --output-format templates. --geo-db and --asn-db are independent and can be combined.

Only country data is bundled. Region/city (via --geo-db City DB) and ASN (via --asn-db) are hooks: the code ships, the data does not.

Feature flags

Zuli ProxyBroker Extended is a library first and a CLI second, so almost everything beyond the core find/check engine is behind a cargo feature. This keeps the default binary lean and lets pure-library users pull in only what they need. Every feature below is declared in Cargo.toml.

The default set

default = ["cli", "server", "geo", "geo-bundled"]

The default build gives you the proxybroker binary, the local rotating server, country geolocation, and the embedded DB-IP database. --no-default-features strips all of that back to the bare library — the find/check engine with no CLI, no server, and no geo.

Every feature

Features that imply others do so through cargo’s dependency edges — enabling tui, for example, automatically turns on cli and server.

Common combinations

CLI only — no server, no geo. The smallest useful binary: find and check proxies, print them, but do not open a listening socket or bundle a geo database.

cargo build --no-default-features --features cli

No geo. Keep the CLI and server, drop the geo code and the bundled database entirely (no attribution duty, a smaller binary):

cargo build --no-default-features --features cli,server

Persistence with a backend. The persist feature is only the Store contract; pair it with a backend:

cargo build --features store-sqlite      # or store-redis

musl-clean by design. The whole default graph is ring-only (rustls with rustls-no-provider + the ring provider installed at startup), so aws-lc-rs — whose -sys C/asm crate is the musl cross-compile blocker — never enters the dependency tree. This is not a feature you toggle; it holds for the default build and the optional backends: store-sqlite compiles SQLite from source (static, no system library), and store-redis uses a ring-only rustls for rediss://. So a --target x86_64-unknown-linux-musl build works with the default features and the stores alike.

Data & licensing

Zuli ProxyBroker Extended keeps a strict line between its code and its bundled data, because they carry different licenses with different obligations. It is a derivative distribution: original upstream proxybroker-rs attribution and the statement of changes remain explicit in NOTICE. Keeping these license scopes separate is deliberate because the code and bundled data have distinct licensing terms and attribution obligations.

Publication status. The Zuli repository is not public yet. The repository links below identify the planned Zuli distribution locations; a local checkout contains the corresponding root files.

Two licenses, two files

The blanket Apache-2.0 grant on the code does not extend to the geo data. LICENSE-DATA covers only the bundled MMDB and nothing else; the two never imply each other. Attribution and the statement of changes for the derivative work live in NOTICE.

Why DB-IP Country Lite, not GeoLite2

The Python original bundles MaxMind’s GeoLite2-Country database inside its distributed package. Zuli ProxyBroker Extended does not redistribute any MaxMind data. It bundles DB-IP Country Lite instead, for a concrete legal reason:

• GeoLite2’s EULA is update-or-destroy. It obliges licensees to destroy superseded copies within 30 days of a new release. A published crates.io version is immutable — it can never be destroyed — so bundling GeoLite2 in a published crate cannot be brought into compliance by attribution, feature flags, or any other means.
• DB-IP Country Lite is CC BY 4.0. It imposes no update-or-destroy duty and carries no ShareAlike obligation. Its only condition is attribution.

The required attribution, which the crate surfaces in --version, the README, and LICENSE-DATA:

IP Geolocation by DB-IP (https://db-ip.com)

The bundled data is country-only

The embedded database resolves an IP to a country and nothing more. Region, city, and ASN are never populated from the bundled data — they are unbundled hooks: the lookup code ships, the data does not.

• Region / city populate only from a user-supplied MaxMind City database via --geo-db.
• ASN (number + organization) populates only from a separate ASN database via --asn-db.

You may lawfully use your own MaxMind GeoLite2 copy under your own license key — pass it with --geo-db / --asn-db. The crate simply never redistributes MaxMind data.

Shipping zero geo data

If you do not want the CC BY 4.0 attribution duty at all, build without the bundled database. The feature flags that control this:

# geo code, but no bundled database (bring your own with --geo-db):
cargo build --no-default-features --features cli,server,geo

# no geo at all — no code, no data, no attribution duty:
cargo build --no-default-features --features cli,server

geo-bundled is the only feature that embeds licensed data; turning it off (or building --no-default-features) ships a crate free of any third-party data. See NOTICE and LICENSE-DATA for the full terms and the statement of changes from the original work.

Observability

Zuli ProxyBroker Extended exposes four ways to see what it is doing at runtime: a Prometheus metrics endpoint, structured JSON logs, a live progress bar, and live-reload of a served pool file. Each is opt-in — some behind a build feature, all behind a flag — so the default binary stays lean and silent.

See feature flags for how to build with the optional features enabled.

Prometheus metrics (--metrics)

When serve is built with the metrics feature, --metrics <ADDR> starts a second listener that serves the live pool state in Prometheus text exposition format (version 0.0.4). Any GET to that address returns the current snapshot.

# Build with metrics, then serve with a scrape endpoint on :9090.
cargo build --release --features metrics
proxybroker serve --types HTTP HTTPS --metrics 127.0.0.1:9090

The exporter is hand-rolled — no prometheus crate is pulled in, since the surface is tiny and stable. It reads a single Pool::snapshot() per request. The exposed series:

Sample output:

# HELP proxybroker_pool_size Proxies currently available in the pool.
# TYPE proxybroker_pool_size gauge
proxybroker_pool_size{scheme="http"} 42
proxybroker_pool_size{scheme="https"} 17
# HELP proxybroker_pool_error_rate_avg Mean proxy error rate over the pool.
# TYPE proxybroker_pool_error_rate_avg gauge
proxybroker_pool_error_rate_avg 0.08
# HELP proxybroker_pool_resp_time_avg_seconds Mean proxy response time over the pool.
# TYPE proxybroker_pool_resp_time_avg_seconds gauge
proxybroker_pool_resp_time_avg_seconds 1.34
# HELP proxybroker_pool_probe_latency_avg_seconds Mean judge-probe latency (check-time) over the pool.
# TYPE proxybroker_pool_probe_latency_avg_seconds gauge
proxybroker_pool_probe_latency_avg_seconds 0.42
# HELP proxybroker_evictions_total Proxies hard-evicted from the pool.
# TYPE proxybroker_evictions_total counter
proxybroker_evictions_total 5
# HELP proxybroker_rotations_total Mid-request rotations to a different proxy.
# TYPE proxybroker_rotations_total counter
proxybroker_rotations_total 12

Error rate is an aggregate gauge, not a per-address one: per-proxy labels would be unbounded cardinality for a constantly-rotating pool. Per-proxy detail lives behind the proxycontrol control API instead.

Library users can render the same text directly:

#![allow(unused)]
fn main() {
use proxybroker::serve_metrics; // async: serve_metrics(addr, pool) -> ServerHandle
// or render the body yourself:
let body = proxybroker::server::render_metrics(&pool);
}

Both render_metrics and serve_metrics are gated on the metrics feature.

Grafana dashboard

After the public Zuli repository is created, a ready-made dashboard will be available at the planned grafana/proxybroker-dashboard.json path. Import it in Grafana (Dashboards → New → Import → Upload JSON file) and pick your Prometheus data source when prompted. It has six panels — pool size by scheme, available proxies, error rate, serve-vs-probe latency, and eviction/rotation rates.

Structured JSON logs (--log-format json)

The global --log-format option (default text) controls how the tracing event stream is rendered. --log-format json emits line-delimited JSON — one object per event — suitable for piping into a log aggregator.

proxybroker find --types HTTP --log info --log-format json 2>events.ndjson

Logs always go to stderr, so they never mix with proxy output on stdout. The verbosity is set by the global --log option (error, warn, info, debug, trace; default warn), or overridden by the standard RUST_LOG-style environment filter if present. In JSON mode the whole stream renders as JSON, including the per-check structured events emitted at each check outcome.

Both --log and --log-format are global options — they apply to every subcommand (grab, find, check, serve, and the others).

Progress bar (--progress)

find accepts --progress to draw a live status line on stderr while checking runs. It renders only when the binary is built with the progress feature; otherwise the flag is a no-op (so scripts stay portable across builds).

cargo build --release --features progress
proxybroker find --types HTTP HTTPS --limit 100 --progress

The bar is a spinner (not a percentage bar — a streaming find has no known total) and shows the running counts polled from the shared stats collector:

⠹ checked 318 · working 74 · avg 1.12s

Like --show-stats, the bar is drawn to stderr so it never contaminates the proxy list on stdout. It clears itself when find completes.

Live-reload a served pool (--watch)

When serve --load <FILE> is built with the watch feature, adding --watch starts a filesystem watcher on the NDJSON pool file. On each change it re-parses the file and reconciles the running pool — additions join, removals drop — without restarting the server.

cargo build --release --features watch
proxybroker serve --load pool.ndjson --watch

Details:

• --watch requires --load (there is nothing to watch when the pool is filled from a live find). Without it, the flag warns and is ignored.
• Write bursts are coalesced with a short debounce, so an editor’s replace-on-save triggers exactly one reconcile.
• A parse error (for example, a half-written file) is logged and the pool left untouched — a bad write never empties a running pool.

The watcher shares the pool’s add/remove seam with the adaptive re-check loop (serve --recheck), so both can safely mutate one live pool concurrently.

Related

• serve reference — every serving flag, including the pool selection strategies whose rotations and evictions the metrics count.
• proxybroker top — a live terminal dashboard over the pool (the tui feature), an interactive alternative to scraping --metrics.
• Feature flags — which build features gate metrics, progress, and watch.

Roadmap & Waves

Historical upstream record. This page documents the original proxybroker-rs project and is retained for provenance. Upstream repository links and historical decisions on this page are intentional and do not describe the current Zuli release plan unless explicitly restated elsewhere.

proxybroker-rs was built past 1.0 in waves: each wave batches features that touch the same module, respect the same dependency order, and can ship as one campaign of one-commit-per-item changes. The full roadmap, with per-item effort estimates and design notes, lives in the repository under docs/roadmap/.

The wave model

Ordering optimizes for four things, in priority order:

1. Dependencies — a feature never precedes what it needs (Deserialize before save/load; SQLite after file-based save/load; retry-failover with status-gating).
2. Module batching — features touching the same file ship together, so server.rs / checker.rs / the output path is opened once, not eight times.
3. Value × feasibility — the biggest genuine gaps and cheapest isolated wins go first.
4. Principle friction last — features that fight the project’s stated principles (ephemeral-by-design, no speculative abstraction, offline-testable, CC BY 4.0 data hygiene) are deferred until demand pulls them.

Every feature must stay offline-testable (constraint C5 — see The Systematic Refactor).

Waves

The committed roadmap (Waves 1–9, all A/B/C/D/E/F items plus store-redis and the top TUI) is shipped, along with C8 (ASN attribution) and P1 (provider expansion).

Feature families

Each feature carries a letter-family prefix. What shipped, by family:

See Observability for the F-family runtime surface, and Feature Flags for how the optional features gate into the build.

Cross-cutting: providers (P1)

Provider expansion is tracked outside the wave sequence because it is ongoing. The bundled registry grew from the Python original’s 12 sources to 50 curated live sources. Its shape is guarded offline by format-archetype fixtures and a registry integrity test; its liveness is guarded by a scheduled audit workflow (see Contributing). The research and expansion notes live in p1-provider-research.md and p1-provider-expansion.md.

What was deliberately not built

Several features were scoped, understood, and consciously deferred with concrete triggers rather than shipped on speculation. See the Deferred Backlog.

Deferred Backlog

Historical upstream record. This page documents the original proxybroker-rs project and is retained for provenance. Upstream repository links and historical decisions on this page are intentional and do not describe the current Zuli release plan unless explicitly restated elsewhere.

Deliberate YAGNI deferrals — features that were scoped, understood, and consciously not built because no consumer needs them yet. Each ships nothing on speculation; each has a concrete trigger that would justify building it. The canonical list lives in the repository at docs/roadmap/deferred-backlog.md.

As of 2026-07-17, the committed roadmap (Waves 1–9, all A/B/C/D/E/F items plus store-redis and the top TUI), C8 (ASN), and P1 (provider expansion) are shipped. A backlog sweep then built four of the six deferrals and consciously kept two deferred (see below). Nothing below blocks anything.

Shipped from the backlog (2026-07-17 sweep)

Consciously kept deferred (reviewed, not built)

Softer ideas

Planning-era notions, not formal roadmap deferrals:

• async Store trait — the persistence trait is synchronous by design (blocking SQLite/Redis behind the observer). An async variant would only pay off for a high-throughput fully-async backend.
• top TUI persisted sparklines — historical timeseries in proxybroker top (today it renders a live snapshot only). Would need a ring-buffer of pool snapshots.

Opportunistic hygiene (done)

• Reject the unspecified IP (0.0.0.0) sentinel — done 2026-07-17 in ProviderSpec::extract (surfaced by the P1 review). canonicalize_ip stays Python-parity-faithful; the filter lives at the provider-candidate layer.

On the deferral discipline

Every kept-deferred entry above is a decision, not an omission. The project’s principle is that speculative abstraction is a liability: an unused feature is code to maintain, a larger dependency tree, and a wider API to keep stable — all paid for before any consumer benefits. Recording the trigger keeps each idea cheap to revive without carrying its cost in the meantime. The same discipline shaped the systematic refactor that produced the port.

The Systematic Refactor

Historical upstream record. This page documents the original proxybroker-rs project and is retained for provenance. Upstream repository links and historical decisions on this page are intentional and do not describe the current Zuli release plan unless explicitly restated elsewhere.

proxybroker-rs is a from-scratch Rust port of Python’s proxybroker2. It was not a line-by-line transliteration — it was a deliberate, evidence-driven refactor that treated the original as a specification to be understood, not a text to be copied. The full working documents live in the repository under docs/systematic-refactor/.

The method: trace → goals → map → port

The port ran through four artifacts before most of the Rust was written, each feeding the next:

1. Trace (trace.md) — an execution-level read of the Python: 55 risks ranked by how likely each was to sink the port. Three of them were confirmed by actually running the Python, not by reading it.
2. Goals (goals.md) — what the port is for: ship a Rust library and a CLI, on stable Rust, with every network path testable offline. Explicit non-goals too: no Python interop, no bug-compatibility, no invented performance target.
3. Map (map.md) — a module-by-module correspondence between Python and Rust, plus a 40-finding critique whose verdict on a naive port was blunt: “not implementable as written.”
4. Port — resolved every conflict in decisions.md, in a strict dependency order: types → error → utils/parse → resolver → proxy → negotiator → judge → checker → provider → broker → server → cli.

When a design document and the compiler disagreed, the compiler won. When two documents disagreed, decisions.md settled it and became authoritative.

Byte-for-byte parity where it matters

The port is idiomatic Rust, not transliterated Python — a faithful port of Proxy.as_json() is impl Serialize, not a method returning a map. But in the places where behaviour is wire-visible or statistically load-bearing, the port matches the original exactly, on purpose:

• Error histogram buckets. ProxyError::Reset deliberately merges the receive and send cases, because Python files both under one errmsg="connection_is_reset" bucket. Splitting them by Rust variant name would silently change the reported error rate, and no test would catch it. Direction lives in the tracing message, exactly where Python put it.
• Header ordering. Request headers use an ordered IndexMap, never a HashMap — iteration order is wire-visible, and a judge’s echo-grep decides whether a proxy passes.
• Display order. CONNECT:80 sorts before CONNECT:25 ('0' < '5'), verified against the Python interpreter rather than assumed.
• Lenient parsing. UTF-8 is decoded lossy-drop (Python’s errors='ignore' drops bytes rather than inserting U+FFFD); base64 is decoded leniently (strict decoding empties three providers); leading-zero IPv4 is handled so real proxies are not silently dropped.

Where the Python is wrong, the Rust is right and the deviation is recorded. The port found and corrected several confirmed upstream bugs — a missing trailing comma that silently dropped both proxyscrape SOCKS sources, a heapq tie-break that raised TypeError on equal response times, and a SMTP-disable path that could IndexError.

Key design decisions

Socket ownership — one answer

Three modules had proposed three different transport designs. One of them, a ProxyConn layer with eight methods, had already been deleted by the two modules that would have called it — a week of work with zero callers, one variant of which could not even compile (you cannot move a field out of a Drop type). The resolution: checker.rs and negotiator.rs win. Proxy is data plus record_attempt(); it owns no socket. The transport lives in the checker and the negotiator, and the shared Stream type has exactly one home.

Judges are probed eagerly, and owned by value

Three independent findings — process-global mutable judge state, a level-vs-edge mismatch between asyncio.Event and tokio::sync::Notify, and a probe-timing question specified three incompatible ways — collapsed into one decision: judges are probed eagerly in the Checker constructor, which owns them by value. That makes Checker::check unconstructible before the baseline exists, so an ordering constraint that was a fragile convention in Python becomes a type fact in Rust.

The licensing pivot

The Python distribution vendors a MaxMind GeoLite2-Country.mmdb inside its package. Whether a crates.io crate may redistribute that data is a real licensing question with a real answer — and the answer is no under MaxMind’s terms. The port pivoted the bundled geo data to DB-IP Country Lite, licensed CC BY 4.0, which is redistributable provided attribution is carried. That attribution is baked into --version output and the NOTICE, and the geo-bundled build feature can be turned off to ship zero geo data and zero attribution duty. The whole crate is Apache-2.0, matching proxybroker2 (a port is a derivative work) and crediting the original authors. See Data & Licensing for the full story.

Why this way

The port was justified by wanting a Rust library, not by a measured Python bottleneck — so no “10× faster” claim was invented, because a claim with no baseline is unfalsifiable fiction. What was committed: no accidental pessimisation (the concurrency model keeps its bounded-queue, capped-in-flight shape), and every network-dependent path testable offline against a local mock server. Those constraints, plus the deferral discipline in the roadmap and deferred backlog, are what kept the rewrite honest.

Contributing

Zuli ProxyBroker Extended is a single-maintainer project with a deliberately strict, fully offline test suite. This page covers what you need to build it, run the checks its CI configuration enforces, and understand the provider liveness audit.

Repository and collaboration

The public Zuli repository, issue tracker, and pull-request surface are not available yet. After the repository is created, clone the planned source and use its focused review workflow:

git clone https://github.com/zuli2021/zuli-proxybroker-extended.git
cd zuli-proxybroker-extended

After publication, report issues at the planned issue tracker and open focused pull requests through the planned pull-request surface. Do not claim a release or tag without repository evidence.

Building

The project builds on the stable Rust toolchain — rust-toolchain.toml pins stable, so a fresh checkout uses it automatically. A Rust library that needs nightly is a library most people cannot use; keeping to stable is a hard constraint.

# Default build: cli + server + geo + geo-bundled.
cargo build

# Release binary.
cargo build --release

Optional functionality lives behind feature flags. Enable the ones you need, for example:

cargo build --features metrics,progress,watch,store-sqlite

Running the checks

CI enforces exactly the checks you should run locally: formatting, clippy with warnings-as-errors, and the full test suite across the feature matrix. Run them before opening a PR after the public repository is available.

cargo fmt --all --check
cargo clippy --all-targets --all-features --locked -- -D warnings
cargo test --all-features --locked
cargo test --no-default-features --locked

clippy runs with -D warnings — a warning fails the build. --locked mirrors CI: it fails rather than silently updating Cargo.lock.

The CI matrix

The CI configuration is .github/workflows/ci.yml in a local checkout. After the public Zuli repository is created, it will be available at the planned CI workflow. It is split into parallel jobs.

The three test configurations

The test job runs three configs because each exercises code the others miss:

1. --all-features — the geo path; since default is all four default features, this also compiles the default library and CLI binary.
2. --no-default-features — the geo-free path a pure-library consumer gets. The broker has genuinely divergent no-geo runtime code (a no-op geo attach, a geo-less builder arm) that the all-features run never hits.
3. --features cli (no server) — a supported combo that neither of the above compiles, so a misplaced #[cfg] gate on a server-only function can slip through both. Build-only.

The whole suite is designed to run fully offline — local mock servers only, no network, no flakiness. That is constraint C5 from the systematic refactor, and it is load-bearing: a test suite that needs the internet is one that fails in CI for reasons unrelated to the code.

Distribution smoke

The dist job proves the shipping artifacts stay self-contained. The static musl binary must build, run with zero runtime data files (everything is embedded), and carry the CC BY 4.0 DB-IP attribution in --version. The FROM scratch Docker image must do the same offline. The install.sh installer must pass shellcheck.

store-redis service

The Redis backend’s atomic upsert can’t be faithfully mocked (it depends on Lua atomicity), so its integration tests run against a real Redis service container. This is the one deliberately non-offline test path; the fold arithmetic and key layout are also pure-tested in the main offline run.

Provider liveness audit

The bundled provider registry (50 curated sources, see the roadmap) is guarded two ways:

• Offline — a registry integrity test and format-archetype fixtures guard the registry’s shape. These run in the normal test job.
• Liveness — after the public repository is created, the configured Provider audit workflow will fetch every bundled source and flag any that yield zero proxies (dead or format-changed).

The audit is scheduled (Mondays 06:00 UTC) and manually triggerable, and deliberately does not run on pull requests or pushes — a source rotting upstream must never block a merge. A red audit run is a maintenance signal (“re-curate the provider data”), not a broken PR. You can run it locally:

cargo test --test provider_audit --locked -- --ignored --nocapture

The audit test is #[ignore]d, so it runs only with --ignored. --nocapture prints the per-source yield table; the test fails (listing the dead URLs) if any source yields nothing.

Commit and review conventions

• One commit per item. A request that bundles several distinct fixes should land as one conventional-style commit each, so history stays reviewable, revertable, and bisectable.
• Fix all red checks before advancing. Never move on with a known-failing test, type error, or lint — a pre-existing failure masks the new ones your change introduces.
• Keep it offline-testable. Any new network-dependent behaviour needs a local mock, not a live dependency.