nwws-rs
A self-hosted NOAA Weather Wire Service (NWWS-OI) platform in one static binary.
Connects to the NWS's lowest-latency public text-product feed, parses every
bulletin strictly (WMO headers, AWIPS/PIL, UGC, VTEC/HVTEC, segments, warning
tags, polygons, storm motion), dedupes, archives, and serves it all back over a
local HTTP API with a live Server-Sent Events stream. Rust core, Python
bindings, CLI.
# the entire setup, start to finish:
docker run -e NWWS_USERNAME=you -e NWWS_PASSWORD=secret \
-v nwws-archive:/archive -p 8080:8080 ghcr.io/fahrenheitresearch/nwws-rs
curl 'http://127.0.0.1:8080/v1/stream?pil=TOR' # tornado warnings, live, as SSE
curl 'http://127.0.0.1:8080/v1/warnings/active' # everything in effect right now
NWWS-OI delivers warnings seconds after the WFO hits send — typically well
ahead of the public CAP/REST mirrors. Credentials are free:
request them from the NWS here.
Why this exists
Running your own NWWS-OI consumer has traditionally meant gluing together an
XMPP library, a parser (usually pyIEM),
hand-rolled reconnect logic, deduplication, and storage. nwws-rs ships that
whole stack as one tool, and the parsing core is fast enough to chew through
years of archives:
| products/sec (full parse) | relative |
|---|
| nwws-rs (via Python bindings) | 31,265 | 19.0x |
| pyIEM 1.x (pure Python) | 1,642 | 1x |
Same bulletins, same machine, full parse path on both sides (headers, UGC,
VTEC, segments, geometry). The nwws-rs number includes the Python boundary
overhead; the pure-Rust path is faster still. Reproduce it:
python tools/bench_pyiem_speed.py (methodology in the script header).
Honest scope: nwws-rs covers NWWS text products. pyIEM also parses METAR,
SHEF, and other formats and is the right tool for those. For NWWS-OI ingest,
warning parsing, alerting, and archive work, nwws-rs is designed to be the
last tool you need.
Quick start
1. Self-hosted API server (the headline feature)
Grab a binary from Releases
(Linux, macOS, Windows) or build with cargo install nwws-rs --features serve,
then:
export NWWS_USERNAME=you NWWS_PASSWORD=secret
nwws serve ./archive --bind 127.0.0.1:8080
That one process connects to NWWS-OI, auto-reconnects forever with jittered
backoff and MUC history backfill, validates and dedupes every product, archives
them under date-partitioned directories, and serves:
| Endpoint | What it returns |
|---|
GET /v1/stream | live products as Server-Sent Events; filter with ?office=KLOT, ?pil=TOR, ?family=tornado |
GET /v1/products/recent | newest archived products (limit, days, office, pil, family) |
GET /v1/products/{fingerprint} | metadata + full raw text of one product |
GET /v1/warnings/active | VTEC warnings in effect at ?at= (default now), collapsed per event |
GET /v1/timeline | warning lifecycle records: issued/canceled/expired, polygons, motion, tags |
GET /healthz | ingest connection state and counters |
CORS is permissive, so a browser dashboard can consume the API directly.
--no-ingest serves an existing archive with zero credentials — useful for
replaying captured data or fronting a shared archive.
A tornado-warning webhook is a shell one-liner:
curl -N 'http://127.0.0.1:8080/v1/stream?pil=TOR' | while read -r line; do
case "$line" in data:*) echo "${line#data: }" | your-notifier ;; esac
done
2. Python
pip install nwws-rs
import nwws_rs
# Parse any NWS text product
msg = nwws_rs.parse_bulletin(open("tor.txt", "rb").read())
print(msg.heading, msg.awips_id, msg.family)
print(msg.segments[0].tornado_tag)
# Or consume NWWS-OI live, no server needed
client = nwws_rs.OiClient.connect("user", "password")
while True:
message = client.next_message()
print(message.wrapper.id, message.heading)
Full surface: parse, parse_bulletin, parse_oi, inspect_*, scan_path,
active_warnings_at, split_pid201_*, archive_import, archive_verify,
Pid201Stream, OiClient. Typed, object-oriented, returns structured objects.
3. Rust
use nwws_rs::NwwsContent;
let bytes = include_bytes!("tests/fixtures/wmo_tornado_warning.txt");
let content = NwwsContent::parse_bulletin(bytes)?;
assert_eq!(content.bulletin.heading.ttaaii(), "WUUS53");
assert_eq!(content.bulletin.heading.cccc(), "KLOT");
assert_eq!(content.bulletin.awips_id.unwrap().raw(), "TORLOT");
# Ok::<(), Box>(())
The supervised ingest loop is a library API too (daemon):
use std::sync::atomic::AtomicBool;
use nwws_rs::{
ArchiveStore, DaemonOptions, DedupeStore, IngestService, MessageRouter, OiClientConfig,
run_oi_daemon,
};
let config = OiClientConfig::new("user", "password");
let router = MessageRouter::new(Some(ArchiveStore::new("archive")));
let dedupe = DedupeStore::open("archive/state/dedupe.txt")?;
let mut service = IngestService::new(router, dedupe);
let shutdown = AtomicBool::new(false);
run_oi_daemon(&config, &mut service, &DaemonOptions::default(), |_event| {}, &shutdown);
# Ok::<(), std::io::Error>(())
4. Headless ingest (no HTTP)
nwws oi daemon ./archive # credentials from NWWS_USERNAME/NWWS_PASSWORD
Same supervision as serve (reconnect, backfill, dedupe), just without the API.
A hardened systemd unit and env template live in deploy/.
How it compares
| nwws-rs | pyIEM + slixmpp | api.weather.gov |
|---|
| Latency | seconds (NWWS-OI direct) | seconds (NWWS-OI direct) | tens of seconds to minutes |
| Setup | one binary / docker run | assemble client, parser, reconnect, storage yourself | none (hosted) |
| Self-hosted / offline archive | yes | DIY | no |
| Live push | SSE out of the box | DIY | no (poll) |
| Bulk reparse throughput | ~31k products/sec | ~1.6k products/sec | n/a |
| Non-NWWS formats (METAR, SHEF...) | no | yes | partial |
| Credentials needed | free NWS signup | free NWS signup | none |
CLI reference
The nwws binary also covers inspection, replay, and research workflows over
files, directories, and archives:
nwws inspect parse + validate one input (WMO, NWWS-OI XML, PID201)
nwws replay stream a captured corpus through the parser
nwws summary source/transport/family counts
nwws active-at --at VTEC warnings active at a moment
nwws timeline [--at ] warning lifecycle records (issued/canceled/expired, polygons)
nwws lead-time --event-at --lat --lon point-event warning lead-time metrics
nwws oi connect [--count n] print live NWWS-OI messages
nwws oi archive [--duration] bounded live capture into an archive
nwws oi daemon supervised always-on ingest (auto-reconnect + backfill)
nwws serve [--bind addr] [--no-ingest] ingest daemon + HTTP API (build feature: serve)
nwws pid201 inspect|split|archive ... PID201 framed-stream (NOAAPORT/EMWIN-style) tooling
nwws archive import|verify|active-at|timeline canonical archive workflows
Most commands accept --format json|jsonl|tool-result for machine-readable
output. tool-result wraps reports in a wx.tool_result.v1 envelope with
artifacts, evidence, limitations, and provenance — built for AI-agent
consumers.
Design notes
Parsing model
NWWS is treated as multiple transport surfaces over one bulletin semantics:
raw WMO text, NWWS-OI XMPP stanzas, and PID201 framed streams. The rule
throughout: never trust the wrapper more than the bulletin. NWWS-OI
metadata is validated against the embedded WMO bulletin instead of being
accepted at face value.
Library layers
src/wmo.rs — WMO bulletin framing and headerssrc/oi.rs — NWWS-OI messages, wrapper-vs-bulletin validationsrc/product.rs — product families, segments, warning tagssrc/ugc.rs / src/vtec.rs — UGC expansion, P-VTEC/H-VTECsrc/geo.rs — LAT...LON polygons, TIME...MOT...LOC motionsrc/pid201.rs — incremental framed-stream ingestsrc/runtime.rs — dedupe, archive store, routingsrc/oi_client.rs — blocking NWWS-OI XMPP client (rustls, no OpenSSL)src/daemon.rs — supervised ingest: jittered exponential backoff
(equal-jitter; Brooker 2015, AWS Architecture Blog), XMPP whitespace keepalives
(RFC 6120 §4.6), MUC history backfill on reconnectsrc/serve.rs — axum HTTP API + SSE (feature serve)src/warning.rs — warning timelines, lead-time and
area-time-polygon verification metricssrc/api.rs — inspect/scan/split/archive surfacesrc/python.rs — PyO3 bindings (abi3, one wheel per platform)
Archive layout
oi daemon / oi archive / serve write
archive/yyyy/mm/dd/////.{xml,json} —
raw stanza plus metadata sidecar, deduplicated by normalized bulletin content
(BLAKE3). Date partitioning keeps HTTP query cost bounded by the lookback
window (?days=), not by total archive size. The dedupe index survives
restarts, so reconnect backfill never double-archives.
Verification
- 140+ unit/integration/property/CLI/HTTP tests, including SSE end-to-end
- differential comparison against pyIEM on overlapping raw-bulletin semantics
(
tools/compare_pyiem.py, tools/compare_pyiem_corpus.py)
- Python API tests against the built wheel
- CI: 3-OS test matrix, clippy
-D warnings, MSRV check, Docker smoke test
.\tools\verify.ps1 # fmt + clippy + tests + python suite
.\tools\verify.ps1 -Corpus # plus the pyIEM corpus comparison
Accuracy scope
Strict and heavily verified, but bounded to what is implemented and tested:
WMO/NWWS-OI/PID201 parsing, AWIPS/UGC/VTEC/HVTEC/segments/tags/geometry,
archive ingest and verification. It is not a NOAAPORT demodulator, not a
satellite receiver, and not proof against every malformed message ever emitted.
License
MIT or Apache-2.0, at your option.