novel-downloader
Rust command-line crawler for Vietnamese novel sites. It downloads chapter
ranges as clean local HTML files and can package those saved chapters into an
EPUB with metadata, cover image, and embedded font support.
Supported hosts:
metruyenhotvn.commetruyenhotne.comkhodocsach.comxtruyen.vn
You never pick a source. Pass a novel URL and the host decides which adapter
handles it: the metruyenhot hosts are scraped from HTML, khodocsach.com is
read through its JSON API, and xtruyen.vn is scraped from HTML whose chapter
text arrives encoded and is decoded locally. URLs from any other host are
rejected with a clear error before any network fetch. The hidden
--allow-any-host flag bypasses the check for local mock fixtures and
integration tests; it is not intended for normal use.
A khodocsach chapter costs two requests, a short-lived ticket followed by the
chapter content, so its adapter does more work per chapter than the metruyenhot
scraper does. Interrupting a run costs nothing on any host: re-run the same
command and already-saved chapters are skipped.
xtruyen.vn enforces a request limit per client address, so its adapter sets
the pace: at most 2 concurrent requests with half a second between them, and
a refusal is retried after however long the site says to wait. That is applied
whatever --workers and --delay say, and the run tells you when it overrides
either. The interactive wizard does not ask for those two values for
this host at all, since the answers would be discarded, and the confirmation
screen shows the pacing the run will actually use. Expect roughly two minutes
for a 200-chapter novel. Chapter numbering follows the site's own reading order
rather than the number printed on each chapter, because some chapters are
published as extensions of an earlier one (chuong-12-1 following
chuong-12); on a novel like that, --start and --end count positions, so
they can drift from the labels the site shows.
Features
- Crawl one chapter or a chapter range from a supported novel URL.
- Automatically discover the latest available chapter when
--end is omitted.
- Save chapters under a per-novel output directory as
chapter_NNNN.html.
- Build an EPUB from newly crawled chapters or from an existing chapter folder.
- Run sequentially or with multiple download workers.
- Choose how to handle existing chapter files: ask, skip, or overwrite.
- Use an interactive TUI wizard with live download and EPUB build screens.
- Watch elapsed time and a running estimate on the download screen.
- Pick from recently used custom EPUB fonts, remembered between wizard runs.
- Open each EPUB chapter with a drop-capped first character.
- Build release binaries for Linux, Windows, macOS Intel, and macOS ARM through GitHub Actions.
Requirements
- Rust stable toolchain
- Network access to fetch novel pages and cover images
Quick Start
cargo build --release
Run the binary from Cargo:
cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 10
Or run the compiled binary:
./target/release/novel-downloader "https://metruyenhotvn.com/your-novel" --start 1 --end 10
Output is written to output//chapter_NNNN.html by default.
Common Commands
Start the interactive wizard:
cargo run -- --interactive
Crawl chapters and build an EPUB:
cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 50 --epub
Let the crawler discover the latest chapter:
cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --epub
Use four parallel workers and skip files that already exist:
cargo run -- "https://metruyenhotvn.com/your-novel" --start 1 --end 100 --workers 4 --if-exists skip
Build an EPUB from an existing chapter directory:
cargo run -- "https://metruyenhotvn.com/your-novel" --epub-only --chapter-dir output/your_novel
Embed a custom font in the EPUB:
cargo run -- "https://metruyenhotvn.com/your-novel" --epub --font-path /path/to/font.ttf
CLI Options
novel-downloader [OPTIONS] [BASE_URL]
Options:
--start Start chapter number, inclusive
--end End chapter number, inclusive
--output-root Root output directory [default: output]
--delay Extra pause after each chapter is written [default: 0.5]
--workers Number of concurrent download workers [default: 1]
--epub Build an EPUB after crawling
--epub-only Build an EPUB from existing saved chapter files
--chapter-dir Existing chapter directory for --epub-only
--font-path Font file to embed instead of the bundled font
--if-exists ask, skip, or overwrite [default: ask]
--fast-skip Skip remote checks when the destination file already exists
-i, --interactive Launch the interactive TUI
-h, --help Show help
-V, --version Show version
--workers > 1 requires --if-exists skip or --if-exists overwrite, because
interactive per-file prompts are only safe in the sequential path.
--delay is your own pause after each chapter is written, and it applies to
every worker. Each source additionally enforces a minimum spacing between
requests that --delay cannot lower, and may cap --workers; the run tells you
when that happens.
How It Works
flowchart TD
A["User runs novel-downloader"] --> B["cli parses arguments"]
B --> C{"interactive or missing URL?"}
C -->|yes| D["ui wizard collects plan"]
C -->|no| E["build non-interactive plan"]
D --> F["registry resolves the host to a SiteAdapter"]
E --> F
F --> G["adapter fetches metadata and the chapter index"]
G --> H["selected chapter refs"]
H --> I{"mode"}
I -->|crawl| J["runner walks the chapter refs"]
I -->|crawl + epub| J
I -->|epub only| N["use existing chapter directory"]
J --> K{"workers"}
K -->|1| L["sequential runner"]
K -->|many| M["parallel runner"]
L --> O["adapter fetches one chapter"]
M --> O
O --> Q["write chapter_NNNN.html"]
Q --> R{"build EPUB?"}
N --> R
R -->|no| S["finish"]
R -->|yes| T["epub reads saved chapters"]
T --> U["download the cover, embed the font"]
U --> V["package EPUB zip"]
V --> S
Code Layout
src/cli.rs: argument parsing and option validation.src/bin/novel-downloader.rs: process entry point and top-level orchestration.src/source/: the site seam. The SiteAdapter trait, the shared Novel, ChapterRef, RatePolicy and SourceError types, the host registry, and one module per site (metruyenhot scrapes HTML, khodocsach reads a JSON API, xtruyen scrapes HTML and decodes an encoded chapter payload).src/crawler/: the on-disk chapter document format, plus the fetch-write-skip flow and its existing-file policy.src/runner.rs: sequential and parallel chapter runners with progress events, concurrency clamping, and rate-policy pacing.src/epub/: saved-chapter reading, cover handling, EPUB XML/XHTML generation, and archive writing.src/ui/: interactive TUI widgets, screens, wizard state, and plan summary.src/utils.rs: shared HTTP, filesystem, text-cleaning, and slug helpers.src/font.rs: embedded font metadata extraction.src/recent_fonts.rs: the recently used custom EPUB fonts remembered for the wizard.
Testing
Run the full test suite:
cargo test
Run a focused test target:
cargo test --test epub
CI and Releases
GitHub Actions are configured for:
- PR and
main branch CI: run tests and build all supported platforms.
- Tag releases: when a
v* tag is pushed, build and upload release artifacts for:
- Linux x86_64
- Windows x86_64
- macOS Intel
- macOS ARM
Create a release by pushing a version tag:
git tag v0.1.0
git push origin v0.1.0