HuggingFace Model Downloader bodaay

Use this command to install HuggingFace Model Downloader:

winget install --id=bodaay.hfdownloader -e

The HuggingFace Model Downloader is a utility tool for downloading models and datasets from the HuggingFace website. It offers multithreaded downloading for LFS files and ensures the integrity of downloaded models with SHA256 checksum verification.

The HuggingFace Model Downloader is a utility tool designed to streamline the process of downloading models and datasets from the HuggingFace platform. It simplifies access to machine learning resources by offering efficient and reliable retrieval mechanisms.

Key Features:

Multithreaded LFS File Downloads: Enhances download speeds for large files, making it easier to handle extensive model and dataset files.
SHA256 Checksum Verification: Ensures the integrity of downloaded models, confirming they are unchanged and valid.
Cross-Platform Compatibility: Operates seamlessly across various operating systems, including Linux, macOS, and Windows WSL2.
Configuration File Support: Allows users to set default values for command flags, enhancing customization and ease of use.
Download Resumption: Continues interrupted downloads without starting over, saving time and bandwidth.
HuggingFace Access Token Integration: Supports secure access to restricted models and datasets using tokens.

Audience & Benefit:

Ideal for researchers, data scientists, and developers working with machine learning models. This tool enables efficient retrieval of resources, ensuring reliability and integrity while handling large-scale downloads. Its flexibility and robust features make it a valuable asset in managing machine learning projects effectively.

The HuggingFace Model Downloader can be installed via winget, offering a straightforward setup process to integrate into your workflow seamlessly.

README

HuggingFace Model Downloader

Reason

Git LFS was slow for me, and I couldn't find a single binary for easy model downloading. This tool may also be integrated into future projects for inference using a Go/Python combination.

One Line Installer (Linux/Mac/Windows WSL2)

The script downloads the correct version based on your OS/architecture and saves the binary as "hfdownloader" in the current folder.

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -h

To install it to the default OS bin folder:

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -i

It will automatically request higher 'sudo' privileges if required. You can specify the install destination with -p.

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -i -p ~/.local/bin/

Quick Download and Run Examples (Linux/Mac/Windows WSL2)

The bash script just downloads the binary based on your OS/architecture and runs it.

Download Model: TheBloke/orca_mini_7B-GPTQ

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -m TheBloke/orca_mini_7B-GPTQ

Download Model: TheBloke/vicuna-13b-v1.3.0-GGML and Get GGML Variant: q4_0

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -m TheBloke/vicuna-13b-v1.3.0-GGML:q4_0

Download Model: TheBloke/vicuna-13b-v1.3.0-GGML and Get GGML Variants: q4_0,q5_0 in Separate Folders

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -f -m TheBloke/vicuna-13b-v1.3.0-GGML:q4_0,q5_0

Download Model with 8 Connections and Save into /workspace/

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -m TheBloke/vicuna-13b-v1.3.0-GGML:q4_0,q4_K_S -c 8 -s /workspace/

Download Model to the Current Directory

bash &lt;(curl -sSL https://g.bodaay.io/hfd) -j TheBloke/vicuna-13b-v1.3.0-GGML:q4_0

Usage

hfdownloader [flags]

Flags

-m, --model string: Model/Dataset name (required if dataset not set). You can supply filters for required LFS model files. Filters will discard any LFS file ending with .bin, .act, .safetensors, .zip that are missing the supplied filtered out.