Inferrs is a high-performance inference engine designed for large language models (LLMs), optimized for memory efficiency and resource utilization. It enables users to run LLMs with minimal overhead while maintaining performance and compatibility across various hardware backends.
Key Features:
Hardware Support: Inferrs supports multiple hardware acceleration options, including CUDA, ROCm, Metal, Hexagon, OpenVino, MUSA, CANN, Vulkan, and CPU.
API Compatibility: It offers OpenAI-compatible endpoints (/v1/completions, /v1/chat/completions), Anthropic-compatible APIs (/v1/messages for streaming and non-streaming), and Ollama-compatible interfaces (/api/generate, /api/chat).
TurboQuant Optimization: Inferrs leverages TurboQuant to improve inference efficiency on lower-end hardware while maintaining accuracy.
Single-Binary Deployment: The tool is distributed as a single binary, simplifying setup and deployment across different environments.
Model Flexibility: It supports running models in various formats, including vLLM-style and llama.cpp-style execution.
Audience & Benefit:
Ideal for developers, data scientists, and organizations seeking to deploy LLMs efficiently. Inferrs provides a lightweight yet powerful solution that minimizes resource usage while maintaining compatibility with popular AI frameworks and APIs. It is particularly beneficial for users working with constrained hardware environments or those requiring fast, efficient inference without compromising on model performance.
Inferrs can be installed via winget for seamless setup and integration into existing workflows.
README
llmman
A command-line tool for managing and serving LLM models using OCI registries.
Models are packaged as standard OCI artifacts and stored in any compatible registry (Docker Hub, GHCR, quay, self-hosted, etc.).
llmman serve exposes Ollama-, OpenAI-, and Anthropic-compatible HTTP APIs.
Commands
Command
Description
serve
Start an inference server (Ollama / OpenAI / Anthropic APIs)
launch
Launch an integration (Claude Code, OpenCode, …)
run
Run a model interactively or with a one-shot prompt
pull
Pull a model from a registry or HuggingFace
resolve
Pull (if needed) and print the local path of a model, as JSON — for other tools to consume
list
List locally stored models
ps
List models currently loaded
build
Package model files into a local OCI image
push
Push a local image to a registry
transfer
Transfer an image directly from one location to another (e.g. HuggingFace to an OCI registry)
rm
Remove a local image
tag
Create a new local tag pointing to an existing image
inspect
Show the manifest of a local or remote image
login
Log in to a container registry
logout
Log out from a container registry
Install
Linux, macOS:
curl -fsSL https://raw.githubusercontent.com/llmmanorg/llmman/main/install.sh | sh
Transfer an image directly from a source to a destination without storing
it locally first — e.g. HuggingFace straight to an OCI registry:
llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest
Any source llmman pull understands (an OCI registry, hf://, ms://, ...) can be paired with any OCI registry destination.
Serve
Start the inference server. Uses llama-server from llama.cpp if it's already on PATH; otherwise llmman downloads and caches a prebuilt release matching your OS/arch/GPU automatically (see --llama-cpp-version to pin a specific release).
llmman serve
The server listens on 127.0.0.1:17434 and exposes:
/v1/responses implements the OpenAI Responses API (the dialect OpenAI
Codex requires), including streaming SSE
and function-tool-call re-mapping. This is a plain pass-through to
llama-server's own native /v1/responses support, so a recent enough
llama-server build is required for it to work.
Use it as an Ollama-compatible server:
OLLAMA_HOST=127.0.0.1:17434 ollama run unsloth/Qwen3.5-0.8B-GGUF
Or with any Ollama, Anthropic or OpenAI-compatible client.
Models are loaded on demand. Each model gets its own llama-server subprocess on a random loopback port; subsequent requests reuse the running process.
Launch an integration
Point an integration at a model in one step. llmman launch starts serve in the background if it isn't already running (preloading the requested model), then sets the right environment variables and execs the integration:
llmman launch claude --model gemma4
Run llmman launch with no arguments to list the supported integrations (Claude Code, OpenCode) and whether each is installed. Any extra arguments after -- are forwarded to the integration's own CLI.
Short names work wherever a model reference is accepted.
Use with vLLM directly
llmman serve already spawns vllm itself as a backend for safetensors
models. llmman resolve is the inverse: it pulls and extracts a model
without starting any server, printing the resulting local path so another
tool can load it instead. This is what the
vllm-llmman vLLM plugin uses so
vllm serve oci:// can pull a CNCF ModelPack image
directly, instead of a HuggingFace repo:
format is "safetensors" (a directory) or "gguf" (a single file).
--no-pull fails instead of pulling if the reference isn't already in the
local store; --store/--cache override the default locations below.
Store location
Default locations:
OS
Path
Linux, macOS
~/.local/share/llmman/store
Windows
%LOCALAPPDATA%\llmman\store
Commands that read or write the local store directly (list, rm, tag, inspect, build, resolve, serve) accept --store to override it. Commands that go through the background daemon instead (pull, push, run, launch, ps) always use whichever store that daemon was started with — pass --store to llmman serve to change it for all of them. transfer, login, and logout never touch a local store at all.