winget install --id=KoalaFacts.HeroParser -e HeroParser CLI is an AI-native, high-performance command-line utility for working with tabular datasets, powered by the HeroParser library. It supports validation, schema inference, converting between formats (CSV, Excel, Fixed-Width, JSONL, HTB), profiling, querying, and AI translation.
HeroParser is a zero-allocation, SIMD-accelerated tabular data parser and writer for .NET 8, 9, and 10. Designed for extreme speed, memory efficiency, and Native AOT compatibility, it also offers first-class integrations for AI agents, vector embeddings, and LLM pipelines.
Tested under .NET 10.0 on an AMD Ryzen AI 9 HX PRO 370 CPU:
byte[]) read paths on both quoted and unquoted data.View live performance graphs and history on the HeroParser Performance Portal.
dotnet add package HeroParser
dotnet add package HeroParser.Console
dotnet tool install --global HeroParser.Cli --version 2.5.1
Install the native binary using Homebrew:
brew tap KoalaFacts/tap
brew install heroparser
Install the native binary without Homebrew:
curl -fsSL https://raw.githubusercontent.com/KoalaFacts/HeroParser/main/install.sh | sh
On Windows, you can install via WinGet (pending registry merge):
winget install KoalaFacts.HeroParser
Define your record type. Decorate it with [GenerateBinder] to enable source-generated, reflection-free, and Native AOT-safe binding:
using HeroParser;
[GenerateBinder]
public class Product
{
public int Id { get; set; }
public string Name { get; set; } = "";
[Validate(RangeMin = 0)]
public decimal Price { get; set; }
[Parse(Format = "yyyy-MM-dd")]
public DateTime ReleaseDate { get; set; }
}
// Read a file in one line (zero-allocation binding)
List products = Csv.Read().FromFile("products.csv").ToList();
// Async stream millions of rows without buffering
await foreach (Product p in Csv.Read().FromFileAsync("products.csv"))
{
Console.WriteLine($"{p.Name}: {p.Price:C}");
}
// Write collection to a file
Csv.Write().ToFile("out.csv", products);
Reads and writes Excel workbooks with zero external dependencies (utilizes only standard .NET compression and XML packages).
using HeroParser.Excels.Core;
// Read Excel files
List products = Excel.Read().FromFile("products.xlsx");
// Write Excel workbook with custom header styles, column styles, and auto-merged duplicate values
var headerStyle = ExcelStyle.Create()
.WithFont(f => f.WithName("Arial").WithSize(12).WithBold().WithColor("FFFFFF"))
.WithFill(fill => fill.WithSolidColor("007ACC")); // Blue background
Excel.Write()
.WithHeaderStyle(headerStyle)
.WithMergeDuplicates(p => p.Category) // Vertically merge contiguous duplicate Categories
.ToFile("out.xlsx", products);
Map properties to specific character boundaries using the [PositionalMap] attribute:
[GenerateBinder]
public class Employee
{
[PositionalMap(Start = 0, Length = 10)]
public string Id { get; set; } = "";
[PositionalMap(Start = 10, Length = 30)]
public string Name { get; set; } = "";
[PositionalMap(Start = 40, Length = 10, Alignment = FieldAlignment.Right)]
public decimal Salary { get; set; }
}
// Read positional records
var employees = FixedWidth.Read().FromFile("employees.dat").Records;
// Write positional records
FixedWidth.Write().ToFile("out.dat", employees);
Perfect for AI fine-tuning datasets, streamed LLM responses, and database bulk loading.
// Read JSONL files (AOT-safe)
var records = Jsonl.Read().FromFile("products.jsonl").ToList();
// Write JSONL files
Jsonl.Write().ToFile("out.jsonl", records);
A custom, high-speed binary serialization format optimized for zero allocations, platform independence, and vector embedding storage (supporting float[] arrays).
[GenerateBinder]) to map properties directly with zero-boxing and zero-reflection overhead.float[]), enabling ultra-fast vector embedding serialization without string parsing overhead.// Read HTB binary files (AOT-safe)
List products = Htb.Read().FromFile("products.htb").ToList();
// Async stream HTB records
await foreach (Product p in Htb.Read().FromFileAsync("products.htb"))
{
Console.WriteLine($"{p.Name}: {p.Price:C}");
}
// Write records to an HTB file
Htb.Write().ToFile("out.htb", products);
// Direct, allocation-free CSV ↔ HTB conversions
Htb.ConvertFromCsv("products.csv", "products.htb", HtbSchema.FromType());
A zero-allocation, reflection-free, and 100% Native AOT-compatible library designed to replace or drop-in substitute Spectre.Console in high-performance terminal applications.
using HeroParser.Console;
// Render styled ANSI markup
AnsiConsole.MarkupLine("[bold green]Success:[/] Row validation completed in [yellow]4.2ms[/].");
// Render highly styled tables and panels with zero allocations
var table = new Table().Border(TableBorder.Rounded);
table.AddColumn("[blue]Filename[/]");
table.AddColumn("[blue]Records[/]");
table.AddRow("data.csv", "10,240");
table.AddRow("data.jsonl", "102,400");
AnsiConsole.Write(table);
Annotate a single record class once, and read or write it across multiple formats:
| Attribute | Purpose | CSV | Excel | Fixed-Width | HTB |
|---|---|---|---|---|---|
[GenerateBinder] | Emits Roslyn source-generated, reflection-free mapping binder | Yes | Yes | Yes | Yes |
[TabularMap(Name, Index)] | Maps property to column header or index | Yes | Yes | No | Yes |
[PositionalMap(Start, Length...)] | Declares character position, alignment, and pad characters | No | No | Yes | No |
[Parse(Format)] | Converts raw values to custom types (e.g. DateTime format) | Yes | Yes | Yes | No |
[Format(WriteFormat...)] | Customizes output formatting during serialization | Yes | Yes | Yes | No |
[Validate(Range, Pattern...)] | Validates properties bidirectionally (Strict/Lenient modes) | Yes | Yes | Yes | Yes |
HeroParser includes first-class support for LLM, vector search, and RAG pipelines:
LlmRepair)Repairs truncated final rows (unclosed quotes/escapes) and strips markdown code blocks from raw LLM text streams.
using HeroParser.AI;
// Repaired on-the-fly and parsed directly into strongly-typed records
await foreach (var dev in LlmRepair.ReadFromTextAsync(rawLlmResponse))
{
Console.WriteLine($"{dev.Name} is a {dev.Role}");
}
ToLlmEmbeddingsAsync)Batches streamed records and pairs them with vector embeddings with a zero-allocation, token-budgeted streaming wrapper.
using HeroParser.AI;
await foreach (var chunk in developers.ToLlmEmbeddingsAsync(
async (texts, ct) => await GetEmbeddingsFromApiAsync(texts, ct),
options: new LlmChunkOptions { MaxTokensPerChunk = 250 },
batchSize: 16))
{
Console.WriteLine($"Chunk of {chunk.Chunk.EndRow - chunk.Chunk.StartRow + 1} rows embedded.");
}
SchemaMetadata.MapFromToolCall)Maps flat dictionaries of case-insensitive arguments returned by tool calling models into typed record models, executing validation constraints and raising rich validation feedback.
using HeroParser.AI;
Developer dev = SchemaMetadata.MapFromToolCall(arguments);
TabularContextProfiler)Generates structured statistical profile cards in markdown directly from datasets to inject into LLM system prompts.
string contextCard = developers.GenerateContextCard(datasetName: "Engineering Team");
JsonLlmChunker)Chunks datasets into valid, token-bounded JSON array blocks, optimized for ingestion into RAG context windows.
await foreach (var jsonChunk in developers.ToJsonLlmChunksAsync(options)) { ... }
ArrayPool for buffers[RequiresUnreferencedCode] where reflection is unavoidableIAsyncEnumerable for CSV, Fixed-Width, Excel, JSONL, and HTB; true non-blocking I/O with sync fast paths.xlsx using only System.IO.Compression and System.XmlJsonl.Read() / Jsonl.Write() mirror the CSV builder pattern; AOT-safe via JsonTypeInfo; CsvToJsonlConverter projects tabular data into OpenAI/Anthropic fine-tuning shapesIAsyncEnumerable.BatchAsync(size) groups streamed records into fixed-size batches for OpenAI/Voyage/Cohere/Anthropic embedding callsVectorParser.ParseFloats(span) handles pre-computed embeddings ("[0.1,0.2,…]", comma/semicolon/whitespace separators, culture-aware)Csv.CreateDataReader(), FixedWidth.CreateDataReader(), Excel.CreateDataReader(), Jsonl.CreateDataReader() for database bulk loading via SqlBulkCopyCsv.ReadFromPipeReaderAsync(pipe) for network streaming without buffering the entire payload[Validate] constraints (NotNull, NotEmpty, Range, Pattern) collected lazily; result.ThrowIfAnyError() for fail-fastFor advanced features and full API guides, see the files under the docs folder:
# Build all projects
dotnet build
# Run unit tests
dotnet test --filter Category=Unit
# Run integration tests
dotnet test --filter Category=Integration
# Run all tests
dotnet test
# Check code formatting
dotnet format --verify-no-changes
# Run benchmarks
dotnet run -c Release --project benchmarks/HeroParser.Benchmarks
MIT