HDiffPatch is a C/C++ library and command-line tools designed for efficient diff and patch operations between binary files or directories. It supports cross-platform functionality, runs quickly, creates small delta files, and handles large files while minimizing memory usage during operations.
Key Features:
Cross-platform compatibility with support for various operating systems.
Efficient creation of small, differential patches (delta files).
Fast performance optimized for handling large binary files.
Low memory requirements during diff and patch operations.
Compatibility with multiple patch formats, including bsdiff4, endsley/bsdiff, open-vcdiff, and xdelta3 (VCDIFF RFC 3284).
Support for both file-level and directory-level operations.
Audience & Benefit:
HDiffPatch is ideal for developers working on embedded systems, IoT devices, Android app updates, or any scenario requiring efficient binary patching. It enables the creation of smaller, faster-to-apply patches, reducing storage and bandwidth requirements while supporting memory-constrained environments. The toolset also offers compatibility with existing patch formats, making it a versatile solution for various use cases.
This software can be installed via winget, ensuring seamless integration into your development workflow.
HDiffPatch is a C\C++ library and command-line tools for diff & patch between binary files or directories(folder); cross-platform; fast running; create small delta/differential; support large files and limit memory requires when diff & patch.
if need patch (OTA) on embedded systems,MCU,NB-IoT..., see demo HPatchLite, with tinyuz decompressor can run on 1KB RAM devices! HPatchLite also supports a simple inplace-patch implementation to support storage-constrained devices.
update your own Android Apk? Jar or Zip file diff & patch? try ApkDiffPatch, to create smaller delta/differential! NOTE: ApkDiffPatch can't be used by Android app store, because it requires re-signing apks before diff.
sfpatcher not require re-signing apks (like archive-patcher), is designed for Android app store, create smaller patch file for apk, patch speed up by a factor of xx than archive-patcher & run with O(1) memory.
if you not have the old versions(too many or not obtain or have been modified), thus cannot create the delta in advance.
you can see sync demo hsynz (like zsync), the new version is only need released once,
and the owners of the old version get the information about the new version and do the diff&patch themselves.
hsynz support zstd compressor & run much faster than zsync; also compatible with the zsync's file format.
additional, if you have the new file locally & not the old file, but can get a hash certificate file(.hsyni) of the old file,
you can also create a hpatchz format patch file(usage scenario like rsync); see the demo cmdline hsign_diff in hsynz.
NOTE: This library does not deal with file metadata, such as file last write time, permissions, link file, etc... To this library, a file is just as a stream of bytes; You can extend this library or use other tools.
Releases/Binaries
Download from latest release : Command line app for Windows, Linux, MacOS; and .so patch lib for Android.
use cmdline to create a delta: $hdiffz oldPath newPath outDiffFile -WD -s-64
Tip: If you want a more precise match (which might use more memory), you could try changing -s-64 to -m-4
apply the delta: $hpatchz oldPath diffFile outNewPath
Tip: -WD format supports multi-thread to speed up patching, for example add -p-5 parameter (multi-thread not recommended on HDD)
Build it yourself
git clone --recurse-submodules https://github.com/sisong/HDiffPatch5.git
cd
$ build_libs.sh (or $ build_libs.bat on windows, then got *.so files)
import file com/github/sisong/HPatch.java (from HDiffPatch/builds/android_ndk_jni_mk/java/) & .so files, java code can call the patch function in libhpatchz.so
oldPath newPath inputPath can be file or directory(folder),
oldPath can empty, and input parameter ""
options:
-m[-matchScore]
DEFAULT; all file load into Memory; best diffFileSize;
requires (newFileSize+ oldFileSize*5(or *9 when oldFileSize>=2GB))+O(1)
bytes of memory;
matchScore>=0, DEFAULT -m-4, recommended bin: 0--4 text: 4--9 etc...
-s[-matchBlockSize]
all file load as Stream; fast;
requires O(oldFileSize*16/matchBlockSize+matchBlockSize*5*parallelThreadNumber)bytes of memory;
matchBlockSize>=4, DEFAULT -s-64, recommended 16,32,48,1k,64k,1m etc...
-block-fastMatchBlockSize
must run with -m;
set block match befor slow byte-by-byte match, DEFAULT -block-1k;
if set -block-0, means don't use block match;
fastMatchBlockSize>=4, recommended 128,4k,64k, etc...
if newData similar to oldData then diff speed++ & diff memory--,
but small possibility outDiffFile size+
-cache
must run with -m;
set is use a big cache for slow match, DEFAULT false;
if newData not similar to oldData then diff speed++,
big cache max used O(oldFileSize) memory, and build slow(diff speed--)
-w[-oldWinSize-segSize]
diff by window mode; optimize read old data when patch;
in -m mode, outDiffFile size+, in -s mode, outDiffFile size--;
if oldWinSize+ or segSize-, outDiffFile size-, but diff & patch speed--
oldWinSize: max window bytes on old data, DEFAULT -w-2m;
for big file, recommended -w-4m-128k, -w-16m-256k, etc...
segSize: initial data granularity during window matching, DEFAULT oldWinSize/64;
recommended oldWinSize/128<=segSize<=oldWinSize/4;
-WD[-stepSize] (need v5.0 patcher)
create window diffData(HDIFFW26), optimize read old data when patch;
recommended as the primary diff format; supports step-by-step patching
during step-by-step downloading, and supports multi-thread patching!
requires diff by window mode, auto enable -w-2m if not set;
default compress by zstd, can set -c-no for no compress;
default checksum by xxh128, can set -C-no for no checksum;
stepSize>=4096, DEFAULT -WD-256k, recommended 128k,512k etc...
-SD[-stepSize]
create single compressed diffData(HDIFFSF20 format), only need one decompress buffer
when patch, and support step by step patching when step by step downloading!
and supports multi-thread patching!
stepSize>=4096, DEFAULT -SD-256k, recommended 64k,2m etc...
-BSD
create diffFile compatible with bsdiff4, unsupport input directory(folder).
also support run with -SD (not used stepSize), then create single compressed
diffFile compatible with endsley/bsdiff (https://github.com/mendsley/bsdiff).
-VCD[-compressLevel[-dictSize]]
create diffFile compatible with VCDIFF, unsupport input directory(folder).
DEFAULT no compress, out format same as $open-vcdiff ... or $xdelta3 -S -e -n ...
if set compressLevel, out format same as $xdelta3 -S lzma -e -n ...
compress by 7zXZ(xz,lzma), compressLevel in {0..9}, DEFAULT level 7;
dictSize can like 4096 or 4k or 4m or 16m etc..., DEFAULT 8m
support compress by multi-thread parallel.
NOTE: default out diffFile used large source window size!
need used -w[-oldWinSize-segSize-newWinSize] to set source/segment/target window size.
-p-parallelThreadNumber
if parallelThreadNumber>1 then open multi-thread Parallel mode;
DEFAULT -p-4; requires more memory!
-p-search-searchThreadNumber
DEFAULT searchThreadNumber same as parallelThreadNumber;
old file on HDD hard drives WARNING: multi-thread search need frequent random
disk reads when -s-matchBlockSize or -block-fastMatchBlockSize(run with -m),
causes slowdown; at this time, need to close(searchThreadNumber<=1) multi-thread
search mode or reduce the number of searchThreadNumber!
-c-compressType[-compressLevel]
set outDiffFile Compress type, DEFAULT uncompress;
for -WD, DEFAULT compress by zstd;
for resave diffFile,recompress diffFile to outDiffFile by new set;
support compress type & level & dict:
-c-no no compress
-c-zlib[-{1..9}[-dictBits]] DEFAULT level 9
dictBits can 9--15, DEFAULT 15.
support run by multi-thread parallel, fast!
-c-ldef[-{1..12}] DEFAULT level 12
compatible with -c-zlib, faster or better compress than zlib;
used libdeflate compressor, & dictBits always 15.
support run by multi-thread parallel, fast!
-c-bzip2[-{1..9}] (or -bz2) DEFAULT level 9
-c-pbzip2[-{1..9}] (or -pbz2) DEFAULT level 8
support run by multi-thread parallel, fast!
NOTE: code not compatible with it compressed by -c-bzip2!
-c-lzma[-{0..9}[-dictSize]] DEFAULT level 7
dictSize can like 4096 or 4k or 4m or 128m etc..., DEFAULT 8m
support run by 2-thread parallel.
-c-lzma2[-{0..9}[-dictSize[-blockSize]]] DEFAULT level 7
dictSize can like 4096 or 4k or 4m or 128m etc..., DEFAULT 8m;
1m<=blockSize<=256m, DEFAULT 0 (AUTO) ...;
recommend blockSize==dictSize when need decompress by multi-thread;
support run by multi-thread parallel, fast!
NOTE: code not compatible with it compressed by -c-lzma!
-c-zstd[-{0..22}[-dictBits]] DEFAULT level 20
dictBits can 10--30, DEFAULT 23.
support run by multi-thread parallel, fast!
-C-checksumType
set outDiffFile Checksum type, for directory diff, DEFAULT -C-fadler64; for -WD, DEFAULT -C-xxh128;
support checksum type:
-C-no no checksum
-C-crc32
-C-fadler64
-C-md5
-C-xxh3 (need v4.12 patcher)
-C-xxh128 recommended (need v4.12 patcher)
-n-maxOpenFileNumber
limit Number of open files at same time when stream directory diff;
maxOpenFileNumber>=16, DEFAULT -n-48, the best limit value by different
operating system.
-g#ignorePath[#ignorePath#...]
set iGnore path list when Directory Diff; ignore path list such as:
#.DS_Store#desktop.ini#*thumbs*.db#.git*#.svn/#cache_*/00*11/*.tmp
# means separator between names; (if char # in name, need write #: )
* means can match any chars in name; (if char * in name, need write *: );
/ at the end of name means match the directory (including its contents);
-g-old#ignorePath[#ignorePath#...]
set iGnore path list in oldPath when Directory Diff;
if oldFile can be changed, need add it in old ignore list;
-g-new#ignorePath[#ignorePath#...]
set iGnore path list in newPath when Directory Diff;
in general, new ignore list should is empty;
-M#outManifestTxtFile
create a Manifest file for inputPath; it is a text file, saved infos of
all files and directoriy list in inputPath; this file while be used in
manifest diff, support re-checksum data by manifest diff;
can be used to protect historical versions be modified!
-M-old#oldManifestFile
oldManifestFile is created from oldPath; if no oldPath not need -M-old;
-M-new#newManifestFile
newManifestFile is created from newPath;
-D force run Directory diff between two files; DEFAULT (no -D) run
directory diff need oldPath or newPath is directory.
-neq
open check: if newPath & oldPath's all datas are equal, then return error;
DEFAULT not check equal.
-d Diff only, don't run patch check, DEFAULT run patch check.
-t Test only, run patch check, patch(oldPath,testDiffFile)==newPath ?
-f Force overwrite, ignore write path already exists;
DEFAULT (no -f) not overwrite and then return error;
if used -f and write path is exist directory, will always return error.
--patch
swap to hpatchz mode.
-info
print infos of diffFile.
-v print Version info.
-h (or -?)
print usage info.
if oldPath is empty input parameter ""
options:
-s[-cacheSize]
DEFAULT -s-8m; oldPath loaded as Stream;
cacheSize can like 262144 or 256k or 64m or 512m etc....
if diffFile is window diffData(created by hdiffz -WD-stepSize -w-oldWinSize), then requires
(cacheSize+ oldWinSize+stepSize + 1*decompress buffer size)+O(1) bytes of memory;
if diffFile is single compressed diffData(created by hdiffz -SD-stepSize), then requires
(cacheSize+ stepSize + 1*decompress buffer size)+O(1) bytes of memory;
if diffFile is compressed diffData(created by hdiffz), then requires
(cacheSize + 4*decompress buffer size)+O(1) bytes of memory.
if diffFile is created by hdiffz -BSD,bsdiff4, hdiffz -VCD, then requires
(cacheSize+ 3*decompress buffer size)+O(1) bytes of memory;
if diffFile is created by hdiffz -VCD -w,xdelta3,open-vcdiff, then requires
(cacheSize+ sourceWindowSize+targetWindowSize + 3*decompress buffer size)+O(1) bytes of memory.
-m oldPath all loaded into Memory;
if diffFile is window diffData(created by hdiffz -WD-stepSize -w-oldWinSize), then requires
(oldWinSize + stepSize + 1*decompress buffer size)+O(1) bytes of memory;
if diffFile is single compressed diffData(created by hdiffz -SD-stepSize), then requires
(oldFileSize + stepSize + 1*decompress buffer size)+O(1) bytes of memory.
if diffFile is compressed diffData(created by hdiffz), then requires
(oldFileSize + 4*decompress buffer size)+O(1) bytes of memory.
if diffFile is created by hdiffz -BSD,bsdiff4, then requires
(oldFileSize + 3*decompress buffer size)+O(1) bytes of memory.
if diffFile is VCDIFF(created by hdiffz -VCD,hdiffz -VCD -w,xdelta3,open-vcdiff), then requires
(sourceWindowSize+targetWindowSize + 3*decompress buffer size)+O(1) bytes of memory.
-p-parallelThreadNumber
if parallelThreadNumber>1 then open multi-thread Parallel mode;
now support window diffData(created by hdiffz -WD) and single compressed diffData(created by hdiffz -SD);
can set 1..5, DEFAULT -p-1!
-p-dec-decThreadNumber
if decThreadNumber>1 then open multi-thread decompress mode, DEFAULT -p-dec-1!
now only support lzma2 with blockSize; NOTE: need more memory!
-C-checksumSets
set Checksum data for window patch & directory patch & VCDIFF patch, DEFAULT -C-new-copy;
checksumSets support (can choose multiple):
-C-no no checksum;
-C-new checksum new files edited from old reference files;
-C-diff checksum diffFile; (VCDIFF unsupport)
-C-old checksum old reference files; (VCDIFF unsupport)
-C-copy checksum new files copy from old same files, for directory patch;
-C-all same as: -C-new-copy-diff-old;
-n-maxOpenFileNumber
limit Number of open files at same time when stream directory patch;
maxOpenFileNumber>=8, DEFAULT -n-24, the best limit value by different
operating system.
-f Force overwrite, ignore write path already exists;
DEFAULT (no -f) not overwrite and then return error;
support oldPath outNewPath same path!(patch to tempPath and overwrite old)
if used -f and outNewPath is exist file:
if patch output file, will overwrite;
if patch output directory, will always return error;
if used -f and outNewPath is exist directory:
if patch output file, will always return error;
if patch output directory, will overwrite, but not delete
needless existing files in directory.
-info
print infos of diffFile.
-v print Version info.
-h (or -?)
print usage info.
library API usage:
diff&patch function in file: libHDiffPatch/HDiff/diff.h & libHDiffPatch/HPatch/patch.h dir_diff() & dir patch in: dirDiffPatch/dir_diff/dir_diff.h & dirDiffPatch/dir_patch/dir_patch.h
manual:
create diff(in newData,in oldData,out diffData);
release the diffData for update oldData.
patch(out newData,in oldData,in diffData);
ok , get the newData.
v1 API, uncompressed diffData:
create_diff()
patch()
patch_stream()
patch_stream_with_cache()
v2 API, compressed diffData (HDIFF13):
create_compressed_diff()
create_compressed_diff_stream()
resave_compressed_diff()
patch_decompress()
patch_decompress_with_cache()
patch_decompress_mem()
v3 API, diff&patch between directories (HDIFF19):
dir_diff()
TDirPatcher_*() functions with struct TDirPatcher
v4 API, single compressed diffData (HDIFFSF20):
create_single_compressed_diff()
create_single_compressed_diff_stream()
resave_single_compressed_diff()
patch_single_stream()
patch_single_stream_mem()
patch_single_compressed_diff()
patch_single_stream_diff()
v5 API, window diffData (HDIFFW26):
create_window_diff()
patch_window_diff()
resave_window_diff()
hpatch lite API, optimized hpatch on MCU,NB-IoT... (demo HPatchLite):
