Unified Nim library for creating/extracting archives and compress/decompress files
SIMD-accelerated, zero-copy I/O core, walk virtual file system
Supporting .tar, .zip, .gz, .zz, .lz4, .bz2, .br

clue install archive

API reference

Note

This is an LLM-accelerated Nim package ported from mholt/archives (Go). It has diverged substantially from the original: own DEFLATE/gzip/zip/tar codecs in pure Nim with no C bindings, split CompressionFormat/ArchiveFormat enums, no global registry, and a zero-copy I/O core. As elsewhere in nimbase, it is written with heavy LLM assistance — APIs may still change, and contributions (bug reports, test vectors, benchmarks) are welcome.

Features

• Archive and compress across tar, zip, gzip, zlib (.zz), lz4, bzip2 and brotli with one uniform API
• Split CompressionFormat/ArchiveFormat string enums plus a MediaType enum: misuse fails at compile time, no runtime registry
• High-level one-shots: compress/decompress (strings, files, buffers), archiveDir/archiveData, auto-identifying extractFile/extractAll/listEntries
• Own DEFLATE/gzip/zip/tar/crc32, no C bindings, brotli via pkg/nbrotli
• Zero-copy I/O core powered by std/memfiles: memory-mapped inputs, caller-reused output buffers, borrowed spans — memory-efficient with bounded working sets, no streams anywhere
• Format detection by filename and magic bytes without consuming input, plus tar.gz-style two-layer composition
• Virtual filesystem with uniform reads over dirs, files, archives and compressed archives
• Per-entry progress callbacks plus a cancellable context, threaded through every operation
• Optional nimsimd feature (AVX2 copies, match finding, SSE4.2 CRC32) behind a runtime silicon self-test
• Per-block CRCs and stream checksums verified on every decode; corrupt inputs raise descriptive errors
• Interop-tested both directions against system gzip/unzip, bzip2, lz4 and CPython zlib/bz2

Examples

Runnable versions live in examples/ (nim.cfg points at ../src, so no install needed — just clue build). Snippets below: High-level API first (import archive covers types and dispatch too); zero-copy Input/Output plumbing is still there when you need it:

import std/[options, strutils]
import archive

# one-shot compression with per-call level (0 = codec default)
let enc = compress(cfBz2, "hello ".repeat(100), level = 9)
assert decompress(cfBz2, enc) == "hello ".repeat(100)

# directory -> backup.tar.gz -> read back (files memory-mapped lazily,
# nothing is read up front; context is internal unless you pass one)
archiveDir(afTar, "my-docs", "backup.tar.gz", "docs",
  compression = some(cfGz))

# extractAll is an iterator of (archive path, content) pairs
for e in extractAll("backup.tar.gz"):
  if e.name.endsWith(".txt"):
    echo e.name, ": ", e.data.len, " bytes"

Archive scattered files by path (stored as given), or rename on the way in with (disk path, archive path) pairs:

let blob = archivePaths(afTar, @["notes.txt", "src/main.nim"])
let renamed = archivePaths(afTar, @[("run.sh", "bin/run")],
  compression = some(cfGz))

Archive in-memory files straight to disk, as zip or compressed tar:

import archive/files   # filesFromEntries, FsEntry: describe files in code

# entries pair an on-disk path with its path inside the archive;
# here both live in memory via the data field
var ctx: ArchiveContext
let entries = filesFromEntries(ctx, FromFSOptions(), [
  FsEntry(path: "a.txt", meta: FileMeta(name: "a.txt", size: 5,
    isRegular: true), data: "hello"),
  FsEntry(path: "sub/b.txt", meta: FileMeta(name: "b.txt", size: 5,
    isRegular: true), data: "world"),
], ".", "")

# plain zip, then tar compressed with gzip (some(cfGz))
archiveFile(afZip, entries, "out.zip", ctx)
archiveFile(afTar, entries, "out.tar.gz", ctx,
  compression = some(cfGz))

# the one-shots above also work without ctx:
archiveFile(afZip, entries, "out.zip")

Inspect an archive, then extract only what you need:

import archive/io   # Input/Output plumbing for entry bodies

# peek at entry paths without decompressing file contents
for name in listEntries("out.tar.gz"):
  echo name

# auto-identifies the format; the callback runs once per entry,
# so filter inside it and read bodies lazily via info.open()
extractFile("out.tar.gz", proc(c: ArchiveContext; info: FileInfo) =
  if info.isRegular() and info.nameInArchive.endsWith(".txt"):
    echo(info.nameInArchive & ": " & $readAll(info.open()).len & " bytes"))

Or extract the whole tree to disk in one call (creates directories, recreates symlinks, rejects entries escaping the destination):

extractFile("out.tar.gz", "restored")

Same work with reusable buffers instead of fresh strings:

import archive/io   # Input/Output: zero-copy cursors, reusable buffers

# caller-owned buffer reused across calls: no per-call allocation
let dst = newOutput()

# zero-copy in (shares the string), compressed bytes appended to dst
compressInto(cfGz, openInputString("hello ".repeat(100)), dst)
let back = newOutput()

# toInput borrows dst's bytes; decompressed output lands in back
decompressInto(cfGz, toInput(dst), back)

# take moves the bytes out as an exact-size string
assert take(back) == "hello ".repeat(100)

Browse an archive through the virtual filesystem:

import archive/vfs  

for path, e in walkFs("out.tar.gz"):
  echo (if e.isDir: "dir  " else: "file ") & path

Roadmap

• tar (ustar/pax, symlinks, hardlinks, insert/append)
• zip (deflate/store, insert/replace)
• gzip (.gz), zlib (.zz), lz4 (.lz4), bzip2 (.bz2)
• brotli (.br) via nbrotli (decode full, store-only encode)
• Identify by name and magic bytes, CompressedArchive composition (tar.gz, tar.bz2, ...)
• Virtual filesystem (DirFS, ArchiveFS, DeepFS, path helpers)
• SIMD acceleration behind the nimsimd feature with silicon self-test
• Encrypted archives via pkg/nimcypher (XChaCha20-Poly1305 AEAD, envelope around the whole blob, format-agnostic like CompressedArchive): EncryptedArchive composition (archive → compress → encrypt on write, decrypt → decompress → extract on read), password keys via Argon2 (deriveKeyFromPassword + stored salt) plus a raw-key API, streaming AEAD (aeadStreamWrite/aeadStreamRead) so file-to-file paths stay bounded-memory, own framing magic + salt + nonce header
• xz (.xz, LZMA2 framing + range coder)
• zstd (.zst)
• snappy/s2 (.sz/.s2, own port)
• lzip (.lz), minlz (.mz)
• rar / 7z read-only (likely needs C bindings or new ports)

❤ Contributions & Support

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

🎩 License

MIT license | Nim Community.