virtualfs

# VirtualFs

A virtual filesystem for Crystal. Mount directories, `.vpak` archives, `.zip`, and `.tar` into a single merged tree. Read files through a stdlib-style API — `File`, `Dir`, `IO` — without caring where they physically live.

## Why

Game engines, desktop apps, and tools need to ship assets alongside a binary. You want to:

- **Patch without rebuilding** — mount a patch archive at higher priority, it shadows the originals
- **Ship fewer files** — pack thousands of small files into one `.vpak` with deduplication and compression
- **Encrypt assets** — per-entry AES-256-GCM or ChaCha20-Poly1305 encryption
- **Read efficiently** — content-defined chunking (FastCDC) enables range reads, mmap, and partial decompression
- **Mount once, read everywhere** — same code reads from disk, `.vpak`, `.zip`, `.tar`, or in-memory data

## Install

```yaml
# shard.yml
dependencies:
  virtualfs:
    github: naqvis/virtualfs
```

```bash
shards install
```

## Quick start

```crystal
require "virtualfs"

vfs = VirtualFs::Vfs.new

# Mount by path — container type inferred from extension
vfs.mount("assets/",    "/", priority: 0)   # directory
vfs.mount("patch.vpak", "/", priority: 10)  # shadows assets/ on conflict

# Read through the merged tree
vfs.exists?("config.json")          # => true
vfs.read("config.json")             # => String
vfs.open("data.bin") do |file|
  buf = Bytes.new(256)
  file.read(buf)
  file.read_range(64, 16)           # direct range read
end
vfs.ls("maps/")                     # => ["forest.map", "cave.map"]
vfs.walk("maps")                    # => recursive listing
vfs.glob("**/*.ogg")                # => ["audio/music.ogg", ...]
```

## Build a `.vpak`

### Via API

```crystal
VirtualFs::PakFormat.build("assets.vpak", package_id: "game:1.0") do |b|
  b.add_file("src/logo.png", "logo.png", mime: "image/png")
  b.add_dir("src/audio/", mount_point: "audio")
  b.compress(:deflate)
  b.encrypt(key_provider)
end
```

### Via CLI

```bash
# Build
vpack build assets/ --out game.vpak --compress deflate
vpack build assets/ --out secure.vpak --encrypt key.bin

# Inspect
vpack list game.vpak

# Verify
vpack verify game.vpak
```

Supported codecs: `none`, `deflate`, `gzip`, `snappy`, `zlib`. Supported ciphers: `aes-256-gcm`, `chacha20-poly1305`.

## Features

| Feature | Description |
|---------|-------------|
| **Multiple containers** | `OsDir`, `.vpak`, `.zip`, `.tar`, `InMemory` |
| **Priority mounting** | Higher-priority mounts shadow lower ones, per-file |
| **Compression** | Pluggable codecs: none, deflate, gzip, snappy, zlib |
| **Encryption** | Per-entry AEAD encryption with per-chunk deterministic nonces |
| **Content-defined chunking** | FastCDC splits original files at natural boundaries; each chunk is transformed independently |
| **Deduplication** | Identical transformed chunks stored once, referenced by SHA-256 hash |
| **mmap** | Zero-copy memory mapping for uncompressed OsDir entries and single-chunk `.vpak` entries |
| **Async loading** | `open_async` and `prefetch` for background I/O |
| **Range reads** | Read byte ranges without decompressing the whole file |
| **HTTP serving** | `StaticHandler` serves VFS files over HTTP with Range support |
| **Thread safe** | Concurrent mount/read/write without external locking |

> **Note on random access:** `.vpak` entries, uncompressed `OsDir` files, and uncompressed `.tar`/`.zip` entries support true random access. Compressed `.tar.gz` archives are fully decompressed at mount time, and compressed `.zip` entries must be decompressed from the start for each seek, so these formats are best for full-file reads.

## Mounting

```crystal
# Priority order: highest first, first match wins
vfs.mount("base.vpak",  "/",    priority: 0)
vfs.mount("patch.vpak", "/",    priority: 10)  # shadows base on conflict
vfs.mount("dlc.vpak",   "/dlc", priority: 0)   # scoped to /dlc/ prefix

# Case-insensitive matching
vfs.mount("assets/", "/", priority: 0, case_insensitive: true)

# Direct container instances for explicit control
vfs.mount(VirtualFs::OsDir.new("assets/"), "/", priority: 0)
vfs.mount(VirtualFs::InMemory.new, "/", priority: 10)
```

## Encryption

```crystal
key = Bytes.new(32) { |i| (i + 1).to_u8 }
vfs.mount("secure.vpak", "/", priority: 0,
  key_provider: VirtualFs::Crypto::StaticKey.new(key))
```

Nonces are derived from `(package_id, entry_hash, chunk_index)` — no nonce storage, no reuse. AEAD tags verified on read; mismatch raises `VirtualFs::CorruptEntry`.

## Compression

Codecs are stored per-entry in the `.vpak` format. `BlockStore` auto-detects on read — no need to specify `codec:` at mount time.

Register custom codecs:

```crystal
VirtualFs::Codec.register("lz4", MyLz4Codec.new)
```

## `.vpak` format

Binary, little-endian, designed for random access. See [`vpak-format.md`](./vpak-format.md) for the full byte-level specification.

```
┌─────────────────────┐ offset 0
│ Header (128 bytes)  │  magic "VPK1\0\0\0\0", version, flags,
│                     │  index offset/length, file count, chunk size,
│                     │  dedup offset/length, tool version
├─────────────────────┤
│ Blob data           │  original file contents chunked via FastCDC;
│                     │  each chunk is compressed and/or encrypted independently
├─────────────────────┤
│ Dedup table         │  per chunk: SHA-256, offset, original length,
│                     │  compressed/encrypted length, CRC32
├─────────────────────┤
│ Index               │  per entry: path, flags, size, chunk refs,
│                     │    chunk original lengths, chunk CRC32s, mime,
│                     │    tags, attrs, timestamps, codec, cipher, mode
├─────────────────────┤
│ SHA-256 (32 bytes)  │  integrity hash of the index
├─────────────────────┤
│ Magic (8 bytes)     │  truncation check
└─────────────────────┘
```

Integers in the index use LEB128 varint encoding — flags, size, chunk count, tag/attr counts, timestamps, and mode all pack into 1–9 bytes. Only CRC32 values and the entry length prefix remain fixed-width.

Every unique **transformed** chunk is stored once. Multiple entries sharing identical compressed/encrypted content reference the same chunk data via SHA-256 hash. This is automatic — zero configuration.

Encrypted chunks use a **random 12-byte nonce per chunk**, stored as `nonce || ciphertext || AEAD tag`. The nonce is not derived from the path or chunk index, so identical plaintext does not produce identical ciphertext.

Three CRCs are stored: a whole-entry blob CRC (on-disk data), a whole-entry original data CRC, and a per-chunk CRC. The whole-entry CRCs are verified on full reads; the per-chunk CRCs are verified on random-access reads.

## HTTP handler

```crystal
vfs = VirtualFs::Vfs.new
vfs.mount("app.vpak", "/", priority: 10)
vfs.mount("dist/",    "/", priority: 20)

server = HTTP::Server.new([VirtualFs::StaticHandler.new(vfs)])
server.listen("127.0.0.1", 3000)
```

GET/HEAD, `index.html` for directories, `Range` requests for streaming, MIME detection from stdlib registry.

## Architecture

Six layers, bottom-up. Cross-cutting features (compression, encryption, mmap, cache) live only in `BlockStore` — container drivers never decompress or decrypt.

```
┌─────────────────────────────────────────────────────────────────┐
│ 6. Vfs façade        mirrors Crystal stdlib: File / Dir / IO    │
├─────────────────────────────────────────────────────────────────┤
│ 5. Async + Events     open_async / prefetch;                    │
│                       on_progress / on_error callbacks on Handle│
├─────────────────────────────────────────────────────────────────┤
│ 4. MountStack         ordered read mounts + single write mount; │
│                       path to (Mount, Entry); per-file override  │
├─────────────────────────────────────────────────────────────────┤
│ 3. Index              per-mount file table: path, size, blob    │
│                       location, flags, checksums, metadata      │
├─────────────────────────────────────────────────────────────────┤
│ 2. BlockStore         the cross-cutting I/O engine:             │
│                       mmap, decompress, decrypt, dedup,          │
│                       prefetch, range/stream reads                │
├─────────────────────────────────────────────────────────────────┤
│ 1. Container drivers  Pak | Zip | Tar | OsDir | InMemory        │
│                       each yields an Index + raw byte ranges    │
└─────────────────────────────────────────────────────────────────┘
```

Adding a new container is cheap: produce an `Index` and serve raw byte ranges. The hard parts live in one testable place (`BlockStore`).

## Error handling

All errors inherit from `VirtualFs::Error < Exception` and carry `path` and `mount_id` for diagnostics.

| Error | When |
|-------|------|
| `NotFound` | path not in any mount |
| `NotWritable` | write to read-only mount / no write dir |
| `CorruptPackage` | header, footer, or index SHA mismatch |
| `CorruptEntry` | per-chunk CRC or AEAD tag failure |
| `UnsupportedCodec` | entry flags name a codec not configured |
| `KeyMissing` | `KeyProvider` returned nil for encrypted entry |
| `PathInvalid` | invalid path (`../`, empty, etc.) |

## License

MIT

## Contributing

1. Fork it (<https://github.com/naqvis/virtualfs/fork>)
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request

## Contributors

- [Ali Naqvi](https://github.com/naqvis) - creator and maintainer