This README is the one indexed from the repository at its latest ref, not from the tag for this version.

C0DATA

A data system built on ASCII C0 control codes. One vocabulary, multiple shapes -- tabular data, hierarchical documents, key-value config, diffs, and more.

C0DATA sits between human-readable text formats (JSON, YAML, TOML) and opaque binary formats (protobuf, msgpack). Values are plain UTF-8 text. Structure is expressed through single-byte control codes -- compact, zero-copy friendly, and inspectable with minimal tooling.

The Idea

ASCII has 32 control codes (0x00-0x1F) that were designed in the 1960s for structuring data transmissions. Four of them -- FS, GS, RS, US -- are literal data separators at four hierarchical levels. C0DATA modernizes these codes as a universal grammar for structured data.

␜mydb
  ␝users
    ␁name␟amount␟type
    ␞Alice␟1502.30␟DEPOSIT
    ␞Bob␟340.00␟WITHDRAWAL
  ␝products
    ␁id␟product␟qty
    ␞01␟Widget␟100
    ␞02␟Gadget␟250
␄

That's a complete database with two tables, headers, and typed rows. Each glyph (␜ ␝ ␞ ␟ ␁ ␄) is a single-byte control code. No braces, no quotes, no escaping. Whitespace around control codes is trimmed -- the indentation is just for readability.

Control Codes

Byte	Abbr	Role
0x01	SOH	Header (declares field names)
0x02	STX	Open nested sub-structure
0x03	ETX	Close nested sub-structure
0x04	EOT	End of document
0x05	ENQ	Reference (look up named data)
0x10	DLE	Escape (next byte is literal)
0x17	ETB	Commit marker (stream mode)
0x1A	SUB	Substitution (C0DIFF)
0x1C	FS	File / Database separator
0x1D	GS	Group / Table / Section separator
0x1E	RS	Record / Row separator
0x1F	US	Unit / Field separator

Data Shapes

The same codes express multiple common data shapes:

Tabular (CSV, SQL)

␝users
  ␁name␟amount␟type
  ␞Alice␟1502.30␟DEPOSIT
  ␞Bob␟340.00␟WITHDRAWAL

Key-Value (TOML, INI)

␝database
  ␞host␟localhost
  ␞port␟5432
␝server
  ␞host␟0.0.0.0
  ␞port␟8080

Document (Markdown)

GS repeated indicates depth level (like # in Markdown):

␜My Document
  ␝Chapter 1
    ␞First paragraph.
    ␞A list:
      ␟Item one
      ␟Item two
    ␝␝Section 1.1
      ␞Nested content.
  ␝Chapter 2
    ␞And so on.

Diff (Atomic Multi-File Edits)

C0DIFF uses anchored patterns for safe search-and-replace:

␜foo.txt
  ␝Hello ␟world␚universe␟!

Means: in foo.txt, find Hello world!, replace world with universe.

Stream (Append-Only Logs)

For event logs and write-ahead logs, ETB (␗) commits each appended block. A crashed append never writes its commit marker, so replay skips the torn tail instead of folding corrupt state:

␞create␟a1b2␗
␞name␟draft-2␗

A block may hold several records under one commit — an atomic batch.

Whitespace and Quoting

In pretty form, whitespace around control codes is trimmed. To preserve significant whitespace, use STX/ETX as quotes:

␞␂  leading spaces  ␃␟normal value

Performance

The scanner's hot loop is a single comparison: byte < 0x20. This makes C0DATA inherently fast to parse -- single-byte delimiters, zero-copy friendly, and SIMD-acceleratable.

Benchmark on 10 MB document (Crystal, --release):

  avg         4.88 ms       2048.0 MB/s
  best        4.09 ms       2447.7 MB/s

For comparison, a fast YAML scanner in C achieves ~420 MB/s on equivalent data.

Installation

Add the dependency to your shard.yml:

dependencies:
  c0:
    github: c0data/c0-cr

Then run shards install.

Usage

require "c0"

Serializable

Like JSON::Serializable, include C0::Serializable in a class to get to_c0 and from_c0 methods:

class User
  include C0::Serializable

  property name : String
  property amount : String

  def initialize(@name = "", @amount = "")
  end
end

# Serialize
user = User.new("Alice", "1502.30")
buf = user.to_c0               # => Bytes (compact form)
str = user.to_c0_pretty        # => String (pretty form)

# Deserialize
user = User.from_c0(buf)
user = User.from_c0(pretty_string)

# Collections
users = [User.new("Alice", "100"), User.new("Bob", "200")]
buf = users.to_c0              # multi-record group
restored = User.array_from_c0(buf)

Field annotations:

class Product
  include C0::Serializable

  @[C0::Field(key: "product_id")]
  property id : Int32 = 0

  @[C0::Field(ignore: true)]
  property internal : String = ""

  property name : String = ""

  def initialize(@id = 0, @name = "", @internal = "")
  end
end

Hashes and named tuples serialize as key-value groups:

{"host" => "localhost", "port" => "5432"}.to_c0("database")
{host: "localhost", port: "5432"}.to_c0("config")

Building

buf = C0::Builder.build do |b|
  b.file("mydb") do
    b.group("users", headers: ["name", "amount", "type"]) do
      b.record("Alice", "1502.30", "DEPOSIT")
      b.record("Bob", "340.00", "WITHDRAWAL")
    end
  end
  b.eot
end

Reading

doc = C0::Document.new(buf)
doc.name                                # => "mydb"
doc["users"].table.record(0).field(0)   # => "Alice" (zero-copy slice)
doc["users"].table.record(0).field(1)   # => "1502.30"

A field whose value is a flat list (US-separated items inside STX/ETX) is written with list_field and read back with Record#list:

buf = C0::Builder.build do |b|
  b.group("users") do
    b.record("Alice")
    b.list_field(["Admin", "Editor"])   # one field: ␂Admin␟Editor␃
  end
end
C0::Table.new(buf).record(0).list(1)  # => ["Admin", "Editor"] (unescaped slices)

Pretty-Printing

puts C0::Pretty.format(buf)
# ␜mydb
#   ␝users
#     ␁name␟amount␟type
#     ␞Alice␟1502.30␟DEPOSIT
#     ␞Bob␟340.00␟WITHDRAWAL
#   ␄

Round-Trip

pretty = C0::Pretty.format(buf)
compact = C0::Pretty.parse(pretty)
# compact is identical to the original buf

CSV Conversion

# CSV → C0DATA
buf = C0::CSV.from_csv(csv_string, group_name: "users")

# C0DATA → CSV
csv = C0::CSV.to_csv(buf)

JSON/YAML Conversion

# JSON → C0DATA
buf = C0::JSON.from_json(json_string)

# YAML → C0DATA
buf = C0::JSON.from_yaml(yaml_string, group_name: "config")

# C0DATA → JSON
json = C0::JSON.to_json(buf)

# C0DATA → YAML
yaml = C0::JSON.to_yaml(buf)

Tables become arrays of objects, key-value groups become flat objects. Nested JSON/YAML structures are preserved using STX/ETX scoping.

C0DIFF

diff = C0::Diff.build do |b|
  b.file("src/app.cr") do
    b.section do |s|
      s.anchor("class App\n  def ")
      s.sub("run", "start")
    end
  end
end

files = {"src/app.cr" => source_code}
result = C0::Diff.apply(diff, files)

Stream Logs

C0::Stream::Writer.open("claims.c0") do |log|
  log.record("create", nonce, ts)
  log.batch do |b|              # atomic multi-record commit
    b.record("name", label, ts)
    b.record("tag", tag, ts)
  end
end

reader = C0::Stream::Reader.read("claims.c0")
reader.torn?                          # uncommitted tail present?
reader.each_record { |rec| ... }      # committed records only

Writer.open repairs a torn tail before appending. c0fmt validate -s checks a log from the command line.

Two Forms

C0DATA has two representations:

• Compact -- canonical wire/storage format. Every byte is literal. No whitespace is ignored.
• Pretty -- human-readable. Uses Unicode Control Pictures (U+2400 block) for visible glyphs. Newlines and indentation are ignored by the parser.

Compact form is canonical in the strong sense: the same logical value encodes to exactly one byte sequence, so compact bytes can be hashed for content addressing (see the spec's "Canonical Form"). C0.canonical?(buf) checks a buffer; the shared conformance vectors live in the c0-spec submodule at c0-spec/ (run git submodule update --init after cloning).

The c0fmt command-line tool converts between them and more.

c0fmt

The c0fmt command-line tool (import/export CSV, JSON, YAML; pretty and compact forms; validation) lives in its own repository: https://github.com/c0data/c0-fmt. It is built on this library.

Design

See the specification (in the c0-spec submodule) for the full definition, including open questions and future directions.

Development

crystal spec        # run tests
crystal build bench/bench_tokenizer.cr -o bench/bench_tokenizer --release
./bench/bench_tokenizer 10   # benchmark with 10 MB document

License

MIT