obelisk

Obelisk is a Crystal syntax highlighting library inspired by Chroma. It provides a performant, type-safe, and extensible syntax highlighting solution with support for multiple languages, themes, and output formats.

Features

• Multi-language support: Built-in lexers for Crystal, JSON, YAML, and more
• Multiple output formats: HTML, ANSI terminal colors, plain text, and JSON
• Theme ecosystem: Compatible with TextMate, Sublime Text, VS Code, and Chroma themes
• Advanced lexing: State machines, context-sensitive parsing, and delegating lexers
• Theme serialization: Import/export themes in JSON, TextMate (.tmTheme), and Chroma XML formats
• Extensible architecture: Easy to add new languages, formatters, and themes
• Type-safe design: Leverages Crystal's type system for reliability
• Performance focused: Hand-written lexers with compile-time optimizations

Installation

Add this to your application's shard.yml:

dependencies:
  obelisk:
    github: watzon/obelisk

Then run:

shards install

Quick Start

require "obelisk"

# Simple syntax highlighting
code = %q(
  def hello(name : String) : String
    "Hello, #{name}!"
  end
)

# Highlight Crystal code as HTML
html = Obelisk.highlight(code, "crystal", "html", "github")
puts html

# Highlight with ANSI colors for terminal
ansi = Obelisk.highlight(code, "crystal", "terminal", "monokai")
puts ansi

Usage

Basic Highlighting

# Quick highlighting
result = Obelisk.highlight(source_code, language, formatter, style)

# Available options
languages = Obelisk.lexer_names      # ["crystal", "json", "yaml", "text"]
formatters = Obelisk.formatter_names # ["html", "html-classes", "terminal", "text", "json"]
styles = Obelisk.style_names         # ["github", "monokai", "bw"]

Manual Tokenization

# Get a lexer and tokenize manually
lexer = Obelisk.lexer("crystal")
tokens = lexer.tokenize(source_code) if lexer

# Process tokens
tokens.each do |token|
  puts "#{token.type}: #{token.value.inspect}"
end

Custom Formatting

# Create formatters with options
html_formatter = Obelisk::HTMLFormatter.new(
  with_classes: true,
  class_prefix: "syntax-",
  with_line_numbers: true
)

# Get a style
style = Obelisk::Styles.github

# Format manually
output = html_formatter.format(tokens, style)

CSS Generation

# Generate CSS for HTML with classes
formatter = Obelisk::HTMLFormatter.new(with_classes: true)
css = formatter.css(Obelisk::Styles.github)
puts css

Theme Import/Export

# Load themes from different formats
tmtheme_style = Obelisk.load_theme("path/to/theme.tmTheme")     # TextMate theme
chroma_style = Obelisk.load_theme("path/to/theme.xml")          # Chroma XML theme
json_style = Obelisk.load_theme("path/to/theme.json")           # Obelisk JSON theme

# Export themes to different formats
json_output = Obelisk.export_theme_json(style, pretty: true)
tmtheme_output = Obelisk.export_theme_tmtheme(style)
chroma_output = Obelisk.export_theme_chroma(style)

# Save themes to files (format auto-detected from extension)
Obelisk.save_theme(style, "exported.json")      # JSON format
Obelisk.save_theme(style, "exported.tmtheme")   # TextMate format
Obelisk.save_theme(style, "exported.xml")       # Chroma XML format

Supported Languages

Core Languages

• Crystal (.cr) - Full syntax support including string interpolation, annotations, and Crystal-specific types
• JSON (.json) - Complete JSON syntax with proper escape handling
• YAML (.yaml, .yml) - YAML syntax including documents, anchors, and tags
• Plain Text - Fallback for unsupported file types

Additional Languages

More languages are being actively developed. See the Roadmap for upcoming language support.

Available Themes

• GitHub - Light theme matching GitHub's syntax highlighting
• Monokai - Dark theme with vibrant colors
• Black & White - Simple monochrome theme

Supported Theme Formats

• JSON - Obelisk's native theme format
• TextMate (.tmTheme) - Compatible with Sublime Text, VS Code, and other editors
• Chroma XML - Compatible with Go's Chroma syntax highlighter

Output Formats

• HTML - Standard HTML with inline styles or CSS classes
• Terminal - ANSI escape codes for colored terminal output
• Plain Text - Strips all formatting
• JSON - Structured token data for analysis

Architecture

Obelisk follows a clean, modular architecture:

• Tokens: Hierarchical token types with CSS class mapping
• Lexers: Regex-based lexers with state machines for complex languages
• Formatters: Pluggable output formatters for different targets
• Styles: Theme system with inheritance and customizable colors
• Registry: Central management of lexers, formatters, and styles

Token Types

Token types are organized hierarchically for style inheritance:

# Root categories
TokenType::Keyword
TokenType::Name
TokenType::Literal
TokenType::Comment

# Specific types inherit from parents
TokenType::KeywordDeclaration  # inherits from Keyword
TokenType::NameFunction        # inherits from Name
TokenType::LiteralString       # inherits from Literal

Creating Custom Lexers

class MyLanguageLexer < Obelisk::RegexLexer
  def config : Obelisk::LexerConfig
    Obelisk::LexerConfig.new(
      name: "mylang",
      aliases: ["my", "mylang"],
      filenames: ["*.my"],
      mime_types: ["text/x-mylang"]
    )
  end

  def rules : Hash(String, Array(Obelisk::LexerRule))
    {
      "root" => [
        Obelisk::LexerRule.new(/\bdef\b/, Obelisk::TokenType::Keyword),
        Obelisk::LexerRule.new(/[a-zA-Z_]\w*/, Obelisk::TokenType::Name),
        # ... more rules
      ]
    }
  end
end

# Register the lexer
Obelisk::Registry.lexers.register(MyLanguageLexer.new)

Creating Custom Styles

custom_style = Obelisk::Style.new("custom", Obelisk::Color::WHITE)
custom_style.set(Obelisk::TokenType::Keyword,
  Obelisk::StyleEntry.new(
    color: Obelisk::Color.from_hex("#ff0000"),
    bold: Obelisk::Trilean::Yes
  )
)

Obelisk::Registry.styles.register(custom_style)

Performance

Obelisk is designed for speed:

• Lazy compilation of regex rules
• Efficient token streaming with iterators
• Minimal memory allocation during lexing
• Fast color calculations and CSS generation

Examples

Check out the examples/ directory for comprehensive usage examples:

• 00_quickstart.cr - Simple, quick examples to get started
• 01_basic_usage.cr - Basic syntax highlighting usage
• 02_html_output.cr - HTML output options and configurations
• 03_terminal_colors.cr - Terminal/ANSI color output
• 04_multi_language.cr - Multiple language support
• 05_custom_formatter.cr - Creating custom formatters
• 06_css_generation.cr - CSS stylesheet generation
• 07_file_highlighting.cr - Highlighting source files
• 08_theme_comparison.cr - Comparing different themes
• 09_custom_style.cr - Creating custom syntax highlighting styles
• 10_theme_serialization.cr - Theme import/export examples
• 14_chroma_xml_themes.cr - Working with Chroma XML stylesheets

Run any example with:

crystal run examples/00_quickstart.cr

Roadmap

Core Architecture

• [x] Token System
  • [x] Hierarchical token types with parent/child relationships
  • [x] CSS class mapping for web output
  • [x] Token categories (keywords, literals, names, etc.)
• [x] Lexer System
  • [x] Abstract base lexer interface
  • [x] Regex-based lexer implementation
  • [x] State machine support
  • [x] Iterator-based token streaming
  • [x] Basic state mutations (push/pop)
  • [x] Advanced state mutations (include, combined states)
  • [x] Delegating lexers for embedded languages
  • [x] Lexer composition and chaining
• [x] Registry Pattern
  • [x] Centralized management of lexers, formatters, and styles
  • [x] Name and alias support
  • [x] Dynamic registration
  • [x] Priority-based lexer selection
  • [x] Content analysis for auto-detection
  • [x] MIME type and filename pattern matching

Language Support

• [x] Built-in Languages
  • [x] Crystal
  • [x] JSON
  • [x] YAML
  • [x] Plain text
  • [x] Ruby
  • [x] Python
  • [x] JavaScript/TypeScript
  • [x] Go
  • [x] Rust
  • [x] C/C++
  • [x] HTML/CSS
  • [x] Markdown
  • [x] SQL
  • [x] Shell/Bash
• [x] Language Features
  • [x] Keywords and operators
  • [x] String literals with escape sequences
  • [x] Comments (single and multi-line)
  • [x] Numbers (integers, floats, hex, binary, octal)
  • [x] String interpolation (Crystal)
  • [x] Embedded language support (via DelegatingLexer)
  • [x] Context-sensitive parsing

Output Formatters

• [x] HTML Formatter
  • [x] Inline styles
  • [x] CSS classes
  • [x] Custom class prefixes
  • [x] Line numbers
  • [x] Line highlighting (specific ranges)
  • [x] Linkable line numbers
  • [x] Table-based line number layout
• [x] Terminal Formatter
  • [x] ANSI color codes
  • [x] 24-bit true color support
  • [x] Bold, italic, underline styles
• [x] Plain Text Formatter
  • [x] Strip all formatting
• [x] JSON Formatter
  • [x] Token data as JSON
• [x] Custom Formatters
  • [x] Extensible formatter interface
  • [x] Examples (Markdown, BBCode, etc.)

Styling System

• [x] Style Engine
  • [x] RGB color support
  • [x] Style attributes (bold, italic, underline)
  • [x] Background colors
  • [x] Style builder API
• [x] Built-in Themes
  • [x] GitHub (light)
  • [x] Monokai (dark)
  • [x] Black & White
• [x] Theme Features
  • [x] CSS generation
  • [x] Per-token styling
  • [x] Full style inheritance
  • [x] Theme serialization/deserialization
  • [x] TextMate (.tmTheme) import/export
  • [x] Chroma XML import/export
  • [x] Multi-format theme support

Performance & Optimization

• [x] Memory Efficiency
  • [x] Iterator-based token streaming
  • [x] Lazy regex compilation
  • [x] Safe token iterator adapter (Crystal bug workaround)
• [x] Performance Features
  • [x] Token coalescing
  • [x] Token splitting utilities
  • [ ] Token size limits
  • [ ] Streaming optimizations
  • [ ] Parallel lexing support

Advanced Features

• [x] Token Processing
  • [ ] Token filtering
  • [ ] Token transformation pipelines
  • [x] Token coalescing
  • [x] Line splitting utilities
• [x] Language Detection
  • [x] Content-based analysis
  • [x] Confidence scoring
  • [x] Multi-lexer fallback
• [x] Import/Export
  • [x] TextMate (.tmTheme) theme support
  • [x] Chroma XML stylesheet support
  • [x] JSON theme serialization
  • [ ] Pygments compatibility
  • [ ] TextMate grammar support
  • [ ] Chroma XML lexer definitions (see #1)

Developer Experience

• [x] API Design
  • [x] Simple one-liner usage
  • [x] Progressive complexity
  • [x] Type-safe interfaces
• [x] Documentation
  • [x] Comprehensive examples
  • [x] API documentation
  • [x] Architecture overview
• [x] Testing
  • [x] Unit tests for all components
  • [x] Integration tests
  • [ ] Performance benchmarks
  • [ ] Fuzzing tests

Known Issues & Future Work

• [ ] Crystal Compiler Bug Workarounds
  • [x] Iterator memory corruption workaround (#2)
  • [ ] Remove workarounds when Crystal bug #14317 is fixed
• [ ] Performance Improvements
  • [ ] Re-enable token coalescing
  • [ ] Optimize large file handling
• [ ] Extended Language Support
  • [ ] XML lexer definitions support (Tartrazine compatibility)
  • [ ] Hybrid approach: hand-written + XML lexers
  • [ ] Code generation from XML definitions

Development

Running Tests

# Run tests without coverage
crystal spec

# Run tests with coverage (requires kcov)
./scripts/test.sh with-coverage

# Auto-detect kcov and run with coverage if available
./scripts/test.sh auto

Install kcov for coverage reports:

# Ubuntu/Debian
sudo apt-get install kcov

# macOS
brew install kcov

Development Commands

See CLAUDE.md for the complete list of development commands including:

• crystal spec - Run all tests
• crystal run examples/00_quickstart.cr - Run examples
• crystal build src/obelisk.cr --no-codegen - Type check
• crystal tool format - Format code

Contributing

1. Fork it (https://github.com/watzon/obelisk/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Inspired by Chroma for Go
• Token type hierarchy based on Pygments
• Built with ❤️ for the Crystal community