onnxruntime

This README is the one indexed from the repository at its latest ref, not from the tag for this version.

# onnxruntime.cr

[![build](https://github.com/kojix2/onnxruntime.cr/actions/workflows/test.yml/badge.svg)](https://github.com/kojix2/onnxruntime.cr/actions/workflows/test.yml)
[![Lines of Code](https://img.shields.io/endpoint?url=https%3A%2F%2Ftokei.kojix2.net%2Fbadge%2Fgithub%2Fkojix2%2Fonnxruntime.cr%2Flines)](https://tokei.kojix2.net/github/kojix2/onnxruntime.cr)

[ONNX Runtime](https://github.com/Microsoft/onnxruntime) bindings for Crystal

## Installation

1. Install ONNX Runtime

   Download and install the ONNX Runtime from the [official releases](https://github.com/microsoft/onnxruntime/releases).

   **Option A: System-wide installation (Recommended)**

   For Linux:

   ```bash
   VERSION_TAG=$(cat ONNXRUNTIME_VERSION)
   VERSION=${VERSION_TAG#v}
   wget https://github.com/microsoft/onnxruntime/releases/download/$VERSION_TAG/onnxruntime-linux-x64-$VERSION.tgz
   tar -xzf onnxruntime-linux-x64-$VERSION.tgz
   
   # Install to system directories
   sudo cp onnxruntime-linux-x64-$VERSION/lib/* /usr/local/lib/
   sudo cp -r onnxruntime-linux-x64-$VERSION/include/* /usr/local/include/
   sudo ldconfig
   ```

   For macOS:

   ```bash
   VERSION_TAG=$(cat ONNXRUNTIME_VERSION)
   VERSION=${VERSION_TAG#v}
   curl -L https://github.com/microsoft/onnxruntime/releases/download/$VERSION_TAG/onnxruntime-osx-arm64-$VERSION.tgz -o onnxruntime-osx-arm64-$VERSION.tgz
   tar -xzf onnxruntime-osx-arm64-$VERSION.tgz
   
   # Install to system directories
   sudo cp onnxruntime-osx-arm64-$VERSION/lib/* /usr/local/lib/
   sudo cp -r onnxruntime-osx-arm64-$VERSION/include/* /usr/local/include/
   ```

   **Option B: Using local installation**

   If you prefer not to install system-wide, set library paths:

   ```bash
   # Download and extract as above, then:
   export LIBRARY_PATH=/path/to/onnxruntime-linux-x64-$VERSION/lib:$LIBRARY_PATH  # For build time
   export LD_LIBRARY_PATH=/path/to/onnxruntime-linux-x64-$VERSION/lib:$LD_LIBRARY_PATH  # For runtime
   
   # Build and run your Crystal program
   crystal build your_program.cr
   ./your_program
   ```

   Alternatively, use `--link-flags`:

   ```bash
   # Set path variable for convenience
   ORT_LIB=/path/to/onnxruntime-linux-x64-$VERSION/lib
   
   # Build with embedded rpath
   crystal build your_program.cr --link-flags="-L$ORT_LIB -Wl,-rpath,$ORT_LIB"
   
   # Run without LD_LIBRARY_PATH
   ./your_program
   ```

2. Add the dependency to your `shard.yml`:

   ```yaml
   dependencies:
     onnxruntime:
       github: kojix2/onnxruntime.cr
   ```

3. Run `shards install`

## Usage

```crystal
require "onnxruntime"

# Recommended: RAII block style
OnnxRuntime::InferenceSession.open("path/to/model.onnx", release_env: true) do |session|
  # Print model inputs and outputs
  puts "Inputs:"
  session.inputs.each do |input|
    puts "  #{input.name}: #{input.type} #{input.shape}"
  end

  puts "Outputs:"
  session.outputs.each do |output|
    puts "  #{output.name}: #{output.type} #{output.shape}"
  end

  # Prepare input data
  input_data = {
    "input_name" => [1.0_f32, 2.0_f32, 3.0_f32]
  }

  # Run inference
  result = session.run(input_data)

  # Process results
  result.each do |name, data|
    puts "#{name}: #{data}"
  end
end
```

## MNIST Example

Download the MNIST model: [mnist-12.onnx](https://github.com/onnx/models/blob/main/validated/vision/classification/mnist/model/mnist-12.onnx) ([raw](https://github.com/onnx/models/raw/refs/heads/main/validated/vision/classification/mnist/model/mnist-12.onnx)

```crystal
require "onnxruntime"

# Load the MNIST model
session = OnnxRuntime::InferenceSession.new("mnist-12.onnx")

# Create a dummy input (28x28 image draw 1)
input_data = Array(Float32).new(28 * 28) { |i| (i % 14 == 0 ? 1.0 : 0.0).to_f32 }

# Run inference
result = session.run({"Input3" => input_data}, ["Plus214_Output_0"], shape: {"Input3" => [1_i64, 1_i64, 28_i64, 28_i64]})

# Get the output probabilities
probabilities = result["Plus214_Output_0"].as(Array(Float32))

# Find the digit with highest probability
predicted_digit = probabilities.index(probabilities.max)
puts "Predicted digit: #{predicted_digit}"

# Explicitly release resources
session.release
OnnxRuntime::InferenceSession.release_env
```

## Memory Management

You can use either explicit release or block-based RAII.

Recommended for short scripts: block-based RAII

```crystal
OnnxRuntime::InferenceSession.open("path/to/model.onnx", release_env: true) do |session|
  result = session.run(input_data)
  # session is always released, even if an error is raised
end
```

Explicit release (useful for long-running apps):

```crystal
# Create and use session
session = OnnxRuntime::InferenceSession.new("path/to/model.onnx")
result = session.run(input_data)

# When finished, explicitly release resources
session.release
OnnxRuntime::InferenceSession.release_env
```

`release_session` remains available for backward compatibility.

For long-running applications like web servers, use explicit release with signal handlers:

```crystal
Signal::INT.trap do
  puts "Shutting down..."
  session.release
  OnnxRuntime::InferenceSession.release_env
  exit
end
```

See the examples directory for more detailed implementations.

Why do you need to manually free memory?

Previously, the following error was displayed on macOS.

```
libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Program received and didn't handle signal ABRT (6)
```

This seems to be related to multithreading and mutex. According to the AI, this is difficult to solve with `finalize`, so we tried to solve it by creating a reference counter, but we were unable to solve it in the end. If you can solve this problem, please create a pull request!

## Development

The code is generated by AI and may not be perfect.
Please feel free to contribute and improve it.

## Contributing

1. Fork it (<https://github.com/kojix2/onnxruntime.cr/fork>)
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request