This README is the one indexed from the repository at its latest ref, not from the tag for this version.

awscr-s3

A Crystal shard for S3 and compatible services.

Installation

Add this to your application's shard.yml:

dependencies:
  awscr-s3:
    github: taylorfinnell/awscr-s3

Examples

Examples

Documentation

Documentation

Usage

require "awscr-s3"

Creating a Client

client = Awscr::S3::Client.new("us-east-1", "key", "secret")

For S3 compatible services, like DigitalOcean Spaces or Minio, you'll need to set a custom endpoint:

client = Awscr::S3::Client.new("nyc3", "key", "secret", endpoint: "https://nyc3.digitaloceanspaces.com")

If you wish you wish to you version 2 request signing you may specify the signer

client = Awscr::S3::Client.new("us-east-1", "key", "secret", signer: :v2)

List Buckets

resp = client.list_buckets
resp.buckets # => ["bucket1", "bucket2"]

Delete a bucket

client = Client.new("region", "key", "secret")
resp = client.delete_bucket("test")
resp # => true

Create a bucket

client = Client.new("region", "key", "secret")
resp = client.put_bucket("test")
resp # => true

Put Object

resp = client.put_object("bucket_name", "object_key", "myobjectbody")
resp.etag # => ...

You can also pass additional headers (e.g. metadata):

client.put_object("bucket_name", "object_key", "myobjectbody", {"x-amz-meta-name" => "myobject"})

Delete Object

resp = client.delete_object("bucket_name", "object_key")
resp # => true

Check Bucket Existence

resp = client.head_bucket("bucket_name")
resp # => true

Raises an exception if bucket does not exist.

Batch Delete Objects

resp = client.batch_delete("bucket_name", ["key1", "key2"])
resp.success? # => true

Get Object

resp = client.get_object("bucket_name", "object_key")
resp.body # => myobjectbody

# Or stream the object (recommended for large objects)
client.get_object("bucket_name", "object_key") do |obj|
  IO.copy(obj.body_io, STDOUT) # => myobjectbody
end

List Objects

client.list_objects("bucket_name").each do |resp|
  p resp.contents.map(&.key)
end

Upload a file

uploader = Awscr::S3::FileUploader.new(client)

File.open(File.expand_path("myfile"), "r") do |file|
  puts uploader.upload("bucket_name", "someobjectkey", file)
end

You can also pass additional headers (e.g. metadata):

uploader = Awscr::S3::FileUploader.new(client)

File.open(File.expand_path("myfile"), "r") do |file|
  puts uploader.upload("bucket_name", "someobjectkey", file, {"x-amz-meta-name" => "myobject"})
end

Creating a Presigned::Form.

form = Awscr::S3::Presigned::Form.build("us-east-1", "access key", "secret key") do |form|
  form.expiration(Time.unix(Time.now.to_unix + 1000))
  form.condition("bucket", "mybucket")
  form.condition("acl", "public-read")
  form.condition("key", SecureRandom.uuid)
  form.condition("Content-Type", "text/plain")
  form.condition("success_action_status", "201")
end

You may use version 2 request signing via

form = Awscr::S3::Presigned::Form.build("us-east-1", "access key", "secret key", signer: :v2) do |form|
  ...
end

Converting the form to raw HTML (for browser uploads, etc).

puts form.to_html

Submitting the form.

data = IO::Memory.new("Hello, S3!")
form.submit(data)

Creating a Presigned::Url.

options = Awscr::S3::Presigned::Url::Options.new(
   aws_access_key: "key",
   aws_secret_key: "secret",
   region: "us-east-1",
   object: "test.txt",
   bucket: "mybucket",
   additional_options: {
  "Content-Type" => "image/png"
})

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:put)

You may use version 2 request signing via

options = Awscr::S3::Presigned::Url::Options.new(
  aws_access_key: "key",
  aws_secret_key: "secret",
  region: "us-east-1",
  object: "test.txt",
  bucket: "mybucket",
  signer: :v2
)

Support 3rd party services

For S3-compatible services like DigitalOcean Spaces, Minio, or Backblaze B2, you'll need to set a custom endpoint.

Minio (local)

To use Minio locally, ensure your Minio server is running and configure the endpoint as follows:

options = Awscr::S3::Presigned::Url::Options.new(
  endpoint: "http://127.0.0.1:9000",
  region: "unused",
  aws_access_key: "admin",
  aws_secret_key: "password",
  bucket: "foo",
  force_path_style: true,
  object: "/test.txt"
)

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:get) # => "http://127.0.0.1:9000/foo/test.txt?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=admin%2F20250313%2Funused%2Fs3%2Faws4_request&X-Amz-Date=20250313T235624Z&X-Amz-SignedHeaders=host&X-Amz-Signature=ce2b33af1dbfd0e132f31d3f9d716eb5d66f4d19fbdcb691e816f7033e345bce"

DigitalOcean Spaces

For DigitalOcean Spaces, the bucket is included in the subdomain:

options = Awscr::S3::Presigned::Url::Options.new(
  endpoint: "https://ams3.digitaloceanspaces.com",
  region: "unused",
  aws_access_key: "ACCESSKEYEXAMPLE",
  aws_secret_key: "SECRETKEYEXAMPLE",
  bucket: "foo",
  object: "/test.txt"
)

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:get) # => "https://foo.ams3.digitaloceanspaces.com/test.txt?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESSKEYEXAMPLE%2F20250313%2Funused%2Fs3%2Faws4_request&X-Amz-Date=20250313T235511Z&X-Amz-SignedHeaders=host&X-Amz-Signature=e7b17f3c335ed615bf68845f81c2091814f856b61d05cb5aae3ad664de0f1b6e"

Using a Custom HTTP Provider

By default, awscr-s3 creates a new standard HTTP::Client for each request. However, you may want more advanced connection management features, such as pooling or connection reuse. You can accomplish this by supplying your own HttpClientFactory to Awscr::S3::Http or Awscr::S3::Client.

Below is a minimal (and not production-ready) demonstration:

require "awscr-s3"
require "http"

class PoolingHttpClientFactory < Awscr::S3::HttpClientFactory
  getter pool : Array(HTTP::Client)
  @created_count : Int32 = 0

  def initialize(@pool_size : Int32 = 3)
    @pool = [] of HTTP::Client
  end

  def acquire_raw_client(endpoint : URI) : HTTP::Client
    if @pool.size > 0
      @pool.pop
    elsif @created_count < @pool_size
      @created_count += 1
      HTTP::Client.new(endpoint)
    else
      raise "No available clients in pool (limit of #{@pool_size} reached)"
    end
  end

  def release(client : HTTP::Client?)
    return unless client
    @pool << client
  end
end

Developing awscr-s3

To develop and run tests, use the following commands:

Run All Tests

To run all types of tests (unit and integration) by default, use:

$ crystal spec

Unit Tests

To run only the unit tests and exclude integration tests, use the --tag filter to exclude the integration tests:

$ crystal spec --tag '~integration'

Integration Tests

Integration tests test awscr-s3 against local S3 implementations, such as MinIO. Before running the integration tests, ensure that the necessary dependencies (like the MinIO server) are running.

1. Start the MinIO server and any required services using Docker Compose:

   $ docker compose -f spec/integration/compose.yml up -d

2. Once the dependencies are up, run the integration tests:

   $ crystal spec --tag 'integration'