maildir

This README is the one indexed from the repository at its latest ref, not from the tag for this version.

**Build status**:
[![Build Status](https://github.com/crystallabs/maildir.cr/actions/workflows/ci.yml/badge.svg)](https://github.com/crystallabs/maildir.cr/actions/workflows/ci.yml)
[![Version](https://img.shields.io/github/tag/crystallabs/maildir.cr.svg?maxAge=360)](https://github.com/crystallabs/maildir.cr/releases/latest)
[![License](https://img.shields.io/github/license/crystallabs/maildir.cr.svg)](https://github.com/crystallabs/maildir.cr/blob/master/LICENSE)

# Maildir

A Crystal library for reading and writing files in the "Maildir" file and directory structure.
Even though this format is mainly used for email messages, the Maildir structure and the implementation of this module are general - they do not require the file contents to be related to email.

## What's so Great About the Maildir Format

See http://cr.yp.to/proto/maildir.html and http://en.wikipedia.org/wiki/Maildir

"Two words: no locks." -- Daniel J. Bernstein

The maildir format allows multiple processes to read and write arbitrary messages without file locks.

New messages are initially written to a "tmp/" directory with an automatically-generated unique filename. Once they are written, they are atomically moved to the "new/" directory where other processes can see and use them.

While the maildir format was created for email, it works well for arbitrary data. This library can read and write any contents using the Maildir file and directory structure. And if you want the contents to be automatically serialized/deserialized objects, pluggable serializers are supported as well.

## Installation

Add the following to your application's `shard.yml`:

```yaml
dependencies:
  maildir:
    github: crystallabs/maildir.cr
    version: ~> 6.0
```

And run `shards install`.

## Usage

Initialize a Maildir and create a Maildir directory structure in `/tmp/maildir_test`:

```crystal
require "maildir"
maildir = Maildir.new("/tmp/maildir_test") # creates tmp, new, and cur dirs
# To skip directory creation, call Maildir.new("/tmp/maildir_test", false)
```

Add a new message. This will create a new file with the contents "Hello, Crystal!" and return the message. As mentioned, messages are written to the "tmp/" directory and then moved to "new/".

```crystal
message = maildir.add("Hello, Crystal!")
```

List new messages:

```crystal
maildir.list("new") # => [message]
```

Move the message from "new" to "cur" to indicate that some process has retrieved and/or processed the message.

```crystal
message.process
```

Indeed, the message is now in "cur/", not "new/".

```crystal
maildir.list("new") # => []
maildir.list("cur") # => [message]
```

Add some flags to the message to indicate state.
See "What can I put in info" at http://cr.yp.to/proto/maildir.html for flag conventions.
The library has convenience methods like `seen!` and `seen?` for all of the 6 standard flags, but arbitrary flags can be set.

```crystal
message.add_flag("S")    # Mark the message as "seen"
message.add_flag("F")    # Mark the message as "flagged"
message.remove_flag("F") # Unflag the message
message.add_flag("DPR")  # Mark the message as "draft", "passed" and "replied"
message.remove_flag("DPR") # Remove the three flags
message.add_flag("T")    # Mark the message as "trashed"
message.add_flag("X")    # Mark with arbitrary-letter flag
```

List "cur/" messages based on flags. Flags must be specified in ascending ASCII order (`"ST"` and not `"TS"`).

```crystal
maildir.list("cur", {:flags => ""})   # => lists all messages without any flags
maildir.list("cur", {:flags => "F"})  # => lists all messages with flag "F"
maildir.list("cur", {:flags => "FS"}) # => lists all messages with flags "F" and "S"
maildir.list("cur", {:flags => "ST"}) # => lists all messages with flags "S" and "T"
```

Retrieve the key that uniquely identifies the message:

```crystal
key = message.key
```

Read/load the contents of the message:

```crystal
data = message.data
```

Find the message based on key:

```crystal
message_copy = maildir.get(key)
message == message_copy # => true
```

Delete the message from disk:

```crystal
message.destroy
maildir.list("cur") # => []
```

### Cleaning up Orphaned Messages

An expected (though rare) behavior is for partially-written messages to be
orphaned in the "tmp/" folder (when clients fail before fully writing a message).

Find messages in "tmp/" that haven't been changed in 36 hours:

```crystal
maildir.get_stale_tmp
```

Clean them up:

```crystal
maildir.get_stale_tmp.each { |msg| msg.destroy }
```

For more usage examples, please see files in the library's `spec/` folder.

### Durability

The maildir format promises that a message which is visible in "new/" is complete.
For that to hold across a crash or a power loss, the message has to reach the disk before it is moved out of "tmp/", so by default every message is `fsync`ed on write.

If your data is cheap to reproduce and you care more about delivery throughput than about surviving a crash, you can turn this off globally:

```crystal
Maildir.fsync = false
```

A custom serializer that manages durability itself can override `Maildir::Serializer::Base#sync` instead.

### Folders

Maildirs can contain folders, in the Maildir++ layout used by Courier and Dovecot.
A folder is a dot-prefixed directory inside the root maildir (`.a`), nesting is expressed by joining the name components with a dot (`.a.x`), and each folder carries an empty `maildirfolder` file so that other tools recognize it as a folder.
All of that is handled for you:

```crystal
maildir = Maildir.new("/tmp/maildir_test")

folder = maildir.folder("a")        # => the maildir at /tmp/maildir_test/.a/
nested = maildir.folder("a.x")      # => the maildir at /tmp/maildir_test/.a.x/
nested == folder.folder("x")        # => true; naming by dot and by chaining agree
maildir.folder("b", false)          # => a folder object without creating it on disk
```

Folders are maildirs like any other, so they add, list and process messages the same way:

```crystal
folder.add("Hello from a folder!")
folder.list("new") # => [message]
```

Navigate the tree with `#folders` (immediate subfolders, sorted), `#parent` and `#root`:

```crystal
maildir.folders.map(&.folder_name) # => ["a", "b"]
folder.folders.map(&.folder_name)  # => ["a.x"]
nested.parent == folder            # => true
nested.root == maildir             # => true
maildir.parent                     # => nil
```

`#folder?` and `#folder_name` tell a folder from a root maildir. A maildir which merely lives at a dot-path, such as `~/.maildir`, is correctly treated as a root and not as a folder of its parent directory.

```crystal
maildir.folder? # => false
nested.folder?  # => true
nested.folder_name # => "a.x"
```

## Pluggable Serializers

By default, message data are written and read from disk as a string. However, it may be desirable to automatically process strings into useful objects. This library supports configurable serializers to convert objects to strings and back.

The following serializers are included:

- `Maildir::Serializer::Base` (default - no serialization, writes and reads contents as string)
- `Maildir::Serializer::JSON` (uses `#to_json` and `JSON#parse`)
- `Maildir::Serializer::YAML` (uses `#to_yaml` and `YAML#parse`)

`Maildir.serializer` and `Maildir.serializer=` let you set the default serializer.

```crystal
Maildir.serializer # => Maildir::Serializer::Base.new (default serializer - strings)
message = maildir.add("Hello, Crystal!") # writes "Hello, Crystal!" to disk
message.data # => "Hello, Crystal!"
```

You can also set the serializer per individual maildir:

```crystal
maildir = Maildir.new("Maildir")
maildir.serializer = Maildir::Serializer::JSON.new
```

The JSON and YAML serializers work similarly, e.g.:

```crystal
maildir.serializer = Maildir::Serializer::JSON.new
my_data = {"foo" => nil, "my_array" => [1, 2, 3]}
message = maildir.add(my_data) # writes {"foo":null,"my_array":[1,2,3]}
message.data == my_data # => true
```

It is trivial to create a custom serializer. Just implement the following two methods:

```crystal
load(path)
dump(data, path)
```

## Similar projects

- https://github.com/ktheory/maildir - Ruby implementation