This README is the one indexed from the repository at its latest ref, not from the tag for this version.

Fetcher

A standalone Crystal library for fetching RSS feeds, Reddit posts, JSON Feeds, and software release data.

⚠️ Unstable API: This library is undergoing active development. The API may change until v1.0.0.

v0.9.21 upgrades to Crystal 1.19.1 (Time.instant migration), adds a GitHub Atom release fallback, a DNS cache size cap, and resource-cleanup helpers (Cache#close, RequestConfig#close). v0.9.4 adds Reddit OAuth2 authentication to bypass datacenter IP blocks. v0.9.3 adds GitLab/Codeberg token support and thread safety improvements. v0.9.0 introduced structured RequestConfig (replacing flat parameters) and moves Cache to a class-based API with a singleton. See Breaking Changes below.

Features

• RSS/Atom Feeds - Standard RSS and Atom feed parsing with content extraction
• JSON Feed - Full JSON Feed v1.1 support
• Reddit - Fetch posts from subreddits
• YouTube - Fetch videos from YouTube channels
• Software Releases - Track GitHub, GitLab, and Codeberg releases
• Content Extraction - Extract full content, authors, categories, and attachments
• Feed Metadata - Extract feed-level information (title, description, language, authors)
• Automatic Driver Detection - Automatically selects the right parser based on URL
• Caching Support - ETag and Last-Modified header support
• Retry Logic - Built-in retry with exponential backoff
• Configurable Timeouts - Customize connection and read timeouts
• Rate Limiting - Optional per-domain rate limiting to prevent API abuse
• Circuit Breaker - Per-domain circuit breaker for resilience when fetching thousands of feeds
• HTTP Compression - Automatic gzip/deflate support
• Secure URL Detection - Blocks private IP ranges to prevent SSRF attacks
• Type-Safe Errors - Structured error types for better error handling

Performance Notes

Connection pooling was removed in v0.2.0 for simplicity. Each request creates a new HTTP::Client instance. For most use cases this is fine, but high-frequency fetching may experience slight performance overhead from repeated TCP connections.

v0.3.0 adds configurable timeouts to handle slow feeds.

v0.4.0 adds optional rate limiting to prevent API abuse.

v0.5.0 adds streaming processing for memory safety with large feeds and token bucket rate limiting for better scalability.

v0.6.0 adds circuit breaker support for resilience when fetching thousands of feeds.

v0.7.0 removes async API methods and fixes thread-safety issues with HTTP client reuse.

Installation

Add to your shard.yml:

dependencies:
  fetcher:
    github: kritoke/fetcher.cr
    version: "~> 0.9"

Usage

Simple Fetch

require "fetcher"

# Simple fetch (auto-detects feed type - RSS, Atom, or JSON Feed)
result = Fetcher.pull("https://example.com/feed.xml")

if error = result.error_message
  puts "Error: #{error}"
else
  result.entries.each { |entry| puts entry.title }
end

Accessing Extracted Content

result = Fetcher.pull("https://example.com/feed.xml")

# Access feed-level metadata
puts "Feed: #{result.feed_title}" if result.feed_title
puts "Description: #{result.feed_description}" if result.feed_description

result.entries.each do |entry|
  # Basic fields (available in all versions)
  puts "Title: #{entry.title}"
  puts "URL: #{entry.url}"
  puts "Published: #{entry.published_at}" if entry.published_at
  
  # Rich content extraction (v0.3.0+)
  puts "Content: #{entry.content}" if !entry.content.empty?
  puts "Author: #{entry.author}" if entry.author
  puts "Categories: #{entry.categories.join(", ")}" unless entry.categories.empty?
  
  # Attachments (podcasts, downloads)
  entry.attachments.each do |att|
    puts "Attachment: #{att.url} (#{att.mime_type})"
    puts "Size: #{att.size_in_bytes / 1024}KB" if att.size_in_bytes
  end
end

Custom Timeouts

# Configure custom timeouts for slow feeds
config = Fetcher::RequestConfig.new(
  timeout: Fetcher::TimeoutConfig.new(connect: 30.seconds, read: 60.seconds)
)

result = Fetcher.pull("https://slow.example.com/feed.xml", config: config)

Rate Limiting (v0.4.0+)

# Configure per-domain rate limiting to prevent API abuse
config = Fetcher::RequestConfig.new(
  rate_limit: Fetcher::RateLimitConfig.new(requests_per_second: 10)
)

result = Fetcher.pull("https://api.example.com/feed.xml", config: config)

Circuit Breaker (v0.6.0+)

For high-volume applications fetching thousands of feeds, the circuit breaker prevents cascading failures by automatically stopping requests to failing domains.

# Configure circuit breaker (enabled by default)
config = Fetcher::RequestConfig.new(
  circuit_breaker: Fetcher::CircuitBreakerConfig.new(
    enabled: true,               # Enable/disable (default: true)
    failure_threshold: 5,        # Open circuit after 5 failures
    recovery_timeout: 60.seconds # Wait 60s before testing recovery
  )
)

result = Fetcher.pull("https://failing-domain.com/feed.xml", config: config)

When a circuit breaker is open, requests to that domain are immediately rejected with a CircuitOpenError without making HTTP requests, reducing load on failing services.

The circuit breaker follows a standard state machine:

• Closed: Normal operation
• Open: After threshold failures, reject all requests
• Half-Open: After recovery timeout, allow one test request
• Closed: If test request succeeds, return to normal operation

Reddit OAuth (v0.9.4+)

Reddit blocks requests from datacenter/VPS IP addresses. To bypass this, register a "script" app at https://www.reddit.com/prefs/apps and provide the credentials via RequestConfig. The OAuth token is cached and auto-refreshed. When credentials are not configured, unauthenticated requests are used as a fallback.

config = Fetcher::RequestConfig.new(
  reddit_client_id: "your_client_id",
  reddit_client_secret: "your_client_secret",
  reddit_username: "your_bot_username",
  reddit_password: "your_bot_password"
)

result = Fetcher.pull_reddit("https://www.reddit.com/r/worldnews", config: config)

Error Handling (v0.4.0+)

result = Fetcher.pull("https://example.com/feed.xml")

# New type-safe error handling
if !result.success?
  error = result.error
  puts "Error: #{error.message}"
  puts "Kind: #{error.kind}"  # ErrorKind enum
  
  case error.kind
  when .timeout?
    # Handle timeout
  when .rate_limited?
    # Handle rate limiting
  when .http_error?
    puts "HTTP #{error.status_code}"
  end
end

# Backward compatible - still works
if msg = result.error_message
  puts "Error: #{msg}"
end

With Caching Headers

result = Fetcher.pull(
  "https://example.com/feed.xml",
  headers: HTTP::Headers.new,
  etag: "abc123",
  last_modified: "Wed, 01 Jan 2025 00:00:00 GMT",
  limit: 50
)

Custom Headers

headers = HTTP::Headers{
  "Authorization" => "Bearer token",
  "X-Custom" => "value"
}

result = Fetcher.pull("https://example.com/feed.xml", headers: headers)

Breaking Changes in v0.9.0

RequestConfig now uses structured sub-configs instead of flat parameters. The flat parameter constructors are removed.

Structured configuration is required:

# New (v0.9.0+)
config = Fetcher::RequestConfig.new(
  timeout: Fetcher::TimeoutConfig.new(connect: 30.seconds, read: 60.seconds),
  retry: Fetcher::RetryConfig.new(max_retries: 5),
  circuit_breaker: Fetcher::CircuitBreakerConfig.new(enabled: true, failure_threshold: 5),
  rate_limit: Fetcher::RateLimitConfig.new(requests_per_second: 10),
  streaming: Fetcher::StreamingConfig.new(enabled: true),
  cache_config: Fetcher::CacheConfig.new(max_size: 500)
)

Accessors changed:

• config.connect_timeout → config.timeout.connect
• config.read_timeout → config.timeout.read
• config.max_retries → config.retry.max_retries
• config.rate_limit_capacity → config.rate_limit.capacity
• config.circuit_breaker_enabled → config.circuit_breaker.enabled
• config.use_streaming_parser → config.streaming.enabled
• config.cache_enabled → config.cache_config.enabled

Cache is now a class with a singleton (Cache.default). Backward-compatible class methods (Cache.get, Cache.set, Cache.clear, Cache.stats) still work. For isolated caches, create instances with Cache.new(max_size: 100).

Cache Injection (non-breaking)

The Cache API allows optional injection of a CacheStore into Cache instances. This is backward-compatible and opt-in:

• Preserve existing behavior (per-instance isolated caches):

  cache = Fetcher::Cache.new(100, true)

• Share a store across multiple Cache instances (new):

  shared_store = Fetcher::CacheStore.new(100, true) cache1 = Fetcher::Cache.new(100, true, shared_store) cache2 = Fetcher::Cache.new(100, true, shared_store)

• Configure the global shared (class-level) store used by Cache.get/set:

  Fetcher::Cache.set_default_store(shared_store)

This approach preserves existing behavior for library consumers while making store sharing explicit and testable.

Cache key format changed from fetcher:reddit:* to reddit:*.

Reddit-specific cache helpers moved to Reddit module. Backward-compatible aliases (Cache.generate_key, Cache.ttl_for_sort, Cache.clear_subreddit) still work but delegate to Reddit.

Breaking Changes in v0.7.0

The *_async methods have been removed. Crystal's native fiber support makes these wrapper methods redundant. Users who need async behavior should wrap calls in fibers directly:

# Before (v0.6.x)
channel = Fetcher.pull_async(url)
result = channel.receive

# After (v0.7.0+)
channel = Channel(Fetcher::Result).new
spawn { channel << Fetcher.pull(url) }
result = channel.receive

Removed methods:

• Fetcher.pull_async(url, ...)
• Fetcher.pull_rss_async(url, ...)
• Fetcher.pull_reddit_async(url, ...)
• Fetcher.pull_software_async(url, ...)
• Fetcher.pull_json_feed_async(url, ...)

This change reduces API surface area and eliminates potential resource leaks from unreceived channels.

# Force a specific driver instead of auto-detection
result = Fetcher.pull_rss("https://example.com/feed.xml")
result = Fetcher.pull_reddit("https://reddit.com/r/crystal")
result = Fetcher.pull_software("https://github.com/crystal-lang/crystal/releases")
result = Fetcher.pull_json_feed("https://example.com/feed.json")
result = Fetcher.pull_youtube("https://www.youtube.com/channel/UCxxxxxxxxxxxxxxxxxx")

Explicit Driver Selection

# Force a specific driver instead of auto-detection
result = Fetcher.pull_rss("https://example.com/feed.xml")
result = Fetcher.pull_reddit("https://reddit.com/r/crystal")
result = Fetcher.pull_software("https://github.com/crystal-lang/crystal/releases")
result = Fetcher.pull_json_feed("https://example.com/feed.json")
result = Fetcher.pull_youtube("https://www.youtube.com/channel/UCxxxxxxxxxxxxxxxxxx")

Structured Configuration (v0.8.0+)

For better organization, you can use the structured configuration approach with sub-configs:

# Using structured configuration
config = Fetcher::RequestConfig.new(
  timeout: Fetcher::TimeoutConfig.new(connect: 30.seconds, read: 60.seconds),
  retry: Fetcher::RetryConfig.new(max_retries: 5, base_delay: 2.seconds),
  circuit_breaker: Fetcher::CircuitBreakerConfig.new(enabled: true, failure_threshold: 10),
  rate_limit: Fetcher::RateLimitConfig.new(requests_per_second: 10),
  driver_detection_mode: Fetcher::DriverDetectionMode::UrlOnly,
  error_detail_level: Fetcher::ErrorDetailLevel::Normal
)

result = Fetcher.pull("https://example.com/feed.xml", config: config)

The structured configuration is fully backward compatible - you can continue using the flat parameter style if preferred.

Automatic Driver Detection

The library automatically detects the feed type based on the URL:

URL Pattern	Driver
youtube.com/channel/	YouTube
reddit.com/r/	Reddit
github.com/.../releases	Software
any-domain/.../-/releases	Software (GitLab, including self-hosted)
codeberg.org/.../releases	Software
.json, /feed.json, /feeds/json	JSON Feed
All others	RSS

Software Releases

The Software driver fetches releases from GitHub, GitLab (including self-hosted), and Codeberg with the following features:

Supported Platforms

Platform	API Used	Fallback
GitHub	REST API (JSON)	None
GitLab	REST API (JSON)	releases.atom → tags.atom
Codeberg	REST API (JSON)	releases.atom

Features

• Self-hosted GitLab: Automatically detects any GitLab instance (e.g., gitlab.company.com/owner/repo/-/releases)
• GitLab fallback chain: If releases API fails, falls back to Atom feed, then to tags Atom feed
• Release body extraction: Extracts release notes/description into entry.content and entry.content_html
• Version extraction: Release version available in entry.version

Example

# GitHub releases
result = Fetcher.pull("https://github.com/crystal-lang/crystal/releases")
result.entries.first.content  # => "## Changes\n- Fixed bug..."

# Self-hosted GitLab
result = Fetcher.pull("https://gitlab.company.com/team/project/-/releases")
result.entries.first.version  # => "v1.2.3"

# Codeberg
result = Fetcher.pull("https://codeberg.org/user/repo/releases")

YouTube Channels

The YouTube driver fetches videos from YouTube channels using their RSS feed.

Supported URL Format

Only direct channel ID URLs are supported:

• youtube.com/channel/UC... ✓

Other formats (handles, custom URLs, usernames) are not supported.

Example

# Auto-detected
result = Fetcher.pull("https://www.youtube.com/channel/UCxxxxxxxxxxxxxxxxxx")

# Explicit driver
result = Fetcher.pull_youtube("https://www.youtube.com/channel/UCxxxxxxxxxxxxxxxxxx")

result.entries.each do |entry|
  puts "Title: #{entry.title}"
  puts "URL: #{entry.url}"
  puts "Published: #{entry.published_at}"
  puts "Author: #{entry.author}"  # Channel name
end

Result Structure

Result Record

record Result,
  # Core fields
  entries : Array(Entry),
  etag : String?,
  last_modified : String?,
  site_link : String?,
  favicon : String?,
  error : Error?,           # Structured error (v0.4.0+)
  error_message : String?,  # Backward compatible accessor
  
  # Feed metadata (v0.3.0+)
  feed_title : String?,
  feed_description : String?,
  feed_language : String?,
  feed_authors : Array(Author)

# Check success/failure
result.success?  # Returns true if no error

record Author,
  name : String?,
  url : String?,
  avatar : String?

Entry Record

record Entry,
  title : String,
  url : String,
  source_type : SourceType,  # Type-safe enum (v0.4.0+), was String
   
  # Rich content (v0.3.0+)
  content : String,           # Full content
  content_html : String?,     # HTML version
  author : String?,           # Author name
  author_url : String?,       # Author URL
  categories : Array(String), # Tags/categories
  attachments : Array(Attachment), # Media files
  
  # Existing fields
  published_at : Time?,
  version : String?           # For software releases

record Attachment,
  url : String,
  mime_type : String,
  title : String?,
  size_in_bytes : Int64?,
  duration_in_seconds : Int32?

RequestConfig (v0.9.0+)

Fetcher::RequestConfig.new(
  timeout: Fetcher::TimeoutConfig.new(connect: 10.seconds, read: 30.seconds),
  retry: Fetcher::RetryConfig.new(max_retries: 3, base_delay: 1.second),
  circuit_breaker: Fetcher::CircuitBreakerConfig.new(enabled: true, failure_threshold: 5),
  rate_limit: Fetcher::RateLimitConfig.new(requests_per_second: nil),
  streaming: Fetcher::StreamingConfig.new(enabled: false, max_memory: 10_485_760),
  cache_config: Fetcher::CacheConfig.new(enabled: true, max_size: 1000),
  max_redirects: 5,
  driver_detection_mode: Fetcher::DriverDetectionMode::Auto,
  error_detail_level: Fetcher::ErrorDetailLevel::Debug
)

Supported Feed Formats

RSS 2.0

• Standard RSS 2.0 elements (title, link, description, pubDate)
• Content-encoded module (content:encoded)
• Dublin Core (dc:creator for author)
• Enclosures (podcasts, downloads)
• Categories
• RSS 1.0/RDF (basic support)

Atom 1.0

• Standard Atom elements (title, link, published, updated)
• Content element (HTML, text, xhtml types)
• Summary element
• Author element (name, uri)
• Categories (term attribute)

JSON Feed 1.0/1.1

• Full JSON Feed v1.0 and v1.1 support
• content_html and content_text
• authors array (feed and item level)
• tags as categories
• attachments for podcasts/media
• date_published and date_modified
• Feed metadata (title, description, language, icon, favicon)

Debugging

Debug Logging

Set the FETCHER_DEBUG environment variable to enable detailed debug logging for troubleshooting:

FETCHER_DEBUG=1 ./your_application

This will output:

• Driver selection for each URL
• Specific exception details when errors occur
• Error context for better diagnosis

Debug logging is particularly useful for diagnosing issues with:

• Reddit API failures
• Feed parsing errors
• Network connectivity problems
• Rate limiting scenarios

Monitoring Circuit Breaker State

For high-volume applications, monitor circuit breaker states to understand which domains are failing:

# Check all circuit breaker states
states = Fetcher::CircuitBreaker::Registry.all_states
states.each do |domain, (state, failure_count)|
  puts "#{domain}: #{state} (#{failure_count} failures)"
end

Development

Requires Crystal >= 1.19.0.

crystal deps
crystal spec  # 423 examples

Contributing

1. Fork it
2. Create your feature branch (git checkout -b feature/my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin feature/my-new-feature)
5. Create a new Pull Request

License

MIT