robots

master branch1 version
  • master branchlatestNov 26, 2021

github.com/grkek/robots.cr

Simple robots.txt parser for Crystal

1 stars
0 dependents
License: MIT

Installation

# Add this to your shard.yml
dependencies:
  robots:
    github: grkek/robots.cr
    branch: master

master is a branch, not a release, so this tracks it rather than pinning a version.

Then run:

shards install

shard.yml

Crystal
>= 0.36.0
License
MIT
Authors
Chris Watson, Giorgi Kavrelishvili

Dependencies

Runtime Dependencies

  • crest*github: mamantoha/crest

Development Dependencies

  • ameba~> 0.13.0github: crystal-ameba/amebadev

README

robots.cr

Simple robots.txt parsing for Crystal

Installation

  1. Add the dependency to your shard.yml:

    dependencies:
  robots:
    github: grkek/robots

  2. Run shards install

Usage

require "robots"

# From reddit
robotstxt = <<-TEXT
# 80legs
User-agent: 008
Disallow: /

User-Agent: *
Disallow: /goto
Disallow: /*after=
Disallow: /*before=
Disallow: /domain/*t=
Disallow: /login
Disallow: /reddits/search
Disallow: /search
Disallow: /r/*/search
Allow: /
TEXT

reddit = Robots.new(robotstxt, "Googlebot")

reddit.allowed?("/login")
# => false

reddit.allowed?("/r/SushiChain")
# => true

reddit.allowed?("/r/SushiChain/search")
# => false

Contributing

  1. Fork it (https://github.com/grkek/robots/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors

Documentation

Read the API docs

Built from the current release. The first visit to a release nobody has asked for starts its build.

Links

This branch

Branch
master
Seen
Nov 26, 2021
Crystal
>= 0.36.0
Indexed
yes

Dependents

No indexed shard depends on this one yet.

Repository

github.com/grkek/robots.cr

Metadata

Created
Aug 16, 2026
Updated
Aug 16, 2026
Synced
Aug 16, 2026
Versions
1