squirm

Version, currently 0.2.03 versions

github.com/squirm-framework/squirm

This was the night of the crawling terror!

32 stars
0 dependents
License: MIT

Installation

# Add this to your shard.yml
dependencies:
  squirm:
    github: squirm-framework/squirm
    version: ~> 0.2.0

Then run:

shards install

shard.yml

Crystal
~> 1.2.0
License
MIT
Author
Giorgi Kavrelishvili

Dependencies

Runtime Dependencies

  • crest*github: mamantoha/crest
  • redis*github: stefanwille/crystal-redis
  • lexbor*github: kostya/lexbor
  • robots*github: grkek/robots.cr, branch: master
  • rocksdb1.1.0github: maiha/rocksdb.cr
  • marionette*github: watzon/marionette, branch: master

Development Dependencies

  • ameba*github: crystal-ameba/amebadev

README

Squirm is a framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Installation

  1. Add the dependency to your shard.yml:

    dependencies:
      squirm:
        github: squirm-framework/squirm
  2. Run shards install

Usage

View the usage in the examples directory.

Contributing

  1. Fork it (https://github.com/squirm-framework/squirm/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors