Recluse

Recluse is a web crawler meant to ease quality assurance. Currently, it has three crawling tests:

• Status—checks the HTTP status codes of links on the site. Good for detecting broken links.
• Find—finds pages with links matching the pattern. Good for ensuring that references to a page are removed or renamed.
• Assert—checks pages for the existence of HTML elements. Good for asserting that things are consistent across pages.

Add this line to your application's Gemfile:

gem 'recluse'

And then execute:

$ bundle

Or install it yourself as:

$ gem install recluse

Recluse depends on creating profiles for your sites. This way, the configuration can be reusable for frequent quality assurance checks. Profiles are saved as YAML files (.yaml) in ~/.recluse/ and have the following format:

---
name: profile_name
roots:
- http://example.com/
- http://anotherroot.biz/subdir
email: email@domain.com
blacklist:
- http://example.com/dontgohere/*
whitelist:
- http://example.com/dontgohere/unlessitshere/*
internal_only: false
scheme_squash: false
redirect: false

After installation, the recluse executable should be available for your command line.

Spiders through the profile and reports the HTTP status codes of the links. If the profile is not internal only, external links will also have their statuses checked.

$ recluse status csv_path profile1 [profile2] ... [options]

Status code,URL,On page,With error

Spiders through the profiles and checks if a link matching one of the provided patterns is found. Will only go over internal pages.

$ recluse find csv_path profile1 [profile2] ... --globs pattern1 [pattern2] ... [options]

Matching URLs,Pages

Page,Matching URLs

Asserts the existence of an HTML element using CSS-style selectors. Will only check internal pages.

$ recluse assert csv_path profile1 [profile2] ... --exists selector1 [selector2] ...

Selector,Exists,On page

Path where the profiles are stored for manual edits.

$ recluse where

Create a profile.

$ recluse profile create [options] name email root1 [root2] ...

For further description of the arguments, check the Profile options section.

Edit profile options. Any option not provided will stay as it was.

$ recluse profile edit name [options]

More powerful blacklist and whitelist editing. All examples are interchangeable between the three list types. However, if the profile has no roots, it will not run.

Add patterns/roots to the profile's list.

$ recluse profile blacklist add name new_thing1 [new_thing2] ...

Remove patterns/roots from the profile's list.

$ recluse profile blacklist remove name thing1 [thing2] ...

Remove all patterns/roots from the profile's list.

$ recluse profile blacklist clear name

List the patterns/roots in the profile's list.

$ recluse profile blacklist list name

Delete a profile.

$ recluse profile remove name

Rename a profile.

$ recluse profile rename old_name new_name

List all profiles.

$ recluse profile list

List the YAML info of the profile.

$ recluse profile info name

Bug reports and pull requests are welcome on GitHub.

Recluse is modular so you can add tasks if you want. Below is an example of adding your own task to Recluse.

require 'recluse'

module MyModule
  ##
  # Create a task object
  class MyTask < Recluse::Tasks::Task
    ##
    # First argument must be the profile. The rest are hash arguments specific for the task.
    def initialize(profile, option1: false, option2: true, results: nil)
      # Sets up everything based on the profile, queue-specific options, and can also prepopulate results.
      super(profile, queue_options, results: results)
      @queue.run_if do |link|
      	# Run a link if this function returns true.
      	# Link is a Recluse::Link object.
      end
      @queue.on_complete do |link, response|
        # Run this function after the page has either successfully been retrieved, or failed to be retrieved.
        # Link is a Recluse::Link object.
        # Response is a Recluse::Response object.
      end
    end
  end
end

# Add your task to the task list under the key 'my_task'.
Recluse::Tasks.add_task(:my_task, MyModule::MyTask)

# You can now access 'my_task' like you would the default Recluse tasks.
my_profile = Recluse::Profile.load('my_profile')
my_profile.test(:my_task, option1: true, option2: true)
results = my_profile.results[:my_task]

The gem is available as open source under the terms of the MIT License.