dynamic_sitemaps

DynamicSitemaps

Dynamic Sitemaps is a plugin for Ruby on Rails that enables you to easily create flexible, dynamic sitemaps. It creates sitemaps in the sitemaps.org standard which is supported by several crawlers including Google, Bing, and Yahoo.

Dynamic Sitemaps is designed to be very (very) simple so there's a lot you cannot do, but possibly don't need (I didn't). If you need an advanced sitemap generator, please see Karl Varga's SitemapGenerator.

Version 2.0

Version 2.0 makes it possible to make very large sitemaps (up to 2.5 billion URLs) in a fast and memory efficient way; it is built for large amounts of data, i.e. millions of URLs without pushing your server to the limit, memory and CPU wise.

Version 2.0 is not compatible with version 1.0 (although the configuration DSL looks somewhat the same) as version 2.0 generates static sitemap XML files whereas 1.0 generated them dynamically on each request (slow for large sitemaps).

Requirements

DynamicSitemaps is tested in Rails 3.2.13 using Ruby 1.9.3 and 2.0.0, but should work in other versions of Rails 3 and above and Ruby 1.9 and above. Please create an issue if you encounter any problems.

Installation

Add this line to your application's Gemfile:

gem "dynamic_sitemaps", "2.0.0.beta2"

And then execute:

$ bundle

Or install it yourself as:

$ gem install dynamic_sitemaps

To generate a simple example config file in config/sitemap.rb:

$ rails generate dynamic_sitemaps:install

If you want to use version 1.0 (v1.0.8) of DynamicSitemaps, please see v1.0.8 of the project. Please note that this version isn't good for large sitemaps as it generates them dynamically on each request.

Basic usage

The configuration file in config/sitemap.rb goes like this (also see the production example below for more advance usage like multiple sites / hosts, etc.):

host "www.example.com"

# Basic sitemap – you can change the name :site as you wish
sitemap :site do
  url root_url, last_mod: Time.now, change_freq: "daily", priority: 1.0
end

# Pings search engines after generation has finished
ping_with "http://#{host}/sitemap.xml"

The host is needed to generate the URLs because the rake task doesn't know anything about the host being used.

Then, to generate the sitemap:

$ rake sitemap:generate

This will, by default, generate a sitemap.xml file in <project root>/public/sitemaps that will look like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://www.example.com/</loc>
    <lastmod>2013-07-08T17:02:45+02:00</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
</urlset>

You then need to symlink from public/sitemap.xml (or whatever you choose) to public/sitemaps/sitemap.xml:

$ ln -s /path/to/project/public/sitemaps/sitemap.xml /path/to/project/public/sitemap.xml

See the below production example for inspiration on how to do this with Capistrano, and other things like multiple sites / hosts, etc.

If a sitemap contains over 50,000 URLs, then by default, as specified by the sitemaps.org standard, DynamicSitemaps will split it into multiple sitemaps and generate an index file that will also be named public/sitemaps/sitemap.xml by default. The sitemap files will then be named site.xml, site2.xml, site3.xml, and so on, and the index file will link to these files using the host set with host.

Automatic sitemaps for resourceful routes

DynamicSitemaps can automatically generate sitemaps for ActiveRecord models with the built-in Rails resourceful routes (the ones you create using routes :model_name).

Example:

host "www.example.com"

# Basic sitemap
sitemap :site do
  url root_url, last_mod: Time.now, change_freq: "daily", priority: 1.0
end

# Automatically link to all pages using the routes specified
# using "resources :pages" in config/routes.rb. This will also
# automatically set <lastmod> to the date and time in page.updated_at.
sitemap_for Page.scoped

# For products with special sitemap name and priority, and link to comments
sitemap_for Product.published, name: :published_products do |product|
  url product, last_mod: product.updated_at, priority: (product.featured? ? 1.0 : 0.7)
  url product_comments_url(product)
end

This generates the sitemap files site.xml, pages.xml, and published_products.xml and links them together in the sitemap.xml index file, splitting them into multiple sitemap files if the number of URLs exceeds 50,000.

The argument passed to sitemap_for needs to respond to #find_each, like an ActiveRecord Relation. This is to ensure that the records from the database are lazy loaded 1,000 at a time, so that it doesn't accidentally load millions of records in one call when the configuration file is read. Therefore we use Page.scoped instead of the normal Page.all.

Custom configuration

You can configure different options of how DynamicSitemaps behaves, including the sitemap path and index file name.

In an initializer, e.g. config/initializers/dynamic_sitemaps.rb:

# These are the built-in defaults, so you don't need to specify them.
DynamicSitemaps.configure do |config|
  config.path = Rails.root.join("public")
  config.folder = "sitemaps" # This folder is emptied on each sitemap generation
  config.index_file_name = "sitemap.xml"
  config.always_generate_index = false # Makes sitemap.xml contain the sitemap
                                       # (e.g. site.xml) when only one sitemap
                                       #  file has been generated
  config.config_path = Rails.root.join("config", "sitemap.rb")
  config.per_page = 50000
end

Pinging search engines

DynamicSitemaps can automatically ping Google and Bing (and other search engines you specify) with the sitemap when the generation finishes.

In config/sitemap.rb:

host "www.example.com"

sitemap :site do
  url root_url
end

ping_with "http://#{host}/sitemap.xml"

To customize it, in e.g. config/initializers/dynamic_sitemaps.rb:

DynamicSitemaps.configure do |config|
  # Default is Google and Bing
  config.search_engine_ping_urls << "http://customsearchengine.com/ping?url=%s"

  # Default is pinging only in production
  config.ping_environments << "staging"
end

In case of failure

DynamicSitemaps generates to a temporary directory (<rails root>/tmp/dynamic_sitemaps) first and, when finished, moves the files into the destination (by default public/sitemaps). So in case you have generated a sitemap succesfully and the next sitemap generation fails, your sitemap files will remain untouched and available.

Production example with multiple domains, Capistrano, and Whenever

This is an example of a real production app that uses DynamicSitemaps with multiple sites and domains in one app, Capistrano for deployment, and Whenever for crontab scheduling.

Sitemap setup

In config/sitemap.rb:

Site.all.each do |site|
  folder "sitemaps/#{site.key}"
  host site.domain

  sitemap :site do
    url root_url, priority: 1.0, change_freq: "daily"
    url blog_posts_url
    url tags_url
  end

  sitemap_for site.pages.where("slug != 'home'")
  sitemap_for site.blog_posts.published
  sitemap_for site.tags.scoped

  sitemap_for site.products.where("type_id != ?", ProductType.find_by_key("unknown").id) do |product|
    url product, last_mod: product.updated_at, priority: (product.featured? ? 1.0 : 0.7)
  end

  ping_with "http://#{host}/sitemap.xml"
end

Routing the default sitemap

Route for sitemap.xml and robots.txt

In config/routes.rb:

get "sitemap.xml" => "home#sitemap", format: :xml, as: :sitemap
get "robots.txt" => "home#robots", format: :text, as: :robots

Controller

In app/controllers/home_controller.rb:

class HomeController < ApplicationController
  # ...

  def sitemap
    path = Rails.root.join("public", "sitemaps", current_site.key, "sitemap.xml")
    if File.exists?(path)
      render xml: open(path).read
    else
      render text: "Sitemap not found.", status: :not_found
    end
  end
  
  def robots
  end
end

View for robots.txt

In app/views/home/robots.text.erb:

Sitemap: <%= sitemap_url %>

Deployment with Capistrano

Capistrano deployment configuration in config/deploy.rb:

after "deploy:update_code", "sitemaps:create_symlink"

namespace :sitemaps do
  task :create_symlink, roles: :app do
    run "mkdir -p #{shared_path}/sitemaps"
    run "rm -rf #{release_path}/public/sitemaps"
    run "ln -s #{shared_path}/sitemaps #{release_path}/public/sitemaps"
  end
end

For automatic crontab scheduling with Whenever, in config/schedule.rb:

every 1.day, at: "6am" do
  rake "sitemap:generate"
end

This will automatically generate the sitemaps and ping Google and Bing every day at 6am using the sitemap URLs configured above.

Problems?

If you encounter any problems with DynamicSitemaps, please create an issue. If you want to fix the problem (please do :smile:), please see below.

Contributing

Help is always appreciated whether it be improvement of the code, testing, or adding new relevant features. Please create an issue before implementing a new feature, so we can discuss it in advance. Thanks.

1. Fork the repo
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request