A simple command-line Sitemap generator tool. Useful for quickly auditing a website.
Distributed as a Ruby Gem [https://rubygems.org/gems/sitemap-generator], it is not intended to be a Search Engine sitemap or integrated CMS/Rails/etc. - there are plenty of other gems that do that well.
gem install sitemap-generator
The following command will generate a basic sitemap, listing all links recursively from the site, containing only URIs from the specified domain name (in this case, onegeek.com.au) and will save to a file named sitemap.csv
sitemap generate http://www.onegeek.com.au/ sitemap.csv
This command deliberately doesn't write to file in order to allow unix-style pipelining
sitemap generate --format=json http://www.onegeek.com.au/
sitemap generate --depth=3 http://www.onegeek.com.au/ sitemap.csv
sitemap generate --no-recursion http://www.onegeek.com.au/ sitemap.csv
By default, URI fragments like foo.com/#!/some-page and query strings like foo.com/?bar=baz are ignored - they are generally duplicitous so sitemap-generator strips them off entirely. This lets them back in:
sitemap generate --query-strings --fragments http://www.onegeek.com.au/ sitemap.csv
sitemap
sitemap generate --help
So of course, after spending a few hours writing this I forgot that wget can do this for you, well basically anyway:
wget -r --delete-after <todo>
FAQs
Unknown package
We found that sitemap-generator demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Socket Firewall Enterprise is now available with flexible deployment, configurable policies, and expanded language support.
Security News
Open source dashboard CNAPulse tracks CVE Numbering Authorities’ publishing activity, highlighting trends and transparency across the CVE ecosystem.
Product
Detect malware, unsafe data flows, and license issues in GitHub Actions with Socket’s new workflow scanning support.