ronin-web
![Gem Version](https://badge.fury.io/rb/ronin-web.svg)
Description
ronin-web is a Ruby library that provides common web security commands and
additional libraries.
ronin-web is part of the ronin-rb project, a Ruby toolkit for security
research and development.
Features
- HTML/XML parsing/building (using Nokogiri).
- Also provides additional extensions to Nokogiri using
[nokogiri-ext].
- Supports diffing HTML/XML documents using nokogiri-diff.
- Supports random
User-Agent
generation using ronin-web-user_agents. - Provides an easy to use Sinatra based web server using
ronin-web-server.
- Provides an easy to use web spider using ronin-web-spider.
- Provides a CLI for common web related tasks.
- Has 98% documentation coverage.
- Has 89% test coverage.
Synopsis
Usage: ronin-web [options] [COMMAND [ARGS...]]
Options:
-h, --help Print help information
Arguments:
[COMMAND] The command name to run
[ARGS ...] Additional arguments for the command
Commands:
completion
diff
help
html
irb
new
reverse-proxy
screenshot
server
session-cookie
spider
user-agent
vulns
wordlist
xml
Open the ronin-web
Ruby REPL:
$ ronin-web irb
Diff two HTML files:
$ ronin-web diff index1.html index2.html
+
+ <div>hax</div>
Diff two URLs:
$ ronin-web diff http://example.com/index.html http://example.com/index2.html
Perform an XPath query on an HTML file:
$ ronin-web html --xpath //meta index.html
<meta charset="utf-8">
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
Perform an XPath query on a URL:
$ ronin-web html --xpath //meta https://example.com/
<meta charset="utf-8">
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
Dump all links from a web page:
$ ronin-web html --links https://www.google.com/
https://www.google.com/imghp?hl=en&tab=wi
https://maps.google.com/maps?hl=en&tab=wl
https://play.google.com/?hl=en&tab=w8
https://www.youtube.com/?tab=w1
https://news.google.com/?tab=wn
https://mail.google.com/mail/?tab=wm
https://drive.google.com/?tab=wo
https://www.google.com/intl/en/about/products?tab=wh
http://www.google.com/history/optout?hl=en
/preferences?hl=en
https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ
/advanced_search?hl=en&authuser=0
https://www.google.com/url?q=https://blog.google/products/search/google-search-new-fact-checking-misinformation/&source=hpp&id=19034203&ct=3&usg=AOvVaw3UxG35a-5UX1Rl8M_VwPbd&sa=X&ved=0ahUKEwjM4Iq--JD-AhVtGTQIHXMBBaYQ8IcBCAU
/intl/en/ads/
/services/
/intl/en/about.html
/intl/en/policies/privacy/
/intl/en/policies/terms/
Spiders a host and print all visited URLs:
$ ronin-web spider --host www.ruby-lang.org
http://www.ruby-lang.org/
http://www.ruby-lang.org/en/
http://www.ruby-lang.org/en/downloads/
http://www.ruby-lang.org/en/documentation/
http://www.ruby-lang.org/en/libraries/
http://www.ruby-lang.org/en/community/
https://www.ruby-lang.org/en/news/
https://www.ruby-lang.org/en/security/
https://www.ruby-lang.org/en/about/
...
Spiders the domain and sub-domains and print every visited URL:
$ ronin-web spider --domain ruby-lang.org
http://ruby-lang.org/
https://www.ruby-lang.org/
https://www.ruby-lang.org/en/
https://www.ruby-lang.org/en/downloads/
https://www.ruby-lang.org/en/documentation/
https://www.ruby-lang.org/en/libraries/
https://www.ruby-lang.org/en/community/
https://www.ruby-lang.org/en/news/
https://www.ruby-lang.org/en/security/
https://www.ruby-lang.org/en/about/
...
Spiders a specific web-site and print every visited URL:
$ ronin-web spider --site https://www.ruby-lang.org/
https://www.ruby-lang.org/
https://www.ruby-lang.org/en/
https://www.ruby-lang.org/en/downloads/
https://www.ruby-lang.org/en/documentation/
https://www.ruby-lang.org/en/libraries/
https://www.ruby-lang.org/en/community/
https://www.ruby-lang.org/en/news/
https://www.ruby-lang.org/en/security/
https://www.ruby-lang.org/en/about/
...
Spider a host and print the response statuses:
200 http://www.ruby-lang.org/
200 http://www.ruby-lang.org/en/
200 http://www.ruby-lang.org/en/downloads/
200 http://www.ruby-lang.org/en/documentation/
200 http://www.ruby-lang.org/en/libraries/
200 http://www.ruby-lang.org/en/community/
200 http://www.ruby-lang.org/en/news/
200 http://www.ruby-lang.org/en/security/
200 http://www.ruby-lang.org/en/about/
...
Spider a host and print the response status and headers:
ronin-web spider --print-headers --host www.ruby-lang.org
200 http://www.ruby-lang.org/
Connection: close
Content-Length: 887
Server: Cowboy
Strict-Transport-Security: max-age=31536000
Content-Type: text/html
Etag: W/"496ac7fab29a6094e490da28025c5857"
X-Frame-Options: SAMEORIGIN
Via: 1.1 vegur, 1.1 varnish
Accept-Ranges: bytes
Date: Tue, 04 Apr 2023 19:42:51 GMT
Age: 155
X-Served-By: cache-pdx12330-PDX
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1680637372.808609,VS0,VE1
Vary: Accept-Encoding
...
Start a debug web server on http://localhost:8000/:
$ ronin-web server
[2023-04-04 12:26:59] INFO WEBrick 1.7.0
[2023-04-04 12:26:59] INFO ruby 3.1.3 (2022-11-24) [x86_64-linux]
== Sinatra (v3.0.4) has taken the stage on 8000 for development with backup from WEBrick
[2023-04-04 12:26:59] INFO WEBrick::HTTPServer#start: pid=8966 port=8000
Start a debug web server on a different address and port:
$ ronin-web server --host 0.0.0.0 --port 1337
Host the files in a directory on http://localhost:8000/:
$ ronin-web server --root .
Mount a specific file at a specific HTTP path:
$ ronin-web server --dir /dir/index.html:./index.html
Mount a specific directory at a specific HTTP path:
$ ronin-web server --dir /dir:.
Add a redirect to the web server:
$ ronin-web server --redirect /redirect:https://example.com/
Start a HTTP reverse proxy that rewrites HTTP responses on http://localhost:8080:
$ ronin-web reverse-proxy --rewrite-response Example:Hax
Test the reverse proxy:
$ curl -H "Host: example.com" http://localhost:8080/
Generate a new Ruby script for parsing HTML/XML:
$ ronin-web new nokogiri parse.rb
erb nokogiri.rb.erb parse.rb
chmod parse.rb
Generate a new web spider script:
$ ronin-web new spider --host=www.example.com spider.rb
erb spider.rb.erb spider.rb
chmod spider.rb
Generate a new web server script:
$ ronin-web new server server.rb
erb server.rb.erb server.rb
chmod server.rb
Generate a new web app:
$ ronin-web new app myapp
mkdir myapp
mkdir myapp/lib
mkdir myapp/views
mkdir myapp/public
erb .ruby-version.erb myapp/.ruby-version
cp Gemfile myapp
erb app.rb.erb myapp/app.rb
cp config.ru myapp
Open the Ronin Web Ruby REPL:
$ ronin-web irb
, Jµ ▓▓█▓
J▌ ▐▓██▌ ████ ██ ▐███D
╓▄▓▓█████▌ ██µ ████ ▄███ÖJ██▌ ███▌
,╓µ▄▄▄▄▄▄▄▄µ;, ,▄▓██████████ ▐███ ▐███▀ ███▌ ████µ ▄███
¬∞MÆ▓███████████████████████▓M ▄██████▀▀╙████▌ ████▌ ████ ▄███ J█████ ███▌
`█████▀▀▀▀▀███████ -████▀└ ████ ▐█████n ▄███O ███▌ ██████████
▓████L ████▀ ▓████ ▓███Ö ███████ ███▌ ▓███ ▐█████████▀
▄████▀ ,╓▄▄▄█████ J████Ü ,███▌ ▄███████████ J███▀ ████ █████
J█████████████████─ ████▌ ████ ████`██████▌ ████ ▐███Ü ▐███Ü
███████████▀▀▀╙└ ▐████ J███▌ ▓███▌ ²█████ J███Ü ███▌ ▀█▌
▓██████████▌ ████▌ ████ ;████ ▀███▀ ███▌ J▀▀▀- █
▄█████▀ ▀█████µ ▐████ ,▄▓████▀ ████▀ ███ J███ `
J█████- ╙▀███▄ ████████████▀╙ J█▀▀▀ █U ▀█▌
████▀ ▀███ ▄████████▀▀ ╨ █
▓██▀ ²▀█▄ █▀▀▀╙└
▄██╜ ╙W
J█▀
▌└
┘
irb(ronin-web)>
Examples
HTML
Parse an HTML string:
doc = html_parse("<html>\n <body>\n <p>Hello world</p>\n </body>\n</html>\n")
Parse a HTML file:
doc = html_open("index.html")
Searching an HTML document using XPath or CSS-path:
nodes = doc.search('//div/p')
nodes = doc.search('div p.class')
node = doc.at('#id')
Build a HTML document:
doc = html_build do
html {
head {
script(type: 'text/javascript', src: 'redirect.js')
}
}
end
puts doc.to_html
XML
Parse an XML response body:
xml_parse("<?xml version=\"1.0\"?>\n<users>\n <user>\n <name>admin</name>\n <password>0mni</password>\n <user>\n</users>\n")
Parse a XML file:
doc = html_open("data.xml")
Searching an XML document using XPath:
users = doc.search('//user')
admin = doc.at('//user[@name="admin"]')
Build a XML document:
doc = xml_build do
playlist {
mp3 {
file { text('02 THE WAIT.mp3') }
artist { text('Evil Nine') }
track { text('The Wait feat David Autokratz') }
duration { text('1000000000') }
}
}
end
puts doc.to_xml
Web Requests
Gets a URL and follows any redirects:
get 'https://example.com/'
Gets a URL and parses the HTML response:
get_html 'https://example.com/'
Gets a URL and parses the XML response:
get_xml 'https://example.com/sitemap.xml'
Gets a URL and parses the JSON response:
get_json 'https://example.com/api/endpoint.json'
POSTs to a URL and follows any redirects:
post 'https://example.com/form', form_data: {'foo' => 'bar'}
POSTs to a URL and parses the HTML response:
post_html 'https://example.com/form', form_data: {'foo' => 'bar'}
POSTs to a URL and parses the XML response:
post_xml 'https://example.com/form', form_data: {'foo' => 'bar'}
POSTs to a URL and parses the JSON response:
post_json 'https://example.com/api/endpoint.json', json: {foo: 'bar'}
User Agents
Get a random User-Agent
string:
user_agent = UserAgents.random
For more examples, see ronin-web-user_agents.
Session Cookie
Parse a Django JSON session cookie:
SessionCookie.parse('sessionid=eyJmb28iOiJiYXIifQ:1pQcTx:UufiSnuPIjNs7zOAJS0UpqnyvRt7KET7BVes0I8LYbA')
For more examples, see ronin-web-session_cookie.
Spider
Spider a website and print out visited URLs:
Spider.site('http://www.rubyinside.com/') do |spider|
spider.every_url { |url| puts url }
end
For more examples, see ronin-web-spider.
Browser
Open a visible web browser and intercept all requests:
browser = Ronin::Web::Browser.new(visible: true)
browser.every_request do |request|
puts "> #{request.method} #{request.url}"
end
browser.go_to("https://twitter.com/login")
For more examples, see ronin-web-browser.
Requirements
Install
$ gem install ronin-web
Development
- Fork It!
- Clone It!
cd ronin-web
./scripts/setup
git checkout -b my_feature
- Code It!
bundle exec rake spec
git push origin my_feature
License
ronin-web - A collection of useful web helper methods and commands.
Copyright (c) 2006-2024 Hal Brodigan (postmodern.mod3 at gmail.com)
ronin-web is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
ronin-web is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with ronin-web. If not, see https://www.gnu.org/licenses/.