Security News
Cloudflare Adds Security.txt Setup Wizard
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
robots-txt-parse
Advanced tools
Streaming robots.txt parser
var parse = require('robots-txt-parse'),
fs = require('fs');
parse(fs.createReadStream(__dirname + '/robots.txt'))
.then(function (robots) {
console.log(robots)
});
assuming this file
user-agent: *
user-agent: googlebot
disallow: /
user-agent: twitterbot
disallow: /
allow: /twitter
Sitemap: http://www.example.com/sitemap.xml
produces following output
{
"groups": [{
"agents": [ "*", "googlebot" ],
"rules": [
{ "rule": "disallow", "path": "/" }
]
}, {
"agents": [ "twitterbot" ],
"rules": [
{ "rule": "disallow", "path": "/" },
{ "rule": "allow", "path": "/twitter" }
]
}],
"extensions": [
{ "extension": "sitemap", "value": "http://www.example.com/sitemap.xml" }
]
}
FAQs
Streaming parser for robots.txt files
The npm package robots-txt-parse receives a total of 7,022 weekly downloads. As such, robots-txt-parse popularity was classified as popular.
We found that robots-txt-parse demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 4 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
Security News
The Socket Research team breaks down a malicious npm package targeting the legitimate DOMPurify library. It uses obfuscated code to hide that it is exfiltrating browser and crypto wallet data.
Security News
ENISA’s 2024 report highlights the EU’s top cybersecurity threats, including rising DDoS attacks, ransomware, supply chain vulnerabilities, and weaponized AI.