
Security News
Open Source CAI Framework Handles Pen Testing Tasks up to 3,600× Faster Than Humans
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
Floodesh is middleware based web spider written with Nodejs. "Floodesh" is a combination of two words, flood
and mesh
.
Make sure g++
, make
, libboost-all-dev
, gperf
, libevent-dev
and uuid-dev
have been installed.
$ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz | tar xvf
$ cd gearmand-1.1.12
$ ./configure
$ make
$ make install
$ npm install -g floodesh-cli
Generate new app from templates by only one command.
$ mkdir demo
$ cd demo
$ floodesh-cli init # all necessary files will be generated in your directory.
Please make sure you have /data/tests and /var/log/bda/tests created and have Write access before use, you can customize path by modifying logBaseDir in config/[env]/index.js
A context instance is a kind of Finite-State Machine implemented by Generators
which is ECMAScript 6 feature. By context, we can access almost all fields in response
and request
, like:
worker.use( (ctx,next) => {
ctx.content = ctx.body.toString(); // totally do not care about the body
return next();
})
Get querystring.
Check if the request is idempotent.
Get the search string. It includes the leading "?" compare to querystring.
Get request method.
Get parsed query-string.
Get the request pathname
Return request url, the same as ctx.href.
Get the origin of URL, for instance, "https://www.google.com".
Return the protocol string "http:" or "https:".
Parse the "Host" header field host and support X-Forwarded-Host when a proxy is enabled.
Parse the "Host" header field hostname and support X-Forwarded-Host when a proxy is enabled.
Check if protocol is https.
Get status code from response.
Get status message from response.
Get the response body in Buffer.
Get length of response body.
Get the response mime type, for instance, "text/html"
Get the Last-Modified date in Date form, if it exists.
Get the ETag of a response.
Return the response header.
key
<String>Get value by key in response headers
type
s <String>|Array>Check if the incoming response contains the "Content-Type" header field, and it contains any of the give mime type
s.If there is no response body, null
is returned.If there is no content type, false
is returned.Otherwise, it returns the first type
that matches.
Array of generated tasks. A task is an object consists of Options and next
, next
is a function name in your spider you want to call in next task , Supported format:
[{
opt:<Options>,
next:<String>
}]
A map to store result, that will be parsed and saved by floodesh.
retry
<Integer>: Retry times at worker side, default 3
logBaseDir
<String>: Directory where project's log directory exists, default '/var/log/bda/'parsers
<Array>: Array of parsers, which are file names in parser directory without '.js'defaultCfg
<Object>
rate
<Integer>: Number of milliseconds to delay between each requestsconcurrent
<Integer>: Size of the worker poolpriorityRange
<Integer>: Range of acceptable priorities starting from 0, default 3
defaultPriority
<Integer>: priority of the requesthomogenous
<Boolean>:trueheaders
<Object>: HTTP headersjobs
<Integer>: Max number of jobs per worker, default 1
srvQueueSize
<Integer>: Max number of jobs queued to gearman server, default 1000
mongodb
<String>: Mongodb Connection String URI,worker
<Object>:
servers
<Array>: Array of server list, server should be an object like {'host':'gearman-server'}
client
<Object>:
servers
<Array>: Same as above,loadBalancing
<String>: 'RoundRobin'retry
<Integer>: Retry times at client sidemongodb
<String>: Mongodb Connection String URIrepo
<String>: [redis|mongodb] default use memory as repo.removeKeys
<Array>:Array of keys in query string to skip when test if an url is seenserver
<String>: Remote service originJust throw an Error
in a synced middleware, otherwise return a rejected Promise. err.stack
will be logged and err.code
will be sent to client to persist.
// sync
module.exports = (ctx, next) => {
// balabala
throw new Error('crash here');
}
// async
module.exports = (ctx, next) => {
return new Promise( (resolve, reject) => {
// balabala
reject(new Error('got error'));
});
}
Cheerio
.Charset
in response headers.iconv
or iconv-lite
.Request.js
, with some default options.bottleneckp
which is asynchronous rate limiter with priority.whacko
, which is a fork of cheerio that uses parse5 as an underlying platform.statsd-client
, which enables you send metrics to a statsd daemon.User-Agent
header automatically from a local file.seenreq
.<
and json body {
.FAQs
Floodesh is a distributed web spider/crawler written with Nodejs.
The npm package floodesh receives a total of 2 weekly downloads. As such, floodesh popularity was classified as not popular.
We found that floodesh demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
Security News
Deno 2.4 brings back bundling, improves dependency updates and telemetry, and makes the runtime more practical for real-world JavaScript projects.
Security News
CVEForecast.org uses machine learning to project a record-breaking surge in vulnerability disclosures in 2025.