com.codemettle:udp-stream-util_2.12
Advanced tools
A fork of wasted/netflow to provide netflow parsing
as a library on top of akka-stream.
The original code was a standalone service using Netty. Since so much code was dependent on
netty-buffer, it's still a dependency, but the network feature is now built on
Alpakka/UDP.
Include in build:
libraryDependencies += "com.codemettle" %% "netflow-stream-lib" % "(version)"
Get a flow that parses tuples of InetSocketAddresses (representing the netflow source) and
akka.util.ByteStrings as NetflowPackets:
package io.netflow {
def netflowParser[In, Mat](
incomingPackets: Flow[In, (InetSocketAddress, ByteString), Mat],
v9TemplateDAO: NetFlowV9TemplateDAO
)(implicit system: ActorSystem): Flow[In, FlowPacket, Mat]
}
Include in build:
libraryDependencies += "com.codemettle" %% "netflow-receiver" % "(version)"
Start a listener bound to the given interface / port which parses incoming UDP netflow packets and passes them along to the given flow:
import java.net.InetSocketAddress
import io.netflow.lib.FlowPacket
import io.netflow.NetFlowV9TemplateDAO
import com.codemettle.netflow.NetflowReceiver
import com.codemettle.streamutil.IngestingResult
implicit val system: ActorSystem = ???
val bindAddress: InetSocketAddress = ???
val packetHandler: Flow[FlowPacket, _, _] = ???
val templateDAO: NetFlowV9TemplateDAO = ???
val resultF: Future[IngestingResult] = NetflowReceiver(bindAddress, packetHandler, templateDAO)
// IngestingResult is one of:
// case object BindFailure
// case class OtherFailure(error: Throwable)
// case class Ingesting(boundTo: InetSocketAddress, ks: KillSwitch)
//
// Assuming everything was successful, the Ingesting.boundTo address will inform the caller
// of what port was bound if a random (0) port was specified, and Ingesting.ks allows the
// stream to be terminated.
=======
This project aims to provide an extensible flow collector written in Scala, using Netty as IO Framework. It is actively being developed by wasted.io, so don't forget to follow us on Twitter. :)
Databases supported:
If we get a pull-request, we won't refuse it. ;)
Can both be found in the Issues section up top.
./sbt compile
./sbt run
First, setup Redis or Cassandra in the configuration file. After, start netflow to create the keyspace and required tables.
Go inside the project's directory and run the sbt command:
./sbt ~run
If you think it's ready for deployment, you can make yourself a .jar-file by running:
./sbt assembly
There are paramters for the configuration file and the logback config. To run the application, try something like ths:
java -server \
-XX:+AggressiveOpts \
-Dconfig.file=application.conf \
-Dlogback.configurationFile=logback.xml \
-Dio.netty.epollBugWorkaround=true \ # only useful on Linux as it is bugged
-jar netflow.jar
A more optimized version can be found in the run shellscript.
We are open to suggestions for some more optimal JVM parameters. Please consider opening a pull request if you think you've got an optimization.
Once it has successfully started, you can start adding authorized senders using the HTTP REST API. To do this, you need to use the admin.authKey and admin.signKey provided by the config. In case you haven't configured them, netflow.io will generate random-keys on every start. The authKey is like a public-key, the signKey more like a private key, so keep it safe.
In order to authenticate against the API, you have 2 possibilities:
To generate the SHA256 HMAC on your console:
echo -n "<authKey>" | openssl dgst -sha256 -hmac "<signKey>"
It's easiest to use like this:
export NF_AUTH_KEY='<authKey>'
export NF_SIGN_KEY='<signKey>'
export NF_SIGNED_KEY=$( echo -n "${NF_AUTH_KEY}" | openssl dgst -sha256 -hmac "${NF_SIGN_KEY}" )
This will setup the NetFlow sender 172.16.1.1 which is monitoring 172.16.1.0/24 and 10.0.0.0/24
curl -X PUT -d '{"prefixes":[{"prefix":"172.16.1.0","prefixLen":24}]}' \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders/172.16.1.1
Of course we also support IPv6!
curl -X PUT -d '{"prefixes":[{"prefix":"2001:db8::","prefixLen":32}]}' \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders/172.16.1.1
Please make sure to always use the first Address of the prefix (being 0 or whatever matches your lowest bit).
To remove a subnet from the sender, just issue a DELETE instead of PUT with the subnet you want to delete from this sender. This also works with multiples, just like PUT.
curl -X DELETE -d '{"prefixes":[{"prefix":"2001:db8::","prefixLen":32}]}' \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders/172.16.1.1
To remove a whole sender, just issue a DELETE without any subnet.
curl -X DELETE \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders/172.16.1.1
For a list of all configured senders
curl -X GET \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders
For information about a specific sender
curl -X GET \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/senders/172.16.1.1
Querying for statistics (more to come here)
curl -X POST -d '{
"2001:db8::/32":{
"years":["2013", "2014"],
"months":["2013-01", "2013-02"],
"days":["2014-02-02"],
"hours":["2014-02-02 05"],
"minutes":[
"2014-02-02 05:00",
"2014-02-02 05:01",
"2014-02-02 05:02",
"2014-02-02 05:03",
"2014-02-02 05:04"
]
}}' \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/stats/172.16.1.1
Querying for more detailed statistics (about dst AS 34868 and src AS 34868)
Tracked fields:
curl -X POST -d '{
"2001:db8::/32":{
"years":["2013", "2014"],
"months":["2013-01", "2013-02"],
"days":["2014-02-02"],
"hours":["2014-02-02 05"],
"minutes":[
{"2014-02-02 05:00": ["all", "dstas:34868", "srcas:34868"]},
{"2014-02-02 05:01": ["all", "dstas:34868", "srcas:34868"]},
{"2014-02-02 05:02": ["all", "dstas:34868", "srcas:34868"]},
{"2014-02-02 05:03": ["all", "dstas:34868", "srcas:34868"]},
{"2014-02-02 05:04": ["all", "dstas:34868", "srcas:34868"]}
]
}}' \
-H "X-Io-Auth: ${NF_AUTH_KEY}" \
-H "X-Io-Sign: ${NF_SIGNED_KEY}" \
-v http://127.0.0.1:8080/stats/172.16.1.1
NetFlow v9 and v10 (IPFIX) consist of two packet types, FlowSet Templates and DataFlows. Templates are defined on a per-router/exporter basis so each has their own. In order to work through DataFlows, you need to have received the Template first to make sense of the data. The issue is usually that your exporter might need a few minutes (10-60) to send you the according Template. If you use IPv4 and IPv6 (NetFlow v9 or IPFIX), the router is likely to send you templates for both protocols. If you want to know more about disecting NetFlow v9, be sure to check out RFC3954.
Basically the same as above, while your collector was down, your sender/exporter might have updated its template. If that happens, your netflow.io misses the update and cannot parse current packets. You will have to wait until the next template arrives.
We encourage everyone to use FreeBSD ng_netflow or OpenBSD pflow (which is a little bit broken in regards to exporting AS-numbers which are in the kernel through OpenBGPd). We advice against all pcap based exporters and collectors since they tend to drop long-living connections (like WebSockets) which exceed ~10 minutes in time.
Our suggestion would be to check your Switch's capability for port mirroring.
Mirror your upstream port to a FreeBSD machine which does the actual NetFlow collection and exporting.
This is also beneficial since the NetFlow collection does not impact your router's performance.
Not yet, but we are heavily developing towards our first stable public release!
We had it implemented into two classes before (LegacyFlow and TemplateFlow), but we were unhappy with "per-flow-version" debugging. We believe that handling each flow separately gives us more maintainability in the end than having lots of dispatching in between.
First rule for debugging: use tcpdump to verify you are receiving flows on your server.
If you need a little guidance for using tcpdump, here is what you do as root or with sudo:
# tcpdump -i <your ethernet device> host <your collector ip>
As a working example for Linux:
# tcpdump -i eth0 host 10.0.0.5
If you suspect the UDP Packet coming from a whole network, you can tell tcpdump to filter for it.
You might want to subtitute the default port 2055 with the port your netflow.io collector is running on.
# tcpdump -i eth0 net 10.0.0.0/24 and port 2055
Just grab the source-ip where packets are coming from and add it into the database.
By the way, tcpdump has an awesome manual!
Copyright 2012, 2013, 2014 wasted.io Ltd <really@wasted.io>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
FAQs
udp-stream-util
We found that com.codemettle:udp-stream-util_2.12 demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
An advanced npm supply chain attack is leveraging Ethereum smart contracts for decentralized, persistent malware control, evading traditional defenses.
Security News
Research
Attackers are impersonating Sindre Sorhus on npm with a fake 'chalk-node' package containing a malicious backdoor to compromise developers' projects.
Security News
The npm package for the LottieFiles Player web component was hit with a supply chain attack after a software engineer's npmjs credentials were compromised.