Here we go with more network analysis tool building. Not content with the Wireshark on Centos post that seems quite popular, I’ve decided to look at some NetFlow tools. I looked around for good, well supported and documented open-source software.. And I find it in SiLK.
Silk is the System for internet Level Knowledge, and is designed to support large datasets. It provides tools which allow folks like me to analyse, query and summarise network traffic flows. The NetSA team do this from a security perspective. I do it from a requirement to understand network traffic flows in new environments.
The documentation is phenomenal, with a good howto manual, “Using SiLK for Network Traffic Analysis” as a 171-page colour PDF file. And the Installation Handbook (which I’ve heavily cribbed from) is clear and detailed. It’s built by the Network Situational Awareness (NetSA) team that are part of the US CERT organisation, based out of Carnegie Mellon University.
Preparation
You can find the software yourself at the NetSA SiLK downloads page. I grabbed version 3.16.0, as it was the latest, and I didn’t really want to muck about with the 4.0 beta version. (I’d like my time focused on outputs, and not on debugging). I need NetFlow v9 records, as I’m looking at some IPv6 work using the toolkit as well as IPv4.
Looking through that comprehensive documentation I notice in Section 2.3.2 of the Installation Handbook:
“… Many SiLK applications support collecting, storing, and querying flow records that contain IPv6 addresses. Because of the overhead of storing IPv6 addresses, this capability is disabled by default. To enable this behavior, specify the –enable-ipv6 switch on the configure command line. If you want SiLK to be able to collect IPv6 flow records, you must include support for libfixbuf …”
libfixbuf
We’re going to need some more code.. firstly libfixbuf, but we’re also going to need some other common CentOS libraries. But let’s do libfixbuf first..
Firstly we’ll take that downloaded file, extract it, and run ./configure
Now we’ll move a copy of the download file to ~/rpmbuild/SOURCES and then rpmbuild -ba libfixbuf.spec
Now we need to install these local libraries, and build SiLK, and then add these to the repository that will be on the Internet after this process.
Firstly you’ll need to do an sudo rpm --install ~/rpmbuild/RPMS/x86_64/libfixbuf-devel-1.7.1-1.el7.centos.x86_64.rpm
You’ll need the libfixbuf-devel libraries, as we’re going to be building SiLK which needs access to the headers, as well as the library itself. On final devices, you’ll just need libfixbuf, and not the header information.
Now let’s move on to SiLK
We’ll need to add some libraries for TLS connections between nodes (I’m hoping not to need this, but I’ll build it in case I have paranoid overlords), and a compression library to shrink the filesizes for storage and transfer.
So we’ll need to:
sudo yum install gnutls-devel lzo-devel
Now to build that SiLK stuff, but first, we need to configure it correctly.
Let’s add IPv6 flow record support, enable the output file compression, and set the file compatibility to use a record format that will store IPv4 and IPv6 information.
Configuring SiLK
The configure command below defaults compression to lzox1 now we’ve added the library, and this applies even to the files output raw from the collector. This is good, as I’m expecting lots of records in the final system.
./configure --enable-ipv6 --enable-ipset-compatibility=3.16.0 --enable-output-compression ... * Configured package: SiLK 3.16.0 * Host type: x86_64-pc-linux-gnu * Source files ($top_srcdir): . * Install directory: /usr/local * Root of packed data tree: /data * Packing logic: via run-time plugin * Timezone support: UTC * Default compression method: SK_COMPMETHOD_LZO1X * IPv6 network connections: YES * IPv6 flow record support: YES * IPset file compatibility: SiLK 3.14.0 (record-version=5) * IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lpthread -lgthread-2.0 -lglib-2.0) * NetFlow9 collection support: YES * sFlow collection support: YES * Fixbuf compatibility: libfixbuf-1.7.1 >= 1.7.0 * Transport encryption support: YES (-lgnutls) * IPA support: NO * ZLIB support: YES (-lz) * LZO support: YES (-llzo2) * SNAPPY support: YES (-lsnappy) * LIBPCAP support: YES (-lpcap) * C-ARES support: YES (-lcares) * ADNS support: NO * Python interpreter: /usr/bin/python * Python support: NO * Build analysis tools: YES * Build packing tools: YES * Compiler (CC): gcc * Compiler flags (CFLAGS): -I$(srcdir) -I$(top_builddir)/src/include -I$(top_srcdir)/src/include -DNDEBUG -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -Wall -W -Wmissing-prototypes -Wformat=2 -Wdeclaration-after-statement -Wpointer-arith -fno-strict-aliasing -O3 * Linker flags (LDFLAGS): * Libraries (LIBS): -lsnappy -llzo2 -lz -ldl -lm
Now to create the RPM files (and the supporting repodata directory, as I’m going to publish this too). Make sure you have a copy SiLK downloaded file in ~/rpmbuild/SOURCES
Building SiLK
rpmbuild -ba silk-3.16.0.spec cd ~/rpmbuild/RPMS/x86_64 rpmsign --resign * createrepo --update .
I’ve created a SiLK.repo file, which you can download from the repository directory. You’ll need to add this to your /etc/yum.repos.d/ directory. Now one last command and we’ll get it added to our system.
yum install silk-rwflowpack silk-analysis
Yea, an installed bit of code. Now to get it to accept data.. but that’s a post for later.