Nov 13, 2015

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1


Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2

Overview

Source: https://en.wikipedia.org/wiki/NetFlow
By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:
  • Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.
  • Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.
  • Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.
In this tutorial, we will use:
  • NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
  • nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
  • Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data
    System Diagram
    The diagram above shows how Netflow data are processed.

    A simple network diagram is created for this tutorial
    Network Diagram
    ELK and nProbe will be installed on 192.168.1.50, and sample NetFlow data will be generated from 192.168.1.60.

    Let's start by setting up an ELK stack on Windows Server 2012 R2

    Ref: https://www.ulyaoth.net/resources/tutorial-install-logstash-and-kibana-on-a-windows-server.34/. I eliminate some extra steps to simplify the tutorial.

    Install Prerequisites

    1. Download and install latest JRE from Oracle (currently jre-8u66-windows-x64.exe). You may install either JDK or JRE.
    2. Add JAVA_HOME environment variable. Open System Properties

    Create a new system variable

    Set JRE folder path, change this path if you update JRE later


    Install ELK

    1. Download ELK for Windows from https://www.elastic.co
    2. Create a folder for ELK at C:\ELK, extract all ELK packages
    3. Open Command Prompt and change to elasticsearch\bin folder (Tip: You can use Shift + Right Click on a folder to open Command Prompt at that folder)

    To install Elasticsearch, run
    service install

    If you plan to install more than one Elasticsearch instance on the same server, edit the file elasticsearch\bin\service.bat and change
    set SERVICE_ID=elasticsearch-service-x64
    to
    set SERVICE_ID=elasticsearch-service-x64-<custom-name>
    Open Elasticsearch service manager, start Elasticsearch service
    service manager
    Update path to new JRE if necessary
    Start service

    Open a web browser, go to http://localhost:9200. We should see as the screenshot below


    4. NSSM (http://nssm.cc) is used to install and run both Logstash and Kibana as a Windows service
    Download
    http://nssm.cc/release/nssm-2.24.zip
    Extract and copy file nssm.exe from nssm-2.24\win64 to
    C:\ELK\logstash\bin
    C:\ELK\kibana\bin


    5. To install Kibana, open Command Prompt at C:\ELK\kibana\bin and run
    nssm install kibana

    Locate kibana.bat

    Install service


    Go to http://localhost:5601 to see if Kibana is working

    6. To install Logstash, 
    Go to C:\ELK\Logstash\bin, create a Logstash config file named logstash.conf and paste the following lines
    input {
    tcp {
    port => 5544
      type => "netflow"
    }
    }

    filter {
    json {
    source => "message"
    }
    }

    output {
    stdout { 
    codec => rubydebug 
    }
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "netflow-%{+YYYY.MM.dd}"
    }
    }


    Create a run.bat file in the same folder and paste:
    logstash.bat agent -f logstash.conf
    Open Command Prompt at C:\ELK\logstash\bin and run
    nssm install logstash

    Locate run.bat



    Start service
    Check if Logstash is listening on port 5544

    Summary

    Our ELK stack is ready to receive NetFlow and other data. In the next part, we will install nProbe, generate NetFlow data, and play with Kibana.
    Part 2: Send NetFlow data to ELK, create searches, visualizations, and dashboards in Kibana