Jul 23, 2016

Elastic Stack: Process IIS Logs


In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance
  • IIS Average time-taken: shows overall site performance/response time
  • IIS Requests over Time: shows site load
  • IIS Average time-taken per site: shows site performance/response time per cs-host
  • IIS Average time-taken per server: shows site performance/response time per s-computer
  • IIS Response Codes: 200, 301, 403, etc.
More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.

Some abbreviations:
  • Logstash: LS
  • Elasticsearch: ES
  • Kibana: KB
If you are new to Elastic Stack, you should start with this.


Let's start by looking the following diagram:
IIS Log Processing Diagram
There are many tools to read and forward logs in real time, but I prefer nxlog  for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.

Jul 17, 2016

Logstash Config: Check if a field exists or not

There are times when we want to check if a field exists or not before performing an action.

To check if a field named field_1 exists

if [field_1] {
    mutate {}
    do something else

To check if a field named field_1 does not exist

if ![field_1] {
    mutate {}
    do something else

Nov 20, 2015

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 4

Part 1: http://sysad.me/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://sysad.me/2015/11/process-netflow-with-nprobe-and_13.html
Part 3: http://sysad.me/2015/11/process-netflow-with-nprobe-and_91.html

Map User Location within ELK stack

Install Sense on Kibana

Before we create GeoIP fields into Elasticsearch (ES), let's install Sense on Kibana so that we have a great UI to interact with Elasticsearch instead of using curl.
Open a Command Prompt and go to
kibana plugin --install elastic/sense
Restart Kibana service and open Kibana.
Sense UI

Nov 14, 2015

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 3

Part 1: http://sysad.me/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://sysad.me/2015/11/process-netflow-with-nprobe-and_13.html

Customize nProbe and Logstash configuration


In the previous part, we have created a basic visualization and a dashboard in Kibana for NetFlow data; however, do we really need all NetFlow fields? NetFlow v9 has more than 50 field types, so it is better if we export only meaningful fields.

As Logstash received NetFlow fields from nProbe, we can configure it to process those data and add more fields or tags to analyze our network traffic better

Configure nProbe to export only significant NetFlow fields

We can run nProbe with option -T followed by a template to export only the fields we are interested in. The following template is a good start
Note: %IN_SRC_MAC %OUT_DST_MAC only shows data when we run nProbe with a mirrored port.
We can start nProbe with a template by running
nprobe.exe /c -b 1 -V 9 --collector-port 2055 -i none -n none --json-label --tcp -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %SRC_MASK %DST_MASK %IN_SRC_MAC %OUT_DST_MAC %L7_PROTO_NAME %PROTOCOL_MAP %PROTOCOL"
We should now see those fields in Kibana Discover
New NetFlow fields

Nov 13, 2015

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 2

Send NetFlow data to ELK, create searches, visualizations, and dashboards in Kibana

If you haven't read part 1, please visit http://sysad.me/2015/11/process-netflow-with-nprobe-and.html

In this part, we will try to create a nice dashboard in Kibana as below

A view of traffic
<br />

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1

Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2


Source: https://en.wikipedia.org/wiki/NetFlow
By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:
  • Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.
  • Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.
  • Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.
In this tutorial, we will use:
  • NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
  • nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
  • Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data
    System Diagram
    The diagram above shows how Netflow data are processed.

    A simple network diagram is created for this tutorial
    Network Diagram
    ELK and nProbe will be installed on, and sample NetFlow data will be generated from

    Let's start by setting up an ELK stack on Windows Server 2012 R2