ELK

Elasticsearch - dokumentowa baza danych indeksującą.

https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

Logstash - AWS na sterydach

https://www.elastic.co/downloads/logstash

Kibana - gui

https://www.elastic.co/downloads/kibana

Instalacja logstash

Na maszynie 1: Logstash jak powyżej z pakietów zainstalować.

input { stdin {} }
output { stdout { codec => rubydebug } }

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

input { stdin {} }
filter { grok { match => { "message" => "%{IP:ipek} %{URI:uri} %{GREEDYDATA:mesg}" }}}
output { stdout { codec => rubydebug } }

input {
  file {
    type => "apache"
    path => "/home/ubuntu/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Aby przeparsować plik na nowo, rm .since*.

Instalacja elasticsearch

Na maszynie 2:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update && sudo apt-get install elasticsearch default-jre
wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
tar xvf kibana-4.3.1-linux-x64.tar.gz
cd kibana-4.3.1-linux-x64
./bin/kibana

vim /etc/elasticsearch/elasticsearch.xml .. code-block:: xml

network.host: 0.0.0.0

Połączenie

Na logstash, config produkcyjny: w /etc/logstash/conf.d/costam.conf.

 path =>
...

output {
 elasticsearch {
     hosts => ["X.x.x.X"]
 }

...

Restart. Kibana.

Geoip

cd /etc/logstash
sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"

 geoip {
   source => "clientip"
   target => "geoip"
   database => "/etc/logstash/GeoLiteCity.dat"
   add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
   add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
             }
mutate {
   convert => [ "[geoip][coordinates]", "float"]
 }