ELK¶
Elasticsearch - dokumentowa baza danych indeksującą.
https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
Logstash - AWS na sterydach
https://www.elastic.co/downloads/logstash
Kibana - gui
https://www.elastic.co/downloads/kibana
Instalacja logstash¶
Na maszynie 1: Logstash jak powyżej z pakietów zainstalować.
input { stdin {} }
output { stdout { codec => rubydebug } }
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
input { stdin {} }
filter { grok { match => { "message" => "%{IP:ipek} %{URI:uri} %{GREEDYDATA:mesg}" }}}
output { stdout { codec => rubydebug } }
input {
file {
type => "apache"
path => "/home/ubuntu/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output {
stdout {
codec => rubydebug
}
}
Aby przeparsować plik na nowo, rm .since*.
Instalacja elasticsearch¶
Na maszynie 2:
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update && sudo apt-get install elasticsearch default-jre
wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
tar xvf kibana-4.3.1-linux-x64.tar.gz
cd kibana-4.3.1-linux-x64
./bin/kibana
vim /etc/elasticsearch/elasticsearch.xml .. code-block:: xml
network.host: 0.0.0.0
Połączenie¶
Na logstash, config produkcyjny: w /etc/logstash/conf.d/costam.conf.
path =>
...
output {
elasticsearch {
hosts => ["X.x.x.X"]
}
...
Restart. Kibana.
Geoip¶
cd /etc/logstash
sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}