ELK
===


Elasticsearch - dokumentowa baza danych indeksującą. 

https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

Logstash - AWS na sterydach

https://www.elastic.co/downloads/logstash

Kibana - gui

https://www.elastic.co/downloads/kibana

Instalacja logstash
-------------------

Na maszynie 1: Logstash jak powyżej z pakietów zainstalować. 


.. code-block:: bash
  /opt/logstash/bin/logstash -f test.conf 

.. code-block:: json

  input { stdin {} }
  output { stdout { codec => rubydebug } }

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

.. code-block:: json

  input { stdin {} }
  filter { grok { match => { "message" => "%{IP:ipek} %{URI:uri} %{GREEDYDATA:mesg}" }}}
  output { stdout { codec => rubydebug } }

.. code-block:: json 

  input {
    file {
      type => "apache"
      path => "/home/ubuntu/access.log"
      start_position => "beginning"
    }
  }

  filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }

  output {
    stdout {
      codec => rubydebug
    }
  }



Aby przeparsować plik na nowo, `rm .since*`.


Instalacja elasticsearch
------------------------

Na maszynie 2:

.. code-block:: bash

  wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
  sudo apt-get update && sudo apt-get install elasticsearch default-jre
  wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
  tar xvf kibana-4.3.1-linux-x64.tar.gz
  cd kibana-4.3.1-linux-x64
  ./bin/kibana 


vim /etc/elasticsearch/elasticsearch.xml
.. code-block:: xml
  
   network.host: 0.0.0.0 
   
   

Połączenie
----------

Na logstash, config produkcyjny: w `/etc/logstash/conf.d/costam.conf`. 


.. code-block:: json 

    path => 
   ... 

   output {
    elasticsearch {
        hosts => ["X.x.x.X"]
    }
   
   ...


Restart.  
Kibana. 

Geoip
-----

.. code-block:: json 

  cd /etc/logstash
  sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"


.. code-block:: json 

    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
		}
   mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }