ELK Stack

5 Notes
+ beats (May 19, 2019, 7:35 p.m.)

This input plugin enables Logstash to receive events from the Elastic Beats framework. The following example shows how to configure Logstash to listen on port 5044 for incoming Beats connections and to index into Elasticsearch: input { beats { port => 5044 } } output { elasticsearch { hosts => "localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } }

+ Difference between Logstash and Beats (May 19, 2019, 7:31 p.m.)

Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Beats have a small footprint and use fewer system resources than Logstash. Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.

+ Elasticsearch cat APIs (April 21, 2019, 11:54 p.m.)

To check the cluster health, we will be using the _cat API. cat APIs JSON is great… for computers. Even if it’s pretty-printed, trying to find relationships in the data is tedious. Human eyes, especially when looking at a terminal, need compact and aligned text. The cat API aims to meet this need. ------------------------------------------------------------- curl '127.0.0.1:9200/_cat/master?v' _cat/master?help ------------------------------------------------------------- List All Indices: curl '127.0.0.1:9200/_cat/indices?v' -------------------------------------------------------------

+ Installation (April 19, 2019, 8:55 p.m.)

apt install openjdk-8-jdk apt-transport-https curl nginx libpcre3-dev ---------------------------------------------------------------------- Elasticsearch ----------------- 1- wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 2- echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list 3- apt update 4- apt install elasticsearch 5- Uncomment the following options from the file "/etc/elasticsearch/elasticsearch.yml" network.host: localhost http.port: 9200 6- systemctl restart elasticsearch systemctl enable elasticsearch 7- Check the status of the elasticsearch server: (Its server takes time to start listening.) curl -X GET http://localhost:9200 ---------------------------------------------------------------------- Kibana --------- 1- apt install kibana 2- systemctl enable kibana 3- echo "admin:$(openssl passwd -apr1 my_password)" | sudo tee -a /etc/nginx/htpasswd.kibana 4- vim /etc/nginx/sites-enabled/kibana server { listen 80; server_name logs.mhass.ir logs.mohsenhassani.com; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.kibana; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } 5- systemctl restart nginx ---------------------------------------------------------------------- Logstash ----------- 1- apt install logstash 2- Create a logstash filter config file in "/etc/logstash/conf.d/logstash.conf", with this content: input { tcp { port => 4300 # optional port number codec => json } } filter { } output { elasticsearch { } stdout { } # or stdout {codec => json} in case you want to see the data in logs for debugging } 3- Restart logstash services: systemctl restart logstash systemctl enable logstash ---------------------------------------------------------------------- For debugging: tcpdump -nti any port 4300 tail -f /var/log/syslog tail -f /var/log/logstash/logstash*.log ----------------------------------------------------------------------

+ Introduction / Definitions (April 19, 2019, 8:54 p.m.)

First Underlying Layer: Logstash + Beats Upper Layer: Elasticsearch Upper Layer: Kibabana ------------------------------------------------------ "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. ------------------------------------------------------ Elasticsearch is a distributed, RESTful search and analytics NoSQL engine based on Lucene. Logstash is a light-weight data processing pipeline for managing events and logs from a wide variety of sources. Kibana is a web application for visualizing data that works on top of Elasticsearch. ------------------------------------------------------ The Elastic Stack is the next evolution of the ELK Stack. ------------------------------------------------------