DevOps Se­ries De­ploy­ing Gray­log Us­ing An­si­ble

This 11th ar­ti­cle in the DevOps se­ries is a tu­to­rial on in­stalling Gray­log soft­ware us­ing An­si­ble.

OpenSource For You - - Contents - By: Shak­thi Kan­nan The au­thor is a free soft­ware en­thu­si­ast and blogs at shak­thi­maan.com.

Gray­log is a free and open source log man­age­ment soft­ware that al­lows you to store and an­a­lyse all your logs from a cen­tral lo­ca­tion. It re­quires Mon­goDB (a doc­u­ment-ori­ented, NoSQL data­base) to store meta in­for­ma­tion and con­fig­u­ra­tion in­for­ma­tion. The ac­tual log mes­sages are stored in Elas­tic­search. It is writ­ten us­ing the Java pro­gram­ming lan­guage and re­leased un­der the GNU Gen­eral Pub­lic Li­cense (GPL) v3.0.

Ac­cess con­trol man­age­ment is built into the soft­ware, and you can create roles and user ac­counts with dif­fer­ent per­mis­sions. If you al­ready have an LDAP server, its user ac­counts can be used with the Gray­log soft­ware. It also pro­vides a REST API, which al­lows you to fetch data to build your own dash­boards. You can create alerts to take ac­tions based on the log mes­sages, and also forward the log data to other out­put streams. In this ar­ti­cle, we will in­stall the Gray­log soft­ware and its de­pen­den­cies us­ing An­si­ble.

GNU/Linux

An Ubuntu 16.04.3 LTS guest virtual ma­chine (VM) in­stance will be used to set up Gray­log us­ing KVM/QEMU. The host sys­tem is a Parabola GNU/Linux-li­bre x86_64 sys­tem. An­si­ble is in­stalled on the host sys­tem us­ing the dis­tri­bu­tion pack­age man­ager. The ver­sion of An­si­ble used is:

$ an­si­ble --ver­sion an­si­ble 2.4.1.0 con­fig file = /etc/an­si­ble/an­si­ble.cfg con­fig­ured mod­ule search path = [u’/home/shak­thi/.an­si­ble/ plug­ins/mod­ules’, u’/usr/share/an­si­ble/plug­ins/mod­ules’]

an­si­ble python mod­ule lo­ca­tion = /usr/lib/python2.7/sitepack­ages/an­si­ble ex­e­cutable lo­ca­tion = /usr/bin/an­si­ble python ver­sion = 2.7.14 (de­fault, Sep 20 2017, 01:25:59) [GCC 7.2.0]

Add an en­try to the /etc/hosts file for the guest ‘ubuntu’ VM as in­di­cated be­low:

192.168.122.25 ubuntu

On the host sys­tem, let’s create a pro­ject di­rec­tory struc­ture to store the An­si­ble play­books:

an­si­ble/in­ven­tory/kvm/ /play­books/con­fig­u­ra­tion/ /play­books/ad­min/

An ‘in­ven­tory’ file is cre­ated in­side the in­ven­tory/kvm folder that con­tains the fol­low­ing code:

ubuntu an­si­ble_host=192.168.122.25 an­si­ble_­con­nec­tion=ssh an­si­ble_user=ubuntu an­si­ble_­pass­word=pass­word

You should be able to is­sue com­mands us­ing An­si­ble to the guest OS. For ex­am­ple:

$ an­si­ble ­i in­ven­tory/kvm/in­ven­tory ubuntu ­m ping

ubuntu | SUC­CESS => { “changed”: false, “failed”: false, “ping”: “pong” }

Pre-req­ui­sites

The Gray­log soft­ware has a few de­pen­dency pack­ages that need to be in­stalled as pre-req­ui­sites. The APT pack­age repos­i­tory is up­dated and up­graded be­fore in­stalling the pre­req­ui­site soft­ware pack­ages.

--name: Pre-req­ui­sites hosts: ubuntu be­come: yes be­come_method: sudo gath­er_­facts: true tags: [pre­req­ui­site]

tasks:

- name: Up­date the soft­ware pack­age repos­i­tory apt: up­date_­cache: yes

- name: Up­date all the pack­ages apt: up­grade: dist

- name: In­stall pre-req­ui­site pack­ages pack­age: name: “{{ item }}” state: lat­est with­_items:

- apt-trans­port-https - open­jdk-8-jre-head­less - uuid-run­time

- pw­gen

The above play­book can be in­voked as fol­lows:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/gray­log.yml --tags pre­req­ui­site -K

The ‘-K’ op­tion prompts for the sudo pass­word for the ‘ubuntu’ user. You can ap­pend mul­ti­ple ‘-v’ to the end of the play­book in­vo­ca­tion to get a more ver­bose out­put.

Mon­goDB

Gray­log uses Mon­goDB to store meta in­for­ma­tion and con­fig­u­ra­tion changes. The Mon­goDB soft­ware pack­age that ships with Ubuntu 16.04 is sup­ported by the lat­est Gray­log soft­ware. The An­si­ble play­book to in­stall the same is as fol­lows: - name: In­stall Mon­godb hosts: ubuntu be­come: yes be­come_method: sudo gath­er_­facts: true tags: [mon­godb]

tasks:

- name: In­stall Mon­goDB pack­age: name: mon­godb-server state: lat­est

- name: Start the server ser­vice: name: mon­godb state: started

- wait­_­for: port: 27017

The Ubuntu soft­ware pack­age for Mon­goDB is called the ‘mon­godb-server’. It is in­stalled, and the data­base server is started. The An­si­ble play­book waits for the Mon­goDB server to start and lis­ten on the de­fault port 27017. The above play­book can be in­voked us­ing the fol­low­ing com­mand:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/gray­log.yml --tags mon­godb -K

Elas­tic­search

Elas­tic­search is a search en­gine that is writ­ten in Java and re­leased un­der the Apache li­cence. It is based on Lucene (an in­for­ma­tion retrieval soft­ware li­brary) and pro­vides a full-text search fea­ture. The elas­tic.co web­site pro­vides .deb pack­ages that can be used to in­stall the same on Ubuntu. The An­si­ble play­book for this is pro­vided be­low:

- name: In­stall Elas­tic­search hosts: ubuntu be­come: yes be­come_method: sudo gath­er_­facts: true tags: [elas­tic]

tasks:

- name: Add key ap­t_key: url: https://ar­ti­facts.elas­tic.co/GPG-KEYe­las­tic­search state: present

- name: Add elas­tic deb sources line­in­file:

path: /etc/apt/sources.list.d/elas­tic-5.x.list create: yes line: ‘deb https://ar­ti­facts.elas­tic.co/pack­ages/5.x/ apt sta­ble main’

- name: Up­date the soft­ware pack­age repos­i­tory apt: up­date_­cache: yes

- name: In­stall Elas­tic­search pack­age: name: elas­tic­search state: lat­est

- name: Up­date cluster name line­in­file: path: /etc/elas­tic­search/elas­tisearch.yml create: yes reg­exp: ‘^#cluster.name: my-ap­pli­ca­tion’ line: ‘cluster.name: gray­log’

- name: Dae­mon reload sys­temd: dae­mon_reload=yes

- name: Start elas­tic­search ser­vice ser­vice: name: elas­tic­search.ser­vice state: started

- wait­_­for: port: 9200

- name: Test Curl query shell: curl -XGET ‘lo­cal­host:9200/?pretty’

The sta­ble elas­tic.co repos­i­tory pack­age is in­stalled be­fore in­stalling Elas­tic­search. The cluster name is then up­dated in the /etc/elas­tic­search/elas­tic­search.yml con­fig­u­ra­tion file. The sys­tem dae­mon ser­vices are reloaded, and the Elas­tic­search ser­vice is started. The An­si­ble play­book waits for the ser­vice to run and lis­ten on port 9200.

The above play­book can be in­voked as fol­lows:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/gray­log.yml --tags elas­tic -K

You can per­form a man­ual query to ver­ify that Elas­tic­search is run­ning us­ing the fol­low­ing Curl com­mand:

$ curl -XGET ‘lo­cal­host:9200/?pretty’

{

“name” : “cFn-3YD”, “clus­ter_­name” : “elas­tic­search”, “clus­ter_u­uid” : “nuBTSlFBTk6PDGyr­fDCr3A”, “ver­sion” : {

“num­ber” : “5.6.5”,

“build_hash” : “6a37571”,

“build_­date” : “2017-12-04T07:50:10.466Z”, “build_s­nap­shot” : false,

“lucene_ver­sion” : “6.6.1”

},

“tagline” : “You Know, for Search”

}

Gray­log

The fi­nal step is to in­stall Gray­log it­self. The .deb pack­age avail­able from the gray­log2.org web­site is in­stalled and then the ac­tual ‘gray­log-server’ pack­age is in­stalled. The con­fig­u­ra­tion file is up­dated with cre­den­tials for the ‘ad­min’ user with a hashed string for the pass­word ‘osfy’. The Web in­ter­face is also en­abled with the de­fault IP ad­dress of the guest VM. The Gray­log ser­vice is fi­nally started. The An­si­ble play­book to in­stall Gray­log is as fol­lows:

- name: In­stall Gray­log hosts: ubuntu be­come: yes be­come_method: sudo gath­er_­facts: true tags: [gray­log]

tasks:

- name: In­stall Gray­log repo deb apt: deb: https://pack­ages.gray­log2.org/repo/pack­ages/ gray­log-2.3-repos­i­to­ry_lat­est.deb

- name: Up­date the soft­ware pack­age repos­i­tory apt: up­date_­cache: yes

- name: In­stall Gray­log pack­age: name: gray­log-server state: lat­est

- name: Up­date data­base cre­den­tials in the file re­place: dest: “/etc/gray­log/server/server.conf” reg­exp: “{{ item.reg­exp }}” re­place: “{{ item.re­place }}” with­_items:

- { reg­exp: ‘pass­word_se­cret =’, re­place: ‘pass­word_ se­cret = QXHg3Eqvsu PmFxUY2aKl­gimUF05­plMPXQ Hy1stUiQ1uaxgIG27 K3t2MviRiFLNot09U1ako T30n­jK3G69KIzqIoYqdY3oLUP’ }

- { reg­exp: ‘#root_user­name = ad­min’, re­place: ‘root_ user­name = ad­min’ }

- { reg­exp: ‘root_­pass­word_sha2 =’, re­place: ‘root_­pass­word_sha2 = eab­b9b­b2efa089223 d4f54d55bf2333ebf04a29094bf­f00753536d7488629399’}

- { reg­exp: ‘#we­b_en­able = false’, re­place: ‘web_ en­able = true’ }

- { reg­exp: ‘#we­b_lis­ten_uri = http://127.0.0.1:9000/’, re­place: “we­b_lis­ten_uri = http://{{ an­si­ble_de­fault­_ipv4.ad­dress }}:9000/” }

- { reg­exp: ‘rest_lis­ten_uri = http://127.0.0.1:9000/ api/’, re­place: “rest_lis­ten_uri = http://{{ an­si­ble_de­fault_ ipv4.ad­dress }}:9000/api/” }

- name: Start gray­log ser­vice ser­vice: name: gray­log-server.ser­vice state: started

The above play­book can be run us­ing the fol­low­ing com­mand:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/gray­log.yml --tags gray­log -K

Web in­ter­face

You can now open the URL http://192.168.122.25:9000 in a browser on the host sys­tem to see the de­fault Gray­log lo­gin page as shown in Fig­ure 1. Fig­ure 1: Gray­log lo­gin page

The user name is ‘ad­min’ and the pass­word is ‘osfy’. You will then be taken to the Gray­log home page as shown in Fig­ure 2.

The guest VM is a sin­gle node, and hence if you tra­verse to Sys­tem -> Nodes, you will see this node in­for­ma­tion as il­lus­trated in Fig­ure 3.

You can now test the Gray­log in­stal­la­tion by adding a data source as in­put by travers­ing Sys­tem -> In­put in the Web in­ter­face. The ‘ran­dom HTTP mes­sage gen­er­a­tor’ is used as a lo­cal in­put, as shown in Fig­ure 4.

The newly cre­ated in­put source is now run­ning and vis­i­ble as a lo­cal in­put in the Web page as shown in Fig­ure 5.

Af­ter a few min­utes, you can ob­serve the cre­ated mes­sages in the Search link as shown in Fig­ure 6.

Unin­stalling Gray­log

An An­si­ble play­book to stop the dif­fer­ent ser­vices, and to unin­stall Gray­log and its de­pen­dency soft­ware pack­ages, is given be­low for ref­er­ence:

--name: Unin­stall Gray­log hosts: ubuntu be­come: yes be­come_method: sudo gath­er_­facts: true tags: [unin­stall]

tasks:

- name: Stop the gray­log ser­vice ser­vice: name: gray­log-server.ser­vice state: stopped

- name: Unin­stall gray­log server pack­age: name: gray­log-server state: ab­sent

- name: Stop the Elas­tic­search server ser­vice: name: elas­tic­search.ser­vice state: stopped

- name: Unin­stall Elas­tic­search pack­age: name: elas­tic­search state: ab­sent

- name: Stop the Mon­goDB server ser­vice: name: mon­godb state: stopped

- name: Unin­stall Mon­goDB pack­age: name: mon­godb-server state: ab­sent

- name: Unin­stall pre-req­ui­sites pack­age: name: “{{ item }}” state: ab­sent with­_items:

- pw­gen

- uuid-run­time - open­jdk-8-jre-head­less - apt-trans­port-https

The above play­book can be in­voked us­ing:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ ad­min/unin­stall-gray­log.yml -K

Fig­ure 4: Ran­dom HTTP mes­sage gen­er­a­tor

Fig­ure 2: Gray­log home page

Fig­ure 3: Gray­log node ac­ti­vated

Fig­ure 6: Gray­log ran­dom HTTP mes­sages

Fig­ure 5: Gray­log in­put ran­dom HTTP mes­sage gen­er­a­tor

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.