DevOps Se­ries An­si­ble De­ploy­ment of Nginx to Serve Static Files and to Use Goac­cess for Log Anal­y­sis

This is the 15th ar­ti­cle in the DevOps se­ries, in which the au­thor demon­strates how an Nginx server can be set up to serve static file con­tent (HTML, CSS, JavaScript), and to use Goac­cess for log anal­y­sis.

OpenSource For You - - Contents - By: Shak­thi Kan­nan The au­thor is a free soft­ware en­thu­si­ast and blogs at shak­thi­maan.com.

Nginx is a free and open source Web server writ­ten in C by Igor Sysoev. It is de­signed to han­dle thou­sands of client con­nec­tions. It is also pop­u­larly used as a load bal­ancer. It does not re­quire much mem­ory and can also be used as an HTTP cache server or a mail proxy server. It was re­leased in 2004 and uses a BSD-like li­cence.

Goac­cess is also free and open source soft­ware. It is a real-time Web log anal­yser writ­ten in C by Ger­ardo Orel­lana. You can run it re­motely on a *nix ter­mi­nal or ac­cess it through a browser. It re­quires only ncurses as a de­pen­dency when used from a ter­mi­nal. It sup­ports a num­ber of Web log for­mats such as Apache, Nginx, Elas­tic load balanc­ing, etc. You can also use a ter­mi­nal dash­board with it and ex­port re­ports to HTML. It was first re­leased in 2010 and uses the MIT li­cence.

GNU/Linux

A De­bian 9 (x86_64) guest vir­tual ma­chine (VM) in­stance us­ing KVM/QEMU is cho­sen to in­stall Nginx.

The host sys­tem is a Parabola GNU/ Linux-li­bre x86_64 sys­tem and An­si­ble is in­stalled on it us­ing the dis­tri­bu­tion pack­age man­ager. The ver­sion of An­si­ble used is 2.5.0 as in­di­cated below:

$ an­si­ble --ver­sion an­si­ble 2.5.0 con­fig file = /etc/an­si­ble/an­si­ble.cfg con­fig­ured mod­ule search path = [u’/home/guest/.an­si­ble/ plug­ins/mod­ules’, u’/usr/share/an­si­ble/plug­ins/mod­ules’]

an­si­ble python mod­ule lo­ca­tion = /usr/lib/python2.7/sitepack­ages/an­si­ble ex­e­cutable lo­ca­tion = /usr/bin/an­si­ble python ver­sion = 2.7.14 (de­fault, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]

You should add an en­try to the /etc/hosts file for the guest ‘de­bian’ VM as fol­lows:

192.168.122.140 de­bian

On the host sys­tem, we will cre­ate a project di­rec­tory struc­ture to store the An­si­ble play­books, the in­ven­tory and static Web files:

an­si­ble/in­ven­tory/kvm/ /play­books/con­fig­u­ra­tion/ /files/

The in­ven­tory/kvm/in­ven­tory file con­tains the fol­low­ing:

de­bian an­si­ble_host=192.168.122.140 an­si­ble_­con­nec­tion=ssh an­si­ble_user=de­bian

The an­si­ble/files/ di­rec­tory has the fol­low­ing:

an­si­ble/files/_site /ex­am­ple.do­main /goac­cessrc

The de­fault De­bian 9 in­stal­la­tion does not in­stall the sudo pack­age. Start the new De­bian VM, log in as the root user, and in­stall the sudo pack­age. You should also pro­vide sudo ac­cess to the user ac­count, which is ‘de­bian’ (in this ex­am­ple).

root@de­bian:~# apt-get in­stall sudo

root@de­bian:~# ad­duser de­bian sudo Adding user `de­bian' to group `sudo'... Adding user de­bian to group sudo

Done.

You should now be able to is­sue com­mands from the host sys­tem to the guest OS, us­ing An­si­ble. For ex­am­ple:

$ an­si­ble -i in­ven­tory/kvm/in­ven­tory de­bian -m ping

de­bian | SUC­CESS => { "changed": false, "ping": "pong" }

Nginx

The first step is to in­stall the Nginx Web server. The soft­ware pack­age re­pos­i­tory is up­dated, and then the Nginx pack­age is in­stalled. The Nginx Web server is started and we wait for the server to lis­ten on port 80. The play­book to in­stall Nginx is as fol­lows:

--name: In­stall nginx hosts: de­bian be­come: yes be­come_method: sudo gath­er_­facts: true tags: [nginx]

tasks: - name: Update the soft­ware pack­age re­pos­i­tory apt: up­date_­cache: yes

- name: In­stall nginx pack­age: name: “{{ item }}” state: lat­est with­_items:

- nginx

- name: Start nginx ser­vice: name: nginx state: started

- wait­_­for: port: 80

The above play­book can be in­voked us­ing the com­mand shown below:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/nginx.yml --tags nginx -vv -K

The -vv rep­re­sents the ver­bosity in the An­si­ble out­put. You can use up to four v’s for a more de­tailed out­put. The

-K op­tion prompts for the sudo pass­word for the de­bian user ac­count. You can now open a Web browser on the lo­cal host with the URL http://192.168.122.140 and you should see the de­fault Nginx home page as shown in Fig­ure 1. Fig­ure 1: Nginx de­fault home page

Serv­ing static files

You can use any static site gen­er­a­tor to cre­ate your HTML, CSS and JavaScript files. You will need to copy the gen­er­ated files to the files/_site di­rec­tory. The An­si­ble play­book to copy the static files to the Nginx Web server lo­ca­tion is given below:

- name: Copy static files hosts: de­bian be­come: yes be­come_method: sudo gath­er_­facts: true tags: [static]

tasks:

- name: Stop nginx server ser­vice: name: nginx state: stopped

- name: Re­move ex­ist­ing di­rec­tory shell: /bin/rm -rf /var/www/html/home/*

- name: Home di­rec­tory should ex­ist file: path: /var/www/html/home state: di­rec­tory mode: 0755

- name: Copy lat­est files copy: src: ../../files/_site/ dest: /var/www/html/home/ di­rec­to­ry_­mode: yes

- name: Copy ex­am­ple.do­main nginx file copy: src: ../../files/ex­am­ple.do­main dest: /etc/nginx/sites-avail­able/

- name: Cre­ate link file file: src: /etc/nginx/sites-avail­able/ex­am­ple.do­main dest: /etc/nginx/sites-en­abled/ex­am­ple.do­main owner: root group: root force: yes state: link

- name: Start nginx ser­vice: name: nginx state: started

- wait­_­for: port: 80

The Nginx Web server is first stopped, as we are chang­ing the Web con­tent. If you have mul­ti­ple in­stances of the Web server run­ning, you can run the play­book in a rolling up­grade fash­ion. In this way, if client re­quests oc­cur dur­ing the up­grade, you will not have a down­time. In this play­book, we first re­move the con­tents of the old di­rec­tory, and then copy the new con­tents. De­pend­ing on the vol­ume of con­tent, this may or may not suit your re­quire­ments. You can also use the syn­chro­nise mod­ule in An­si­ble (http://docs. an­si­ble.com/an­si­ble/lat­est/mod­ules/syn­chro­nize_­mod­ule. html). This uses rsync to syn­chro­nise files be­tween your lo­cal host and the re­mote server.

The ex­am­ple.do­main is the Nginx con­fig­u­ra­tion file for the web­site. As per the Nginx con­fig­u­ra­tion, it is copied to the /etc/nginx/sites-avail­able di­rec­tory, and a sym­bolic link file to it is cre­ated in the /etc/nginx/sites-en­abled di­rec­tory. The con­tent of the ex­am­ple.do­main is as fol­lows:

server { lis­ten 80; lis­ten [::]:80; server_­name ex­am­ple.do­main www.ex­am­ple.do­main;

root /var/www/html;

# Add in­dex.php to the list if you are us­ing PHP in­dex in­dex.html in­dex.htm in­dex.nginx-de­bian.html;

lo­ca­tion / { re­turn 301 $scheme://www.ex­am­ple.do­main/ home$re­quest_uri;

}

lo­ca­tion /home { try_­files $uri $uri/ =404; } }

The Nginx Web server is then started to serve the newly copied con­tent. The An­si­ble play­book for copy­ing and serv­ing the static files can be in­voked as fol­lows:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/nginx.yml --tags static -vv -K

You can now re­fresh the http://192.168.122.140 page on your lo­cal host to see your static Web con­tent.

Goac­cess

The Goac­cess in­ter­ac­tive, real-time Web log anal­yser is avail­able in the De­bian pack­age re­pos­i­tory. Af­ter up­dat­ing the soft­ware APT re­pos­i­tory, Goac­cess is in­stalled. The con­fig­u­ra­tion for­mat to parse the Nginx logs is made avail­able in the goac­cessrc file. Its con­tents are as fol­lows:

log-for­mat %h %^[%d:%t %^] “%r” %s %b “%R” “%u” %T %^ time-for­mat %H:%M:%S date-for­mat %d/%b/%Y

The above code is copied to ~/.goac­cessrc. The An­si­ble play­book to in­stall Goac­cess is given below:

- name: In­stall goac­cess hosts: de­bian be­come: yes

be­come_method: sudo gath­er_­facts: true tags: [goac­cess]

tasks:

- name: Update the soft­ware pack­age re­pos­i­tory apt: up­date_­cache: yes

- name: In­stall goac­cess pack­age: name: “{{ item }}” state: lat­est with­_items:

- goac­cess

- name: Copy goac­cessrc copy: src: ../../files/goac­cessrc dest: “/home/{{ an­si­ble_ssh_user }}/.goac­cessrc”

The above play­book’s in­vo­ca­tion and sam­ple out­put is shown below for ref­er­ence:

$ an­si­ble-play­book -i in­ven­tory/kvm/in­ven­tory play­books/ con­fig­u­ra­tion/nginx.yml --tags goac­cess -vv -K an­si­ble-play­book 2.5.0 con­fig file = /etc/an­si­ble/an­si­ble.cfg con­fig­ured mod­ule search path = [u’/home/guest/.an­si­ble/ plug­ins/mod­ules’, u’/usr/share/an­si­ble/plug­ins/mod­ules’]

an­si­ble python mod­ule lo­ca­tion = /usr/lib/python2.7/sitepack­ages/an­si­ble ex­e­cutable lo­ca­tion = /usr/bin/an­si­ble-play­book python ver­sion = 2.7.14 (de­fault, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]

Us­ing /etc/an­si­ble/an­si­ble.cfg as con­fig file

SUDO pass­word:

PLAY­BOOK: nginx.yml ***************************************** ************************************************************* 3 plays in play­books/con­fig­u­ra­tion/nginx.yml

PLAY [In­stall nginx] **************************************** ************************************************************* TASK [Gath­er­ing Facts] ************************************** *********************************************************** task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:2 ok: [de­bian]

META: ran han­dlers

META: ran han­dlers

META: ran han­dlers

PLAY [Copy static files] ************************************* ************************************************************* TASK [Gath­er­ing Facts] ************************************** ************************************************************* task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:29 ok: [de­bian]

META: ran han­dlers

META: ran han­dlers

META: ran han­dlers

PLAY [In­stall goac­cess] ************************************* *************************************************************

TASK [Gath­er­ing Facts] ************************************** ************************************************************* task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:79 ok: [de­bian]

META: ran han­dlers

TASK [Update the soft­ware pack­age re­pos­i­tory] *************** ************************************************************* task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:87 changed: [de­bian] => {“cache_up­date_­time”: 1523274608, “cache_up­dated”: true, “changed”: true}

TASK [In­stall goac­cess] ************************************* ************************************************************* task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:91 ok: [de­bian] => (item=goac­cess) => {“cache_up­date_­time”: 1523274608, “cache_up­dated”: false, “changed”: false, “item”: “goac­cess”}

TASK [Copy goac­cessrc] ************************************** ************************************************************* task path: /home/guest/an­si­ble/play­books/con­fig­u­ra­tion/nginx. yml:98 changed: [de­bian] => {“changed”: true, “check­sum”: “b8981b­b97a7727d8f­b­d7d92cd1730b9­cac19a2b0”, “dest”: “/home/ de­bian/.goac­cessrc”, “gid”: 0, “group”: “root”, “md5­sum”: “f85cd220742c­c6735e74327388744f3d”, “mode”: “0644”, “owner”: “root”, “size”: 98, “src”: “/home/de­bian/.an­si­ble/tmp/ an­si­ble-tmp-1523274615.53-165679868036633/source”, “state”: “file”, “uid”: 0}

META: ran han­dlers

META: ran han­dlers

PLAY RE­CAP ************************************************** ************************************************************* de­bian : ok=6 changed=2 un­reach­able=0 failed=0

You can now log in to the guest VM and run Goac­cess

from the ter­mi­nal for the Nginx ac­cess.log file as fol­lows: $ sudo goac­cess -f /var/log/nginx/ac­cess.log -p ~/.goac­cessrc -a

You will see the Goac­cess dash­board as il­lus­trated in Fig­ure 2.

You can also ex­port the dash­board to an HTML re­port that you can view in a browser: $ sudo goac­cess -f /var/log/nginx/ac­cess.log -p ~/.goac­cessrc -a > re­port.html

A sam­ple screen­shot of the HTML re­port is shown in Fig­ure 3.

You are en­cour­aged to read the Goac­cess man­ual page at https://goac­cess.io/man to learn more about its us­age.

Fig­ure 2: Goac­cess dash­board

Fig­ure 3: Goac­cess HTML re­port

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.