DevOps Series Ansible Deployment of Nginx to Serve Static Files and to Use Goaccess for Log Analysis
This is the 15th article in the DevOps series, in which the author demonstrates how an Nginx server can be set up to serve static file content (HTML, CSS, JavaScript), and to use Goaccess for log analysis.
Nginx is a free and open source Web server written in C by Igor Sysoev. It is designed to handle thousands of client connections. It is also popularly used as a load balancer. It does not require much memory and can also be used as an HTTP cache server or a mail proxy server. It was released in 2004 and uses a BSD-like licence.
Goaccess is also free and open source software. It is a real-time Web log analyser written in C by Gerardo Orellana. You can run it remotely on a *nix terminal or access it through a browser. It requires only ncurses as a dependency when used from a terminal. It supports a number of Web log formats such as Apache, Nginx, Elastic load balancing, etc. You can also use a terminal dashboard with it and export reports to HTML. It was first released in 2010 and uses the MIT licence.
GNU/Linux
A Debian 9 (x86_64) guest virtual machine (VM) instance using KVM/QEMU is chosen to install Nginx.
The host system is a Parabola GNU/ Linux-libre x86_64 system and Ansible is installed on it using the distribution package manager. The version of Ansible used is 2.5.0 as indicated below:
$ ansible --version ansible 2.5.0 config file = /etc/ansible/ansible.cfg configured module search path = [u’/home/guest/.ansible/ plugins/modules’, u’/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python2.7/sitepackages/ansible executable location = /usr/bin/ansible python version = 2.7.14 (default, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]
You should add an entry to the /etc/hosts file for the guest ‘debian’ VM as follows:
192.168.122.140 debian
On the host system, we will create a project directory structure to store the Ansible playbooks, the inventory and static Web files:
ansible/inventory/kvm/ /playbooks/configuration/ /files/
The inventory/kvm/inventory file contains the following:
debian ansible_host=192.168.122.140 ansible_connection=ssh ansible_user=debian
The ansible/files/ directory has the following:
ansible/files/_site /example.domain /goaccessrc
The default Debian 9 installation does not install the sudo package. Start the new Debian VM, log in as the root user, and install the sudo package. You should also provide sudo access to the user account, which is ‘debian’ (in this example).
root@debian:~# apt-get install sudo
root@debian:~# adduser debian sudo Adding user `debian' to group `sudo'... Adding user debian to group sudo
Done.
You should now be able to issue commands from the host system to the guest OS, using Ansible. For example:
$ ansible -i inventory/kvm/inventory debian -m ping
debian | SUCCESS => { "changed": false, "ping": "pong" }
Nginx
The first step is to install the Nginx Web server. The software package repository is updated, and then the Nginx package is installed. The Nginx Web server is started and we wait for the server to listen on port 80. The playbook to install Nginx is as follows:
--name: Install nginx hosts: debian become: yes become_method: sudo gather_facts: true tags: [nginx]
tasks: - name: Update the software package repository apt: update_cache: yes
- name: Install nginx package: name: “{{ item }}” state: latest with_items:
- nginx
- name: Start nginx service: name: nginx state: started
- wait_for: port: 80
The above playbook can be invoked using the command shown below:
$ ansible-playbook -i inventory/kvm/inventory playbooks/ configuration/nginx.yml --tags nginx -vv -K
The -vv represents the verbosity in the Ansible output. You can use up to four v’s for a more detailed output. The
-K option prompts for the sudo password for the debian user account. You can now open a Web browser on the local host with the URL http://192.168.122.140 and you should see the default Nginx home page as shown in Figure 1. Figure 1: Nginx default home page
Serving static files
You can use any static site generator to create your HTML, CSS and JavaScript files. You will need to copy the generated files to the files/_site directory. The Ansible playbook to copy the static files to the Nginx Web server location is given below:
- name: Copy static files hosts: debian become: yes become_method: sudo gather_facts: true tags: [static]
tasks:
- name: Stop nginx server service: name: nginx state: stopped
- name: Remove existing directory shell: /bin/rm -rf /var/www/html/home/*
- name: Home directory should exist file: path: /var/www/html/home state: directory mode: 0755
- name: Copy latest files copy: src: ../../files/_site/ dest: /var/www/html/home/ directory_mode: yes
- name: Copy example.domain nginx file copy: src: ../../files/example.domain dest: /etc/nginx/sites-available/
- name: Create link file file: src: /etc/nginx/sites-available/example.domain dest: /etc/nginx/sites-enabled/example.domain owner: root group: root force: yes state: link
- name: Start nginx service: name: nginx state: started
- wait_for: port: 80
The Nginx Web server is first stopped, as we are changing the Web content. If you have multiple instances of the Web server running, you can run the playbook in a rolling upgrade fashion. In this way, if client requests occur during the upgrade, you will not have a downtime. In this playbook, we first remove the contents of the old directory, and then copy the new contents. Depending on the volume of content, this may or may not suit your requirements. You can also use the synchronise module in Ansible (http://docs. ansible.com/ansible/latest/modules/synchronize_module. html). This uses rsync to synchronise files between your local host and the remote server.
The example.domain is the Nginx configuration file for the website. As per the Nginx configuration, it is copied to the /etc/nginx/sites-available directory, and a symbolic link file to it is created in the /etc/nginx/sites-enabled directory. The content of the example.domain is as follows:
server { listen 80; listen [::]:80; server_name example.domain www.example.domain;
root /var/www/html;
# Add index.php to the list if you are using PHP index index.html index.htm index.nginx-debian.html;
location / { return 301 $scheme://www.example.domain/ home$request_uri;
}
location /home { try_files $uri $uri/ =404; } }
The Nginx Web server is then started to serve the newly copied content. The Ansible playbook for copying and serving the static files can be invoked as follows:
$ ansible-playbook -i inventory/kvm/inventory playbooks/ configuration/nginx.yml --tags static -vv -K
You can now refresh the http://192.168.122.140 page on your local host to see your static Web content.
Goaccess
The Goaccess interactive, real-time Web log analyser is available in the Debian package repository. After updating the software APT repository, Goaccess is installed. The configuration format to parse the Nginx logs is made available in the goaccessrc file. Its contents are as follows:
log-format %h %^[%d:%t %^] “%r” %s %b “%R” “%u” %T %^ time-format %H:%M:%S date-format %d/%b/%Y
The above code is copied to ~/.goaccessrc. The Ansible playbook to install Goaccess is given below:
- name: Install goaccess hosts: debian become: yes
become_method: sudo gather_facts: true tags: [goaccess]
tasks:
- name: Update the software package repository apt: update_cache: yes
- name: Install goaccess package: name: “{{ item }}” state: latest with_items:
- goaccess
- name: Copy goaccessrc copy: src: ../../files/goaccessrc dest: “/home/{{ ansible_ssh_user }}/.goaccessrc”
The above playbook’s invocation and sample output is shown below for reference:
$ ansible-playbook -i inventory/kvm/inventory playbooks/ configuration/nginx.yml --tags goaccess -vv -K ansible-playbook 2.5.0 config file = /etc/ansible/ansible.cfg configured module search path = [u’/home/guest/.ansible/ plugins/modules’, u’/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python2.7/sitepackages/ansible executable location = /usr/bin/ansible-playbook python version = 2.7.14 (default, Jan 5 2018, 10:41:29) [GCC 7.2.1 20171224]
Using /etc/ansible/ansible.cfg as config file
SUDO password:
PLAYBOOK: nginx.yml ***************************************** ************************************************************* 3 plays in playbooks/configuration/nginx.yml
PLAY [Install nginx] **************************************** ************************************************************* TASK [Gathering Facts] ************************************** *********************************************************** task path: /home/guest/ansible/playbooks/configuration/nginx. yml:2 ok: [debian]
META: ran handlers
META: ran handlers
META: ran handlers
PLAY [Copy static files] ************************************* ************************************************************* TASK [Gathering Facts] ************************************** ************************************************************* task path: /home/guest/ansible/playbooks/configuration/nginx. yml:29 ok: [debian]
META: ran handlers
META: ran handlers
META: ran handlers
PLAY [Install goaccess] ************************************* *************************************************************
TASK [Gathering Facts] ************************************** ************************************************************* task path: /home/guest/ansible/playbooks/configuration/nginx. yml:79 ok: [debian]
META: ran handlers
TASK [Update the software package repository] *************** ************************************************************* task path: /home/guest/ansible/playbooks/configuration/nginx. yml:87 changed: [debian] => {“cache_update_time”: 1523274608, “cache_updated”: true, “changed”: true}
TASK [Install goaccess] ************************************* ************************************************************* task path: /home/guest/ansible/playbooks/configuration/nginx. yml:91 ok: [debian] => (item=goaccess) => {“cache_update_time”: 1523274608, “cache_updated”: false, “changed”: false, “item”: “goaccess”}
TASK [Copy goaccessrc] ************************************** ************************************************************* task path: /home/guest/ansible/playbooks/configuration/nginx. yml:98 changed: [debian] => {“changed”: true, “checksum”: “b8981bb97a7727d8fbd7d92cd1730b9cac19a2b0”, “dest”: “/home/ debian/.goaccessrc”, “gid”: 0, “group”: “root”, “md5sum”: “f85cd220742cc6735e74327388744f3d”, “mode”: “0644”, “owner”: “root”, “size”: 98, “src”: “/home/debian/.ansible/tmp/ ansible-tmp-1523274615.53-165679868036633/source”, “state”: “file”, “uid”: 0}
META: ran handlers
META: ran handlers
PLAY RECAP ************************************************** ************************************************************* debian : ok=6 changed=2 unreachable=0 failed=0
You can now log in to the guest VM and run Goaccess
from the terminal for the Nginx access.log file as follows: $ sudo goaccess -f /var/log/nginx/access.log -p ~/.goaccessrc -a
You will see the Goaccess dashboard as illustrated in Figure 2.
You can also export the dashboard to an HTML report that you can view in a browser: $ sudo goaccess -f /var/log/nginx/access.log -p ~/.goaccessrc -a > report.html
A sample screenshot of the HTML report is shown in Figure 3.
You are encouraged to read the Goaccess manual page at https://goaccess.io/man to learn more about its usage.