Configure the default.vcl file
The default.vcl file is where you will have to make most of the configuration changes in order to tell Varnish about your Web servers, assets that shouldn’t be cached, etc. Open the default.vcl file in your favourite editor:
[root@bookingwire sridhar]# nano /etc/varnish/default.vcl
Since we expect to have two NGINX servers running our application, we want Varnish to distribute the http requests between these two servers. If, for any reason, one of the servers fails, then all requests should be routed to the healthy server. To do this, add the following to your default. vcl file: backend bw1 { .host = “146.185.129.131”;
.probe = { .url = “/google0ccdbf1e9571f6ef.
html”; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; }} backend bw2 { .host = “37.139.24.12”;
.probe = { .url = “/google0ccdbf1e9571f6ef.
html”; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; }} backend bw1ssl { .host = “146.185.129.131”; .port = “443”; .probe = { .url = “/google0ccdbf1e9571f6ef.
html”; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; }} backend bw2ssl { .host = “37.139.24.12”; .port = “443”; .probe = { .url = “/google0ccdbf1e9571f6ef.
html”; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; }} director default_director round-robin { { .backend = bw1; } { .backend = bw2; }
} director ssl_director round-robin { { .backend = bw1ssl; } { .backend = bw2ssl; }
} sub vcl_recv { if (server.port == 443) {
set req.backend = ssl_director; } else { set req.backend = default_director; }
}
You might have noticed that we have used public IP
addresses since we had not enabled private networking within our servers. You should define the ‘backends’ —one each for the type of traffic you want to handle. Hence, we have one set to handle http requests and another to handle the https requests.
It’s a good practice to perform a health check to see if the NGINX Web servers are up. In our case, we kept it simple by checking if the Google webmaster file was present in the document root. If it isn’t present, then Varnish will not include the Web server in the round robin league and won’t redirect traffic to it.
.probe = { .url = “/google0ccdbf1e9571f6ef.html”;
The above command checks the existence of this file at each backend. You can use this to take an NGINX server out intentionally either to update the version of the application or to run scheduled maintenance checks. All you have to do is to rename this file so that the check fails!
In spite of our best efforts to keep our servers sterile, there are a number of reasons that can cause a server to go down. Two weeks back, we had one of our servers go down, taking more than a dozen sites with it because the master boot record of Centos was corrupted. In such cases, Varnish can handle the incoming requests even if your Web server is down. The NGINX Web server sets an expires header (HTTP 1.0) and the max-age (HTTP 1.1) for each page that it serves. If set, the max-age takes precedence over the expires header. Varnish is designed to request the backend Web servers for new content every time the content in its cache goes stale. However, in a scenario like the one we faced, it’s impossible for Varnish to obtain fresh content. In this case, setting the ‘Grace’ in the configuration file allows Varnish to serve content (stale) even if the Web server is down. To have Varnish serve the (stale) content, add the following lines to your default. vcl: sub vcl_recv {
set req.grace = 6h;
} sub vcl_fetch {
set beresp.grace = 6h;
}
if (!req.backend.healthy) {
unset req.http.Cookie;
}
The last segment tells Varnish to strip all cookies for an authenticated user and serve an anonymous version of the page if all the NGINX backends are down.
Most browsers support encoding but report it differently. NGINX sets the encoding as Vary: Cookie, Accept-Encoding. If you don’t handle this, Varnish will cache the same page once each, for each type of encoding, thus wasting server resources. In our case, it would gobble up memory. So add the following commands to the vcl_recv to have Varnish cache the content only once: if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ “gzip”) {
# If the browser supports it, we’ll use gzip. set req.http.Accept-Encoding = “gzip”;
}
else if (req.http.Accept-Encoding ~ “deflate”) { # Next, try deflate if it is supported. set req.http.Accept-Encoding = “deflate”;
}
else { # Unknown algorithm. Remove it and send unencoded. unset req.http.Accept-Encoding;
}
}
Now, restart Varnish.
[root@bookingwire sridhar]# service varnish restart
Additional configuration for content management systems, especially Drupal
A CMS like Drupal throws up additional challenges when configuring the VCL file. We’ll need to include additional directives to handle the various quirks. You can modify the directives below to suit the CMS that you are using. When using CMSs like Drupal if there are files that you don’t want cached for some reason, add the following commands to your default.vcl file in the vcl_recv section:
if (req.url ~ “^/status\.php$” || req.url ~ “^/update\.php$” || req.url ~ “^/ooyala/ping$” || req.url ~ “^/admin/build/features” || req.url ~ “^/info/.*$” || req.url ~ “^/flag/.*$” || req.url ~ “^.*/ajax/.*$” || req.url ~ “^.*/ahah/.*$”) { return (pass); }
Varnish sends the length of the content ( see the Varnish log output above) so that browsers can display the progress bar. However, in some cases when Varnish is unable to tell the browser the specified content- length ( like streaming audio) you will have to pass the request directly to the Web server. To do this, add the following command to your default. vcl:
if (req.url ~ “^/content/music/$”) {
return (pipe);
}
Drupal has certain files that shouldn’t be accessible to the outside world, e.g., Cron.php or Install.php. However, you should be able to access these files from a set of IPs that your development team uses. At the top of default.vcl include the following by replacing the IP address block with that of your own:
acl internal {
“192.168.1.38”/46;
}
Now to prevent the outside world from accessing these pages we’ll throw an error. So inside of the vcl_recv function include the following:
if (req.url ~ “^/(cron|install)\.php$” && !client.ip ~ internal) {
error 404 “Page not found.”;
}
If you prefer to redirect to an error page, then use this instead:
if (req.url ~ “^/(cron|install)\.php$” && !client.ip ~ internal) { set req.url = “/404”;
}
Our approach is to cache all assets like images, JavaScript and CSS for both anonymous and authenticated users. So include this snippet inside vcl_recv to unset the cookie set by Drupal for these assets:
if (req.url ~ “(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html| htm)(\?[a-z0-9]+)?$”) {
unset req.http.Cookie;
}
Drupal throws up a challenge especially when you have enabled several contributed modules. These modules set cookies, thus preventing Varnish from caching assets. Google analytics, a very popular module, sets a cookie. To remove this, include the following in your default. vcl: set req.http.Cookie = regsuball(req.http.Cookie, “(^|;\s*) (__[a-z]+|has_js)=[^;]*
If there are other modules that set JavaScript cookies, then Varnish will cease to cache those pages; in which case, you should track down the cookie and update the regex above to strip it.
Once you have done that, head to /admin/config/ development/performance, enable the Page Cache setting and set a non-zero time for ‘Expiration of cached pages’.
Then update the settings.php with the following snippet by replacing the IP address with that of your machine running Varnish. $conf[‘reverse_proxy’] = TRUE;
$conf[‘reverse_proxy_addresses’] = array(‘37.139.8.42’);
$conf[‘page_cache_invoke_hooks’] = FALSE; $conf[‘cache’] = 1; $conf[‘cache_lifetime’] = 0;
$conf[‘page_cache_maximum_age’] = 21600;
You can install the Drupal varnish module (http://www. drupal.org/project/varnish), which provides better integration with Varnish and include the following lines in your settings.php: $conf[‘cache_backends’] = array(‘sites/all/modules/varnish/ varnish.cache.inc’);
$conf[‘cache_class_cache_page’] = ‘VarnishCache’;
Checking if Varnish is running and serving requests
Instead of logging to a normal log file, Varnish logs to a shared memory segment. Run varnishlog from the command line, access your IP address/ URL from the browser and view the Varnish messages. It is not uncommon to see a ‘503 service unavailable’ message. This means that Varnish is unable to connect to NGINX. In which case, you will see an error line in the log (only the relevant portion of the log is reproduced for clarity).
[root@bookingwire sridhar]# Varnishlog 12 StatSess c 122.164.232.107 34869 0 1 0 0 0 0 0 0
12 SessionOpen c 122.164.232.107 34870 :80
12 ReqStart c 122.164.232.107 34870 1343640981 12 RxRequest c GET
12 RxURL c/ 12 RxProtocol c HTTP/1.1 12 RxHeader c Host: 37.139.8.42
12 RxHeader c User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:27.0) Gecko/20100101 Firefox/27.0
12 RxHeader c Accept: text/html,application/ xhtml+xml,application/xml;q=0.9,*/*;q=0.8
12 RxHeader c Accept-Language: en-US,en;q=0.5
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c Referer: http://37.139.8.42/
12 RxHeader c Cookie: __zlcmid=OAdeVVXMB32GuW
12 RxHeader c Connection: keep-alive
12 FetchError c no backend connection
12 VCL_call c error
12 TxProtocol c HTTP/1.1
12 TxStatus c 503
12 TxResponse c Service Unavailable
12 TxHeader c Server: Varnish
12 TxHeader c Retry-After: 0
12 TxHeader c Content-Type: text/html; charset=utf-8
12 TxHeader c Content-Length: 686
12 TxHeader c Date: Thu, 03 Apr 2014 09:08:16 GMT
12 TxHeader c X-Varnish: 1343640981
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: close
12 Length c 686
Resolve the error and you should have Varnish running. But that isn’t enough—we should check if it’s caching the pages. Fortunately, the folks at the following URL have made it simple for us.
Check if Varnish is serving pages
Visit http://www.isvarnishworking.com/, provide your URL/ IP address and you should see your Gold Star! (See Figure 3.) If you don’t, but instead see other messages, it means that Varnish is running but not caching.
Then you should look at your code and ensure that it sends the appropriate headers. If you are using a content management system, particularly Drupal, you can check the additional parameters in the VCL file and set them correctly. You have to enable caching in the performance page.
Running the tests
Running Pingdom tests showed improved response times of 2.14 seconds. If you noticed, there was an improvement in the response time in spite of having the payload of the page increasing from 2.9MB to 4.1MB. If you are wondering why it increased, remember, we switched the site to a new theme.
Apache Bench reports better figures at 744.722 ms.
Configuring client IP forwarding
Check the IP address for each request in the access logs of your Web servers. For NGINX, the access logs are available at /var/log/nginx and for Apache, they are available at /var/ log/httpd or /var/log/apache2, depending on whether you are running Centos or Ubuntu.
It’s not surprising to see the same IP address (of the Varnish machine) for each request. Such a configuration will throw all Web analytics out of gear. However, there is a way out. If you run NGINX, try out the following procedure. Determine the NGINX configuration that you currently run by executing the command below in your command line:
[root@bookingwire sridhar]# nginx -V
Look for the –with-http_realip_module. If this is available, add the following to your NGINX configuration file in the http section. Remember to replace the IP address with that of your Varnish machine. If Varnish and NGINX run on the same machine, do not make any changes. set_real_ip_from 127.0.0.1; real_ip_header X-Forwarded-For;
Restart NGINX and check the logs once again. You will see the client IP addresses.
If you are using Drupal then include the following line in settings.php:
$conf[‘reverse_proxy_header’] = ‘HTTP_X_FORWARDED_FOR’;
Other Varnish tools
Varnish includes several tools to help you as an administrator.
varnishstat -1 -f n_lru_nuked: This shows the number of
objects nuked from the cache.
Varnishtop: This reads the logs and displays the most frequently accessed URLs. With a number of optional flags, it can display a lot more information.
Varnishhist: Reads the shared memory logs, and displays a histogram showing the distribution of the last N requests on the basis of their processing. Varnishadm: A command line utility for Varnish. Varnishstat: Displays the statistics.
Dealing with SSL: SSL-offloader, SSLaccelerator and SSL-terminator
SSL termination is probably the most misunderstood term in the whole mix. The mechanism of SSL termination is employed in situations where the Web traffic is heavy. Administrators usually have a proxy to handle SSL requests before they hit Varnish. The SSL requests are decrypted and the unencrypted requests are passed to the Web servers. This is employed to reduce the load on the Web servers by moving the decryption and other cryptographic processing upstream.
Since Varnish by itself does not process or understand SSL, administrators employ additional mechanisms to terminate SSL requests before they reach Varnish. Pound (http://www.apsis.ch/pound) and Stud (https://github. com/bumptech/stud) are reverse proxies that handle SSL termination. Stunnel (https://www.stunnel.org/) is a program that acts as a wrapper that can be deployed in front of Varnish. Alternatively, you could also use another NGINX in front of Varnish to terminate SSL.
However, in our case, since only the sign-in pages required SSL connections, we let Varnish pass all SSL requests to our backend Web server.
Additional repositories
There are other repositories from where you can get the latest release of Varnish: wget repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/ varnish-release/varnish-release-3.0-1.el6.noarch.rpm rpm –nosignature -i varnish-release-3.0-1.el6.noarch.rpm
If you have the Remi repo enabled and the Varnish cache repo enabled, install them by specifying the defined repository. Yum install varnish –enablerepo=epel Yum install varnish –enablerepo=varnish-3.0
Our experience has been that Varnish reduces the number of requests sent to the NGINX server by caching assets, thus improving page response times. It also acts as a failover mechanism if the Web server fails.
We had over 55 JavaScript files (two as part of the theme and the others as part of the modules) in Drupal and we aggregated JavaScript by setting the flag in the Performance page. We found a 50 per cent drop in the number of requests; however, we found that some of the JavaScript files were not loaded on a few pages and had to disable the aggregation. This is something we are investigating. Our recommendation is not to choose the aggregate JavaScript files in your Drupal CMS. Instead, use the Varnish module (https://drupal.org/ project/varnish).
The module allows you to set long object lifetimes (Drupal doesn’t set it beyond 24 hours), and use Drupal’s existing cache expiration logic to dynamically purge Varnish when things change.
You can scale this architecture to handle higher loads either vertically or horizontally. For vertical scaling, resize your VPS to include additional memory and make that available to Varnish using the -s directive.
To scale horizontally, i.e., to distribute the requests between several machines, you could add additional Web servers and update the round robin directives in the VCL file.
You can take it a bit further by including HAProxy right upstream and have HAProxy route requests to Varnish, which then serves the content or passes it downstream to NGINX.
To remove a Web server from the round robin league, you can improve upon the example that we have mentioned by writing a small PHP snippet to automatically shut down or exit() if some checks fail.