Non-nonsense way to configure Apache for SSL termination to Varnish and Letsencrypt on CentOS 7

First things first. If you’ve come here via a search engine results then you most probably know what Varnish is. If not, then you’ve been wasting some serious hosting money. Basically, it makes sites go fast. I mean, really fast.
But it does have a caveat. It won’t work over SSL. And it most likely never will. At least in it’s opensource edition. Why? According to it’s creator, these are the reasons:
https://varnish-cache.org/docs/trunk/phk/ssl_again.html

So if Varnish accepts only HTTP requests coming to port 80, while Apache listens on some other port, we either need a load balancer or some kind of reverse-proxy software which will terminate our SSL requests as we have decided to go with the “everything over SSL” paradigm.

Currently there are a few options available out there which would solve the SSL termination issue: Nginx, HAProxy, pound, even Varnishes own reverse-proxy program called – hitch.
But we already do have Apache installed, right? And the word out there is that Apache is quite fast for serving static content. If that is the case, then let’s see what we can do with it.

For the purpose of this tutorial we’ll assume that you have already installed the following packages: apache, mod_ssl, mod_proxy, varnish, certbot, certbot-apache and have already procured SSL certificate via Letsencrypt. So basically what you’ll have is Apache serving non-secured content on port 80, secured content on port 443 and Varnish is not participating for the time being.

Apache configuration

Let us begin by forcing Apache to listen on port 8080. This is done by modifying the Listen directive in httpd.conf:

Listen 8080

Next, we change our site’s VirtualHost directive to listen to the same port:

<VirtualHost *:8080>
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/html/example
    ErrorLog /var/log/httpd/example-error.log
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvarnish
    CustomLog /var/log/httpd/example-access.log combinedvarnish
</VirtualHost>

Don’t forget to take note of the LogFormat directive. It is quite important. As you can see the first parameter there is %{X-Forwarded-For}i. This is a special method for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer. Without it, our access log file will show 127.0.0.1 as the originating IP address of all our requests and we won’t be able to distinguish our visitors.

Now let’s check our VirtualHost file which serves secured content:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName example.com
    ServerAlias www.example.com

    SSLProxyEngine On
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:80/
    ProxyPassReverse / http://127.0.0.1:80/
    RequestHeader set X-Forwarded-Port "443"
    RequestHeader set X-Forwarded-Proto "https"

    ErrorLog /var/log/httpd/example-error.log

    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvarnish
    CustomLog /var/log/httpd/example-access.log combinedvarnish

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/example.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem
</VirtualHost>
</IfModule>

As you can see, there are few lines which are added to the default configuration. To make the long story short, when Apache receives a connection on port 443, it will proxy the request to 127.0.0.1:80 where Varnish will take it over.

Varnish configuration

Let’s move to our Varnish configuration. First we’ll open /etc/varnish/varnish.params and change the VARNISH_LISTEN_PORT from 6081 to 80 as Varnish will be intercepting all HTTP traffic. Also we will add a variable called VARNISH_PROXY_PORT which will hold the value of 6081.

VARNISH_LISTEN_PORT=80
VARNISH_PROXY_PORT=6081

# Admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

Next let’s change our Varnish service in /etc/systemd/system. Type:

systemctl edit --full varnish.service

Add the newly created variable VARNISH_PROXY_PORT to the startup command, so the file will look like this:

ExecStart=/usr/sbin/varnishd \
        -P /var/run/varnish.pid \
        -f $VARNISH_VCL_CONF \
        -a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
        -a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_PROXY_PORT},PROXY \
        -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
        -S $VARNISH_SECRET_FILE \
        -s $VARNISH_STORAGE \
        $DAEMON_OPTS

Don’t forget the PROXY attribute at the end of the line, as that will tell Varnish there is a proxy server in front of it and will make him expect the IP X-Forwarded-For method.

Next, we edit the /etc/varnish/default.vcl file and make it look like this:

vcl 4.0;

acl local {
    "127.0.0.1";
}

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Add an X-Forwarded-For header with the client IP address.
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For) {
            set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
        }
	else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }

    # the PROXY protocol allows varnish to see
    # apache's listening port (443) as server.ip
	if (client.ip !~ local) {
		set req.http.location = "https://" + req.http.host + req.url;
		return(synth(301));
	}
}

#http -> https
sub vcl_synth {
	if (resp.status == 301 || resp.status == 302) {
		set resp.http.location = req.http.location;
		return (deliver);
	}
}

sub vcl_backend_response {
}

sub vcl_deliver {
}

So this is what’s going on:

1. As Varnish receives the request, first thing it does is to modify the X-Forwarded-For method with the real IP address of the web site visitor.
2. Then it checks if the request is coming from address different then 127.0.0.1. If the address is coming from the localhost then it’s safe to assume that the request is coming through mod_ssl (our VirtualHost listening on port 443), and there is nothing for Varnish to do except it’s usual magic.
3. But if the request is coming from a different address, then this means that it was received on port 80 and is insecure as such. So, next Varnish changes the url to HTTPS and makes another request by sending HTTP-301 (Permanently moved) to the visitor. This ensures that all traffic coming to and from our site will be secured and made via HTTPS.

Restart and enable both Apache and Varnish services.

systemctl start httpd
systemctl start varnish
systemctl enable httpd
systemctl enable varnish

Letsencrypt configuration

Last thing we want to do is enable certbot to check if the certificate needs to be renewed (each Letsencrypt certificate lasts 3 months). If we go with the default certbot configuration and simply try to renew the certificate via the –dry-run option, we’ll see that certbot complains that it cannot find configured domains to issue a certificate for.

certbot renew --dry-run

Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.. Skipping.

So let’s fix that. All we need to do is edit the certbot-renew service and modify it by adding the http-01-port 8080 parameter to it’s command. This will tell Letsencrypt to use HTTP protocol instead of the default TLS to send it’s challenge and also pass port number 8080.
This does not mean you need to open 8080 in your firewall. So don’t.

systemctl edit --full certbot-renew.service

[Unit]
Description=This service automatically renews any certbot certificates found

[Service]
EnvironmentFile=/etc/sysconfig/certbot
Type=oneshot
ExecStart=/usr/bin/certbot renew --http-01-port 8080 --noninteractive --no-random-sleep-on-renew $PRE_HOOK $POST_HOOK $RENEW_HOOK $CER$

And the only thing that remains left is to activate and enable it’s service:

systemctl start certbot-renew.service && systemctl enable certbot-renew.service

So that’s it. In this tutorial we have learned how to:

1. Configure Apache to serve as a reverse-proxy and terminate SSL requests to Varnish
2. Transfer all traffic to HTTPS with Varnish
3. Configure certbot for automatic renewals in this kind of setup

Drop me a line.