Blog: Ivan Ristić / Apache

Increasing DHE strength on Apache 2.4.x

2013-08-15T00:00:00+01:00

Update (13 May 2015): Apache 2.2.30 (not yet released at the time of writing) will also support configurable DH parameter strength, in the same way Apache 2.4.x does.

Update (26 Nov 2013): Apache 2.4.7, released today, has been improved to automatically select appropriate DH parameters, using the strength of the server key as guidance. Thus, there is no need to modify the source code any more. The improvements are explained in the CHANGES file, and in the SSLCertificateFile documentation.

Update (18 Oct 2013): Since I wrote this post, Apache improved DH parameter handling. The trunk version now automatically selects appropriate DH parameter strength based on the strength of the private key. The bug report now includes a backport to 2.4.x.

Some users of older versions of the Apache web server, who are trying to get their Forward Secrecy configuration just right usually run into a brick wall when they determine that Apache won't use Diffie-Hellman parameters stronger than 1024 bits. ~~This is because Apache leaves the business of DH configuration to OpenSSL, and 1024 bits is just what you get by default (for the stronger suites; the weaker ones use 512 bits).~~

Other web servers expose configuration options that allow you to increase the DH strength, but Apache does not. But, if you went through the hassle of installing Apache from source code (as I described in my previous blog post), there's an easy fix for this problem. Apache Software Foundation's bug #49559 ("Patch to add user-specified Diffie-Hellman parameters") contains a patch that introduces a new configuration directive, called SSLDHParametersFile, that adds the missing functionality to Apache httpd 2.4.x.

The bug is more than 3 years old, even though the code seems straightforward. The latest version of the patch was made against httpd 2.4.2, just over a year ago. I've tested it against httpd 2.4.6, and, with a hiccup, it appears to work as advertised.

The hiccup is that the patch won't compile as-is; a trivial change is required. If you're a programmer, you'll have no problem understanding the error and finding the fix from looking at the rest of the mod_ssl code. And if you don't want to do even that, you can download my version. (Disclaimer: I just made it compile, but didn't otherwise read or review the code. Use at your own risk.)

Assuming the starting point is the pristine source code tree of httpd 2.4.6, apply the patch like this:

$ cd httpd-2.4.6
$ patch -p1 < /path/to/patch-49559-for-2.4.6.diff
patching file modules/ssl/mod_ssl.c
patching file modules/ssl/ssl_engine_config.c
Hunk #2 succeeded at 189 (offset 6 lines).
Hunk #3 succeeded at 318 (offset 15 lines).
Hunk #4 succeeded at 801 (offset 36 lines).
patching file modules/ssl/ssl_engine_init.c
Hunk #1 succeeded at 994 (offset 44 lines).
Hunk #2 succeeded at 1192 (offset -1 lines).
Hunk #3 succeeded at 1205 (offset -1 lines).
Hunk #4 succeeded at 1769 (offset 20 lines).
patching file modules/ssl/ssl_engine_pphrase.c
patching file modules/ssl/ssl_private.h
Hunk #2 succeeded at 546 (offset 18 lines).
Hunk #3 succeeded at 573 (offset 18 lines).
Hunk #4 succeeded at 744 (offset 28 lines).
patching file modules/ssl/ssl_util_ssl.c
patching file modules/ssl/ssl_util_ssl.h

From here, compile and install as usually.

To use stronger DH parameters, you first need to create a special parameter file with the desired strength:

$ openssl dhparam -out dh2048.pem 2048

The process will take some time and you'll see lots of output on your screen. (If you're curious what the various characters mean, have a look here.) Then you need to add this directive to your Apache configuration:

SSLDHParametersFile /path/to/dh2048.pem

That's all. Now you only need to restart the web server and test it using the SSL Labs test.

Update (15 Aug 2013): As Michael Reschly reminded me, increasing DH strength can cause interoperability issues. In particular, Java is known not to support DH parameters above 1024 bits. Here's a Stack Overflow thread that contains more information. The issue has been resolved in the (not yet released) Java 8.

Update (19 Aug 2013): Ryan Hurst also wanted me to tell you that Windows crypto libraries don't support DH parameters stronger than 1024 bits. (Windows 8.1 might appears to actually support stronger DHE keys.) The only browser that could be affected by this is Internet Explorer. Other browsers running on Windows use their own crypto, and do not have this limitation. But even with IE users—as Reschly pointed out via Twitter—the DH limitation should not be a problem, for two reasons. First, if you configure Forward Secrecy correctly using ECDHE suites (as I discussed in my previous post), IE will always use ECDHE. Second, IE does not support DHE in combination with RSA, and will never negotiate a DHE suite, anyway.

Update (13 May 2015): The original post incorrectly stated that Apache supported only 1024-bit DH parameters because it delegated DH parameter handling to OpenSSL. That is incorrect. The limit was in the Apache code. Thanks to Richard Moore for pointing this out.

Defending against the BREACH attack

2013-08-07T00:00:00+01:00

When Juliano and Thai disclosed the CRIME attack last year, it was clear that the same attack technique could be applied to any other compressed data, and compressed response bodies (via HTTP compression) in particular. But it was also clear that—with our exploit-driven culture—browser vendors were not going to do anything about.

Progress will be made now that there is an exploit to worry about because, this year at Black Hat, a group of researched presented BREACH, a variant of CRIME that works exactly where it hurts the most, on HTTP response bodies. If you're not already familiar with the attack I suggest that you go to the researchers' web site, where they have a very nice paper and a set of slides.

If you don't want to read the paper right this moment, I will remind you that the CRIME attack works by having the attacker guess some secret text. The trick is to include the guess in the same context as the actual secret (for example, the same response body). When his guess is 100% wrong, the size of the response increases for the size of the guess. But, when the guess is correct (fully, or partially, meaning there is some overlap between the guess and the secret), compression kicks in, and the response body shrinks slightly. With enough guesses and enough time, you can guess anything on the page.

TLS does not defend against this attack because, when the protocol was originally designed, it was impossible for MITM attackers to submit arbitrary plaintext via victims' browsers. Since then, the threat model evolved, but the protocols remained the same. (Interestingly, there is a draft proposal to deal with this sort of thing at the TLS level: Length Hiding Padding for the Transport Layer Security Protocol.)

Mitigation

Clearly, one option is to address the problem at the TLS level; but that will take some time.

Outside TLS, the paper itself includes a nice list of mitigation options, and, with exceptions, I am not going to repeat them here. In general, it's not going to be easy. When dealing with CRIME, we were fortunate because few people knew TLS compression existed, and only a small number of browsers/users actually supported it. This time, the flaw is exploited in a feature that's not only very widely used, but one which many sites cannot exist without. Just try convincing a large site to turn off compression, at a large financial and performance cost.

Although most discussions about BREACH will no doubt focus on its threat against CSRF tokens, we should understand that the impact is potentially wider. Any sensitive data contained in responses is under threat. The good news is that session tokens don't appear to be exposed. However, a well-placed forged CSRF request can do a lot of damage.

CSRF token defence

For CSRF tokens there is a simple and effective defence, which is to randomize the token by masking it with a different (random) value on every response. The masking does not hide the token (whoever has the token can easily reverse the masking), but it does defeat the attack technique. Guessing is impossible when the secret is changing all the time. Thus, we can expect that most frameworks will adopt this technique. Those who rely on frameworks will only need to upgrade to take advantage of the defence. Those who don't will have to fix their code.

HTTP chunked encoding mitigation

The award for least-intrusive and entirely painless mitigation proposal goes to Paul Querna who, on the httpd-dev mailing list, proposed to use the HTTP chunked encoding to randomize response length. Chunked encoding is a HTTP feature that is typically used when the size of the response body is not known in advance; only the size of the next chunk is known. Because chunks carry some additional information, they affect the size of the response, but not the content. By forcing more chunks than necessary, for example, you can increase the length of the response. To the attacker, who can see only the size of the response body, but not anything else, the chunks are invisible. (Assuming they're not sent in individual TCP packets or TLS records, of course.)

This mitigation technique is very easy to implement at the web server level, which makes it the least expensive option. There is only a question about its effectiveness. No one has done the maths yet, but most seem to agree that response length randomization slows down the attacker, but does not prevent the attack entirely. But, if the attack can be slowed down significantly, perhaps it will be as good as prevented.

Referer check mitigation

A quick, dirty, tricky, and a potentially unreliable mitigation approach you can apply today, is to perform Referer header checks on all incoming requests. Because the attacker cannot inject requests from the web site itself (unless he gains access via XSS, in which case he owns the browser and has no need for further attacks), he must do so from some other web site (a malicious web site, or an innocent site hijacked from a MITM location). In that case, the referrer information will show the request originating from that other web site, and we can easily detect that.

Now, you can't just drop such requests (because then no links to your web site would work any more), but you can drop all the cookies before they reach the application. Without the cookies, your application will not resume the victim's session, and won't place anything secret in the response. Attack mitigated.

There is a catch, however (as Hubert pointed out to me this morning): if your web site relies on being framed by arbitrary 3rd party web sites, or if it exposes public services to others (for browser consumption), then you can't use this defence. I am assuming your services need the cookies. If they do not, you're back in the game. If you do decide to try it, please test your major use cases in a staging environment first.

You can implement this defence with only two Apache directives (replace www.example.com with your own domain name):

SetEnvIfNoCase Referer ^https://www\.example\.com keep_cookies
RequestHeader unset Cookie env=!keep_cookies

The cookies are kept only for requests arriving from the same site. There's a potential problem with users that follow links from other sites and bookmarks and expect to be logged in straight away (either because they are already logged in, or because your site supports auto-log in). For such users you might need to have a welcome page, where you will ask them to click on a link to enter the web site. The cookies will be sent again on the next request.

Just to be clear, there is a long history of attacks focusing on spoofing the Referer header. For example, there was one such problem in Chrome just recently. However, such attacks are addressed quickly after discovery. Further, as Krzysztof Kotowicz pointed out, it is trivial to submit requests that do not contain the Referer header (read about it here and here).

To conclude, I can't really say that I like this approach, but its huge advantage is that you can deploy it very quickly at the web server or reverse proxy level, without having to make any changes to the application code. Even with all the constraints, I imagine there will be a large number of applications for which the trade-offs will be acceptable.

We should really fix browsers

I would really like to see this problem addressed where it should be addressed—at the browser level. At the moment, a MITM attacker can intercept your non-encrypted requests, mess with them, and trick your browser into sending requests with arbitrary content to the sites that you care about. It's this interaction that's making several very interesting attacks possible: BEAST, CRIME, RC4, and now BREACH.

A carefully designed opt-in security measure could do the trick, but I suppose it would take a lot of talking and politics to get it implemented. The idea is that a web site can control which other web sites (cross-origins) can initiate requests to it, even if it is via <script> and <img> tags.

Incidentally, just a couple of days ago, Mike Shema and Vaagn Toukharian (fellow Qualys employees), proposed a new cookie control (update: and there is now a follow-up post from Mike) that would restrict when cookies are sent. Their intention was to deal with CSRF, but the measure would work against BREACH, too. If a 3rd party web site initiates a request to your web site, against your wishes, being able to tell the browser to drop the cookies would mitigate the potential attacks.

Update (8 August 2013): The first version of this blog post included a referrer check defence that allowed empty referrers, with the idea to support auto-log in for the users following bookmarks. But then Krzysztof Kotowicz pointed out that the Referer header can be removed by the attackers. I have now modified the example to drop cookies on all requests not originating from the web site.

Update (14 October 2013): A better idea for mitigation based on referrer checking is to selectively disable compression, rather than drop cookies. This approach comes with a slight performance penalty, but no loss of functionality. Read the original discussion thread for more information how to achieve this with Apache.

Configuring Apache, Nginx, and OpenSSL for Forward Secrecy

2013-08-05T00:00:00+01:00

In my earlier blog post, I gave an overview of Forward Secrecy, as well as some configuration tips. If you're new to the concept, I suggest that you go and read that post first. This time, I am following up with detailed configuration examples for Apache, Nginx, and OpenSSL.

Software Requirements

To deploy Forward Secrecy, you need to have both your web server and the underlying SSL/TLS library support Elliptic Curve (EC) cryptography. For Apache, Nginx, and OpenSSL, the following minimum versions will suffice:

OpenSSL 1.0.1c+
Apache 2.4.x+
nginx 1.0.6+ and 1.1.0+

You will probably want to upgrade to the most recent versions wherever possible, because you don't want to be running old and obsolete and potentially vulnerable software. If your distribution does not support an acceptable version of one of the components, consider installing everything from scratch, as I described in a recent blog post.

You are probably aware that Linux distributions often ship modified packages. The modifications are usually improvements, but could mean feature removal in some cases. For example, Red Hat appears to have no support for Elliptic Curve crypto on their operating systems, because of patent issues. If you're running CentOS, for example, and wish to support Forward Secrecy, you will need to recompile the key packages to put EC support back in. (There appear to be plenty tutorials on the Web for this.)

Once the correct packages are in place, enabling Forward Secrecy requires two steps:

Configure the web server to actively select suites
Activate the correct OpenSSL suite configuration string

The following configuration advice applies only if you are using compatible software components, as described in the previous section. It is not possible to support Forward Secrecy otherwise.

Apache

To configure Apache, you need to have the following lines in your configuration:

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"

Nginx

To configure Nginx, you need to have the following lines in your configuration:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

RC4 versus BEAST

Today, only TLS 1.2 with GCM suites offer fully robust security. All other suites suffer from one problem or another (e.g, RC4, Lucky 13, BEAST), but most are difficult to exploit in practice. Because GCM suites are not yet widely supported, most communication today is carried out using one of the slightly flawed cipher suites. It is not possible to do better if you're running a public web site.

The one choice you can make today is whether to prioritize RC4 in most cases. If you do, you will be safe against the BEAST attack, but vulnerable to the RC4 attacks. On the other hand, if you remove RC4, you will be vulnerable against BEAST, but the risk is quite small. Given that both issues are relatively small, the choice isn't clear.

However, the trend is clear. Over time, RC4 attacks are going to get better, and the number of users vulnerable to the BEAST attack is going to get smaller.

Configuring OpenSSL (with RC4)

This configuration assumes you wish to deploy best-possible configuration supporting Forward Secrecy, and that you have a preference for GCM suites (resistant to timing attacks) and RC4 (resistant to BEAST). To achieve best performance, the faster ECDHE suites are used whenever possible.

EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS

To see what suites the above configuration string uses, invoke the following command (all one line):

$ openssl ciphers -V 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA256 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EDH+aRSA EECDH RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS'

The output will be similar to this:

ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-RSA-RC4-SHA       SSLv3 Kx=ECDH     Au=RSA  Enc=RC4(128)  Mac=SHA1
ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA1
ECDHE-ECDSA-RC4-SHA     SSLv3 Kx=ECDH     Au=ECDSA Enc=RC4(128)  Mac=SHA1
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256) Mac=SHA1
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-SEED-SHA        SSLv3 Kx=DH       Au=RSA  Enc=SEED(128) Mac=SHA1
DHE-RSA-CAMELLIA128-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(128) Mac=SHA1
ECDH-RSA-RC4-SHA        SSLv3 Kx=ECDH/RSA Au=ECDH Enc=RC4(128)  Mac=SHA1
ECDH-ECDSA-RC4-SHA      SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=RC4(128)  Mac=SHA1
RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1

If you deploy your configuration, your SSL Labs test results might look like this.

Notes:

The list begins with a number of strong TLS 1.2 suites (GCM, SHA256, and SHA384); this helps avoid RC4 whenever possible.
ECDSA suites are preferred to the RSA ones, giving you a performance edge in a dual-certificate deployment scenario.
The main 2 suites (for the current browsers, most of which do not support TLS 1.2) are ECDHE-RSA-RC4-SHA and ECDHE-ECDSA-RC4-SHA, used for RSA and ECDSA certificates, respectively.
The DHE suites are there to support those rare browsers that do not support ECDHE. (Opera before version 15 is one such browser. Firefox as shipped on Red Hat systems another.) These suites are slower, but the majority of browsers will offer the faster ECDHE.
The RC4-SHA suite at the end is there to support IE8 running on Windows XP. (And possibly IE6, but I have not tested for it.)

Issues:

Internet Explorer, in all versions, does not support the ECDHE and RC4 combination (which has the benefit of supporting Forward Secrecy and being resistant to BEAST). But IE has long patched the BEAST vulnerability and so we shouldn't worry about it.
Same comment as above for Firefox on Red Hat systems.
It is impossible to support Forward Secrecy for IE8 running on Windows XP, because this browser does not support the necessary suites. The same is probably true for any IE version running on Windows XP. If you'd rather not handshake with such browsers, add !RC4-SHA to the configuration.

Configuring OpenSSL (without RC4)

If you prefer not to use RC4, use the same configuration string (for simplicity), but add !RC4 to the end:

EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4

Alternatively (in order to support a very wide range of browsers, including the IE versions running on Windows XP), you could deploy with RC4 enabled, but used only as a last resort:

EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS +RC4 RC4

Final Remarks

Finally, consider the following:

There's a vast number of SSL/TLS clients in deployment, each potentially supporting a unique list of cipher suites. It is impossible to guarantee consistent results across such a large user base.
The Handshake Simulation feature of the SSL Labs test is of great help when choosing cipher suite configuration. It supports a wide range of desktop browsers (including older versions). However, do note that the support for mobile devices is not very good at the moment.
Configuring OpenSSL can be tricky. I recommend reading the (free) OpenSSL Cookbook, which describes the configuration in detail.

Update (21 August): I have tweaked the suite configuration string to position SHA256 and SHA384 suites (which are TLS 1.2-only) after GCM suites and before RC4 suites. This helps avoid RC4 whenever possible. (This matters now because Google Chrome 29 has just been released, and, for the first time, we have a desktop browser that supports TLS 1.2 by default). Of course, I assume that you have upgraded your OpenSSL to 1.0.1d or better in order to fix the timing issues with CBC suites (the so-called Lucky 13 attack).

Compiling Apache with static OpenSSL libraries

2013-08-03T00:00:00+01:00

Back in 2004, when I was writing Apache Security, it was quite common to install Apache from source code, and I spent a lot of time documenting the process. When the stabilized most people stopped bothering with the source code and relied on the binaries provided by the operating system.

We're back into unstable territories. Today, to get the best SSL/TLS configuration you have to roll your sleeves and do everything the old way. For example, I have a couple of servers running Ubuntu 10.04 LTS; the OpenSSL version installed does not support TLS 1.2, and its Apache 2.2.x does not support the ECDHE suites (which are necessary for Forward Secrecy). If you're running an operating that originated at Red Hat, I hear that you don't get any Elliptic Curve crypto from the default binaries.

The easiest way to run Apache with a recent version of OpenSSL is to compile the crypto code statically, and install everything into a separate location. That way you achieve the goal, but you don't mess with the rest of the operating system.

First, you should get the desired version of OpenSSL and install it at a location where it will not interfere with your system version. I usually choose /opt for this purpose.

$ ./config \
    --prefix=/opt/openssl-1.0.1e \
    --openssldir=/opt/openssl-1.0.1e
$ make
$ sudo make install

Now, get the latest Apache 2.4.x, APR and APR-Util libraries. You will need to unpack all three packages into the same source tree, with the latter two in the location where Apache expects them. For example:

$ tar zxvf httpd-2.4.6.tar.gz
$ cd httpd-2.4.6/srclib/
$ tar zxvf ../../apr-1.4.8.tar.gz
$ ln -s apr-1.4.8/ apr
$ tar zxvf ../../apr-util-1.5.2.tar.gz
$ ln -s apr-util-1.5.2/ apr-util

You are now ready to configure and install Apache. The mod_ssl module will be compiled statically, with all other modules dynamically.

$ ./configure \
    --prefix=/opt/httpd \
    --with-included-apr \
    --enable-ssl \
    --with-ssl=/opt/openssl-1.0.1e \
    --enable-ssl-staticlib-deps \
    --enable-mods-static=ssl
$ make
$ sudo make install

Apache Security 1ed now available from Feisty Duck

2010-04-21T00:00:00+01:00

Apache Security was originally published by O'Reilly in 2005, and it was very well received. Soon after publication, it was heralded as the best Apache security resource, according to many. Contrary to my expectations, it also aged very gracefully, which is probably why it continues to be popular. As much as I wanted to release an update, I struggled for years to justify a second edition. When I finally could, it turned out that O'Reilly was not too keen on the idea.

[Note: This blog post contains the entire preface to the digital reprint edition of Apache Security. More information on the book is available from the Feisty Duck's web site.]

That was an opportunity for me to do things differently. As much as I enjoyed working on Apache Security a few years ago, the process was traditional and slow. It was a new digital age and we had all the advanced technology at our fingertips, yet we were still producing books the old-fashioned way. I wanted more. Above all, I wanted the ability to update my books whenever I felt the need. Unable to find a publisher that supported the process I wanted, I started my own publishing company. Feisty Duck, as my wife and I named it, is a publisher of computer books, with special focus on continuous publishing and digital delivery.

We are now releasing what is pretty much the original Apache Security, in digital format only, in order to establish a starting point for the second edition, which will be published by Feisty Duck at some point in the future. The known errors in the book have been fixed. If further errors are discovered, they will be fixed, too, and the digital version will be updated.

You may wonder whether the first edition of Apache Security is still worth paying for. After all, it has been five years since the first edition. Here's what I think:

The only part of the book that is completely obsolete is the ModSecurity chapter. I have only myself to blame for that, because I completely rewrote ModSecurity itself in 2006. If ModSecurity is what you're after, you should look at my other book, ModSecurity Handbook (Feisty Duck, 2010). You will find more information about it at https://www.feistyduck.com.
Chapter 10, "Web Application Security," was the best introduction to the topic at the time of the original publication. It remains a good introduction, but there have been many advances and discoveries since it was written. These days, you actually have to read several books to get decent coverage of web application security, and complete coverage is virtually impossible.
The same can be said for Chapter 11, "Web Security Assessment": it's still good, but it's just not enough any more.
The rest of the book remains pretty solid. Five years later, some aspects are not entirely accurate, but what is in the book is still very useful. You will find that the general principles of web server security haven't changed at all.

To conclude, Apache Security is still a good book, although it will no longer serve all audiences equally well. To paraphrase a recent Amazon.com reviewer, if you are at the beginner or intermediate levels, it will work for you. If you are an advanced user, it may not. If you are not sure, the best thing to do is decide by looking at the table of contents.

ModSecurity Handbook available for pre-order and early access

2009-11-26T00:00:00+00:00

ModSecurity Handbook, which I announced a couple of days ago, is now available for pre-order and early digital access. We managed to meet our self-imposed deadline and have everything ready for November 24th, actually.

This book is a big deal, in more ways than one:

It took me more than 5 years to gather courage to start writing another book (after Apache Security, which I started writing in 2004).
This book is about ModSecurity, a project that is very dear to my heart. It makes me very happy that I will document everything I know about it.
I am releasing the book early because I want to interact with the readers while the content is still not finalised. With Apache Security I ended up being terribly unhappy because I was writing in isolation and because I couldn't seek feedback from the readers prior to publication.
To publish this book (and all my subsequent books), my wife and I started a publishing company and dealt with all the stuff that publishers have to deal with. The learning curve wasn't very difficult because of my previous experience in publishing, but there was a lot of things to do.
This book will be a living book. I intend to keep it up to date at all times, keeping up with the changes in ModSecurity. We've invested a significant amount of time into polishing a single-source publishing system, where the manuscript is kept as XML (DocBook, stored in a Subversion repository) and automatically converted to any of the supported formats (only PDF at the moment, but several forms of PDF, HTML and ePub in the near future). The system allows me to make changes and push updates instantly to all the readers!

The work is far from done, of course. First, I need to finish the book, first of all. Second, we'll have to figure out how to promote it effectively, and I somehow suspect that will be the hardest part. Perhaps, when it's all done, I'll write a blog post called "Adventures in Computer Book Publishing".

Update: The official Reference Manual and Data Formats Guide guide have been added to the book. There's about 230 pages of material right now, with the final count expected to be close to or over 300.

Announcing ModSecurity Handbook

2009-11-16T00:00:00+00:00

It is a pleasure to announce my next book, ModSecurity Handbook, which features an in-depth coverage of ModSecurity, an open source web application firewall. I am very happy because, finally, ModSecurity will have the documentation it deserves.

The main highlights are the following:

Step-by-step instructions for those just starting out
Detailed explanations of the internals, and advanced techniques for seasoned users
Includes the official ModSecurity Reference Manual and Data Formats Guide
Available in digital format (PDF, HTML and ePub, although not all straight away) and as paperback (once the first edition is complete)
Continually updated as ModSecurity evolves (with the updates included with purchase)
Readers can talk to me to shape the book to work better for them

The complete table of contents is available on the book's web site.

I estimate that the book is about 75% complete. In a week's time (on November 24th) it will be available for early access and pre-order. The idea with the early access is to avoid the problem I experienced with Apache Security -- writing in isolation. This time, I want to engage with my readers before my book is published.

Also, it is pretty important that this book is (and will be) continually updated. I have the entire publishing workflow automated so whenever I make a change to the book, the update is automatically made available to the readers. With this feature, again, I want to avoid the painful experience that I had with Apache Security, where for years I wanted to provide updates but I couldn't. (Apache Security readers, fear not, the second edition is being worked on.) In the future, I hope to evolve the publishing toolchain to enable readers to make comments straight to the HTML version of the book that is kept online.

TLS Server Name Indication now in Apache

2009-07-29T00:00:00+01:00

Apache 2.2.12, which was released yesterday (see the changes), now supports TLS Server Name Indication (SNI), which is an extension to TLS that makes virtual SSL hosting possible. At the moment, every SSL web site requires a separate IP address and that makes SSL deployment more difficult than it could and should be.

Don't get your hopes up, though. Although the extension was defined in 2003, the support for it is still very limited. For example, Apache's adoption means that, for the first time, SNI is supported in a major web server. (For the record, you could have used SNI previously with mod_gnutls, but mod_gnutls is not widely used yet.) It is only now that the extension begins to have a fighting chance.

However, the server-side support is not the main problem, although we are yet going to see it IIS. The lack of client-side SNI support holds it back. Although Internet Explorer supports SNI since 7.0, the fact that it does so only on Vista and more recent versions, means that we will have to wait many years for SNI to become practical.

Apache Security Model

2009-02-18T00:00:00+00:00

The tough part of securing Apache (or anything else, for that matter) is knowing what you need to defend from. Although my book (Apache Security) enumerates the threats, you need to read through hundreds of pages to learn about them, and even then it may be difficult to remember them as you need them. I've wanted for a long time to make this process easier and now, finally, here it is: the Apache Security Model:

Apache-Security-Model.png

At this time the model is only a draft, but I will polish it in the coming months. It will eventually make an important addition to the second edition of Apache Security.

Apache process infection

2007-06-27T00:00:00+01:00

A very interesting research paper titled Apache Prefork MPM Vulnerabilities [PDF] was released a few days ago, as you can see in the corresponding Bugtraq post. The paper describes, in detail, the dangers of allowing third-parties to run code under the same account as the Apache web server. This normally happens when dynamic content is produced using Apache modules (e.g. PHP) or when CGI scripts are configured to run without suEXEC. This topic itself is not new. You will find several articles on runtime process infection following this Google search link. I warn about this problem throughout my book and especially in Chapter 6, which is dedicated to those situations when more than one party is using the one Apache installation. However, it is one thing to know that something is possible and another to demonstrate, step by step, how it is done. Another interesting finding resulting from this paper is that it is possible to send a SIGUSR1 signal, as root, to any process on the system instead of just to Apache children processes. This is an issue that will have to be fixed in one of the future versions of Apache.

This problem with running code as the same identity as the web server is well understood (and has been for years) among the advanced Apache users. The solution is to always execute CGI scripts through suEXEC and to never allow third parties access to any of the modules. The real problem is that, as with any other product, there are few people who understand Apache inside out (and they can protect themselves) but there also those who are using the technology but do not have the luxury of learning everything there is about it (and there are many legitimate reasons for that).

The solution is obvious. Apache must be safe out of the box! We should dispense with the idea of running things in the same process. Process isolation facilities (either suEXEC or something else) should be installed and running by default on all installations. We can and should make provisions for those who know what they are doing to shoot themselves in the foot, of course. But the only reason to attempt to run things in the same process is performance and I suspect, in this day and age, virtually all users will be happy with the performance of their web server doing things in a secure manner.

Apache reverse proxy memory consumption observations

2006-08-14T00:00:00+01:00

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing:

Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.
The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.

Apache Security in Japanese!

2006-06-26T00:00:00+01:00

My book was translated to Japanese and published by O'Reilly Japan! This is, apparently, old news, as they did it back in 2005, but I only found out about it from the three-monthly royalties statement I received in April.

While we are on the subject of writing, I am starting to get the itch again. There are two or three topics I would like to explore further. Topics such as web application firewalls and ModSecurity, web application security, and application security patterns. On the other hand, I have a few compelling reasons against writing another book:

It takes a lot of time (time better spent building Thinking Stone into a stronger business).
It's lonesome (but this can be dealt with by finding someone to co-author the book with me.
My hands and arms haven't fully recovered from the first book. (This one is the most compelling reason of all - I barely managed to finish Apache Security in the first place. If you are using keyboard extensively make sure to read about RSI and always keep Workrave active.)

Apache suEXEC chroot patch

2006-06-23T00:00:00+01:00

I was recently involved with a project where we needed to configure an Apache server that was intended to run multiple web sites/applications. It's a pretty common assignment. To ensure the setup is secure I decided to start by creating a separate user account for each application. This allowed me to correctly configure file permissions to allow Apache to serve the static files directly. To take care of the dynamic content, I configured suExec to execute each application's scripts under its own account. (In case you are wondering, this particular server is fast enough to run the scripts as CGIs. But if process creation becomes a bottleneck we can always seamlessly switch to FastCGI to avoid the performance penalty. Nothing to worry about, then.)

SuExec is a great tool but I'd love it to be capable of jailing (via the chroot system call) the binaries it executes. However, this feature is not present in the stock version. Having been responsible for the internal chroot feature of ModSecurity, I think I have a pretty good idea of why this is the case: unless you know what you're doing it's pretty easy to break applications with chroot. And if that happens you are going to ask for help... from those that created the feature, right? Of course! As it turns out, chrooting is notoriously difficult to debug remotely and that's why the developers would much rather not deal with it.

But, if do you know your way around feel free to use my suExec chroot patch, which I have just added to the Apache Tools project. But, please, don't write to me if it's not working as you are expecting :)

Jailing Apache on Windows

2006-06-13T00:00:00+01:00

Yury Zaytsev wrote to me recently to tell me about his experiences in jailing Apache on Windows. Although, strictly speaking, Windows does not have the chroot system call or an equivalent it is still possible to do a pretty good job restricting its access to the system, as Yury demonstrates. From his email:

All you need is to make a local user, say, called "Apache" (you may even set him a password, don't think that makes any sense, but anyway) and deny him local and network login via group policies. Then you need to explicitly deny this user any access to the local drives (deny just everything: dir listing, read, write, modify etc), that's done via Properties - Security. Now any process spawned with "Apache"'s rights won't be able even to LIST the directories.
Now you've got to grant it the read/list folders access to the Apache Software folder (done via folder properties -> security) and write access to the PID file and log files (hopefully it doesn't need anything else).

And the last thing to do is to edit Apache's system service: you should change it's privileges via My computer - Manage workstation - Services - Apache from System service to "Apache" user (it might prompt you for the password if you've set any).

Reboot, check that Apache process is running as Apache user via the Task Manager, make sure everything is working fine and you're done.

This has also an important positive impact on the scripts security: now even if one manages to hack your poorly coded PHP/Perl script, since PHP/Perl is run via SAPI/mod_perl it couldn't list folders above Apache's root and even change any files you haven't allowed it explicitly inside Apache's root.

As you may see from above, my method is a complete rip off the Unix chroot (and chmod, he-he) technology. It's primitive and efficient (..er how efficient a Windows server can be comparing to Unixes?) Anyway it really saved my butt several times the script kiddies managed to exploit vulnerable PHP scripts.

Apache Security one year after

2006-03-27T00:00:00+01:00

It's been exactly one year since my book, Apache Security, was published. I was very glad to learn Amazon.com is now enabling book authors to talk to their audience. It is unfortunate this feature did not exist at the time - I would have loved the opportunity to point those looking at this page to the book's web site - www.apachesecurity.net.

I have always believed publication is just a first step in the life of a book (a long step in my case, as I spent eight months writing), and that the best stuff comes only after a book has been in use for a year or two. Let's face it, we (the authors) don't know nearly as much as our collective readership does. Therefore I invite you, the reader, to send me your feedback and make the second edition of Apache Security much better!

Apache programming book on the way!

2005-09-12T00:00:00+01:00

I have been involved with Apache programming for several years now. During this time I've been following the main Apache development list and the module programming one. This is how I got to meet Nick Kew, probably the most helpful person on these two lists. (Perhaps on other lists too, but I only follow these two.) Rumours that Nick is writing a book (spread by the author himself) have been circulating for many months now. I am happy to say this is now official; Nick's book, Apache Programming (I am not sure if this is the official title or not) will be published by Prentice Hall in their Open Source Series. Nick has been kind to invite me to help him as a technical reviewer. This is great news for the Apache community! Apache is a fantastic web server but its growth is being slowed down by the lack of proper documentation for programmers. I only wish I had this book a couple of years back when I was starting with ModSecurity!

Apache 2.1.7 Beta released

2005-09-12T00:00:00+01:00

A new beta version of the Apache web server has been released. This release is important because it is a beta version in the 2.1.x development branch, meaning Apache is one step closer to an official 2.2 version. The most important changes are the refactoring of the authentication/authorisation backend, faster and better caching support, and the much improved mod_proxy with support for load balancing.

The PHP chapter from Apache Security available for download

2005-06-13T00:00:00+01:00

I have made the PHP chapter from Apache Security available for free download. When we made the decision to set the installation and configuration chapter free, several months ago, I did not realise this chapter only told one half of the story. Most people need to configure Apache *and* PHP. This is now fixed, and the two chapters together make a valuable resource. My long-term plans are to convert both chapters to DocBook, keep them up-to-date, and publish them as PDF and as HTML. But not yet--the content is still very fresh!

The public life of Apache Security begins

2005-04-26T00:00:00+01:00

My book, Apache Security, is finally out, after a year and a half of hard labour. I began talking to the publisher in the summer of 2003, and began research shortly after. I began writing in March 2004 and finished in December 2004. O'Reilly had it in stock on March 1st 2005, but it only become widely available in late March.

The work itself was great fun. It is a great privilege to be able to explore the boundaries of your own knowledge in detail. I did have periods of despair, of course. But I was prepared for them from reading blog entries of other book writers. I knew how difficult it was going to be. The biggest challenge I had was deciding what to cover, and what to leave out. It was never going to be a book for absolute beginners (officially, it's an "intermediate to advanced" type of book) but I didn't want to write a book that would be understood only by a few people who are already Apache experts. On an another level, it was also not possible to look at the security of the Apache web server in isolation. A book that pretends to provide "everything you need to know to secure your Apache web server" must delve into topics such as networks security, host security, and web application security. As it turned out I had to deal with these questions every single day. It was a struggle to keep the book from growing too much. Initially, the book was supposed to be around 280 pages long. In the end, it grew to over 400 pages.

As my work progressed I began to think more and more about the process. The traditional book writing process restricts the author to his experience, the experience of his immediate peers, and the experience of the technical reviewers. While this may work in some, or even many cases, I came to believe that a piece of technical writing can achieve its best only through the collaboration process with the readers. Now that the book is out, this is exactly what I am looking forward to.

The first step, the book's web site, is already completed. In the following months I plan to put more material from the book online, start adding fresh content, and generally try to engage the public by offering them the stuff they are interested in. The way I see it, my work has just began.

Apache Security cover and beta chapter available!

2005-02-16T00:00:00+00:00

The Apache Security cover will feature a horse, I am happy to say. I knew all along my book was going to be an animal book but the identity of the animal was not known until fairly recently. Now that the animal is known and the tagline ("The Complete Guide to Securing Your Apache Web Server") is sorted I decided to publish the cover for everyone to see. The official launch date is less than two weeks from now so I'll save my "it was a difficult job" speech for then.

A beta chapter, Installation and configuration, is also available for download. A part of me wanted to advertise my knowledge by giving away one of the more exciting chapters. But since properly installing and configuring Apache is very important, in the end I decided to give away the less exciting but probably more useful chapter.