Ivan Ristić

Apache Security 1ed now available from Feisty Duck

Apache Security was originally published by O'Reilly in 2005, and it was very well received. Soon after publication, it was heralded as the best Apache security resource, according to many. Contrary to my expectations, it also aged very gracefully, which is probably why it continues to be popular. As much as I wanted to release an update, I struggled for years to justify a second edition. When I finally could, it turned out that O'Reilly was not too keen on the idea.

[Note: This blog post contains the entire preface to the digital reprint edition of Apache Security. More information on the book is available from the Feisty Duck's web site.]

That was an opportunity for me to do things differently. As much as I enjoyed working on Apache Security a few years ago, the process was traditional and slow. It was a new digital age and we had all the advanced technology at our fingertips, yet we were still producing books the old-fashioned way. I wanted more. Above all, I wanted the ability to update my books whenever I felt the need. Unable to find a publisher that supported the process I wanted, I started my own publishing company. Feisty Duck, as my wife and I named it, is a publisher of computer books, with special focus on continuous publishing and digital delivery.

We are now releasing what is pretty much the original Apache Security, in digital format only, in order to establish a starting point for the second edition, which will be published by Feisty Duck at some point in the future. The known errors in the book have been fixed. If further errors are discovered, they will be fixed, too, and the digital version will be updated.

You may wonder whether the first edition of Apache Security is still worth paying for. After all, it has been five years since the first edition. Here's what I think:

• The only part of the book that is completely obsolete is the ModSecurity chapter. I have only myself to blame for that, because I completely rewrote ModSecurity itself in 2006. If ModSecurity is what you're after, you should look at my other book, ModSecurity Handbook (Feisty Duck, 2010). You will find more information about it at https://www.feistyduck.com.
• Chapter 10, "Web Application Security," was the best introduction to the topic at the time of the original publication. It remains a good introduction, but there have been many advances and discoveries since it was written. These days, you actually have to read several books to get decent coverage of web application security, and complete coverage is virtually impossible.
• The same can be said for Chapter 11, "Web Security Assessment": it's still good, but it's just not enough any more.
• The rest of the book remains pretty solid. Five years later, some aspects are not entirely accurate, but what is in the book is still very useful. You will find that the general principles of web server security haven't changed at all.

To conclude, Apache Security is still a good book, although it will no longer serve all audiences equally well. To paraphrase a recent Amazon.com reviewer, if you are at the beginner or intermediate levels, it will work for you. If you are an advanced user, it may not. If you are not sure, the best thing to do is decide by looking at the table of contents.

ModSecurity Handbook available for pre-order and early access

ModSecurity Handbook, which I announced a couple of days ago, is now available for pre-order and early digital access. We managed to meet our self-imposed deadline and have everything ready for November 24th, actually.

This book is a big deal, in more ways than one:

1. It took me more than 5 years to gather courage to start writing another book (after Apache Security, which I started writing in 2004).
2. This book is about ModSecurity, a project that is very dear to my heart. It makes me very happy that I will document everything I know about it.
3. I am releasing the book early because I want to interact with the readers while the content is still not finalised. With Apache Security I ended up being terribly unhappy because I was writing in isolation and because I couldn't seek feedback from the readers prior to publication.
4. To publish this book (and all my subsequent books), my wife and I started a publishing company and dealt with all the stuff that publishers have to deal with. The learning curve wasn't very difficult because of my previous experience in publishing, but there was a lot of things to do.
5. This book will be a living book. I intend to keep it up to date at all times, keeping up with the changes in ModSecurity. We've invested a significant amount of time into polishing a single-source publishing system, where the manuscript is kept as XML (DocBook, stored in a Subversion repository) and automatically converted to any of the supported formats (only PDF at the moment, but several forms of PDF, HTML and ePub in the near future). The system allows me to make changes and push updates instantly to all the readers!

The work is far from done, of course. First, I need to finish the book, first of all. Second, we'll have to figure out how to promote it effectively, and I somehow suspect that will be the hardest part. Perhaps, when it's all done, I'll write a blog post called "Adventures in Computer Book Publishing".

Update: The official Reference Manual and Data Formats Guide guide have been added to the book. There's about 230 pages of material right now, with the final count expected to be close to or over 300.

Announcing ModSecurity Handbook

It is a pleasure to announce my next book, ModSecurity Handbook, which features an in-depth coverage of ModSecurity, an open source web application firewall. I am very happy because, finally, ModSecurity will have the documentation it deserves.

The main highlights are the following:

• Step-by-step instructions for those just starting out
• Detailed explanations of the internals, and advanced techniques for seasoned users
• Includes the official ModSecurity Reference Manual and Data Formats Guide
• Available in digital format (PDF, HTML and ePub, although not all straight away) and as paperback (once the first edition is complete)
• Continually updated as ModSecurity evolves (with the updates included with purchase)
• Readers can talk to me to shape the book to work better for them

The complete table of contents is available on the book's web site.

I estimate that the book is about 75% complete. In a week's time (on November 24th) it will be available for early access and pre-order. The idea with the early access is to avoid the problem I experienced with Apache Security -- writing in isolation. This time, I want to engage with my readers before my book is published.

Also, it is pretty important that this book is (and will be) continually updated. I have the entire publishing workflow automated so whenever I make a change to the book, the update is automatically made available to the readers. With this feature, again, I want to avoid the painful experience that I had with Apache Security, where for years I wanted to provide updates but I couldn't. (Apache Security readers, fear not, the second edition is being worked on.) In the future, I hope to evolve the publishing toolchain to enable readers to make comments straight to the HTML version of the book that is kept online.

TLS Server Name Indication now in Apache

Apache 2.2.12, which was released yesterday (see the changes), now supports TLS Server Name Indication (SNI), which is an extension to TLS that makes virtual SSL hosting possible. At the moment, every SSL web site requires a separate IP address and that makes SSL deployment more difficult than it could and should be.

Don't get your hopes up, though. Although the extension was defined in 2003, the support for it is still very limited. For example, Apache's adoption means that, for the first time, SNI is supported in a major web server. (For the record, you could have used SNI previously with mod_gnutls, but mod_gnutls is not widely used yet.) It is only now that the extension begins to have a fighting chance.

However, the server-side support is not the main problem, although we are yet going to see it IIS. The lack of client-side SNI support holds it back. Although Internet Explorer supports SNI since 7.0, the fact that it does so only on Vista and more recent versions, means that we will have to wait many years for SNI to become practical.

Apache Security Model

The tough part of securing Apache (or anything else, for that matter) is knowing what you need to defend from. Although my book (Apache Security) enumerates the threats, you need to read through hundreds of pages to learn about them, and even then it may be difficult to remember them as you need them. I've wanted for a long time to make this process easier and now, finally, here it is: the Apache Security Model:

• Apache-Security-Model.png

At this time the model is only a draft, but I will polish it in the coming months. It will eventually make an important addition to the second edition of Apache Security.

Apache process infection

A very interesting research paper titled Apache Prefork MPM Vulnerabilities [PDF] was released a few days ago, as you can see in the corresponding Bugtraq post. The paper describes, in detail, the dangers of allowing third-parties to run code under the same account as the Apache web server. This normally happens when dynamic content is produced using Apache modules (e.g. PHP) or when CGI scripts are configured to run without suEXEC. This topic itself is not new. You will find several articles on runtime process infection following this Google search link. I warn about this problem throughout my book and especially in Chapter 6, which is dedicated to those situations when more than one party is using the one Apache installation. However, it is one thing to know that something is possible and another to demonstrate, step by step, how it is done. Another interesting finding resulting from this paper is that it is possible to send a SIGUSR1 signal, as root, to any process on the system instead of just to Apache children processes. This is an issue that will have to be fixed in one of the future versions of Apache.

This problem with running code as the same identity as the web server is well understood (and has been for years) among the advanced Apache users. The solution is to always execute CGI scripts through suEXEC and to never allow third parties access to any of the modules. The real problem is that, as with any other product, there are few people who understand Apache inside out (and they can protect themselves) but there also those who are using the technology but do not have the luxury of learning everything there is about it (and there are many legitimate reasons for that).

The solution is obvious. Apache must be safe out of the box! We should dispense with the idea of running things in the same process. Process isolation facilities (either suEXEC or something else) should be installed and running by default on all installations. We can and should make provisions for those who know what they are doing to shoot themselves in the foot, of course. But the only reason to attempt to run things in the same process is performance and I suspect, in this day and age, virtually all users will be happy with the performance of their web server doing things in a secure manner.

Apache reverse proxy memory consumption observations

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing:

1. Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.
2. The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using  MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.

Apache Security in Japanese!

My book was translated to Japanese and published by O'Reilly Japan! This is, apparently, old news, as they did it back in 2005, but I only found out about it from the three-monthly royalties statement I received in April.

While we are on the subject of writing, I am starting to get the itch again. There are two or three topics I would like to explore further. Topics such as web application firewalls and ModSecurity, web application security, and application security patterns. On the other hand, I have a few compelling reasons against writing another book:

1. It takes a lot of time (time better spent building Thinking Stone into a stronger business).
2. It's lonesome (but this can be dealt with by finding someone to co-author the book with me.
3. My hands and arms haven't fully recovered from the first book. (This one is the most compelling reason of all - I barely managed to finish Apache Security in the first place. If you are using keyboard extensively make sure to read about RSI and always keep Workrave active.)

Apache suEXEC chroot patch

I was recently involved with a project where we needed to configure an Apache server that was intended to run multiple web sites/applications. It's a pretty common assignment. To ensure the setup is secure I decided to start by creating a separate user account for each application. This allowed me to correctly configure file permissions to allow Apache to serve the static files directly. To take care of the dynamic content, I configured suExec to execute each application's scripts under its own account. (In case you are wondering, this particular server is fast enough to run the scripts as CGIs. But if process creation becomes a bottleneck we can always seamlessly switch to FastCGI to avoid the performance penalty. Nothing to worry about, then.)

SuExec is a great tool but I'd love it to be capable of jailing (via the chroot system call) the binaries it executes. However, this feature is not present in the stock version. Having been responsible for the internal chroot feature of ModSecurity, I think I have a pretty good idea of why this is the case: unless you know what you're doing it's pretty easy to break applications with chroot. And if that happens you are going to ask for help... from those that created the feature, right? Of course! As it turns out, chrooting is notoriously difficult to debug remotely and that's why the developers would much rather not deal with it.

But, if do you know your way around feel free to use my suExec chroot patch, which I have just added to the Apache Tools project. But, please, don't write to me if it's not working as you are expecting :)

Apache Security one year after

It's been exactly one year since my book, Apache Security, was published. I was very glad to learn Amazon.com is now enabling book authors to talk to their audience. It is unfortunate this feature did not exist at the time - I would have loved the opportunity to point those looking at this page to the book's web site - www.apachesecurity.net.

I have always believed publication is just a first step in the life of a book (a long step in my case, as I spent eight months writing), and that the best stuff comes only after a book has been in use for a year or two. Let's face it, we (the authors) don't know nearly as much as our collective readership does. Therefore I invite you, the reader, to send me your feedback and make the second edition of Apache Security much better!