Ivan Ristić

IronBee versus ModSecurity

After spending a couple of weeks talking about IronBee to anyone willing to listen, I have assembled a list of commonly asked questions. Not unexpectedly, the question that tops the list is about the difference between ModSecurity and IronBee.

With IronBee we had a luxury of starting a brand new project with a wealth of experience and a clear idea of what we want to achieve long-term. (This is completely the opposite from where I was when I started ModSecurity.) Thus, we were able to look at our goals and choose the best path to reach them. Because so much of our lives were spent with ModSecurity, the first thing we did was look at its successes and limitations, with the idea that we should keep what's good and improve what's not as good. Two not so good things of ModSecurity stuck out: the lack of a community of developers and the fact that ModSecurity runs only in the Apache web server.

To deal with that, we do two core things differently:

• Community focus. We are making IronBee as open as it can be, not only by using a non-viral open source licence (Apache Software License v2), but also by adopting a transparent community-oriented approach to project management and development. We have also designed IronBee to be highly modular, so that adding to it does not have to mean having to understand the entire architecture and operation. Time will tell, but the idea is that giving up tight control will make for a better open source project in the long run.
• Abstracted data acquisition and host-container interaction model. IronBee is built as a framework from ground up, with focus on portability among web servers and a variety of deployment models (embedded, proxy, passive, batch, etc). Hence the universal application security sensor wording. We want you to have access to IronBee no matter what your platform is.

These two things are actually tightly related. For example, we can't succeed with the second goal without first succeeding with the first one. There are so many platforms (potential host containers) out there, so it is not possible for a small team to support all of them. However, by being open and by structuring the code base to make it easy to add new platforms, we create the right conditions for others to port IronBee to other platforms as the need arises.

Apache Security 1ed now available from Feisty Duck

Apache Security was originally published by O'Reilly in 2005, and it was very well received. Soon after publication, it was heralded as the best Apache security resource, according to many. Contrary to my expectations, it also aged very gracefully, which is probably why it continues to be popular. As much as I wanted to release an update, I struggled for years to justify a second edition. When I finally could, it turned out that O'Reilly was not too keen on the idea.

[Note: This blog post contains the entire preface to the digital reprint edition of Apache Security. More information on the book is available from the Feisty Duck's web site.]

That was an opportunity for me to do things differently. As much as I enjoyed working on Apache Security a few years ago, the process was traditional and slow. It was a new digital age and we had all the advanced technology at our fingertips, yet we were still producing books the old-fashioned way. I wanted more. Above all, I wanted the ability to update my books whenever I felt the need. Unable to find a publisher that supported the process I wanted, I started my own publishing company. Feisty Duck, as my wife and I named it, is a publisher of computer books, with special focus on continuous publishing and digital delivery.

We are now releasing what is pretty much the original Apache Security, in digital format only, in order to establish a starting point for the second edition, which will be published by Feisty Duck at some point in the future. The known errors in the book have been fixed. If further errors are discovered, they will be fixed, too, and the digital version will be updated.

You may wonder whether the first edition of Apache Security is still worth paying for. After all, it has been five years since the first edition. Here's what I think:

• The only part of the book that is completely obsolete is the ModSecurity chapter. I have only myself to blame for that, because I completely rewrote ModSecurity itself in 2006. If ModSecurity is what you're after, you should look at my other book, ModSecurity Handbook (Feisty Duck, 2010). You will find more information about it at https://www.feistyduck.com.
• Chapter 10, "Web Application Security," was the best introduction to the topic at the time of the original publication. It remains a good introduction, but there have been many advances and discoveries since it was written. These days, you actually have to read several books to get decent coverage of web application security, and complete coverage is virtually impossible.
• The same can be said for Chapter 11, "Web Security Assessment": it's still good, but it's just not enough any more.
• The rest of the book remains pretty solid. Five years later, some aspects are not entirely accurate, but what is in the book is still very useful. You will find that the general principles of web server security haven't changed at all.

To conclude, Apache Security is still a good book, although it will no longer serve all audiences equally well. To paraphrase a recent Amazon.com reviewer, if you are at the beginner or intermediate levels, it will work for you. If you are an advanced user, it may not. If you are not sure, the best thing to do is decide by looking at the table of contents.

The state of ModSecurity in March 2010 (Part 2)

By now you should have a good understanding of where ModSecurity has been. (If you don't, you should read part 1 of my analysis of the state of ModSecurity.) I will now focus on where ModSecurity is, and where it can go.

The key issue to consider is that of motivation. A major loss for ModSecurity is the fact that the interests of its owner, Breach Security, are not aligned with the interests of ModSecurity itself. (Disclaimer: I left left Breach Security more than a year ago; my judgement is based entirely on the publicly available information, and I may be wrong. Feel free to correct me.) Despite the misalignment, Breach Security continues to spend significant resources on ModSecurity. Brian Rectanus and Ryan Barnett continue to spend their time improving ModSecurity and Core Rules and support the community.

Let's look at the individual components:

• ModSecurity. ModSecurity is developed by Brian Rectanus and myself. Brian, employed by Breach Security, does most of the work. I contribute only from time to time, mostly when I have an itch to scratch.

  The last major version of ModSecurity, 2.5, was released in February 2008, more than two years ago. At present there is no sign of work on the version 3.0. The good news is that ModSecurity is very well maintained. The current version is 2.5.12 (and there's 2.5.13 in the trunk), which means that there's been a new version released every 2 months. That's very good. The analysis of the CHANGES file shows that some of the versions have had substantial improvements, especially the most recent ones. In addition, from personal experience I can tell you that the work done on ModSecurity (by Brian Rectanus) is more extensive than the CHANGES file make it appear. ModSecurity is under active development, even if there's currently no sign of any major improvements.

• Rules. The second generation of ModSecurity Core Rules was released in mid-2009 as a project of OWASP. Ryan Barnett, who works for Breach Security, leads the Core Rules project. The Core Rules is a substantial project, and it's very encouraging to see it move forward. One of the biggest weaknesses of ModSecurity is the lack of an easy to use rule set.
• Management (GUI). On the management front, things have been quite boring for very long. ModSecurity Community Console, first released in 2006, remains free for up to three sensors. With a bit of tweaking, the console can handle thousands of alerts and it provides adequate monitoring facilities, but there's no management. There's a promising new development from Christian Bockermann: his AuditConsole implements similar functionality to that of ModSecurity Community Console, but has no sensor limits.
• Documentation. Now that my own ModSecurity Handbook has been released, I can, for the first time ever, honestly say that ModSecurity has adequate documentation. My book documents pretty much everything I know about ModSecurity. Furthermore, I've pledged to keep the book up-to-date as ModSecurity evolves. Having good documentation is always very important, but especially so for ModSecurity, which is a very low-level tool.
• Community. My personal perception is that the interest in ModSecurity is on the rise, although a look at the number of messages per month (in the list archives) does not seem to support my view. Still, there's almost 900 people subscribed to the mailing list, and that's not an insignificant number.

So now you know the facts, but what do they tell us? Here's my analysis:

• ModSecurity is a mature product that generally does what it wants to do, and it does it well. There are currently no major features in the works, but there's an ongoing effort to improve the features that are already there. (Although I no longer influence the development, the current approach is consistent with my own philosophy, which is about quality rather than quantity.)
• The primary use cases for ModSecurity are logging, real-time monitoring, access control, and virtual patching. If you need any of those, you need not look any further, because ModSecurity supports them all.
• Because of its low-level nature, ModSecurity is useful to a small number of expert users, but now that the extensive documentation is available, the hope is that it will can be useful to a wider group.
• The major missing features are learning (automated policy generation), easy-to-use rules and management GUIs. I suspect that a well-maintained fairly simple rule set that promises to deal with known worms and exploits could be the killer application for ModSecurity.

The state of ModSecurity in March 2010 (Part 1)

Last night, during the recording of an OWASP Podcast episode, Jim Manico asked me what the state of ModSecurity was. The question was so simple and straightforward, yet it remained with me for long after the recording. Indeed, what is the state of ModSecurity?

To understand where ModSecurity is today you need to understand where it's been. In today's post I will look back at the history of ModSecurity. In my next post I will cover the current state of affairs.

I started to work on ModSecurity in 2002. Initially, it was only a hobby, but in 2004 I started to work on it full time. ModSecurity 2.x, a complete rewrite, came out in 2006, and was a great step forward. In that same year I sold my business (and ModSecurity with it) to Breach Security. (For those interested, here's the blog post I wrote at the time.) By the time Breach Security approached me I was getting seriously frustrated with the slow pace of development. I was working on my own, developing ModSecurity and supporting the community at the same time. I had so many ideas, but there was only so much time I could do alone.

In the months following the acquisition we formed the ModSecurity team, consisting of myself, Ofer Shezaf (who was already at Breach Security), and Brian Rectanus and Ryan Barnett (who were new hires). In retrospective, I don't think we could have assembled a better team. Breach Security kept ModSecurity open, as they had promised, and the hard work of the team greatly improved the quality of the ModSecurity package (the code, documentation, community aspects, and rules). ModSecurity reached maturity, which was further reinforced with the release of 2.5 in 2008.

Ultimately, however, the business interests of Breach Security did not align with the interests of ModSecurity. The team remained in place, but, over time, we found ourselves spending more and more time on other things. In late 2008, after several years of working very hard and having little life outside work, I found myself very tired and decided to leave Breach Security. Above all, I wanted do something else with my life. My unhappiness with the pace of ModSecurity certainly influenced my decision to leave, but it was not the deciding factor.

Whenever a business is acquired and the founder leaves, the inevitable question comes to mind: did he leave because of an internal disagreement? I didn't, and I remain in good relations with everyone at Breach Security. It was a pleasure to work with them -- I learned so much. Sure, the acquisition could have worked out better for ModSecurity, but I can say the same for many other things in my life, and so can you. The acquisition did a lot of good for ModSecurity and the net result is overwhelmingly positive. Breach Security gave so much to ModSecurity, and continues to do so.

ModSecurity Handbook in print

Well, now it is official. Feisty Duck's first book, ModSecurity Handbook, is in print as of March 15, 2010.

ModSecurity Handbook shipping soon!

It's been an adventurous journey, but we are nearing a major milestone: the official publication of the first book published by our publishing company, Feisty Duck! We've just received a batch of ModSecurity Handbook paperbacks and we're enjoying them in all their glory. Two further batches are on their way to our warehouses (one in the US and one in the UK), from where they will be shipped to early adopters. (If you're one of the early adopters, you will soon get an email from us with more information.)

Our work is nowhere near the end, however, because now we need to focus on reaching out to the book's audience to inform them that the book exists.

If you haven't purchased the book yet, now would be a very good time: because the official publication date is the 15th, we'll be maintaining the pre-order discount for a little while longer. You only have about 4-5 days to take advantage of the 25% discount. Buy now!

Programming in Lua 2ed now sold by Feisty Duck (PDF only)

The Feisty Duck book store yesterday increased the number of titles on offer by 100%, adding the digital version of Programming in Lua 2ed, written by Roberto Ierusalimschy.

If you don't know about Lua, it's a very nice embeddable scripting language, with low memory consumption, very fast interpreter, and even faster just-in-time compiler. I loved it so much I added it to ModSecurity, and it is now possible to write rules in a proper programming language. It's great for those times when you have complex requirements. I am seeing Lua slowly but surely taking over the open source world (when embedding and fast and reliable operation is required). It's already in ModSecurity, Snort 3.x is using it, and in the future it will be part of Apache too.

The book itself is very good too, with a 5-star score in Amazon.com reviews.

ModSecurity Handbook available for pre-order and early access

ModSecurity Handbook, which I announced a couple of days ago, is now available for pre-order and early digital access. We managed to meet our self-imposed deadline and have everything ready for November 24th, actually.

This book is a big deal, in more ways than one:

1. It took me more than 5 years to gather courage to start writing another book (after Apache Security, which I started writing in 2004).
2. This book is about ModSecurity, a project that is very dear to my heart. It makes me very happy that I will document everything I know about it.
3. I am releasing the book early because I want to interact with the readers while the content is still not finalised. With Apache Security I ended up being terribly unhappy because I was writing in isolation and because I couldn't seek feedback from the readers prior to publication.
4. To publish this book (and all my subsequent books), my wife and I started a publishing company and dealt with all the stuff that publishers have to deal with. The learning curve wasn't very difficult because of my previous experience in publishing, but there was a lot of things to do.
5. This book will be a living book. I intend to keep it up to date at all times, keeping up with the changes in ModSecurity. We've invested a significant amount of time into polishing a single-source publishing system, where the manuscript is kept as XML (DocBook, stored in a Subversion repository) and automatically converted to any of the supported formats (only PDF at the moment, but several forms of PDF, HTML and ePub in the near future). The system allows me to make changes and push updates instantly to all the readers!

The work is far from done, of course. First, I need to finish the book, first of all. Second, we'll have to figure out how to promote it effectively, and I somehow suspect that will be the hardest part. Perhaps, when it's all done, I'll write a blog post called "Adventures in Computer Book Publishing".

Update: The official Reference Manual and Data Formats Guide guide have been added to the book. There's about 230 pages of material right now, with the final count expected to be close to or over 300.

Announcing ModSecurity Handbook

It is a pleasure to announce my next book, ModSecurity Handbook, which features an in-depth coverage of ModSecurity, an open source web application firewall. I am very happy because, finally, ModSecurity will have the documentation it deserves.

The main highlights are the following:

• Step-by-step instructions for those just starting out
• Detailed explanations of the internals, and advanced techniques for seasoned users
• Includes the official ModSecurity Reference Manual and Data Formats Guide
• Available in digital format (PDF, HTML and ePub, although not all straight away) and as paperback (once the first edition is complete)
• Continually updated as ModSecurity evolves (with the updates included with purchase)
• Readers can talk to me to shape the book to work better for them

The complete table of contents is available on the book's web site.

I estimate that the book is about 75% complete. In a week's time (on November 24th) it will be available for early access and pre-order. The idea with the early access is to avoid the problem I experienced with Apache Security -- writing in isolation. This time, I want to engage with my readers before my book is published.

Also, it is pretty important that this book is (and will be) continually updated. I have the entire publishing workflow automated so whenever I make a change to the book, the update is automatically made available to the readers. With this feature, again, I want to avoid the painful experience that I had with Apache Security, where for years I wanted to provide updates but I couldn't. (Apache Security readers, fear not, the second edition is being worked on.) In the future, I hope to evolve the publishing toolchain to enable readers to make comments straight to the HTML version of the book that is kept online.

Planned usability improvements for ModSecurity 2.6

After leaving Breach Security in February I took a break from ModSecurity. On the one hand, I was tired. On the other, I needed time to figure out what role I will have in the project, or if I wanted to have a role at all. Over time my interest in ModSecurity started to come back. Even more interestingly, I started to look at it with a fresh perspective. What would a newcomer make of it? In doing so I came up with a long list of small problems that need fixing. The beauty of my new position (a contributor) is that I no longer need to think where the project is heading next, but can spend as much (or little) time as I like working on the things that I have interest in.

Here's my list:

1. It is not clear what happens when an operator fails. I think a level 1 error message is emitted, but there is no effect on rule processing. This is a small anomaly that needs fixing. I believe the user should be given an option to be strict (block) or relaxed (not block). [MODSEC-12]
2. The variable REQUEST_URI is used directly from the Apache's structures. As a consequence the value may change between phases. ModSecurity should have one robust variable that refers to a path, to avoid small problems. The MODSEC-98 improvement fixed this issue.
3. At the moment we provide a raw REQUEST_URI and expect the users to use transformation functions to deal with evasion. That is inconveniencing our users; we should have REQUEST_URI pre-normalised. And, because we rely on Apache for normalization at the moment, doing normalization ourself would give use better control over the process and increase consistency. (For example, Apache does not always do the right thing when deployed in proxy mode.)
4. The REQUEST_BODY variable contains request bodies, but the variable is only available when the for urlencoded bodies. This is because in most cases it doesn't make any sense to store the entire request body in memory (think file uploads). There's a hack to force a request body into memory (ctl:forceRequestBody), but we need to provide a consistent way to deal with the inspection of request bodies.
5. Phase information should not be allowed in SecDefaultAction. That is something for rules to be concerned about.
6. Transformation options should not be allowed in SecDefaultAction, because transformations need only be configured on the per-rule basis. In fact, the entire situation with how SecDefaultAction works is murky and needs improvement.
7. SecArgumentSeparator should not be a main-only directive.
8. SecComponentSignature should not be a main-only directive.
9. ModSecurity should refuse to accept the rules that use variables in phases when they are not available. Why wait for the users to discover the problem the hard way?
10. There's no reason that non-existent response types are specified with "(null)". We should change that to "null", keep the old variant, but remove it from the documentation. This had been fixed, but I missed it.
11. The performance-tracking information (Stopwatch) -- which was just copied over from ModSecurity 1.9.x -- does not provide useful performance measurement. Done. Implemented Stopwatch2, which is a great improvement.
12. The users have been complaining about an elusive mod_deflate compatibility issue for ages now. It needs fixing. Done (MODSEC-89).
13. Allow main and virtual host configuration and phase 1 rules to be overridden in <Location> configuration contexts. Done (MODSEC-98).
14. Use international-friendly spelling: change "sanitise" to "sanitize". Done (MODSEC-95).
15. Use international-friendly spelling: change "normalise" to "normalize". Done (MODSEC-103).
16. Remove the obsolete PDF UXSS functionality. Done (MODSEC-96).