Ivan Ristić

HTTP parser for intrusion detection and web application firewalls

For a couple of months now I've been working on a new HTTP parser (library), which I am designing for use in intrusion detection systems and web application firewalls. I suppose that HTTP parser is not really an adequate name for this library, because it sounds narrow in scope. In truth, the library will cover all the protocols and encodings used in web applications.

The first user of the parser will be the Open Information Security Foundation (OISF), which is currently building a new IDS from scratch (first release expected on December 31st). The parser itself is going to be released under an open source licence and supported long term.

The biggest challenge with a parser like that is the desire to support an entirely passive mode. Whereas normal parsers are free to interpret the input stream in any way they're pleased for as long they appear to get the job done, a passive parser must be able to decipher traffic intended for multiple web servers, and thus also needs to be aware of the quirks in their processing. Also, without the ability to terminate traffic, opportunities for evasion are rife. The really interesting part of this project is figuring out all the possible ways to evade the parser. I think this is the first time that I will have the time to think like an attacker for as long as I need to do the job properly. I am currently experimenting with the idea of parser personalities, whereas the user is allow to tweak exactly how the parser behaves on per-connection basis. This approach makes it possible to use one set of rules for an Apache web server, and another for an IIS web server.

For the first release of the parser to goal is to be able to parse HTTP streams reliably. In the subsequent versions I will work in the parser's security properties (such as the ability to see through evasion attacks).

A couple of weeks ago, at DeepSec in Vienna, I gave a lightning talk about my work. Matt Jonkman kindly allowed me to use some of the time of his own talk. I am attaching the slides here:

• Download Ivan Ristic - The Challenges of HTTP Intrusion Detection (PDF)

Announcing ModSecurity Handbook

It is a pleasure to announce my next book, ModSecurity Handbook, which features an in-depth coverage of ModSecurity, an open source web application firewall. I am very happy because, finally, ModSecurity will have the documentation it deserves.

The main highlights are the following:

• Step-by-step instructions for those just starting out
• Detailed explanations of the internals, and advanced techniques for seasoned users
• Includes the official ModSecurity Reference Manual and Data Formats Guide
• Available in digital format (PDF, HTML and ePub, although not all straight away) and as paperback (once the first edition is complete)
• Continually updated as ModSecurity evolves (with the updates included with purchase)
• Readers can talk to me to shape the book to work better for them

The complete table of contents is available on the book's web site.

I estimate that the book is about 75% complete. In a week's time (on November 24th) it will be available for early access and pre-order. The idea with the early access is to avoid the problem I experienced with Apache Security -- writing in isolation. This time, I want to engage with my readers before my book is published.

Also, it is pretty important that this book is (and will be) continually updated. I have the entire publishing workflow automated so whenever I make a change to the book, the update is automatically made available to the readers. With this feature, again, I want to avoid the painful experience that I had with Apache Security, where for years I wanted to provide updates but I couldn't. (Apache Security readers, fear not, the second edition is being worked on.) In the future, I hope to evolve the publishing toolchain to enable readers to make comments straight to the HTML version of the book that is kept online.

The key to successful WAF deployment is getting the ownership right

While pruning my bookmarks, I stumbled across this fantastic post to the Secure Coding mailing list, from almost two years ago:

• Re: [SC-L] darkreading: PCI, web app firewalls, and software security

The first part of the email gives a great insight into PCI. The second part of the email is about web application firewalls, and it's equally interesting, exciting even. This sort of email is extremely important to help understand why web application firewalls aren't nowhere near the popularity as of network-level controls. The main issue is ownership, and almost everyone struggles with it.

Anonymous application security practitioner writes:

7) Web Application Firewalls (WAF) are alluded to in requirement 6.6 of
 PCI DSS v1.1 as an "application layer firewall":

 6.6 Ensure that all web-facing applications are protected
  against known attacks by applying either of the following
  methods:
  - Having all custom application code reviewed for common
  vulnerabilities by an organization that specializes in
  application security
  - Installing an application layer firewall in front of
  web-facing applications.
 Note: This method is considered a best practice until June 30, 2008,
 after which it becomes a requirement.

 Notice that WAF is not required per se, but the alternative of inspecting
 all application code by an organization that specialized in application
 security is very seldom feasible. In fact, InfoSec first approached our
 team to do code inspections, but when we heard that there were more than
 1M LOC, we told them that this was impossible given our current team size.

 Code inspection *might* work for some small company who exclusively uses
 some custom code using CC info with something like a small, custom
 a shopping cart application, but it is not going to be economical for
 most medium to large companies.

 I also think that there is going to be significant push-back by
 companies where PCI audits are required. I wouldn't be surprised if that
 date gets pushed out to Jan, 2009. On the positive side at least, the
 PCI auditor who was interacting with our company said that we could
 configure an Apache proxy together with Ivan Ristic's "mod_security"
 and that would qualify as a legitimate WAF to satisfy section 6.6.
 So at least we can experiment on-the-cheap. (Of course YMMV at other
 companies; check with your assigned PCI auditor to see what WAF is
 right for you! ;-)

8) I personally have mixed feelings about WAF. Specifically, I have some
 anxiety about that app dev teams will start relying on the WAF to
 filter out the PCI auditors pen testing attempts rather than on
 correcting it within the application itself. I don't have too much of a
 problem of using a WAF to address DoS type attacks (6.5.9). (IMO, DoS
 attacks are often very difficult to handle within the application itself,
 especially when the app itself is using some J2EE container; addressing
 DoS at the app level almost always requires some significant amount
 of redesign which only complicates the app, making it more likely to
 be vulnerable to other vulnerabilities.) However, inevitably some
 clever 0-day is going to get past most WAFs, especially if they are
 solely signature based. (Those which are also anomaly-based *might* have a
 prayer.)

9) The dirty little secret that no one is talking about are the operational
 issues with WAF. Our company discussed possible use with WAF/XML firewalls
 years before PCI DSS made it "popular".

 We never got much beyond initial discussions because we couldn't
 identify any existing personnel within our company who had all the
 required skills to troubleshoot it AND would be willing to do so.
 (Think additional "pager duty".) We talked with stakeholders from
 InfoSec, our company's network engineering team who manages our border
 firewalls and routers, some *nix and Windows OS system administration
 teams, and amongst our team. I think there might be only 2, possibly
 3, people who have sufficient skills to operate and troubleshoot a
 WAF, and all of those people are smart enough to know that it would be
 a thankless job and they aren't really looking for additional opportunities
 to carry pagers. Chances are, we would have to probably hire a few people
 fully dedicated to this, but even then, there is the question where would
 they fit organizationally? One possible option not explored--because we had
 no stakeholders represented who could finance such a thing--would be to
 outsource the management of WAF to some managed security company such
 as BT Counterpane, ISS, etc. Anyhow, it's a big problem and one that
 isn't going to go away.

 At first, intuition would suggest at first that usual FW teams would be best
 suited (and that indeed is what most WAF vendors suggest), but for the most
 part, the usual FW teams' understanding of attacks at layer 7 is often
 very limited.

 If you or anyone you know ever comes up with a solution on how to address
 this particular issue, please let me know. I think it is a show stopper.

PCI Council clarifies Requirement 6.6, ends ambiguities

The PCI Council has just published the clarifications for the Requirement 6.6 of the PCI DSS 1.1 standard. The 8-page document dated April 15th (but not yet released in an electronic form, which is expected to happen tomorrow, on the 18th) significantly expands on the 7 lines of text of the original requirement. [Update (Apr 23): the document is now available for download.]

Overall, the clarification document comes across as very useful and well balanced. It basically lays down the options for the organisations looking for compliance, saying they can choose how to handle 6.6 for as long as they do it properly and stay within the spirit of the requirement.

Here are the highlights:

• Vendors of web vulnerability scanners and source code analysis tools are going to benefit from the clarification, because both are now explicitly mentioned as offering an adequate approach.
• Overall, I think the Council has gone to a great length to discuss how they don't want to see the automated tools (of any sort) used to just tick a few boxes away. Throughout the document they emphasize the importance of doing the job properly, and by qualified personnel. That is very encouraging.
• It is now clear that it is all right for an internal resource to perform a scan and/or a review, for as long it is not within the same organisational unit as those in charge of developing and maintaining the application.
• More than half of the document discusses web application firewalls, but doesn't really say much new, except to those who are not familiar with the technology. Although there is plenty of content—some of which is very useful—there is not enough space to cover everything one needs to know about WAFs. I think a better approach would have been to focus on what is expected of a WAF to do in the PCI role, leaving to other efforts (such as WAFEC, which is referenced in the document) to discuss what WAFs are and what they can do.
• By far the best sentence in the web application firewall section demands proper use of the technology, not mere installation: "Note that compliance is not assured by merely implementing a product with the capabilities described in this paper. Proper positioning, configuration, administration, and monitoring are also key aspects of a compliant solution." Well said. I would write it in the document in big bold letters.
• I also very much liked the mentioning of the fact that there are many products out there advertised as containing application firewall functionality, but that the organisation needs to ensure they are doing what is required from them—actually making sense of the application-layer traffic.

Overall, the document is a significant step forward. It expands what was a brief mention of several very diverse technologies into a solid recommendation how they are to be used to achieve compliance.

Is PCI 6.6 good for web application firewalls?

PCI requirement 6.6, which endorses web application firewalls, raises the profile of this technology but leaves a lot to be desired. Requirement 6.6 is a part of Section 6, which deals with development and maintenance of systems and applications. Sections 6.1 through 6.5 are all sound, dealing with issues such as patching, change control and secure development practices. At a glance, Requirement 6.6 seems sound too:

"6.6 Ensure that all web-facing applications are protected against known attacks by applying either of the following methods:

• Having all custom application code reviewed for common vulnerabilities by an organization that specializes in application security
• Installing an application layer firewall in front of web-facing applications.

Note: This method is considered a best practice until June 30, 2008, after which it becomes a requirement. "

One issue with the text is that it puts two very different techniques against each other. Organisations are going to choose only one technique to achieve compliance, where, in reality, they should be using both. Looking past that, the wording works very well for web application firewalls, in the sense that most organisations are likely to choose to deploy a WAF rather than go through a very long and very expensive process of code review.

And that is exactly what causes me to worry: organisations looking for PCI certification are going to choose web application firewalls because they are the less ugly choice. Not because of the merits, of which there are many.

As someone making a living from web application firewalls, I should probably be more optimistic about the whole affair. After all, there is no doubt the level of interest in web application firewalls is rising, in no small part due to the PCI standard. The sales are increasing too. And I am sure many organisations will attempt to use the products they've purchased; some of them will like them. PCI 6.6 may actually lead to a wide adoption of web application firewalls. But it still troubles me that many of the purchases of this remarkable technology are going to be made for the wrong reasons.

At the moment the addition of web application firewalls to the PCI standard looks like an afterthought. There is a slight danger that the products will be bought and installed, but not actively used. The PCI standards needs to give WAFs at least the same treatment it currently gives to intrusion detection systems: prescribe not only the installation, but a continuous use, including various options such as blocking, detection-only deployment, virtual patching and the use of WAFs for traffic logging and monitoring.

Tide is turning for web application firewalls

There is a long-running tradition in the web application firewall space; every year we say: "This year is going to be the one when web application firewalls take off!" So far, every year turned out to be a bit of a disappointment in this respect. This year feels different, and I am not saying this because it's a tradition to do so. Recent months have seen a steady and significant rise in the interest in and the recognition of web application firewalls. But why is it taking so long?

Having been involved with the industry for many years, I come up with many valid theories to explain the apparent slow adoption of web application firewalls. Here are some of them: 

• It's a brand new type of product that requires effort to learn how to use. Articles, books and papers need to be written, conference talks need to be scheduled and best practices need to be established. We need a critical mass of people with access to the technology in order for discussions to take place and for users to start to be comfortable.
• Network security people are the likely ones to be tasked to deal with application security. To deploy a WAF one needs at least a minimal understanding of application security, but to achieve this, in a field where attacks are still evolving at a rapid pace, is not easy.
• Many organisations are yet to assign people to deal with application security full-time, let alone web application firewalls.
• It is often not clear who is supposed to manage the technology. Does it fall under network security or application development? Or should we assign it to the application security team instead (where it exists)? This decision is made more difficult by the fact that some web application firewalls can be deployed inline (e.g. as a bridge or a reverse proxy), where they impact performance (not necessarily in a negative way) and create a point of failure.

Above all, the perception seems to have been that web application firewalls are not something we cannot live without. At the same time, the opposite is true for network firewalls. This has to do with the differences in risk distribution in network security and application security. In the network application space virtually all organisations run off-the-shelf products on their servers. Once vulnerabilities in these products are exposed exploits are written. These exploits are easy to deploy in an automated and indiscriminate manner, and this sort of thing is happening on a massive scale. The likelihood of being hit by such an exploit is very high, although the damage coming from such an attack might not be. When you add to this the fact workstations (whose numbers by far outweigh those of the servers) are also targets, it becomes clear why firewalls are viewed as essential.

In the application security world most attacks are still carried out by hand. One reason for this is that people haven't started automating the attacks yet (but this is changing, as demonstrated by recent automated SQL Injection attacks); the other that most web applications, unlike network security products, are custom-developed and thus require manual exploitation. The net result is that some organisations are hit by application security problems and some others aren't, although most are equally insecure. Lack of mass-scale exploitation is contributing to the feeling there is still more time to act.

This, of course, is an illusion. Organisations without web application firewalls are playing a game not unlike that of Russian roulette, hoping they won't be affected. But in this game you get hit—eventually. It's just a matter of time. We are seeing the increased interest now because people are starting to get fed up with web application security issues appearing left and right. Every new day sees a new type of problem discovered. Every day we hear of a new massive attack with damages running into millions. (The Web Hacking Incidents Database project is particularly good at documenting these.) People are waking up to the fact that addressing their problems before attacks take place is going to be much less painful (and far less costly) than doing the same afterwards.

In other words, and to simplify greatly, we haven't seen mass adoption of web application firewalls so far because that market was too young. The time was not right. But it will be right this year. I think.