Apache module dumpio doesn’t dump null (zero) bytes
I’ve been using the Apache mod_dumpio module for almost as long as I’ve been blogging to capture all of the data received by my web server. Doing so is expensive but very useful for analyzing attacks. What I did not realize until today is that mod_dumpio drops (i.e., omits) nul (i.e., null or zero) bytes in the input when it writes that data to the Apache error log.
Looking at the lines written by mod_dumpio in the Apache error log you’ll find plenty of strings such as “\x03” but no instances of “\x00“. Even where such sequences would be expected in the incoming data. I confirmed this by hand-crafting a HTTP request that included null bytes. When I sent that request to my server and looked at the error log the mod_dumpio data contained every byte except the zero (null) bytes.
Using Google to search for various phrases such as “apache dumpio drops zero bytes” and “apache dumpio omits zero bytes” and “mod_dumpio skips null bytes” and many other variations resulted in no indication that anyone has ever noticed this problem. What. The. Fuck.
I first noticed this when I decided to decode the content that malware hackers were attempting to send (e.g., upload) to my web server. I had no difficulty decoding the base64 encoded data that was typical for PHP malware. I had zero success decoding the application/zip encoded data typical for an attempt to exploit the WordPress revslider plugin vulnerabilities. This was not entirely surprising given that mod_dumpio reported that the client said it was sending x bytes yet the data logged by mod_dumpio was much smaller than x bytes in length.
Comparing the data from those attacks to a Zip archive I created showed that the mod_dumpio data contained no null bytes. Something highly unlikely, perhaps impossible, for a legitimate Zip archive. As I mentioned earlier I confirmed this by deliberately sending a request that included null (zero) bytes and confirming that mod_dumpio did not log those bytes.
Time for me to look at the mod_dumpio code and see if I can fix it and convince the Apache community to accept my fix.
Shortly after writing the previous sentence I looked at the
dumpio.c file and found this comment in the dumpit function:
/* XXX: Seriously flawed; we do not pay attention to embedded * \0's in the request body, these should be escaped; however, * the logging function already performs a significant amount * of escaping, and so any escaping would be double-escaped. * The coding solution is to throw away the current logic * within ap_log_error, and introduce new vformatter %-escapes * for escaping text, and for binary text (fixed len strings). */
The author of this Apache module knew they were omitting data from the log. I applaud them for documenting the fact. I’ve done the same thing many times in my career as a software programmer. Sometimes there just aren’t enough hours in a day to write perfect code. So I cannot hold the author of the module entirely responsible for the problem.
Searching Google returns references to this module as old as nine years ago. In the intervening time no one has felt the need to fix this issue. And that is something I find inexplicable. Surely many people have noticed this shortcoming of the module and yet none of them fixed the problem. To all of those people I say “shame on you.”
OMFG! The problem is that the
mod_dumpio.c module code calls ap_log_cerror() with a %.*s format. That function passes the format and arguments to apr_vsnprintf which, not surprisingly, treats the string referenced by the %.*s format as a null terminated string; even though it was passed a length argument. Which is to say it stops formatting characters when it sees the first null (zero) byte in the buffer.
The C language is the first one I fell in love with. I also hate it because it makes it too easy for seemingly smart programmers to ignore the difference between strings and binary buffers.
apr_vnsprintf function (and its immediate dependencies) assume a null-terminated string so does apr_escape_echo function that performs the escaping. So simply introducing a new “%?” formatting code is insufficient. I’ve concluded that any patch to the APR library to handle printing, with escapes, fixed length buffers will be rejected. So instead I’m going to modify the mod_dumpio.c module to encode null characters. I’ll do this by co-opting another character (probably 0xFF) and use a prefix encoding scheme.