As any readers of the iContact blog may have learned, MogileFS has become an integral part of our infrastructure at iContact. Rather than store the bodies of messages in our database, we moved them to a quick&dirty storage method in our infrastructure long ago. This method was essentially a cheap WebDAV server and on each STORE command it would write to two backend servers and issue a GET from only one. About a month ago, we migrated most of our messages away from this older, less scalable method to our newer MogileFS backend.

Our MogileFS setup allows the disk space on each web server (normally unutilized) to form a cheap storage node, and make use of space that would otherwise go entirely unused.

On Monday 1/21 the database servers behind MogileFS paged with too many connections, which leads to Mogile going very slowly for a while, and sometimes requiring a restart of some of the nodes.

This database issue cascaded into us asking our Mogile client for item A, but receiving item B in response...

The problem turned out to be at the lowest level, that of the socket communication with Mogile. The request chain asked for item A, but since the database was responding very slowly, the "fread" waiting for the response timed out. Unfortunately, the timeout simply passes back up the chain as a "not found". In the background, the data for the response to the request was buffered by TCP. We handled the error, told our daemon to check later for that item, and proceeded to ask for item B. When doing another "fread" for the response, we received the data from the original request. Since all was well, we continued requesting items and receiving a non-matching item in response.

In order to fix this on our end, we simply made the connection terminate when any of these timeouts occurs. Additionally, we'll be adding application-level matching for these requests and responses (the data we store in Mogile will likely include the key so when we pull it back out, it can be check-summed against the requested key).

However, I wonder if it'd make sense for the MogileFS protocol itself to support this matching. It seems key that this type of problem not affect others, so I encourage you to check your client for such a problem, and I plan to file bugs against those who have the bug we found. We were using an old PEAR module as we sought a client with a non-GPL license for inclusion in our codebase. However, a cursory examination of the PHP extension we were coveting seems it might be subject to the same problem.

Other than this issue (and the large headache it caused), which mostly lies on the client side (unless I get any pickup on the protocol-level support above), MogileFS has proved to be a great asset to our infrastructure and I look forward to seeing it scale to much more data.