A few weeks back I wrote about a very unusual bug that returned 500 errors for routes that should have been valid. Recently my team and I, to our great relief, discovered the cause of this bug. Of course, that wasn't before we ended up yelling "WTF?!" a great many times. A great many times. A New Failing Route Most of the our collective WTFs occurred when we discovered the original problem. As I wrote about in the... Read more >
UPDATE (17 May 2017): We've fixed the stupid bug described in this post! Check it out! Bug hunts can take on many forms. Some are simple cleanup jobs, some take hours and end up being something stupid, and some look rather easy and turn out to be impossible. My group recently encountered a fourth kind, a bug where the solution is found but makes no sense, and we could use some help figuring out why this bug exists. Birth of... Read more >
The problems started small, as they often do. But as we've seen many times before, lots of small problems in quick succession tend to make one big problem. In this case, the problem got big fast. It started off easy enough: read the big report, find the bug, fix it, the usual. Our bug-tracking team located the source of the issue right away, and my team set about trying to work out the fix. We found the data source that... Read more >
Within the span of an hour, it had all gone to hell. The first deployment went rather smoothly. It was a fix to an existing web service, and went out with no problems, or so we thought. Within ten minutes of the deployment, the users started complaining of a minor bug, one that was seemingly omnipresent but didn't really stop them from doing meaningful work. The team which had sent out the deployment immediately set to work figuring out what... Read more >
Elena The flickering streetlamp fought valiantly, but in the end Elena was victorious once again. The green mantis/cricket hybrid that had been resisting her invocations finally gave up the ghost, melting into the virtual ground before materializing as a tiny pixelated icon in the upper-right corner of her phone's screen. The evening had been good to them so far; this was the third such avatar they'd captured. The path of broken streetlamps they'd carved through the neighborhood was a... Read more >
Danny Shadows flank me as I march down Jamestown Avenue toward the short, squat building in the distance. The sun's last few rays are off in the distance, casting a lavender twilight into the sky that would be beautiful if I had the time to admire it. Night will soon blanket the campus, punctured only by the streetlamps and the lights of other students' rooms as they cram for finals. That's where I should be, studying, but I can't focus... Read more >
