Posts involving characters, whether original fiction or otherwise.

A Stupid Bug and a Plea for Help

Bug hunts can take on many forms. Some are simple cleanup jobs, some take hours and end up being something stupid, and some look rather easy and turn out to be impossible. My group recently encountered a fourth kind, a bug where the solution is found but makes no sense, and we could use some help figuring out why this bug exists.

Birth of a (Stupid) Bug

Let me set the problem up. I had an ASP.NET Web API controller that looked something like this:

public class AuthController : ApiController  
    public IHttpActionResult Login(LoginUserRequest request)

    public IHttpActionResult TestOracle()

Naturally, this results in URLs that look like these:

  • /auth/login
  • /auth/testOracle

The bug was that any time I attempted to execute the /auth/login action, I got a response of 500 Internal Server Error. For those of you who might not work in the web world, that status message is about as helpful as a tent in a hurricane; it's a generic error message used when we don't know what actually happened (which is to say, it's used almost all the time). Worse, it didn't matter what inputs I POSTed, I always got the 500 error.

Now, it's not really a secret around here that I hate ORACLE, though sometimes unjustly. So when this bug started rearing its ugly head, I just assumed (apparently not learning from my mistakes) that ORACLE had something to do with it. I extracted the ORACLE part of the /auth/login method into the new /auth/testOracle method, and when that failed too, I thought I'd found the problem.

If you've read this blog for a while, you might see where this is going. You're right, and you're also wrong, just as I was.

You're right because I was making bad assumptions. Those assumptions caused me to waste two weeks blaming the problem on the ORACLE team. When we have problems like this, often it actually is ORACLE causing the problem, but not always. I was succumbing to lazy thinking by not investigating sooner than I did.

You are wrong (and, to be fair, so was I) because the actual problem was not something we could have predicted.

Tracking the Monster

My coworker Nate and I sat down early last week to try to figure this out once and for all. This bug appeared to only happen on our dev environment, not locally, so we would make incremental changes to the API, deploy that via our new CI/CD process, hit it with a known-working call to test that and make the app compile, and then hit the known-bad method. We did this over and over, for an entire morning.

Normally, when we hit this app, it has to compile before it can respond to requests (as do many ASP.NET applications). That means there's a delay of about 5-10 seconds before we get a response to the first request, and no such delay for any subsequent requests.

In my frustration, I made a deployment to the dev server and immediately tried to hit the known-bad method in my browser. The route to that method looks like this:

  • /auth/testOracle

While I was turned away from my monitor in disgust, Nate noticed that the method had responded immediately. There had been no compilation delay, which should not happen, but that wasn't even the worst of it. On a whim, we tried several more method calls in the same route:

  • /auth/login
  • /auth/getuserinfo
  • /auth/changepassword

Each of them had the same problem: they would respond immediately after a new deployment, with no compilation delay. Something very strange was going on here.

We conferred with the server admin group, and they swore up and down that there was nothing wrong with the configuration on this server. We inspected the network traffic, checked error logs, and found nothing. This thing completely stumped us.

As is normal when we go bug-hunting, we immediately tried to determine the commonality between all these failing methods. The only thing we could come up with was the route prefix /auth, which we changed to:

  • /authentication/login
  • /authentication/getuserinfo
  • /authentication/changepassword
  • /authentication/testOracle

And, to our considerable surprise, it worked! All our methods would now execute properly.

But WTF? Why would this only happen on the dev server, not locally? I figured it must be that "auth" was some kind of keyword on this server, and IIS was automatically redirecting any requests beginning with "/auth" to another location it couldn't find.

So, change "auth" to "authentication" and all's well, right? Not quite.

We had a lot of route prefixes in this app. Here's just a few of them:

  • /auth
  • /messages
  • /equipment
  • /photos
  • /userdata

All of them worked, except for /photos. The calls to the methods in that route had the exact same issue as the calls to the /auth route; they responded immediately with a 500 error.

A Plea for Help

So now we have two failing routes, /auth and /photos, and no idea how or why they are failing. Changing them to /authentication and /pictures makes them work, but the problem is still basically magic and I am loathe to leave it alone.

So, I'm asking for help from you, dear readers. Has anyone seen an issue like this before? Do you have any links or documents we could read to figure out why /auth and /photos, of all our 20+ route prefixes, are failing? This issue won't prevent us from finishing this project, but we'd truly be grateful for any assistance.

If you have any ideas, even ones you think we might have already tried, let us know in the comments.

Thanks, and Happy Coding!

"I Don't Trust Anything That We Didn't Build"

The problems started small, as they often do. But as we've seen many times before, lots of small problems in quick succession tend to make one big problem.

In this case, the problem got big fast. It started off easy enough: read the big report, find the bug, fix it, the usual. Our bug-tracking team located the source of the issue right away, and my team set about trying to work out the fix. We found the data source that was causing the issue, and it happened to be a web service owned by another team. We couldn't see into it, we could only see the inputs and outputs; it was essentially a black box to us.

Here's where the bad part starts. Due to a lack of efficient communication on all sides, impending deadlines, frustrated coders and general hesitancy to deal with this particular, ahem, time-intensive project, the actual bug fix took about four days to nail down. Yes, four days; we were just as annoyed as you are, and probably more so.

To make a long story short, the project we were attempting to deal with was:

  • old,
  • slow,
  • in desperate need of a rewrite,
  • using a data source which we had no visibility into (the aforementioned service),
  • not written by anyone currently on the team,
  • still our responsibility to fix AND
  • needed to be fixed right friggin now.

You should read that list and cringe a little for each bullet point. I know I did.

All of those problems put together (plus the fact that it took us four days to figure it out) prompted my manager, normally a well-reasoned, thoughtful individual, to say during our bug post-mortem:

"I'm starting to really loathe this project. It's getting to the point where I don't trust anything that we didn't build."

I have to say, it's hard to blame him for wondering if we shouldn't be using things that were not invented here.

It's incredibly easy for a software development team, even an experienced one like mine, to fall into the comfortable trap of believing that everybody else's code is terrible and their own is awesome. We developers often forget (or, quite possibly, want to forget) that most of the time the bug is in our code, no matter how much we wish that it wasn't.

Do this enough and the untamed wild of other people's code starts to look like an intimidating place. It's safer, easier, to believe that your code is correct and everyone else's is wrong, because that means you don't have to spend time trying to understand what the other person was thinking. Or, just as often, spending time figuring out how you are wrong, something nobody enjoys doing.

I've written before that I believe code should exist for a reason. The difficulty in working with other people's code is that not only are you trying to understand what the code does, you're trying to comprehend the reason why it does that. That's a difficult thing to accomplish in the best of times (efficient communication being a feat that usually fails, except by accident), and when you're approaching a deadline and trying to have a meaningful conversation with the original developer who has his own deadlines and responsibilities to deal with, it can be nigh impossible.

Let me be perfectly honest: there are times I completely understand my manager's frustration. It would be SO much easier if the only code I had to deal with was my own, because then the only stupid person in the equation is me and I can fix that. Dealing with other stupid people is infinitely more frustrating than dealing with your own stupidity.

To be clear, I am not calling my coworkers stupid; they are in fact quite the opposite. But it's tempting to fall back to lazy thinking and believe they are stupid merely because they were dealing with requirements and scenarios that I didn't have time to thoroughly understand. That temptation, to believe that things are stupid because I don't understand them, is something I find myself fighting against on a daily basis. It's an innate human quality, and not unique to programmers or other technical people.

Here is a basic fact of life: people, on the whole, are not stupid. Programmers do not write code for no reason, as the best code is no code at all and if we could have our way there would be no code, ever. But because code needs a reason to exist, it almost certainly had a set of requirements, or scenarios, or something which shaped its current form. Even if those requirements were merely thoughts in the original developer's head, they existed. It is not the fault of that developer that some idiot who saunters up to a laptop and is trying to break her code doesn't understand what said code is meant to do.

But it's easy to think that, isn't it? It's easy, it's simple, it's lazy. When we don't have time or energy to think, really think, the lazy thoughts are what we are left with. Given that programming is an almost-entirely-mental task, accepting the lazy thoughts as fact could even be seen as a reprieve from needing to think critically all day, every day.

Resist the lazy thoughts. Resist the idea that your fellow programmers are stupid, or wrong, or only doing a half-done job. Resist Not Invented Here syndrome. Resist the idea that because someone didn't understand you, they're dumb. Resist all these little thoughts that end up with a conclusion of "those other people are stupid," and instead try to answer "what were they trying to accomplish?" There's nothing wrong with digging a little deeper for a better understanding.

That's what I say to you: resist the lazy thoughts, and dig a little deeper. You will eventually have to trust something you didn't build. If you keep digging, you'll find what you are looking for.

Post image is Digging a hole for the digester from Wikimedia Commons, used under license.

How Do You Fix An Impossible Bug?

Within the span of an hour, it had all gone to hell.

The first deployment went rather smoothly. It was a fix to an existing web service, and went out with no problems, or so we thought. Within ten minutes of the deployment, the users started complaining of a minor bug, one that was seemingly omnipresent but didn't really stop them from doing meaningful work. The team which had sent out the deployment immediately set to work figuring out what was going on.

Unrelated to that deployment and forty minutes later, my team was launching a major change to a web site that was consuming that other team's web service. When our change went out, the bug that the users had been complaining about from the web service deployment vanished, replaced by a major bug that caused them to be unable to do any work at all. Naturally we were a little concerned.

The major bug caused an error message that included this statement:

"Unknown server tag 'asp:ScriptManager'."

Now usually when I see an error message like that, I think the project is missing a reference. But no, the references the project needed did, in fact, exist. The second thing I think is that it was using the incorrect version of .NET (in this case, 2.0). Wrong again, it's using the proper version. So now I'm a bit stumped; I pull one of my developers off his project to work on this, and he and I go to our manager and the web service team to try to hash this out.

It took the five of us about an hour to work out where exactly the problems were. As so often happens with major problems like this, it wasn't one problem but several, all intersecting at one time. They snowballed like so:

  • The previous day, my group attempted to roll out the major change to the web site. The roll out didn't work, and another manager (who had previously owned the project, and unbeknownst to me) immediately copied the application to a different location on the same server. He figured this would solve the problem, as it had before with a different app; it didn't.
  • Before the web service change went out, the users had already been notified to use the new location. Consequently they started complaining about the major error.
  • When the web service change was deployed, a different set of users within the same group complained about the minor bug, as word had not reached them to use the new location.
  • When our web site change went out (to the original location), the users of that site noticed and now complained of a "new" bug at the old location, despite the fact that it was the same bug as the bug at the new location.
  • All of this taken together meaning that the fact that our web site and their web service were related was a coincidence. The fact that the two deployments went out so near to each other had nothing to do with what the actual problem was. Coincidences like this are the worst possible thing that can happen when trying to find a bug.

Got all that?

Ultimately we worked out the problem. Well, really, we stumbled onto it. How we got there was such blind luck that I'm not convinced we actually solved the problem so much as lucked into a solution.

A bit of googling for the error message revealed this StackOverflow answer which states that the reason we get the above error is that a piece of the web.config is missing. Here's what it should be.

        <add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

We had previously confirmed that in our application configuration that line did, in fact, exist. Obviously this could not be the problem. (I apparently forgot about what happened the last time I assumed something was true.) Later, when we started getting more desperate to find the source problem, I had our server team give me a copy of the app running on the production servers. This is what I found:

        <!--<add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>-->

Yes, you read that right. For some damn reason, the offending line of configuration code had been commented out.

And we had no idea how this happened.

We tried examining the config transforms; there was nothing that would result in that line being commented out. We tried looking at the server logs, the build logs, the source control history, anything that might give us a scrap of information as to how the hell this happened. We found nothing.

As you might imagine, this was a little frightening. This line got commented out, and we couldn't reproduce how. How can you fix a bug that should have never occurred in the first place? When we deployed the corrected configuration file, it worked, of course. But in the meantime we had wasted nearly an entire day looking for something that should have been impossible.

But was it impossible, or did we miss something? I'm inclined to believe the latter. One of the things I stress to my team when they come to me with bug reports is the important question of what changed? What changed between when the system worked and when it didn't? Was it business rules, data sources, the build process? If we can determine what changed, the time needed to pinpoint the actual bug shrinks dramatically. In this case, either we simply didn't know what had changed (the most likely scenario) or nothing had changed (the far scarier scenario). Either way, something was off and we couldn't determine what it was.

What was even more worrisome was that there had been a minor bug reported before the major bug showed up, the one that was annoying but not work-stopping. That minor bug was not reproducible now, so it's entirely possible it's still lurking out there, waiting for some unsuspecting user to click the wrong button or enter the wrong date and then blow up the whole system. We don't think it will be that serious, but we also don't really know that it won't. We're stuck in bug-induced purgatory.

That's a terrible feeling. Something went wrong, and you can't figure out how. You know what the solution is, but you don't know why.

I suppose I should just be happy we figured out the problem. I am, sort of. And yet I am forced to conclude that there exists a bug which caused a critical part of our application configuration to be commented out for no discernible reason. A part of me still wonders: how can I find and fix that bug, a bug that should have never existed in the first place?

And what if it happens again?

What about you, dear readers? What are some bugs you found that you couldn't source? How did you fix these "impossible" bugs? Share in the comments!

Happy Coding Debugging!

Post image is 2008 broken computer monitor from Wikimedia, used under license.

The BugCatcher Chronicles #2 - Pine Hills


The flickering streetlamp fought valiantly, but in the end Elena was victorious once again.

The green mantis/cricket hybrid that had been resisting her invocations finally gave up the ghost, melting into the virtual ground before materializing as a tiny pixelated icon in the upper-right corner of her phone's screen. The evening had been good to them so far; this was the third such avatar they'd captured. The path of broken streetlamps they'd carved through the neighborhood was a testament to their success.

"Nice inputs, babe." said a voice, hovering over her left shoulder.

"Thanks, Greg. How's yours going?"

"Not too bad." A quick glance at his screen showed a pink rhinoceros beetle clacking its jaws at him. He spun the input selectors again, tapped the invoke button, and the beetle lost one of its jaws. Startled, it attempted to flee, but a quick invocation finally brought it down. The beetle evaporated and rematerialized, and the garage door at the house across the street started to rise of its own accord.

"Shit. Better move!"

Elena quickly followed Greg's fleeing footsteps, and the two scrambled over the next hill, pausing to catch their breath under one of the pine trees that the neighborhood was famous for. They'd gotten away, like they always did. The setting sun framed the outline of the cookie-cutter houses, melting them into an endless skyline of sameness. Elena wondered how any of the residents could find their own home amongst all these identical facades.

"Woo! Got 'em!" cheered Greg. "We're killing it tonight babe!"

"Yeah we are!" affirmed Elena, pulling her hood back up over her head. It was chilly tonight, and while she didn't feel like going home yet, the cold was started to seep through her jacket.

"Any more nearby?" she asked. "I don't see them."

"There may be another over that hill to the west." said Greg. "I mean, we had good luck there yesterday."

"Cool, let's go."

The two slid their phones back into their pockets, volume turned all the way up so as to hear the rhythmic beeps that signify a nearby bug. These bugs are what Elena and Greg are after; software bugs that exist in the real world that, somehow, this app BugCatcher can find and invoke. Once invoked, Elena and Greg get a new, custom-generated avatar for their collection.

Of course, whatever the bug actually does will also happen, hence the trail of broken streetlights and the now-stuck-open garage door.

Elena slid her left hand into Greg's right. He'd been the one to introduce her to the game and, frankly, it had been a lot of fun. Find a bug, try some inputs, capture the bug, see what it did. Everything they'd captured so far had involved broken streetlamps or sprinkler systems gone haywire or suddenly-loud music systems; in other words, nothing too serious. The worst thing that had happened was that they'd accidentally turned a car on, with no way to turn it off. It hadn't been shifted out of park, so Elena figured it was no big deal; it couldn't go anywhere.

As they crested the hill, Elena got a glimpse of the rest of the neighborhood. It was quite nice, to be sure, but it was also so... dull. Every house was the same, with a big green yard, 2.5 children and a dog. Bland, unoriginal, inoffensive, boring.

It didn't really bother her that they were out here catching bugs, breaking streetlamps and sprinkler systems. After all, if anybody could afford the mild chaos she and Greg were causing, it was these people. Goddamn rich people.

A low drone sounded from Greg's pocket, and Elena's started doing the same. A bug was nearby. As if by magic, Greg's phone appeared in his hand as though it had been there all along. Elena was a bit slower, but in a couple seconds they had located the bug. Apprehension turned to amazement as they beheld the monstrosity before them.

It had the body of a caterpillar, but the clacking mandibles of a fire ant. The thing writhed in an unnatural way, as if trying to shake some invisible bond that held it within the phone. It was huge, almost as big as the house nearby, with giant black eyes that seemingly reflected the virtual light which attempted to pass through them. Elena was amazed at the design quality of this particular avatar; whoever coded this up did a hell of a job.

Greg was quicker, as always, and his first few invocations started flying. Elena joined in on her app, working the opposite way Greg was, just like he'd taught her. Between the two of them, they'd always managed to find the right combination of inputs to invoke the bug and capture the monster before it had a chance to flee.

This one proved difficult. Several rounds went by, and nothing was taking. The caterpillar's writhing continued, and it looked to Elena like it was specifically programmed to come leaping out of her phone if she failed to catch it.

Finally something stuck. The leftmost input glowed with a red outline, and the monster's right side started limping. She'd found something!

"Red 47!" she whispered urgently to Greg. "That's the left input!"

A moment later, he whispered back: "Black 12 is the middle."

All they needed now was the right side. Her fingers flew across the screen as she desperately tried combination after combination, willing the app to give her the right answer so she might have this bug for her collection.

Sustaining hit after hit, the caterpillar monster finally collapsed, dead, and faded away. The pixelated avatar appeared, and despite herself she let out a whoop of joy.

"Nice job, babe," said Greg, though not without a hint of jealousy.

"Hey, it was you who figured out the middle input."

"Yeah, but you brought it down. Make sure to show that to Danny when we get back to campus."

The two started back the way they came. Only a moment later, the trees became illuminated in red and blue, and a police cruiser slammed to a stop right next to the pair of hunters. Another cruiser followed on the heels of the first one, and three offices exited the two vehicles.

"Hands up!" shouted the nearest officer. "Don't move."

Greg glanced at Elena, and she could tell what he was thinking. Should we move? She shook her head, almost imperceptibly, and raised her hands. No point. He reluctantly did the same.

The nearest officer took Greg over toward the nearby tree, cuffed him, and slid him into the back of the first cruiser. Another officer cuffed Elena, and as she was being walked to the second cruiser she belatedly realized that the officer that had cuffed her had also been speaking to her, though she couldn't remember what he had been saying.

I'm being arrested! For what?!

As she took a seat in the back of the cruiser, she wondered how the hell it had come to this. Goddamn rich people, thinking they own the world. It was just some broken streetlamps. We didn't do anything!


Twenty minutes before the police showed up way earlier than she'd expected them to, Georgia Huntsworth sat in an easy chair and stared blankly at the TV. The family insisted that having it on helped poor Frank, but Georgia just couldn't see how. The man was a vegetable, with no hope of waking up anytime soon. He'd have no idea it was even on, much less be able to watch it.

She sighed, and got up to fix another cup of coffee. A stronger one, this time. The gentle whirring of Frank's artificial breathing machine faded into the background as Georgia crossed into the house's kitchen. It would be a long night, but the machines would do most of the work. All she had to do was make sure he was comfortable; at least, as comfortable as she could imagine him being.

She placed the pot back on its mantle and leaned against the countertop. The job wasn't bad, per se, it was just boring. There really wasn't much to do for poor old Frank. She made him comfortable, she kept the TV on, and otherwise she chatted with her coworkers or her daughter or her beloved Zachary. It was a good job, just not very exciting.

The coffee was done, so Georgia poured herself a mug and went back to the easy chair. She glanced at Frank; nothing had changed, and she mentally chastised herself for thinking that something might. She sat down, dug out her phone, and starting to type a message for Zachary.

Then she realized that Frank was no longer breathing.

And, even worse, there was no whirring noise.

She checked the machine. It was off, and silent, though she'd been told that if it failed it's alarm would go off. She checked for Frank's pulse, found it, weak but present, and scrambled to dial 911. Once that call was finished, she straddled Frank's still body and began performing CPR on him, hoping to keep his blood pumping long enough to get him to the hospital.

Thirty seconds later, the red-and-blue flashes of the police cruisers arrived, and Georgia was impressed with their promptness, only to be disappointed when they left prematurely. They were followed a minute later by an ambulance. An EMT burst through the open front door, relieved Georgia, and together they loaded Frank into the ambulance.


The room was bright, unnaturally so, and Elena's eyes were having a hard time adjusting. When the burly detective (Elena assumed he was a detective, given that he didn't wear a uniform) slammed the door open and barged into the tiny room, she nearly jumped out of her chair.

"Well now, little missy, mind telling me what you were doing out in Pine Hills tonight?"

"I, uh, I was just playing a game," she stammered. "We both were."

"That's a hell of game, then." The detective's voiced boomed off the walls; Elena thought she saw the one-way mirror rattle. "Four broken street lamps, one garage door stuck open, two sprinkles systems that now need extensive repairs. You know the whole neighborhood saw you two, right?"

Elena hadn't known that, but it seemed blindingly obvious now, so out of fear of embarrassing herself she kept her mouth shut.

The detective wasn't fooled. "Right. So would it surprise you to learn that all that property damage wasn't all you did?"

"What? I swear, all we were doing was catching bugs!" Elena insisted.

"I'm sure. I doubt the family of Frank Doornbos will be so understanding."

What the hell is he talking about? Elena wondered. I don't know any Frank Doornbos. Probably some damn rich guy who lives in that neighborhood.

When she realized what the detective had actually said, Elena felt a chill run down her spine.

"What does that mean?"

"We've already arrested you for destruction of property. Our officers picked you up outside of Mr. Doornbos's home. We know you were playing BugCatcher, despite that game being illegal in this state. You really want to add obstruction of justice to your rap sheet?"

"I swear, I don't know what you are talking about." Elena couldn't believe what she was hearing. BugCatcher was illegal? Why hadn't Greg mentioned that?

"Oh, all right then. I'll spell it out for you. Frank Doornbos is dead. Whatever you two and that app did, it caused his artificial breathing machine to stop working. His nurse called 911, performed CPR, the works, but it was too late. He was dead before they reached the hospital."

Elena's arms and legs went numb. That bug, the last one they'd caught before the police picked them up, the huge jawed caterpillar thing. It had killed someone?

Her face must have given her away, because she looked up from the metal table in front of her to see the detective's eyes boring holes into her skull. Her mind furiously searched for something, anything, to say.

"No, see, you got it all wrong," Elena fought through tears that threatened to overwhelm her. This was all wrong, it was just some fun at the expense of rich people, they weren't doing any real harm. "We were just catching bugs, we didn't kill anyone."

"Oh, but see, you did." The detective smiled, if you could call a mouth full of teeth a smile. "You invoked that bug. That bug shut off the breathing machine, which killed Frank Doornbos. Therefore, you killed Frank Doornbos." He slammed a file down on the desk and Elena jumped. "We are charging you with second-degree murder. Your boyfriend has already confessed; you would be wise to do the same. We will notify your family, and it would be wise to get a lawyer."

The detective picked up the file and walked out of the brightly-lit room, the door slamming behind him. From within the room, sniffles turned to sobs, and the detective strode down the long hallway to his desk.

As he approached, he heard a low drone sounding from his desk drawer, and he grinned.