Today I fixed a hard to find bug in a production system for one of my customers. The scenario is that in our web service we call another existing web service for a big result and then caches that result. The caching is done in our business logic and the code to get the cache object is simple.
Public ReadOnly Property Cache As Cache
Get
If HttpContext.Current Is Nothing Then
Return Nothing
Else
Return HttpContext.Current.Cache
End If
End Get
End Property
The caching code then checks to see if the Cache-property is nothing or not before doing the cache logic. This is not the focus for this post, but still important for understanding the bug and why this bug have been in production for three months by now. No one have complained about the way too long response times, but that fact probably says more about the users expectations of the IT-environment at the company, than anything else. Well back to the bug.
The bug was introduced at the release, three months ago, which was our third release. In the first two releases all that was needed for our web service to return it's result could be found in the big result from the external web service, but now we also needed to get some information from our own database. So we spined up two threads one getting the result from our local database and the other one getting the result from the external web service. When we did this also the caching logic gets executed from the new thread, which made the Cache property to always return Nothing, so we took the performance hit on each and single call.
The lessons learned are one that I got into before, but it's easy to forget, especially when changing existing code. The HttpContext is only available in the thread handling the request. Of course there are other lessons to learn, to prevent these kind of errors to get unnoticed into production, by making the property a little bit smarter. Smarter will probably different, depending on the situations, but here we could for example logged a warning and/or used conditional compiler directives.