Retrying Operations
Here’s something I hacked together last night: I’m writing an app that involves a lot of web requests to a very unreliable server. Maybe the site is down, maybe we only get part of a message back because the stream was interrupted, maybe the network cable is loose and being chewed on by gremlins. Who knows. The application is unattended though, so we need it to wait a little while, and then retry the operation a few time – before ultimately giving up and terminating, leading to other actions.
Also, it would be best if this retrying could happen out of sight of the code making the web request – I want to hide all the network activity behind various facades, for testability and to make the logic using the results of the call much easier to write. So, last night the “RetryOperation” class was written.
The code is pretty simple, and I have a feeling that this is already implemented in the .Net framework somewhere… I’ve attached the code and an example program below.
Basically here’s what happens: you call retryOperation.Try(() => DoSomethingUnreliable()); either that will return (synchronously) with a result, or you’ll get an exception when retryOperation gives up.
public TResult Try<TResult>(Func<TResult> action) { int tries = 0; while (tries < MaxTries) { try { // go do it. return action(); } catch (Exception ex) { // action failed. // log about our failure, and sleep for "a while" and then try again, // if we're out of retries then give up and send the exception // back up the call stack. tries++; string logMsg = "Retry Attempt " + tries; log.Warn(logMsg, ex); if (tries >= MaxTries) { // YOU FAIL! throw; } // note: this could be configurable, pick your favorite // timeout-waiting-strategy! int timeout = 10 * 1000 * tries; log.Warn("Sleeping for " + timeout + " ms"); Thread.Sleep(timeout); } } // this point should "never happen.." // either we get a successful result, or we go through our maximum number // of retries and throw an exception above. throw new RetryException("Error condition in Try() escaped from custody!"); }






Retrying actions is a real life-saver, indeed.
I’m using a similar approach to transfer data between application boundaries (i.e.: from repository to DB, while resolving dead-locks, or between client applications and web services, while handling communication exceptions). Ability to inject the specific policy via the IoC from a single place makes it even more easy to
If you are interested, there is a production-quality open-source library that leverages this concept (with a configuration syntax for retry policies). Here’s the article introducing the retry aspect:
http://abdullin.com/journal/2008/12/1/net-exception-handling-action-policies-application-block.html
BTW, these policies are compatible with policies used in Windows Azure.