@ianfnelson admit it: you use Castle Windsor primarily to highlight and lampoon Google's poor selection of adwords?!
Funny, but not true. I am enamoured with the Castle Windsor project because its power makes it fairly simple for me to develop loosely-coupled systems which are easily maintained and tested. The wide range of Facilities and Contrib projects also integrate nicely with the other parts of my current development stack (NHibernate, WCF, WF, log4net).
Whilst there is a lot of material on the web about the Dependency Injection capabilities of Windsor, the Aspect-Oriented Programming (AOP) features don’t seem to get as much exposure, so I thought I’d quickly blog about one way in which I’ve been making use of those in the system I’m currently developing.
Earlier this year Davy Brion posted an excellent C# implementation of the Circuit Breaker pattern described in Michael Nygard’s equally excellent book Release It! Design and Deploy Production-Ready Software.
For the uninitiated, this pattern advocates protecting your system from issues affecting any remote service on which it depends by wrapping your calls to that service with a circuit breaker component. This component notes any failed service invocations, until some threshold is reached, causing the circuit to trip. Subsequent attempted service invocations then “fail fast”, throwing a custom exception rather than passing the method call on to the remote service. This benefits your system, as it prevents you from tying up valuable threads creating expensive remote service calls which may be slow to timeout. And it benefits the remote system as you avoid piling further pressure on a service which is already down or unresponsive.
For more details of this pattern, and some entertaining war stories of situations in which they would have proved useful, I encourage you to read Michael Nygard’s book.
Now, I like Davy’s implementation of this pattern a lot, particularly since it is implemented as an interceptor for Castle Windsor, which as mentioned I’m already making heavy use of in my current projects. But my only concern is that it is configured to trip after the number of failed invocations reaches a specific figure, irrespective of how long it takes to reach that threshold. I want an implementation which is able to sense the difference between infrequent exceptions over a prolonged period, and a sudden flurry of exceptions – the latter causing the circuit breaker to trip.
To achieve this I’ve tweaked Davy’s implementation slightly by adding an extra parameter to the constructor specifying the historical period over which to total the number of exceptions. That is, you can configure it such that a certain number of failures within the preceding y minutes causes the circuit breaker to trip.
I look forward to seeing what adverts Google sees fit to stick at the bottom of this post… :-)
/// <summary>
/// Castle Interceptor Circuit Breaker.
///
/// Used in an AOP-style to wrap calls to protected methods.
/// If the number of unhandled exceptions that occur within a specified period
/// exceeds a specified threshold, then invocations will cease to be passed to the
/// protected code until a specified timeout has elapsed.
/// Instead, OpenCircuitExceptions will immediately be thrown when invocations are
/// attempted.
///
/// Based on code by Davy Brion (http://davybrion.com) and Ayende Rahien (http://ayende.com).
///
/// For background on the Circuit Breaker pattern, see "Release It! - Design and Deploy
/// Production-Ready Software" by Michael Nygard (Pragmatic Bookshelf, 2007).
/// </summary>
public class CircuitBreaker : IInterceptor
{
private readonly object monitor = new object();
private CircuitBreakerState state;
private List<DateTime> failures;
private TimeSpan period;
private TimeSpan timeout;
private int threshold;
/// <summary>
/// Initializes a new instance of the CircuitBreaker class.
/// </summary>
/// <param name="threshold">Number of permissible exceptions within period before circuit breaker blows.</param>
/// <param name="period">Historical period over which to log exceptions.</param>
/// <param name="timeout">Timeout after blowing of Circuit Breaker for which the circuit will stay open and throw OpenCircuitExceptions.</param>
public CircuitBreaker(int threshold, TimeSpan period, TimeSpan timeout)
: this()
{
this.threshold = threshold;
this.period = period;
this.timeout = timeout;
}
/// <summary>
/// Initializes a new instance of the CircuitBreaker class.
/// </summary>
public CircuitBreaker()
{
this.failures = new List<DateTime>();
this.MoveToClosedState();
}
/// <summary>
/// Intercepts the planned method invocation.
/// </summary>
/// <param name="invocation">Original method call.</param>
public void Intercept(IInvocation invocation)
{
using (TimedLock.Lock(monitor))
{
state.ProtectedCodeIsAboutToBeCalled();
}
try
{
invocation.Proceed();
}
catch (Exception e)
{
using (TimedLock.Lock(monitor))
{
// Add current datetime to list of failed invocations.
failures.Add(DateTime.UtcNow);
state.ActUponException(e);
}
throw;
}
using (TimedLock.Lock(monitor))
{
state.ProtectedCodeHasBeenCalled();
}
}
private void MoveToClosedState()
{
state = new ClosedState(this);
}
private void MoveToOpenState()
{
state = new OpenState(this);
}
private void MoveToHalfOpenState()
{
state = new HalfOpenState(this);
}
private void ResetFailureList()
{
this.failures.Clear();
}
private bool ThresholdReached()
{
// Remove log of any failed invocations that occurred earlier than period in which we are interested.
this.failures.RemoveAll(f => f < DateTime.UtcNow - period);
// Have number of failures breached the allowed threshold?
return failures.Count > threshold;
}
private abstract class CircuitBreakerState
{
protected readonly CircuitBreaker circuitBreaker;
protected CircuitBreakerState(CircuitBreaker circuitBreaker)
{
this.circuitBreaker = circuitBreaker;
}
public virtual void ProtectedCodeIsAboutToBeCalled() { }
public virtual void ProtectedCodeHasBeenCalled() { }
public virtual void ActUponException(Exception e) { }
}
/// <summary>
/// Represents a closed CircuitBreaker - this is the "normal" behaviour,
/// with invocations passed through to the protected code.
/// </summary>
private class ClosedState : CircuitBreakerState
{
public ClosedState(CircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
circuitBreaker.ResetFailureList();
}
/// <summary>
/// Called when an exception occurs in the protected code.
/// </summary>
/// <param name="e"></param>
public override void ActUponException(Exception e)
{
// If the threshold has been breached, open the circuit.
if (circuitBreaker.ThresholdReached()) circuitBreaker.MoveToOpenState();
}
}
/// <summary>
/// Represents an open CircuitBreaker - in this state we do not pass the
/// invocation to the protected code, but instead throw OpenCircuitExceptions
/// until the timeout expires.
/// </summary>
private class OpenState : CircuitBreakerState
{
private readonly DateTime expiry;
public OpenState(CircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
// Keep a note of the time at which the circuit should be moved
// to half-open state.
expiry = DateTime.UtcNow + circuitBreaker.timeout;
}
public override void ProtectedCodeIsAboutToBeCalled()
{
// Has the timeout expired?
if (DateTime.UtcNow < expiry)
{
// Not yet, so throw an eppy.
throw new OpenCircuitException();
}
// yes, timeout has expired, so move to half-open state.
circuitBreaker.MoveToHalfOpenState();
}
}
/// <summary>
/// Represents a CircuitBreaker in the "Half-Open" state.
/// In this state a timeout has just expired and we are tentatively calling
/// the protected code again. If all goes well, we will move to the closed state.
/// However, if this first dipping of our electronic toe into the water is met by
/// the shock of an unhandled exception, we shall quickly open the circuit again
/// for a further timeout period.
/// </summary>
private class HalfOpenState : CircuitBreakerState
{
public HalfOpenState(CircuitBreaker circuitBreaker) : base(circuitBreaker) { }
public override void ActUponException(Exception e)
{
// Eek, problems remain. Open the circuit again quickly.
circuitBreaker.MoveToOpenState();
}
public override void ProtectedCodeHasBeenCalled()
{
// Things seem to be OK now. Close the circuit.
circuitBreaker.MoveToClosedState();
}
}
}
