CQRS – Event Versioning
When using Event Sourcing you store your events in an Event Store. This Event Store can only insert new events and read historical events, nothing more nothing less. So when you change your domain logic and also the events belonging to this behavior, then you cannot go back into the Event Store and do a one time convert of all the historical events belonging to the same behavior. The Event Store needs to stay intact, that is one of its powers.
So you make a new version of the original event, this new version carries more or less information then the original one. Lets take a look at a very simple example:
1 namespace Fohjin.DDD.Events.Account
2 {
3 [Serializable]
4 public class CashWithdrawnEvent : DomainEvent
5 {
6 public decimal Balance { get; private set; }
7 public decimal Amount { get; private set; }
8
9 public CashWithdrawnEvent(decimal balance, decimal amount)
10 {
11 Balance = balance;
12 Amount = amount;
13 }
14 }
15
16 [Serializable]
17 public class CashWithdrawnEvent_v2 : DomainEvent
18 {
19 public decimal Balance { get; private set; }
20 public decimal Amount { get; private set; }
21 public Guid AtmId { get; private set; }
22
23 public CashWithdrawnEvent_v2(decimal balance, decimal amount, Guid atmId)
24 {
25 Balance = balance;
26 Amount = amount;
27 AtmId = atmId;
28 }
29 }
30 }
This to me looks like a natural evolution for this type of event, so how do you deal with this. Because after having used the system, before adding this extension there have been many cash withdrawals. So all these events are in the Event Store, they cannot be altered, and when you retrieve an Aggregate Root from the Event Store all these historical events need to be processed in order to restore the internal state.
Now what you don’t want is to maintain code in the Aggregate Root that knows how to handle these old event versions, sure one version is ok, but what about one hundred different versions? Also we are not just talking about just in the Aggregate Root, also the different event handlers need to be kept and maintained.
The better approach is to have a mechanism that you can hook-up with different event convertors. Then when an event is retrieved from the Event Store it first goes through this pipeline of convertors to be converted to the latest event version.
Now I wanted to do this properly and write some actual code for this, and then blog about it, but someone kept nagging me about it, so here is a very rough spike instead, first some tests:
1 namespace Test.Fohjin.DDD.Spike
2 {
3 public class Spike_test_1 : BaseTestFixture
4 {
5 private object ConvertedEvent;
6
7 protected override void When()
8 {
9 ConvertedEvent = new EventConvertor().Convert(new CashWithdrawnEvent(10.0M, 20.0M));
10 }
11
12 [Then]
13 public void The_converted_event_is_the_latest_version()
14 {
15 ConvertedEvent.WillBeOfType<CashWithdrawnEvent_v4>();
16 }
17
18 [Then]
19 public void The_converted_event_wil_contain_the_correct_data()
20 {
21 ConvertedEvent.As<CashWithdrawnEvent_v4>().Balance.WillBe(10.0M);
22 ConvertedEvent.As<CashWithdrawnEvent_v4>().Amount.WillBe(20.0M);
23 ConvertedEvent.As<CashWithdrawnEvent_v4>().AtmId.WillBe(string.Empty);
24 }
25 }
26
27 public class Spike_test_2 : BaseTestFixture
28 {
29 private object ConvertedEvent;
30
31 protected override void When()
32 {
33 ConvertedEvent = new EventConvertor().Convert(new CashWithdrawnEvent_v2(10.0M, 20.0M, "12345"));
34 }
35
36 [Then]
37 public void The_converted_event_is_the_latest_version()
38 {
39 ConvertedEvent.WillBeOfType<CashWithdrawnEvent_v4>();
40 }
41
42 [Then]
43 public void The_converted_event_wil_contain_the_correct_data()
44 {
45 ConvertedEvent.As<CashWithdrawnEvent_v4>().Balance.WillBe(10.0M);
46 ConvertedEvent.As<CashWithdrawnEvent_v4>().Amount.WillBe(20.0M);
47 ConvertedEvent.As<CashWithdrawnEvent_v4>().AtmId.WillBe("12345");
48 }
49 }
50
51 public class Spike_test_3 : BaseTestFixture
52 {
53 private object ConvertedEvent;
54
55 protected override void When()
56 {
57 ConvertedEvent = new EventConvertor().Convert(new CashWithdrawnEvent_v3(10.0M, 20.0M, "12345"));
58 }
59
60 [Then]
61 public void The_converted_event_is_the_latest_version()
62 {
63 ConvertedEvent.WillBeOfType<CashWithdrawnEvent_v4>();
64 }
65
66 [Then]
67 public void The_converted_event_wil_contain_the_correct_data()
68 {
69 ConvertedEvent.As<CashWithdrawnEvent_v4>().Balance.WillBe(10.0M);
70 ConvertedEvent.As<CashWithdrawnEvent_v4>().Amount.WillBe(20.0M);
71 ConvertedEvent.As<CashWithdrawnEvent_v4>().AtmId.WillBe("12345");
72 }
73 }
74
75 public class Spike_test_4 : BaseTestFixture
76 {
77 private object ConvertedEvent;
78
79 protected override void When()
80 {
81 ConvertedEvent = new EventConvertor().Convert(new CashWithdrawnEvent_v4(10.0M, 20.0M, "12345"));
82 }
83
84 [Then]
85 public void The_converted_event_is_the_latest_version()
86 {
87 ConvertedEvent.WillBeOfType<CashWithdrawnEvent_v4>();
88 }
89
90 [Then]
91 public void The_converted_event_wil_contain_the_correct_data()
92 {
93 ConvertedEvent.As<CashWithdrawnEvent_v4>().Balance.WillBe(10.0M);
94 ConvertedEvent.As<CashWithdrawnEvent_v4>().Amount.WillBe(20.0M);
95 ConvertedEvent.As<CashWithdrawnEvent_v4>().AtmId.WillBe("12345");
96 }
97 }
98 }
So basically some tests to confirm the correct conversion from one event to another event, now below here is the full implementation:
1 namespace Test.Fohjin.DDD.Spike
2 {
3 public class EventConvertor
4 {
5 private readonly Dictionary<Type, Func<object, object>> _convertors;
6
7 public EventConvertor()
8 {
9 _convertors = new Dictionary<Type, Func<object, object>>();
10 RegisterEventConvertors();
11 }
12
13 private void RegisterEventConvertors()
14 {
15 _convertors.Add(typeof(CashWithdrawnEvent), x => new CashWithdrawnEventConvertor().Convert((CashWithdrawnEvent)x));
16 _convertors.Add(typeof(CashWithdrawnEvent_v2), x => new CashWithdrawnEvent_v2Convertor().Convert((CashWithdrawnEvent_v2)x));
17 _convertors.Add(typeof(CashWithdrawnEvent_v3), x => new CashWithdrawnEvent_v3Convertor().Convert((CashWithdrawnEvent_v3)x));
18 }
19
20 public object Convert(object soureEvent)
21 {
22 Func<object, object> convertor;
23 return _convertors.TryGetValue(soureEvent.GetType(), out convertor)
24 ? Convert(convertor(soureEvent))
25 : soureEvent;
26 }
27 }
28
29 public interface IEventConvertor<TSourceEvent, TTargetEvent>
30 where TSourceEvent : IDomainEvent
31 where TTargetEvent : IDomainEvent
32 {
33 TTargetEvent Convert(TSourceEvent sourceEvent);
34 }
35
36 public class CashWithdrawnEventConvertor : IEventConvertor<CashWithdrawnEvent, CashWithdrawnEvent_v4>
37 {
38 public CashWithdrawnEvent_v4 Convert(CashWithdrawnEvent sourceEvent)
39 {
40 var theEvent = new CashWithdrawnEvent_v4(sourceEvent.Balance, sourceEvent.Amount, string.Empty)
41 {
42 AggregateId = sourceEvent.AggregateId
43 };
44 (theEvent as IDomainEvent).Version = (sourceEvent as IDomainEvent).Version;
45 return theEvent;
46 }
47 }
48
49 public class CashWithdrawnEvent_v2Convertor : IEventConvertor<CashWithdrawnEvent_v2, CashWithdrawnEvent_v3>
50 {
51 public CashWithdrawnEvent_v3 Convert(CashWithdrawnEvent_v2 sourceEvent)
52 {
53 var theEvent = new CashWithdrawnEvent_v3(sourceEvent.Balance, sourceEvent.Amount, sourceEvent.AtmId)
54 {
55 AggregateId = sourceEvent.AggregateId
56 };
57 (theEvent as IDomainEvent).Version = (sourceEvent as IDomainEvent).Version;
58 return theEvent;
59 }
60 }
61
62 public class CashWithdrawnEvent_v3Convertor : IEventConvertor<CashWithdrawnEvent_v3, CashWithdrawnEvent_v4>
63 {
64 public CashWithdrawnEvent_v4 Convert(CashWithdrawnEvent_v3 sourceEvent)
65 {
66 var theEvent = new CashWithdrawnEvent_v4(sourceEvent.Balance, sourceEvent.Amount, sourceEvent.AtmId)
67 {
68 AggregateId = sourceEvent.AggregateId
69 };
70 (theEvent as IDomainEvent).Version = (sourceEvent as IDomainEvent).Version;
71 return theEvent;
72 }
73 }
74 }
This implementation is definitely not very elegant (so it doesn’t really belong on this blog) but hey it does show you how a possible solution would work. When building this yourself you might want to use conventions to auto register the convertors and chain them together during configuration so there is no need for the recursive functionality.
Also look at the jump from version 1 to version 4, this is an optimization to speed up the conversion. You would do this after a few versions, not for each version.
I’ll be adding a proper solution to the example in the near future, something that you would just plug the convertors in and the system would figure out how to handle them itself.
I did something alike but not pertaining to event sourcing. I had a data structure (similar to your event) that went through different versions. Contrary to your approach, I did keep the old code in the code base. Well, actually just the part that read the data structure (hefty piece of xml). The data structure was mapped onto an in-memory model (which of course was also just the latest version of that model). The write part was only kept for the latest version of the data structure.
As time passes and end-users migrate old data structures implicitly (because they touch the data structure in one way or another), a one-time fix becomes a possibility. This is important for us – to be able to drop older versions, that is.
@@seagile
In your case you allow yourself to change the existing events (or data), in our case this is not what we want, because then you loose the ability to also use your Event Store as an Audit log. And you add uncertainty I believe because you have no way of proving that your conversion works, especially over time.
But that is my take on this.
Mark,
according to the conversation with @seagile, you want to keep all the events versions in your code, am I getting it right? Wouldn’t xml serialized versions of them be enough for the mentioned audit log?
The second think is a case, when the newly added property, for instance AtmId cannot be null nor empty. What about this case? Or maybe new properties should always allow default(T) values to be set? One of the aspects of event sourcing is the ability to undo and redo the events execution. What if the undo moves the history to the point when an event has a nulls_not_allowed property set to null? Is this kind of situation possible at all?
Who’s the nagger?
I can see how a method like this would work well for simple changes like the one you described. But how would you handle big changes?
For instance, say that we do a big refactor and we rename/move/merge/split some of our events. Having to keep those “old” events around along with the newly refactored events seems like it could get messy fast.
Or say we do something simple like just rename an assembly or a namespace. I’m pretty sure that would break the binary serializer and I wouldn’t want to have to keep that old assembly lying around indefinitely just to support those legacy events in the event store.
In this case, wouldn’t it be ok to modify the event store and migrate the old events to the new events – as long as the semantic meaning of the events is still the same?
And if so, is there an easy way to go about performing such a migration assuming the events are binary serialized somewhere?
@Chad
You keep the events because they offer business value, either in the ability to gather more information from your data, or as an audit log. If you would go and start changing these historical events you introduce the possibility that you make a mistake. And your audit log becomes invalid. So depending on how important those features are for you you might decide to ‘break the rules’.
But before you do; once you have a mechanism to convert events from one version to an other version this becomes easy. You can also issue one time compensating events if you run into a change that you cannot manage naturally. So I would like to argue that it is not very difficult, and that changing existing events might introduce more pain than you may gain.
The binary formatter indeed means that you need to keep the events in the same namespace, but that is only one way of serializing, I recommend creating a custom formatter disconnecting yourself from the namespace issue and also increasing speed and minimizing space. And consider using an other medium to store the events, like using CouchDB or something like that.
But to answer your last question, I wouldn’t go into the historical events and change them, I rather version them and deal with that issue.
How do you handle sensitive data when using event sourcing? Do you store credit card numbers in the event log? What if you have to delete all data related to an entity? Say to comply with data protection rules or court order and you have to remove all historical data for some entities (but not all) in your system. Wouldn’t that mess up your event log and auditing?
@Kaarel
That is an interesting issue! And the obvious answer is “it depends”
But do you want to keep track of the CC# meaning like Amazon to enable fast purchases? Then you just need it in the reporting database, and deleting from there is a non-issue. But in-order to get it there you need to fire of an event. So I would suggest adding an attribute on the event that indicates the event store to skip storing this event. Because the rule is that you don’t touch events once they are in the event store.
Now if you are forced to do change your events in the event store by law, then you don’t have a choice, in that case I would just go in and just change the CC# within those events, so all other information stays intact and you still keep track of what happened when. But it is not optimal, and you would have the same issue with other auditing solutions.
So if upfront you know something like this is a requirement then I would opt for skipping these events, also I would want them to be containing only the absolute minimum needed information, or perhaps even double sent the event one with CC# and one without, with CC# will not be stored, without will be.
But this is just me thinking about it, all situations are different.
Hope that helps
Hi Mark,
As long we add data to events we agree we have to reflect that as a new property in the AR. I don’t see any conflict for the case of appending data to events since the new property will be Null until set. So why the need to create versioned events instead of appending new properties to the same event class?
On the other hand, if we design an event to hold 3 attributes and after system usage we got a requeriment to delete 1… we do will have to change code where that deleted property where set (in the AR), however we might leave the event with those 3 atributes where 1 of those would be redundant and will no afect the AR behaviour since no longer will be reflected into the AR.
However for this case, of deleting attributes I do see the benefit of “prunning” or “clearing” events using converters but I don’t achieve to see the benefit for the reverse case of appending data to events.
Cheers.
@Sebastian
I used adding a property to the event as a simple example, and in this case you might be right it doesn’t need a new event, except if you want to track these things as well. But I am sure there are many more ways these things evolve during the lifetime of a system. Versioning the events this way is a good way of dealing with these changes.
Hi mark,
Thanks for your response. I would really appreciate to get clarified how to deal concurrency issues in CQRS when not using RDMS as the event storage mechanism since I still see this field in color grey
Thanks for your blog and for takin down everything to code examples, that really makes things understandable and gives a clear picture of the whole.
Cheers.