This post is an extension to the session I presented at Devoxx UK in June, alongside my colleague Andrew Harmel-Law. Our insights were driven by the experience of building a client microservice architecture, a project that evolved from a single microservice build into one involving microservice to microservice communication and beyond.
Anyone starting a new microservices project can relate to some of the initial challenges we faced:
- How do we split our application into microservices?
- What is our core model and how do we identify it?
- How do we organize our teams around these?
Being familiar with Domain Driven Design (DDD), we thought this would be an ideal scenario to put those concepts into practice.
We are the integration team and we develop APIs. This might not be a typical situation where one would apply DDD, but even here you can see many benefits to applying strategic and tactical patterns in the right way. It is even more important for us to grasp the concept of bounded content and apply it correctly.
Microservices are awesome for various reasons, not least because they co-exist while simultaneously evolving independently of one another. And they are human sized i.e. a developer starting on a project can hit the ground running on her/his first day. Most importantly, they bring interconnectivity of systems to the surface and make it explicit to the teams.
Know your Boundaries
“Getting service boundaries wrong can result in having to make lots of changes in service-service collaboration. An expensive operation”, Sam Newman, ‘Building Microservices’.
Microservices are HARD. When you split a big thing into multiple smaller things, you begin to realise how many moving parts there are. But, splitting things is even harder. With microservices especially, changes naturally ripple across services. Get the split wrong and the impact of these changes can be huge.
It is important to identify and define boundaries – because boundaries are where the models interrelate and also where interfaces and sharing lie. The boundaries are also important for another reason. We’ve recognized how much the teams need to communicate in order to build these microservices. It can be an expensive operation if you get this wrong.
Decompose and Decompose More
One of the most dangerous things in implementing microservices is to get your decomposition wrong. But it is far less painful if there isn’t enough decomposition to begin with, because you can always split further and further until you get to the right size. Moving functionality from one place to another because decomposition was wrong in the first place can cause significantly more pain than in traditional monolithic architectures most often alluded to.
Domain Driven Design to the Rescue
“Despite microservices being so hyped there is tremendous value in them, probably giving us the best environment we have ever had for doing Domain-Driven Design”, Eric Evans in his DDD Exchange Keynote, London 2015.
DDD introduces concepts such as Bounded Contexts, Ubiquitous Language, Hands-On Modeling etc. so that we can get decomposition right in terms of splitting applications into microservices. We will use a design journey from one of our recent projects to demonstrate this.
A Shared Payment Service that allows customers to subscribe to and pay for products online.
It uses a Commerce System built on top of its Content Management System to provide UI for Checkout journeys, product management and other admin capabilities. It also uses a Payment Gateway to handle payments using credit and debit cards, direct debit and other online payment methods. These two systems are integrated using another system called the Payment and Subscription Processing system. We can see how different systems, each implemented in a different language need to communicate using APIs in order to implement a functionality. Let’s look at some important DDD concepts and how it maps to our case study.
“A system of abstractions that describes selected aspects of a domain and can be used to solve problems related to that domain.”
Our journey begins with identifying the models that are in play within this application. During our first attempt, we identified at least five entities within our Shared Payment Services.
- Subscription – Represents subscription related information such as product, amount, starting date, number of payments etc.
- Mandate – Represents the recurring payment details such as payment method, recurring payment amount etc.
- Payment – Actual payment made by the Customer for his purchases or as an initial payment towards a Subscription.
- Order – Commerce order created by the Commerce system for every purchase made by the Customer.
- Refund – Refund claimed by a Customer for an Order as a whole or for individual line items inside the order
“Cluster the entities and value objects into aggregates, and define boundaries around each. Define properties and invariants for the aggregate as a whole and give enforcement responsibility to the root or some designated framework mechanism.”
From the entities we identified through the above step, we can derive at least two aggregates. We will focus on just one for now, which is the Subscription aggregate where Subscription is the root of the aggregate.
As you can see from the diagram, Subscription and Mandate has one-many relationships. What is not shown is that a Subscription can only have one active mandate at any point of time. That is the invariant imposed on this aggregate.
One of the user flows in our Shared Payment Service is to allow a customer to amend their recurring payment details, for example changing the payment method from credit to debit card. As you may have gathered, the commerce system calls the Payment and Subscription Processing system with a reference to the Subscription (being the root) that looks up its persistent store to fetch the active mandate. It then triggers a call that cancels the active mandate, creates a new one and activates it.
User experience being higher priority meant we had to go down the route of creating a new mandate, activate it and then cancel the old mandate. Now, these are multiple calls to the Payment Gateway and as with any distributed systems, these are bound to fail. Suppose it fails while cancelling the old but still active mandate, there is a chance, however remote, that we may end up with two active mandates at the same time. This is clearly a violation of the invariant on the aggregate. To solve this, we introduced an intermediate status in the Payment and Subscription Processing System and moved the failed mandates to this status. We then implemented a house keeping service (a designated framework mechanism) to pull these failed mandates and retry the call to the Payment Gateway until it succeeded. The thought behind this comes straight from Domain Driven Design principles, which states “apply consistency rules synchronously within the aggregate boundaries and asynchronously outside of those boundaries”.
“The setting in which a word or statement appears that determines its meaning”
To explain contexts from our case study’s perspective, lets look at couple of user flows.
First let’s consider the scenario where a customer wants to make an initial payment towards a Subscription. The Commerce System creates an ‘Order’ for this purchase and calls the Payment and Subscription Processing System to allow the customer to make a payment through Payment Gateway. The Payment Gateway creates a ‘Payment’ and returns a reference to this payment back to the Payment and Subscription Processing System. It’s quite clear that the Commerce System only understands and cares about ‘Order’ and likewise the Payment Gateway cares only about ‘Payment’. What about the Payment and Subscription Processing System? Since it integrates these two systems it should know about both entities.
Let’s consider another scenario where the Payment and Subscription Processing System calls the Payment Gateway to set up a recurring payment for a Subscription. Now, Payment Gateway creates something called Continuous Authority Account to set up a recurring payment. This is Subscription and Mandate entities rolled up into one. The reason we have two different entities in Payment and Subscription Processing System is to allow the Commerce System to amend a recurring payment without changing the Subscription.
Finally, let’s consider the situation where a customer claims refund on his order. This is just negative payment in Payment Gateway. We now know the meaning of each entity we identified earlier in different contexts. Does that help?
Same or Similar?
Let’s examine the Order and Payment entities a bit more closely.
Order and Payment entities have some attributes that are the same such as ‘Product’, ‘Amount’ and ‘Quantity’. Some, that are similar such as Status – which for Order can be Check-Out, Fulfilled or Refunded, but for Payment are Authorized, Settled etc. Then there is ‘Type’ which again has the same name but the values can be quite different.
Worse still, Order has an attribute which Payment doesn’t i.e. ‘Line Items’. Conversely, Payment has Merchant Reference but Order doesn’t. Merchant Reference is a unique reference which Payment Gateway gives back to the Payment And Subscription Processing System whenever a payment is made. Looks like our model is broken!
In Part II of this article we’ll reach into the DDD hat and see if there is a concept that will help us solve this problem.