Microservices have worn deep grooves into the techsphere this year, and even if you’re not ready to blast apart your monolith, you can bet someone in your organisation has sat in a conference talk somewhere, tracing out an imaginary blueprint for a completely modular system. Well, hold your horses. Because Tapjoy Senior Software Engineer Sean Kelly has a real story from the coalface – a tale of what happens when you get all fired up from reading cool blog posts and start carving and slicing out microservices like they were turkey. Find out how Tapjoy dealt with the sting that came in the trail of this experiment, and the business lessons they learnt along the way. Filmed at Ricon 2015.
1.22: Voxxed: Can you tell us how your company’s [experiment with microservices] went?
Kelly: There was a big push in the community, maybe six months or a year or a little longer ago, about microservices. They were everywhere – “everybody’s got to do microservices” – “microservices are the new hot thing.” And we had a huge monolithic Rails application, which I think a lot of people could relate to, and the idea was that we could take that approach of microservices and cleanly carve out sections of that monolith and stick some APIs in front of them and turn them into microservices.
At first, it worked pretty well. We took some low hanging fruit, and we turned that into discrete services. But as we started to get into more of the meat and potatoes of what the monolith actually had to do, we ran into things like nested dependencies and tangled code problems where it was very difficult to take that clean cut, and it started to become wider and wider and more jagged and reaching over into different places just to carve out, for example, a core piece of our stack.
Voxxed: Like pulling a string and then the string is attached to a giant snarled ball of yarn?
Kelly: Yes – it was a lot like that. And so we had a couple of services that we carved out, and that worked pretty well. But then eventually over time, it became obvious that it was very difficult to take something – a system that had been built and evolved naturally over years and years of time – and think that we could cleanly extract core services out of that in a way that could be done very quickly.
Eventually, what we focused on was wrapping up the microservices work that we did, and then we continued with the approach that we had been going with, which was to take the monolith and deploy it to discrete clusters that we could scale independently, and use load balancers at the very edge of the system direct to individual clusters.
For example, some traffic goes directly towards internal API calls, some traffic goes towards serving ads to people, some traffic goes towards other services. But it’s all basically the same monolithic process, just deployed, scaled and tuned differently, based on the traffic pattern that it expected to receive. And that’s served us very, very well.
Voxxed: Do you feel like you’ve recovered from your experiment with microservices?
Kelly: I think we have. I refer to it as “the microservices hangover,” and I think we have. We ran into quite a bit of pain during part of the process, and then we realised that it was more important to get things done and to ship product features and continue to drive than it was to sit there and feel good about the seat that we had carved ourselves in the ivory tower. It was pretty good at first, and then we made the right choice at the right time to say, “This isn’t necessarily working for us right now, and we’re going to shelve that work and continue on with product and feature development.”
Voxxed: I think this is a scenario a lot of shops face, particularly if you’re not that big – you’ve got that crux between, “This could really help us down the line,” and also, you’ve got your day to day business.
Kelly: Right, and that’s the thing. And like I said, because we already had a reasonably effective way of scaling and deploying our services into discrete clusters, it got us some of the benefit of microservices, without a lot of the pain of having to extract things or to worry and to handle with really complicated and distributed transactions, which is one of the main downfalls of the microservice architecture.
Unless you have a really strong sense of how distributed transactions work…how to recover them…and what does it mean if you have six services in sequence and service two and four fail? Do you roll back, do you roll forward, do you care at all? It’s very difficult to reason about a lot of that stuff without spending a lot of time up front. And not having to worry about that as much enables us to ship product and features a lot faster.
Voxxed: So if someone else is dealing with these distributed transactions and they’re thinking about taking the jump into microservices, would your advice be, “Run, run away!”?
Kelly: No, it wouldn’t. I think it’s definitely one of those things where you need to take a look at your product in your company and decide; is this something we need to do? Is this something we can do? Do we properly understand how distributed transactions work? Do we know how to scale services over the network?
It’s definitely not as easy as all the really cool blog posts made it sound! But, I wouldn’t say it’s a bad thing. It’s definitely an architecture that makes a lot of sense for certain use cases.
I hate to give vague non-answers, but you really do need to take a look at what you’re doing and your engineering team and the talent that you have there and decide, is it more important to break things into discrete little composable services and do you guys understand how to reason about that? Or is it better to keep them under one house where they can all talk together, and focus more on code quality issues where you can maintain internal consistencies so that you don’t have dependencies that are reaching across into each other, or are nested or circular in any capacity, and to focus more on internal code architecture, as opposed to external service architecture….
I think a lot of people may have seen the microservices approach as, “Oh, we’re going to carve out our code into discrete services, and along the way, we’re going to clean it up too…We’re going to take care of tech debt, and do all this new shiny stuff.”
But what they realised is that all they did was add more tech debt to their stack, when you could have just focused on internal code cleanup and consistency – at least, that’s the way I’ve seen a lot of this stuff going with some of the blog posts that I’ve read, both before, as the microservices “boom” took off, and then after when people were waking up and realising, “This wasn’t the magical cure all I thought it was! We still have all these problems…we didn’t really solve everything we thought we were going to…and now we actually have more problems in some cases.”