Microservices Quotes

If you are working in an organization that places lots of restrictions on how developers can do their work, then microservices may not be for you.

The golden rule: can you make a change to a service and deploy it by itself without changing anything else?

No matter how it looks at first, it’s always a people problem. Gerry Weinberg, The Second Law of Consulting

if bridge building were like programming, halfway through we’d find out that the far bank was now 50 meters farther out, that it was actually mud rather than granite, and that rather than building a footbridge we were instead building a road bridge.

When services are loosely coupled, a change to one service should not require a change to another.

The code that changes together, stays together.

No matter how it looks at first, it’s always a people problem. ​—​Gerry Weinberg, The Second Law of Consulting

A good architecture will allow a system to be born as a monolith, deployed in a single file, but then to grow into a set of independently deployable units, and then all the way to independent services and/or micro-services.

Once launched into production, our software will continue to evolve as the way it is used changes. For most things we create, we have to accept that once the software gets into the hands of our customers we will have to react and adapt, rather than it being a never-changing artifact. Thus, our architects need to shift their thinking away from creating the perfect end product, and instead focus on helping create a framework in which the right systems can emerge, and continue to grow as we learn more.

As strange as it may seem, this decision is one of the options that a good architect leaves open. A system that is written as a monolith, and that depends on that monolithic structure, cannot easily be upgraded to multiple processes, multiple threads, or micro-services should the need arise. By comparison, an architecture that maintains the proper isolation of its components, and does not assume the means of communication between those components, will be much easier to transition through the spectrum of threads, processes, and services as the operational needs of the system change over time.

So we borrow from other professions. We call ourselves software “engineers,” or “architects.” But we aren’t, are we? Architects and engineers have a rigor and discipline we could only dream of, and their importance in society is well understood. I remember talking to a friend of mine, the day before he became a qualified architect. “Tomorrow,” he said, “if I give you advice down at the pub about how to build some‐ thing and it’s wrong, I get held to account. I could get sued, as in the eyes of the law I am now a qualified architect and I should be held responsible if I get it wrong.” The importance of these jobs to society means that there are required qualifications people have to meet. In the UK, for example, a minimum of seven years study is required before you can be called an architect. But these jobs are also based on a body of knowledge going back thousands of years. And us? Not quite. Which is also why I view most forms of IT certification as worthless, as we know so little about what good looks like

Architects are responsible for a lot of things. They need to ensure there is a set of principles that can guide development, and that these principles match the organization’s strategy. They need to make sure as well that these principles don’t require working practices that make developers miserable. They need to keep up to date with new technology, and know when to make the right trade-offs. This is an awful lot of responsibility. All that, and they also need to carry people with them — that is, to ensure that the colleagues they are working with understand the decisions being made and are brought in to carry them out. Oh, and as we’ve already mentioned: they need to spend some time with the teams to understand the impact of their decisions, and perhaps even code too. A tall order? Absolutely. But I am firmly of the opinion that they shouldn’t do this alone. A properly functioning governance group can work together to share the work and shape the vision.

Postel’s Law (otherwise known as the robustness principle), which states: “Be conservative in what you do, be liberal in what you accept from others.

So my primary guideline would be don't even consider microservices unless you have a system that's too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application. Do pay attention to good modularity within that monolith, but don't try to separate it into separate services.

Most CMSes are pretty bad even at doing page layout, typically providing drag-and-drop tools that don’t cut the mustard. And even then, you end up needing to have someone who understands HTML and CSS to fine-tune the CMS templates. They tend to be terrible platforms on which to build custom code.

Avoid database integration at all costs. Understand the trade-offs between REST and RPC, but strongly consider REST as a good starting point for request/response integration. Prefer choreography over orchestration. Avoid breaking changes and the need to version by understanding Postel’s Law and using tolerant readers.

Neal Ford puts it, many of our working practices around deployment and host management are an attempt to optimize for scarcity of resources.

You’ll also need to think differently about how you scale your systems and ensure that they are resilient. Don’t also be surprised if things like distributed transactions or CAP theorem start giving you headaches, either!

Erik Doernenburg first shared with me the idea that we should think of our role more as town planners than architects for the built environment

To borrow a term from Frank Buschmann, architects have a duty to ensure that the system is habitable for developers too

We can focus our efforts, then, on automating the creation and deployment of these images. This also becomes a really neat way to implement another deployment concept, the immutable server

All too often, considerations about CFRs come far too late. I strongly suggest looking at your CFRs as early as possible, and reviewing them regularly.

So our architects as town planners need to set direction in broad strokes, and only get involved in being highly specific about implementation detail in limited cases.

Making decisions in system design is all about trade-offs, and microservice architectures give us lots of trade-offs to make

Getting integration right is the single most important aspect of the technology associated with microservices in my opinion. Do it well, and your microservices retain their autonomy, allowing you to change and release them independent of the whole. Get it wrong, and disaster awaits. Hopefully once you’ve read this chapter you’ll learn how to avoid some of the biggest pitfalls that have plagued other attempts at SOA and could yet await you in your journey to microservices.

Remember when we talked about the core principles behind good microservices? Strong cohesion and loose coupling — with database integration, we lose both things.

Another principle introduced in REST that can help us avoid the coupling between client and server is the concept of hypermedia as the engine of application state (often abbreviated as HATEOAS, and boy, did it need an abbreviation

The work to shadow production traffic can be complex, though, especially if the events /requests being replayed aren’t idempotent.

The term nonfunctional never sat well with me. Some of the things that get covered by this term seem very functional in nature! One of my colleagues, Sarah Taraporewalla, coined the phrase cross-functional requirements (CFR) instead, which I greatly prefer

Teams now measure themselves not in terms of how many points they complete, but instead optimize for how long it takes for code to get from laptop to live

In his book Antifragile (Random House), Nassim Taleb talks about things that actually benefit from failure and disorder

Embracing and inciting failure through software, and building systems that can handle it, is only part of what Netflix does. It also understands the importance of learning from the failure when it occurs, and adopting a blameless culture when mistakes do happen

Michael Nygard’s book Release It! (Pragmatic Programmers) shows how the same idea can work wonders as a protection mechanism for our software.

Hystrix allows you, for example, to implement bulkheads that actually reject requests in certain conditions to ensure that resources don’t become even more saturated; this is known as load shedding

This can be disastrous. At the start of a new project, we often don’t know exactly what we want to build, nor do we know if it will be successful

You can think of Mountebank as a small software appliance that is programmable via HTTP. The fact that it happens to be written in NodeJS is completely opaque to any calling service. When it launches, you send it commands telling it what port to stub on, what protocol to handle (currently TCP, HTTP, and HTTPS are supported, with more planned), and what responses it should send when requests are sent. It also supports setting expectations if you want to use it as a mock. You can add or remove these stub endpoints at will, making it possible for a single Mountebank instance to stub more than one downstream dependency.

I have also seen many a team’s morale and productivity destroyed by having a mandated framework thrust upon them

Much of the role of the technical leader is about helping grow them — to help them understand the vision themselves — and also ensuring that they can be active participants in shaping and implementing the vision too.

am a strong believer that great software comes from great people

A loosely coupled service knows as little as it needs to about the services with which it collaborates. This also means we probably want to limit the number of different types of calls from one service to another, because beyond the potential performance problem, chatty communication can lead to tight coupling.

I find it odd that people pick JSON because it is nice and lightweight, then try and push concepts into it like hypermedia controls that already exist in XML

I saw used effectively by one of our teams was to delay the implementation of proper persistence for the microservice, until the interface had stabilized enough

I recommend REST in Practice (O’Reilly), which covers the topic of REST over HTTP in depth

Another key to deferring a breaking change is to encourage good behavior in your clients, and avoid them binding too tightly to your services in the first place

This pattern — of implementing a reader able to ignore changes we don’t care about — is what Martin Fowler calls a Tolerant Reader.

This pattern is sometimes referred to as backends for frontends (BFFs)

The danger with this approach is the same as with any aggregating layer; it can take on logic it shouldn

Many implementations of CRM tools I have seen are among the best examples of adhesive (as opposed to cohesive) services.

Much like with our example of fronting the CMS system with our own code, with a strangler you capture and intercept calls to the old system.

In his book Working Effectively with Legacy Code (Prentice-Hall), Michael Feathers defines the concept of a seam —

A great place to start is to use a tool like the freely available SchemaSpy, which can generate graphical representations of the relationships between tables.

using this approach, which it then open sourced as the Aegisthus project

During this time of churn, changes across service boundaries are more likely, and what is in or not in a given service is likely to change frequently. During this period, having all services in a single build to reduce the cost of cross-service changes may make sense.

Packer is a tool designed to make creation of images much easier

But once we run our deployment process, what happens if someone comes along, logs into the box, and changes things independently of what is in source control? This problem is often called configuration drift — the code in source control no longer reflects the configuration of the running host.

This balance, between production-like environments and fast feedback, won’t be static. Keep an eye on the bugs you find further downstream and your feedback times, and adjust this balance as required.

When PaaS solutions work well, they work very well indeed. However, when they don’t quite work for you, you often don’t have much control in terms of getting under the hood to fix things. This is part of the trade-off you make

I would say that in my experience the smarter the PaaS solutions try to be, the more they go wrong

I worked on one monolithic system, for example, where we had 4,000 unit tests, 1,000 service tests, and 60 end-to-end tests. We decided that from a feedback point of view we had way too many service and end-to-end tests (the latter of which were the worst offenders in impacting feedback loops), so we worked hard to replace the test coverage with smaller-scoped tests.

The more moving parts, the more brittle our tests may be, and the less deterministic they are. If you have tests that sometimes fail, but everyone just re-runs them because they may pass again later, then you have flaky tests

This situation worsens if we fall into the trap of adding a new end-to-end test for every piece of functionality we add. Show me a codebase where every new story results in a new end-to-end test, and I’ll show you a bloated test suite that has poor feedback cycles and huge overlaps in test coverage

Originally just for Ruby, Pact now includes JVM and .NET ports

If you are interested in reading more about testing, I recommend Agile Testing by Lisa Crispin and Janet Gregory (Addison-Wesley), which among other things covers the use of the testing quadrant in more detail.

Libraries exist for a number of different platforms that allow our services to send metrics to standard systems. Codahale’s Metrics library is one such example library for the JVM

Some of these libraries, such as Hystrix

Riemann is an event server that allows for fairly advanced aggregation and routing of events and can form part of such a solution

Suro is Netflix’s data pipeline and operates in a similar space. Suro is explicitly used to handle both metrics associated with user behavior, and more operational data like application logs

in my earlier publication Lightweight Systems for Realtime Monitoring (O’Reilly).

At the time of writing, OpenAM and Gluu are two of the very few options available in this space, compared to a wealth of options for SAML (including Active Directory, which seems to be everywhere)

If you’re using HTTP, it could populate headers with this information. Shibboleth is one tool that can do this for you, and I’ve seen it used with Apache to great effect to handle integration with SAML-based identity providers.

Gateway layers tend to take on more and more functionality, which itself can end up being a giant coupling point. And the more functionality something has, the greater the attack surface

This problem, unfortunately, has no simple answer, because it isn’t a simple problem. Be aware that it exists, though.

As you can see, the choices we made about what technology to use were based on an understanding of the nature of the information being secured

There are automated tools that can probe our systems for vulnerabilities, such as by looking for cross-site scripting attacks. The Zed Attack Proxy (aka ZAP) is a good example. Informed by the work of OWASP, ZAP attempts to re-create malicious attacks on your website

As we’ll learn in this chapter, you ignore your company’s organization chart at your peril!

Melvin Conway’s paper How Do Committees Invent, published in Datamation magazine in April 1968,

Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization’s communication structure.

If you have four groups working on a compiler, you’ll get a 4-pass compiler

In their study, in which they matched similar product pairs from each type of organization, the authors found that the more loosely coupled organizations actually created more modular, less coupled systems, whereas the more tightly focused organization’s software was less modularized.

Early on, Amazon started to understand the benefits of teams owning the whole lifecycle of the systems they managed

This model is certainly one I favor. It pushes the decisions to the people best able to make them, giving the team both increased power and autonomy, but also making it accountable for its work

Conway’s law highlights the perils of trying to enforce a system design that doesn’t match the organization. This leads us to trying to align service ownership to colocated teams, which themselves are aligned around the same bounded contexts of the organization

When you get down to it, we discovered the hard way that systems that just act slow are much harder to deal with than systems that just fail fast. In a distributed system, latency kills

Such a setup is called eventually consistent, and if you can handle the temporary inconsistency it is a fairly easy and common way to help scale systems

Be warned, however: this sort of pattern is quite a shift away from a model where a single data store handles all our CRUD operations. I’ve seen more than one experienced development team struggle to get this pattern right!

Be warned, however: this sort of pattern is quite a shift away from a model where a single data store handles all our CRUD operations. I’ve seen more than one experienced development team struggle to get this pattern right

For every public-facing website I’ve worked on, we’ve ended up doing a mix of all three approaches. But for more than one distributed system, I’ve gotten away with no caching at all. But it all comes down to knowing what load you need to handle, how fresh your data needs to be, and what your system can do right now. Knowing that you have a number of different tools at your disposal is just the beginning.

For every public-facing website I’ve worked on, we’ve ended up doing a mix of all three approaches. But for more than one distributed system, I’ve gotten away with no caching at all. But it all comes down to knowing what load you need to handle, how fresh your data needs to be, and what your system can do right now. Knowing that you have a number of different tools at your disposal is just the beginning

For a more in-depth discussion of the various merits, take a look at the book REST In Practice (O’Reilly) or read section 13 of the HTTP 1.1 specification, which describes how both clients and servers are supposed to implement these various controls.

A technique I saw used at the Guardian, and subsequently elsewhere, was to crawl the existing live site periodically to generate a static version of the website that could be served in the event of an outage. Although this crawled version wasn’t as fresh as the cached content served from the live system, in a pinch it could ensure that a version of the site would get displayed

Be careful about caching in too many places! The more caches between you and the source of fresh data, the more stale the data can be, and the harder it can be to determine the freshness of the data that a client eventually sees. This can be especially problematic with a microservice architecture where you have multiple services involved in a call chain. Again, the more caching you have, the harder it will be to assess the freshness of any piece of data. So if you think a cache is a good idea, keep it simple, stick to one, and think carefully before adding more!

Caching can be very powerful indeed, but you need to understand the full path of data that is cached from source to destination to really appreciate its complexities and what can go wrong

Which is right, AP or CP? Well, the reality is it depends. As the people building the system, we know the trade-off exists

You’ll often see posts about people beating the CAP theorem. They haven’t. What they have done is create a system where some capabilities are CP, and some are AP. The mathematical proof behind the CAP theorem holds. Despite many attempts at school, I’ve learned that you don’t beat math.

You’ll often see posts about people beating the CAP theorem. They haven’t. What they have done is create a system where some capabilities are CP, and some are AP. The mathematical proof behind the CAP theorem holds. Despite many attempts at school, I’ve learned that you don’t beat math

Zookeeper itself is fairly generic in what it offers, which is why it is used for so many use cases. You can think of it just as a replicated tree of information that you can be alerted about when it changes. This means that you’ll typically build things on top of it to suit your particular use case. Luckily, client libraries exist for most languages out there.

Two different pieces of technology, Swagger and HAL, try to make this a reality, and both are worth looking at.

Services should also hide their databases to avoid falling into one of the most common sorts of coupling that can appear in traditional service-oriented architectures, and use data pumps or event data pumps to consolidate data across multiple services for reporting purposes.

I get asked this question a lot. My first piece of advice would be that the less well you understand a domain, the harder it will be for you to find proper bounded contexts for your services. As we discussed previously, getting service boundaries wrong can result in having to make lots of changes in service-to-service collaboration — an expensive operation. So if you’re coming to a monolithic system for which you don’t understand the domain, spend some time learning what the system does first, and then look to identify clean module boundaries prior to splitting out services.

right