Devops Handbook Quotes

In high-performing organizations, everyone within the team shares a common goal—quality, availability, and security aren’t the responsibility of individual departments, but are a part of everyone’s job, every day.

Ask a programmer to review ten lines of code, he’ll find ten issues. Ask him to do five hundred lines, and he’ll say it looks good.

By relentless and constant experimentation in their daily work, they were able to continually increase capacity, often without adding any new equipment or hiring more people.

management hints that the person guilty of committing the error will be punished. They then create more processes and approvals to prevent the error from happening again.

Bill Baker, a distinguished engineer at Microsoft, quipped that we used to treat servers like pets: “You name them and when they get sick, you nurse them back to health. [Now] servers are [treated] like cattle. You number them and when they get sick, you shoot them.

It’s difficult to overstate the enormity of this problem—it affects every organization, independent of the industry we operate in, the size of our organization, whether we are profit or non-profit. Now more than ever, how technology work is managed and performed predicts whether our organizations will win in the marketplace, or even survive.

When these accidents affect our customers, we seek to understand why it happened. The root cause is often deemed to be human error, and the all too common management response is to “name, blame, and shame” the person who caused the problem.† And, either subtly or explicitly, management hints that the person guilty of committing the error will be punished. They then create more processes and approvals to prevent the error from happening again. Dr. Sidney Dekker, who codified some of the key elements of safety culture and coined the term just culture, wrote, “Responses to incidents and accidents that are seen as unjust can impede safety investigations, promote fear rather than mindfulness in people who do safety-critical work, make organizations more bureaucratic rather than more careful, and cultivate professional secrecy, evasion, and self-protection.” These issues are especially problematic in the technology value stream—our work is almost always performed within a complex system, and how management chooses to react to failures and accidents leads to a culture of fear, which then makes it unlikely that problems and failure signals are ever reported. The result is that problems remain hidden until a catastrophe occurs.

As has been proven time and again, the further the distance between the person doing the work (i.e., the change implementer) and the person deciding to do the work (i.e., the change authorizer), the worse the outcome.

When new learnings are discovered locally, there must also be some mechanism to enable the rest of the organization to use and benefit from that knowledge.

believe that this is simpler than it sounds. It is about identifying the obstacles in our way and taking today’s best-practice ideas—those found in the Agile Manifesto and in books like Lean Startup, Lean Software Development, Lean Enterprise, The DevOps Handbook, and others on today’s management bookshelves—and applying them to IT leadership.

In order for a field or discipline to progress and mature, it needs to reach a point where it can thoughtfully reflect on its origins, seek out a diverse set of perspectives on those reflections, and place that synthesis into a context that is useful for how the community pictures the future.

For monitoring, alerting and trending you can use VictorOps, PagerDuty or Sensu. Logging provides an interface for analyzing various machine-generated data, so application-issues detection, as well as resolution, is easier to achieve. For logging, you can use Logotries, Sumo Logic, PaperTrail, Loggly, and Splunk.

To have humans executing tests that should be automated is a waste of human potential.

Monitoring is so important that our monitoring systems need to be more available and scalable than the systems being monitored.

we should ensure that all potentially significant application events generate logging entries, including

When we have a tightly-coupled architecture, small changes can result in large scale failures. As

Work is not done when Development completes the implementation of a feature—rather, it is only done when our application is running successfully in production, delivering value to the customer.

Small batch sizes result in less WIP, faster lead times, faster detection of errors, and less rework.

This includes the creation of automated build, integration, and test processes so that we can immediately detect when a change has been introduced that takes us out of a correctly functioning and deployable state.

telemetry is what enables us to assemble our best understanding of reality and detect when our understanding of reality is incorrect.

However, for decades we have ended up with silos of information, where Development only creates logging events that are interesting to developers, and Operations only monitors whether the environments are up or down. As

Once we have centralized our logs, we can transform them into metrics by

In addition to collecting telemetry from our production services and environments, we must also collect telemetry from our deployment pipeline when important events occur, such as when our automated tests pass or fail and when we perform deployments to any environment. We

We should also collect telemetry on how long it takes us to execute our builds and tests. By

all team members as well as passers-by can see the latest information at a glance: count of automated tests, velocity, incident reports, continuous integration status, and so on. This

Gatekeeper, the Facebook feature toggling service.

Much of my career has involved rewrites of critical systems. You would think such a thing is easy—just make the new one do what the old one did. Yet they are always much more complex than they seem, and overflowing with risk. The big cut-over date looms, and the pressure is on. While new features (there are always new features) are liked, old stuff has to remain. Even old bugs often need to be added to the rewritten system.

This is the reality of operating complex systems; no single person can see the whole system and understand how all the pieces fit together.

One of the most significant things they did to help change the outcomes of deployments was to have all Facebook engineers, engineering managers, and architects rotate through on-call duty for the services they built. By

the Third Way focuses on creating a culture of continual learning and experimentation. These

we also design our system of work so that we can multiply the effects of new knowledge, transforming local discoveries into global improvements. Regardless of where someone performs work, they do so with the cumulative and collective experience of everyone in the organization.

Studies have shown that the time to complete even simple tasks, such as sorting geometric shapes, significantly degrades when multitasking. Of

Complex work is managed so that problems in design and operations are revealed.

Feedback and feedforward loops cause components within a system to reinforce or counteract each other.

It is virtually impossible to make any business decision that doesn’t result in at least one IT change.

high performers were twice as likely to exceed profitability, market share, and productivity goals. And,

Feedback loops not only enable quick detection and recovery of problems, they also inform us on how to prevent these problems from occurring again in the future. Doing this increases the quality and safety of our system of work, and creates organizational learning.

Leaders are responsible for the design and operation of processes at a higher level of aggregation where others have less perspective and authority.

every Facebook user was part of a massive load testing program, which

One of the most important parts of any improvement initiative is to define a measurable goal with a clearly defined deadline, between six months and two years in the future. It

Ensure releases can always be performed during normal business hours with zero downtime.

what’s most important is that the information we display is up to date, and that we constantly revise what we measure to make sure it’s helping us understand progress toward our current target conditions.

A significant benefit of this is that when production incidents are shown in the same work systems as development work, it will be obvious when ongoing incidents should halt other work, especially when we have a kanban board.

because everything is being recorded, we may not need to ask someone else for help in the future—we simply search for it.

In the DevOps ideal, developers receive fast, constant feedback on their work, which enables them to quickly and independently implement, integrate, and validate their code, and have the code deployed into the production environment (either

work can bounce between teams endlessly due to incomplete information, or work can be passed onto downstream work centers with problems that remain completely invisible until we are late delivering what we promised to the customer or our application fails in the production environment.

interrupting technology workers is easy, because the consequences are invisible to almost everyone, even though the negative impact to productivity may be far greater than in manufacturing.

Even under the best circumstances, some knowledge is inevitably lost with each handoff. With

We cannot achieve deployments on demand if each of our production code deployments take weeks or months to perform (i.e., each deployment requires 1,300 manual, error-prone steps involving up to three hundred engineers). The countermeasure is to automate our deployments as much as possible, with the goal of being completely automated so they can be done self-service by any developer.

We cannot achieve deployments on demand if every code deployment requires two weeks to set up our test environments and data sets, and another four weeks to manually execute all our regression tests. The countermeasure is to automate our tests so we can execute deployments safely and to parallelize them so the test rate can keep up with our code development rate.

we are no longer able to respond quickly to our changing competitive landscape, nor are we able to provide stable, reliable service to our customers. As

failure results in reflection and genuine inquiry.

Another benefit of having Development and Operations using a shared tool is a unified backlog, where everyone prioritizes improvement projects from a global perspective, selecting

the %C/A can be obtained by asking downstream customers what percentage of the time they receive work that is ‘usable as is,’ meaning that they can do their work without having to correct the information that was provided, add missing information that should have been supplied, or clarify information that should have and could have been clearer.

In technology, our work happens almost entirely within complex systems with a high risk of catastrophic consequences. As

New local knowledge is exploited globally throughout the organization.

The more assumptions we can invalidate, the faster we can find and fix problems, increases our resilience, agility, and ability to learn and innovate.

our goal is to create fast feedback and fast forward loops wherever work is performed, at all stages of the technology value stream, encompassing Product Management, Development, QA, Infosec, and Operations. This

We also create pervasive telemetry so we can see how all our system components are operating in the production environment, so that we can quickly detect when they are not operating as expected. Telemetry

By seeing problems as they occur and swarming them until effective countermeasures are in place, we continually shorten and amplify our feedback loops, a core tenet of virtually all modern process improvement methodologies. This maximizes the opportunities for our organization to learn and improve.

the systems most prone to failure are also our most important and are at the epicenter of our most urgent changes. When

when something does go wrong, we conduct blameless post-mortems, not to punish anyone, but to better understand what caused the accident and how to prevent it. This

for those organizations that provided a stock ticker symbol, we found that high performers had 50% higher market capitalization growth over three years. They also had higher employee job satisfaction, lower rates of employee burnout, and their employees were 2.2 times more likely to recommend their organization to friends as a great place to work.

Many psychologists assert that creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings—we deprive other people of their ability to control their own outcomes and even create a culture where people are afraid to do the right thing because of fear of punishment, failure, or jeopardizing their livelihood. This

Doing this requires disruption and innovation, which puts us at odds with groups who are currently responsible for daily operations and the internal bureaucracies, and who will almost always win.

In order to be able to know if we are making progress toward our goal, it’s essential that everyone in the organization knows the current state of work. There

high-performing manufacturing operations require and actively promote learning—instead of work being rigidly defined, the system of work is dynamic, with line workers performing experiments in their daily work to generate new improvements, enabled by rigorous standardization of work procedures and documentation of the results.

We increase flow by making work visible, by reducing batch sizes and intervals of work, and by building quality in, preventing defects from being passed to downstream work centers. By

doing the same thing twice will not predictably or necessarily lead to the same result. It is this characteristic that makes static checklists and best practices, while valuable, insufficient to prevent catastrophes from occurring. See

Every company is a technology company, regardless of what business they think they’re in. A bank is just an IT company with a banking license.

When people are trapped in this downward spiral for years, especially those who are downstream of Development, they often feel stuck in a system that pre-ordains failure and leaves them powerless to change the outcomes. This powerlessness is often followed by burnout, with the associated feelings of fatigue, cynicism, and even hopelessness and despair.

If adopting DevOps could enable us, through better management and increased operational excellence, to halve that waste and redeploy that human potential into something that’s five times the value (a modest proposal), we could create $2.6 trillion of value per year.

The work done at Netflix highlights one very specific way we can use telemetry to mitigate problems before they impact our customer.

Having developers share responsibility for the quality of the systems they build not only improves outcomes but also accelerates learning.

In Operations, many of our data sets have what we call ‘chi squared’ distribution. Using

Chuck Rossi, Director of Release Engineering at Facebook, described, “All the code supporting every feature we’re planning to launch over the next six months has already been deployed onto our production servers. All we need to do is turn it on.

the architecture that our services operate within dictates how we test and deploy our code. This

we must ensure that the applications we build and operate are creating sufficient telemetry.

creating application and infrastructure telemetry to be one of the highest return investments we’ve made. In

When deciding whether a message should be ERROR or WARN, imagine being woken up at 4 a.m. Low printer toner is not an ERROR.

common practice of using rumor and hearsay, which can lead to the unfortunate metric of mean time until declared innocent—how quickly can we convince everyone else that we didn’t cause the outage.

should be as easy as writing one line of code to create a new metric that shows up in a common dashboard where everyone in the value stream can see it.

Alert fatigue is the single biggest problem we have right now…We need to be more intelligent about our alerts or we’ll all go insane.

we need to design our systems so that they are continually creating telemetry, widely

It was important to us that for a developer, adding production telemetry didn’t feel as difficult as doing a database schema change.

Over-alerting causes Operations engineers to be woken up in the middle of the night for protracted periods of time, even when there are few actions that they can appropriately take. The

our most fragile artifacts support either our most important revenue-generating systems or our most critical projects. In

Even high-profile product and feature releases become routine by using dark launch techniques. Long before the launch date, we put all the required code for the feature into production, invisible to everyone except internal employees and small cohorts of real users, allowing us to test and evolve the feature until it achieves the desired business goal.

The key thing we should care about is not the form, but the outcomes: deployments should be low-risk, push-button events we can perform on demand.

Up-front analysis helps us identify the smallest possible piece of work that will usefully achieve a business outcome using

To even get feedback from our integration process would require twenty-four to thirty-six hours.

One of the inherent challenges with initiatives such as DevOps transformations is that they are inevitably in conflict with ongoing business operations. Part

Testing is done in scarce integration test environments, which often require weeks to obtain and configure. The

developers should be able to understand and update the code of a service without knowing anything about the internals of its peer services. Services

Services interact with their peers strictly through APIs and thus don’t share data structures, database schemata, or other internal representations of objects. Bounded

We cannot achieve deployments on-demand if we always have to wait weeks or months for production or test environments. The countermeasure is to create environments that are on demand and completely self-serviced, so that they are always available when we need them.

In addition to the human suffering that comes with the current way of working, the opportunity cost of the value that we could be creating is staggering—the authors believe that we are missing out on approximately $2.6 trillion of value creation per year, which is, at the time of this writing, equivalent to the annual economic output of France, the sixth-largest economy in the world.

build ever-deeper knowledge about how to manage the systems for doing our work, converting inevitable up-front ignorance into knowledge.

This process of applying stress to increase resilience was named antifragility by author and risk analyst Nassim Nicholas Taleb.

In the technology value stream, we optimize for downstream work centers by designing for operations, where operational non-functional requirements (e.g., architecture, performance, stability, testability, configurability, and security) are prioritized as highly as user features.