Devops Handbook Quotes

In high-performing organizations, everyone within the team shares a common goal—quality, availability, and security aren’t the responsibility of individual departments, but are a part of everyone’s job, every day.

Ask a programmer to review ten lines of code, he’ll find ten issues. Ask him to do five hundred lines, and he’ll say it looks good.

management hints that the person guilty of committing the error will be punished. They then create more processes and approvals to prevent the error from happening again.

By relentless and constant experimentation in their daily work, they were able to continually increase capacity, often without adding any new equipment or hiring more people.

Every company is a technology company, regardless of what business they think they’re in. A bank is just an IT company with a banking license.

integrating the objectives of QA and Operations into everyone's daily work reduces firefighting, hardship, and toil, while making people more productive and increasing joy in the work we do.

Cagan notes that when organizations do not pay their “20% tax,” technical debt will increase to the point where an organization inevitably spends all of its cycles paying down technical debt.

However, interrupting technology workers is easy, because the consequences are invisible to almost everyone, even though the negative impact to productivity may be far greater than in manufacturing.

Organizations which design systems ...are constrained to produce designs which are copies of the communication structures of the organizations.. the larger an organization is, the less flexibility it has and the more pronounced the phenomenon

Bill Baker, a distinguished engineer at Microsoft, quipped that we used to treat servers like pets: “You name them and when they get sick, you nurse them back to health. [Now] servers are [treated] like cattle. You number them and when they get sick, you shoot them.

Broadly speaking, to acheive DevOps outcomes, we need to reduce the effects of functional orientation ("optimizing for cost") and enable market orientation ("optimizing for speed") so we can have many small teams working safely and independently, quickly delivering value to the customer

It’s difficult to overstate the enormity of this problem—it affects every organization, independent of the industry we operate in, the size of our organization, whether we are profit or non-profit. Now more than ever, how technology work is managed and performed predicts whether our organizations will win in the marketplace, or even survive.

Many psychologists assert that creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings—we deprive other people of their ability to control their own outcomes and even create a culture where people are afraid to do the right thing because of fear of punishment, failure, or jeopardizing their livelihood

However, interrupting technology workers is easy, because the consequences are invisible to almost everyone, even though the negative impact to productivity may be far greater than in manufacturing. For instance, an engineer assigned to multiple projects must switch between tasks, incurring all the costs of having to re-establish context, as well as cognitive rules and goals.

high performance starts with organizations whose leadership focuses on building an environment where people from different backgrounds and with different identities, experiences, and perspectives can feel psychologically safe working together, and where teams are given the necessary resources, capacity, and encouragement to experiment and learn together in a safe and systematic way.

DevOps benefits all of us in the technology value stream, whether we are Dev, Ops, QA, Infosec, Product Owners, or customers. It brings joy back to developing great products, with fewer death marches. It enables humane work conditions with fewer weekends worked and fewer missed holidays with our loved ones. It enables teams to work together to survive, learn, thrive, delight our customers, and help our organization succeed.

DevOps requires potentially new cultural and management norms and changes in our technical practices and architecture. This requires a coalition that spans business leadership, Product Management, Development, QA, IT Operations, Information Security, and even Marketing, where many technology initiatives originate. When all these teams work together, we can create a safe system of work, enabling small teams to quickly and independently develop and validate code that can be safely deployed to customers. This results in maximizing developer productivity, organizational learning, high employee satisfaction, and the ability to win in the marketplace.

In typical DevOps transformations, as we progress from deployment lead times measured in months or quarters to lead times measured in minutes, the constraint usually follows this progression: Environment creation: We cannot achieve deployments on-demand if we always have to wait weeks or months for production or test environments. The countermeasure is to create environments that are on demand and completely self-serviced, so that they are always available when we need them. Code deployment: We cannot achieve deployments on demand if each of our production code deployments take weeks or months to perform (i.e., each deployment requires 1,300 manual, error-prone steps, involving up to three hundred engineers). The countermeasure is to automate our deployments as much as possible, with the goal of being completely automated so they can be done self-service by any developer. Test setup and run: We cannot achieve deployments on demand if every code deployment requires two weeks to set up our test environments and data sets, and another four weeks to manually execute all our regression tests. The countermeasure is to automate our tests so we can execute deployments safely and to parallelize them so the test rate can keep up with our code development rate. Overly tight architecture: We cannot achieve deployments on demand if overly tight architecture means that every time we want to make a code change we have to send our engineers to scores of committee meetings in order to get permission to make our changes. Our countermeasure is to create more loosely-coupled architecture so that changes can be made safely and with more autonomy, increasing developer productivity.

DevOps Evangelist An expert Consultant who can now evangelise a DevOps solution for an Enterprise. Typical tasks: Scaling DevOps capabilities across Enterprise Change Management Organisation Alignment This is a high-end Consulting role, with a heavy technology bend. At a DevOps Engineer level, it’s mostly about deep technical skills. As you evolve into a solution architect kind of role, along with the technical acumen, it’s also about leadership skills to build consensus/agreements and making a team work together.

Udemy Coursera Simplilearn Skillshare

Stack Overflow profile

GitHub Profile

The infrastructure that supports our continuous integration and continuous deployment processes also presents a new surface area vulnerable to attack. For instance, if someone compromises the servers running the deployment pipeline that has the credentials for our version control system, it could enable someone to steal source code. Worse, if the deployment pipeline has write access, an attacker could also inject malicious changes into our version control repository and, therefore, inject malicious changes into our application and services.

By speeding up flow through the technology value stream, we reduce the lead time required to fulfill internal or customer requests, especially the time required to deploy code into the production environment. By doing this, we increase the quality of work as well as our throughput and boost our ability to innovate and out-experiment the competition. The resulting practices include continuous build, integration, test, and deployment processes, creating environments on demand, limiting work in process (WIP), and building systems and organizations that are safe to change.

Three Ways: Flow, Feedback, and Continual Learning and Experimentation.

In Operations, we may deal with this problem with the following rule of thumb: When something goes wrong in production, we just reboot the server. If that doesn’t work, reboot the server next to it. If that doesn’t work, reboot all the servers. If that doesn’t work, blame the developers, they’re always causing outages.

Without automated testing, continuous integration is the fastest way to get a big pile of junk that never compiles or runs correctly.

Compounding the issue, the person performing the work often has little visibility or understanding of how their work relates to any value stream goals (e.g., “I’m just configuring servers because someone told me to.”). This places workers in a creativity and motivation vacuum.

measure:9 •deployment frequency •deployment cycle time •change failure rate •development cycle time •number of incidents •mean time to recover (MTTR)

And by 2010, with the introduction of DevOps and the never-ending commoditization of hardware, software, and now the cloud, features (even entire startup companies) could be created in weeks, quickly being deployed into production in just hours or minutes—for these organizations, deployment finally became routine and low risk.

These organizations are able to perform experiments to test business ideas, discovering which ideas create the most value for customers and the organization as a whole, and which are then further developed into features that can be rapidly and safely deployed into production.

Today, organizations adopting DevOps principles and practices often deploy changes hundreds or even thousands of times per day.

In an age where competitive advantage requires fast time to market and relentless experimentation, organizations that are unable to replicate these outcomes are destined to lose in the marketplace to more nimble competitors and could potentially go out of business entirely, much like the manufacturing organizations that did not adopt Lean principles.

regardless of what industry we are competing in, the way we acquire customers and deliver value to them is dependent on the technology value stream.

It’s difficult to overstate the enormity of this problem—it affects every organization, independent of the industry we operate in, the size of our organization, whether we are profit or nonprofit.

Now more than ever, how technology work is managed and performed predicts whether our organizations will win in the marketplace or even survive.

deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

∘Event severity: How severe was this issue? This directly relates to the impact on the service and our customers.

How long after we knew there was a problem did it take for us to restore service?

deployments occur throughout the business day when everyone is already in the office and without our customers even noticing—except when they see new features and bug fixes that delight them.

And, by deploying code in the middle of the workday, IT Operations is working during normal business hours like everyone else for the first time in decades.

Instead of security and compliance activities only being performed at the end of the project, controls are integrated into every stage of daily work in the software development life cycle, resulting in better quality, security, and compliance outcomes.

This book and the DevOps community as a whole have shown time and time again that DevOps practices and processes can take even the most legacy-riddled, old “horse” enterprise organization and turn it into a nimble technology organization.

In 2021, it is clearer than ever before that every business is a technology business and every leader is a technology leader.

Part III describes how to accelerate flow by building the foundations of our deployment pipeline: enabling fast and effective automated testing, continuous integration, continuous delivery, and architecting for low-risk releases.

Part IV discusses how to accelerate and amplify feedback by creating effective production telemetry to see and solve problems, better anticipate problems and achieve goals,

Part V describes how we accelerate continual learning and experimentation by establishing a just culture, converting local discoveries into global improvements,

Three Ways: Flow, Feedback, and Continual Learning

how to properly integrate security and compliance

increase the success of DevOps initiatives,

technology value stream as the process required to convert a business hypothesis into a technology-enabled service or feature that delivers value to the customer.

The Third Way enables the creation of a generative, high-trust culture that supports a dynamic, disciplined, and scientific approach to experimentation and risk-taking, facilitating the creation of organizational learning, both from our successes and failures.

working in small batches (the First Way), fast feedback and monitoring (the Second Way), and a generative culture (the Third Way).

DevOps isn’t about automation, just as astronomy isn’t about telescopes.”2

While many of the DevOps patterns shown in this book require automation, DevOps also requires cultural norms and an architecture that allows for the shared goals to be achieved throughout the IT value stream.

DevOps principles and practices are compatible with Agile,

Agile often serves as an effective enabler of DevOps because of its focus on small teams continually delivering high-quality code to customers.

DevOps and its resulting technical, architectural, and cultural practices represent a convergence of many philosophical and management movements (including): Lean, Theory of Constraints, Toyota production system, resilience engineering, learning organizations, safety culture, Human factors, high-trust management cultures, servant leadership, organizational change management, and Agile methods.

in our

Stop starting. Start finishing.

In any value stream, there is always a direction of flow, and there is always one and only one constraint; any improvement not made at that constraint is an illusion.” If

The deal [between product owners and] engineering goes like this: Product management takes 20% of the team’s capacity right off the top and gives this to engineering to spend as they see fit. They might use it to rewrite, re-architect, or re-factor problematic parts of the code base...whatever they believe is necessary to avoid ever having to come to the team and say, ‘we need to stop and rewrite [all our code].’ If you’re in really bad shape today, you might need to make this 30% or even more of the resources. However, I get nervous when I find teams that think they can get away with much less than 20%.

When every team expedites their work, the net result is that every project ends up moving at the same slow crawl.

In high-performing organizations, everyone within the team shares a common goal—quality, availability, and security aren’t the responsibility of individual departments but are a part of everyone’s job, every day.

Our goal with a product-based funding model is to value the achievement of organizational and customer outcomes, such as revenue, customer lifetime value, or customer adoption rate, ideally with the minimum of output (e.g., amount of effort or time, lines of code). Contrast this to how projects are typically measured, such as whether it was completed within the promised budget, time, and scope.

By doing this, Development and Operations may end up creating a shared work queue, instead of each silo using a different one (e.g., Development uses JIRA while Operations uses ServiceNow). A significant benefit of this is that when production incidents

DevOps isn’t about automation, just as astronomy isn’t about telescopes.

high performers use a disciplined approach to solving problems. This is in contrast to the more common practice of using rumor and hearsay, which can lead to the unfortunate metric of mean time until declared innocent—how quickly can we convince everyone else that we didn’t cause the outage. When there is a culture of blame around outages and problems, groups may avoid documenting changes and displaying telemetry where everyone can see them to avoid being blamed for outages.

The work done at Netflix highlights one very specific way we can use telemetry to mitigate problems before they impact our customer.

Alert fatigue is the single biggest problem we have right now…We need to be more intelligent about our alerts or we’ll all go insane.

In Operations, many of our data sets have what we call ‘chi squared’ distribution. Using

Over-alerting causes Operations engineers to be woken up in the middle of the night for protracted periods of time, even when there are few actions that they can appropriately take. The

Chuck Rossi, Director of Release Engineering at Facebook, described, “All the code supporting every feature we’re planning to launch over the next six months has already been deployed onto our production servers. All we need to do is turn it on.

Much of my career has involved rewrites of critical systems. You would think such a thing is easy—just make the new one do what the old one did. Yet they are always much more complex than they seem, and overflowing with risk. The big cut-over date looms, and the pressure is on. While new features (there are always new features) are liked, old stuff has to remain. Even old bugs often need to be added to the rewritten system.

Up-front analysis helps us identify the smallest possible piece of work that will usefully achieve a business outcome using

To even get feedback from our integration process would require twenty-four to thirty-six hours.

the architecture that our services operate within dictates how we test and deploy our code. This

creating application and infrastructure telemetry to be one of the highest return investments we’ve made. In

Gatekeeper, the Facebook feature toggling service.

The key thing we should care about is not the form, but the outcomes: deployments should be low-risk, push-button events we can perform on demand.

telemetry is what enables us to assemble our best understanding of reality and detect when our understanding of reality is incorrect.

Once we have centralized our logs, we can transform them into metrics by

We should also collect telemetry on how long it takes us to execute our builds and tests. By

Monitoring is so important that our monitoring systems need to be more available and scalable than the systems being monitored.

we must ensure that the applications we build and operate are creating sufficient telemetry.

every Facebook user was part of a massive load testing program, which

This is the reality of operating complex systems; no single person can see the whole system and understand how all the pieces fit together.

we need to design our systems so that they are continually creating telemetry, widely

However, for decades we have ended up with silos of information, where Development only creates logging events that are interesting to developers, and Operations only monitors whether the environments are up or down. As

When deciding whether a message should be ERROR or WARN, imagine being woken up at 4 a.m. Low printer toner is not an ERROR.

we should ensure that all potentially significant application events generate logging entries, including

common practice of using rumor and hearsay, which can lead to the unfortunate metric of mean time until declared innocent—how quickly can we convince everyone else that we didn’t cause the outage.

should be as easy as writing one line of code to create a new metric that shows up in a common dashboard where everyone in the value stream can see it.

It was important to us that for a developer, adding production telemetry didn’t feel as difficult as doing a database schema change.

all team members as well as passers-by can see the latest information at a glance: count of automated tests, velocity, incident reports, continuous integration status, and so on. This

To have humans executing tests that should be automated is a waste of human potential.

In addition to collecting telemetry from our production services and environments, we must also collect telemetry from our deployment pipeline when important events occur, such as when our automated tests pass or fail and when we perform deployments to any environment. We

Having developers share responsibility for the quality of the systems they build not only improves outcomes but also accelerates learning.

INCREASE THE VISIBILITY OF WORK In order to be able to know if we are making progress toward our goal, it’s essential that everyone in the organization knows the current state of work. There are many ways to make the current state visible, but what’s most important is that the information we display is up to date, and that we constantly revise what we measure to make sure it’s helping us understand progress toward our current target conditions.

What these organizations have in common is a high-trust culture that enables all departments to work together effectively, where all work is transparently prioritized and there is sufficient slack in the system to allow high-priority work to be completed quickly. This is, in part, enabled by automated self-service platforms that build quality into the products everyone is building.