Devops Quotes

Improving daily work is even more important than doing daily work.

Any improvements made anywhere besides the bottleneck are an illusion.

Being able to take needless work out of the system is more important than being able to put more work into the system.

A great team doesn’t mean that they had the smartest people. What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists.

Something seems wrong in a world where half the e-mail messages sent are urgent. Can everything really be that important?

To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy.

We need to create a culture that reinforces the value of taking risks and learning from failure and the need for repetition and practice to create mastery.

The only thing more dangerous than a developer is a developer conspiring with Security.

until code is in production, no value is actually being generated, because it’s merely WIP stuck in the system.

technical debt’ that is not being paid down. It comes from taking shortcuts, which may make sense in the short-term. But like financial debt, the compounding interest costs grow over time. If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.

Practice creates habits, and habits create mastery of any process or skill.

Unplanned work is what prevents you from doing it. Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it.

You get what you design for. Chester, your peer in Development, is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful ’itties.

repetition creates habits, and habits are what enable mastery.

CIO stands for “Career Is Over.

Left unchecked, technical debt will ensure that the only work that gets done is unplanned work!

a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.

In high-performing organizations, everyone within the team shares a common goal—quality, availability, and security aren’t the responsibility of individual departments, but are a part of everyone’s job, every day.

High performers understand that they don’t have to trade speed for stability or vice versa, because by building quality in they get both.

ARCHITECTS SHOULD FOCUS ON ENGINEERS AND OUTCOMES, NOT TOOLS OR TECHNOLOGIES

Ask a programmer to review ten lines of code, he’ll find ten issues. Ask him to do five hundred lines, and he’ll say it looks good.

Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful.

Developers should be able to run all automated tests on their workstations in order to triage and fix defects.

Adrian Cockcroft, Netflix’s seminal cloud architect, was once asked by a senior leader in a Fortune 500 company where he got his amazing people from. Cockcroft replied, “I hired them from you!

every work center is made up of four things: the machine, the man, the method, and the measures.

Remember, it goes beyond reducing WIP. Being able to take needless work out of the system is more important than being able to put more work into the system.

The most important characteristic of high-performing teams is that they are never satisfied: they always strive to get better. High performers make improvement part of everybody’s daily work.

we’re hearing more lately: something called “DevOps.” Maybe everyone attending this party is a form of DevOps, but I suspect it’s something much more than that. It’s Product Management, Development, IT Operations, and even Information Security all working together and supporting one another.

We found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.

As Deming said, ’whenever there is fear, you get the wrong numbers

In any value stream, there is always a direction of flow, and there is always one and only constraint; any improvement not made at that constraint is an illusion.

Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place” (Deming 2000).

Knowledge is power, and you should give power to those who have the knowledge.

A key goal of continuous delivery is changing the economics of the software delivery process so the cost of pushing out individual changes is very low.

The small batches principle is part of the DevOps methodology. It comes from the Lean Manufacturing movement, which is often called just-in-time (JIT) manufacturing. It

There is really no bad software development process. There is only how you are doing it today and better.

Situations like this only reinforce my deep suspicion of developers: They’re often carelessly breaking things and then disappearing, leaving Operations to clean up the mess.

There should be absolutely no way that the Dev and QA environments don’t match the production environment.

Our analysis is clear: in today’s fast-moving and competitive world, the best thing you can do for your products, your company, and your people is institute a culture of experimentation and learning, and invest in the technical and management capabilities that enable it.

If you can’t out-experiment and beat your competitors in time to market and agility, you are sunk. Features are always a gamble. If you’re lucky, ten percent will get the desired benefits. So the faster you can get those features to market and test them, the better off you’ll be. Incidentally, you also pay back the business faster for the use of capital, which means the business starts making money faster, too.

Broadly speaking, to acheive DevOps outcomes, we need to reduce the effects of functional orientation ("optimizing for cost") and enable market orientation ("optimizing for speed") so we can have many small teams working safely and independently, quickly delivering value to the customer

Another Forrester report states that DevOps is accelerating technology, but that organizations often overestimate their progress (Klavens et al. 2017). Furthermore, the report points out that executives are especially prone to overestimating their progress when compared to those who are actually doing the work.

Westrum’s description of a rule-oriented culture is perhaps best thought of as one where following the rules is considered more important than achieving the mission—and we have worked with teams in the US Federal Government we would have no issue describing as generative, as well as startups that are clearly pathological.

I’ve learned that while the finance goals are important, they’re not the most important. Finance can hit all our objectives, and the company still can fail. After all, the best accounts receivables team on the planet can’t save us if we’re in the wrong market with the wrong product strategy with an R&D team that can’t deliver.

Five Dysfunctions of a Team, by Patrick Lencioni. He writes that in order to have mutual trust, you need to be vulnerable.

Westrum’s theory posits that organizations with better information flow function more effectively.

much of what has been implemented is faux Agile—people following some of the common practices while failing to address wider organizational culture and processes.

We found that where code deployments are most painful, you’ll find the poorest software delivery performance, organizational performance, and culture.

How can we manage production if we don’t know what the demand, priorities, status of work in process, and resource availability are?

We increase flow by making work visible, by reducing batch sizes and intervals of work, and by building quality in, preventing defects from being passed to downstream work centers.

delivery lead time, deployment frequency, time to restore service, and change fail rate.

two parts to lead time: the time it takes to design and validate a product or feature, and the time to deliver the feature to customers.

We measured product delivery lead time as the time it takes to go from code committed to code successfully running in production

By relentless and constant experimentation in their daily work, they were able to continually increase capacity, often without adding any new equipment or hiring more people.

I’m starting to associate the smell of pizza with the futility of a death march.

How in the hell do you support and secure something that’s written in Microsoft Access? When

History is the collective consciousness of the then intellectuals and simply facts that show some numbers and events.

management hints that the person guilty of committing the error will be punished. They then create more processes and approvals to prevent the error from happening again.

What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists.

It’s not the upfront capital that kills you, it’s the operations and maintenance on the back end.

As a software engineer, how do you feel if your code was running in the production environment being used by millions of customers 30 minutes after you commit it to source control?

Unplanned work is not free. Quite the opposite. It's very expensive, because unplanned work comes at the expense of Planned work.

Business agility is not just about raw speed. It’s about how good you are at detecting and responding to changes in the market and being able to take larger and more calculated risks.

You’ve just described ‘technical debt’ that is not being paid down. It comes from taking shortcuts, which may make sense in the short-term. But like financial debt, the compounding interest costs grow over time. If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.

What got worked on was based on who yelled the loudest or most often, who could engineer the best side deals with the expediters, or who could get the ear of the highest ranking executive.

integrating the objectives of QA and Operations into everyone's daily work reduces firefighting, hardship, and toil, while making people more productive and increasing joy in the work we do.

software delivery is an exercise in continuous improvement, and our research shows that year over year the best keep getting better, and those who fail to improve fall further and further behind.

However, interrupting technology workers is easy, because the consequences are invisible to almost everyone, even though the negative impact to productivity may be far greater than in manufacturing.

we hypothesized that implementing CD would influence organizational culture. Our analysis shows that this is indeed the case. If you want to improve your culture, implementing CD practices will help.

Being a leader doesn’t mean you have people reporting to you on an organizational chart—leadership is about inspiring and motivating those around you. A good leader affects a team’s ability to deliver code, architect good systems, and apply Lean principles to how the team manages its work and develops products. All of these have a measurable impact on an organization’s profitability, productivity, and market share.

Traditionally siloed technical teams interact through complex ticketing systems and ritualistic request procedures, which may require director-level intervention. A team taking a more DevOps approach talks about the product throughout its lifecycle, discussing requirements, features, schedules, resources, and whatever else might come up. The focus is on the product, not building fiefdoms and amassing political power.

where mistakes are routinely punished and scapegoats fired. Punishing failure and “shooting the messenger” only cause people to hide their mistakes, and eventually, all desire to innovate is completely extinguished.

Goldratt taught us that in most plants, there are a very small number of resources, whether it’s men, machines, or materials, that dictates the output of the entire system. We call this the constraint—or bottleneck. Either

In ten years, I’m certain every COO worth their salt will have come from IT. Any COO who doesn’t intimately understand the IT systems that actually run the business is just an empty suit, relying on someone else to do their job.

Organizations which design systems ...are constrained to produce designs which are copies of the communication structures of the organizations.. the larger an organization is, the less flexibility it has and the more pronounced the phenomenon

I’ve seen this movie before. The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.

Bill Baker, a distinguished engineer at Microsoft, quipped that we used to treat servers like pets: “You name them and when they get sick, you nurse them back to health. [Now] servers are [treated] like cattle. You number them and when they get sick, you shoot them.

Brent. Brent, Brent, Brent! Can’t we do anything without him? Look at us! We’re trying to have a management discussion about commitments and resources, and all we do is talk about one guy! I don’t care how talented he is. If you’re telling me that our organization can’t do anything without him, we’ve got a big problem.” Wes

It’s difficult to overstate the enormity of this problem—it affects every organization, independent of the industry we operate in, the size of our organization, whether we are profit or non-profit. Now more than ever, how technology work is managed and performed predicts whether our organizations will win in the marketplace, or even survive.

IT historically goes for perfection. Many times there is the thinking that unless every business requirement, function or feature is implemented the solution will not be acceptable. It is easy to over-architect solutions and build much more than what the business would be happy with. Constructing more than what is really needed is a form of waste.

He saw a presentation given by John Allspaw and his colleague Paul Hammond that flipped the world on its head. Allspaw and Hammond ran the IT Operations and Engineering groups at Flickr. Instead of fighting like cats and dogs, they talked about how they were working together to routinely do ten deploys a day! This is in a world when most IT organizations

Many psychologists assert that creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings—we deprive other people of their ability to control their own outcomes and even create a culture where people are afraid to do the right thing because of fear of punishment, failure, or jeopardizing their livelihood

Many psychologists assert that creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings—we deprive other people of their ability to control their own outcomes and even create a culture where people are afraid to do the right thing because of fear of punishment, failure, or jeopardizing their livelihood.

I made a lot of mistakes along the way and wish I had access to the information in this book back then. Common traps were stepped in—like trying a top-down mandate to adopt Agile, thinking it was one size fits all, not focusing on measurement (or the right things to measure), leadership behavior not changing, and treating the transformation like a program instead of creating a learning organization (never done).

Technology managers, like so many other well-meaning managers, often try to fix the person while ignoring the work environment, even though changing the environment is far more vital for long-term success. Managers who want to avert employee burnout should concentrate their attention and efforts on: Fostering a respectful, supportive work environment that emphasizes learning from failures rather than blaming Communicating a strong sense of purpose Investing in employee development Asking employees what is preventing them from achieving their objectives and then fixing those things Giving employees time, space, and resources to experiment and learn Last but not least, employees must be given the authority to make decisions that affect their work and their jobs, particularly in areas where they are responsible for the outcomes.

Lean defines two types of customers that we must design for: the external customer (who most likely pays for the service we are delivering) and the internal customer (who receives and processes the work immediately after us). According to Lean, our most important customer is our next step downstream. Optimizing our work for them requires that we have empathy for their problems in order to better identify the design problems that prevent fast and smooth flow.

In addition to regular stand-ups with squads, product owners, IT-area leads, and chapter leads, the tribe lead also regularly visits the squads to ask questions—not the traditional questions like “Why isn’t this getting done?” but, rather, “Help me better understand the problems you’re encountering,” “Help me see what you’re learning,” and “What can I do to better support you and the team?” This kind of coaching behavior does not come easily to some leaders and managers. It takes real effort,

We make our system of work safer by creating fast, frequent, high quality information flow throughout our value stream and our organization, which includes feedback and feedforward loops. This allows us to detect and remediate problems while they are smaller, cheaper, and easier to fix; avert problems before they cause catastrophe; and create organizational learning that we integrate into future work. When failures and accidents occur, we treat them as opportunities for learning, as opposed to a cause for punishment and blame.

They’d be responsible for documenting what they learned, and Brent would never be allowed to work on the same problem twice. I’d review each of the issues weekly, and if I find out that Brent worked a problem twice, there will be hell to pay. For both the level 3s and Brent.” I add, “Based on Wes’ story, we shouldn’t even let Brent touch the keyboard. He’s allowed to tell people what to type and shoulder-surf, but under no condition will we allow him to do something that we can’t document afterward. Is that clear?

Do change management boards actually improve delivery performance?” (Spoiler alert: they do not; they are negatively correlated with tempo and stability.)

we also discovered that it is possible to influence and improve culture by implementing DevOps practices.

Our research also found that developing off trunk/master rather than on long-lived feature branches was correlated with higher delivery performance.

Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods.

The fact that low performers were more likely to be using—or integrating against—custom software developed by another company underlines the importance of bringing this capability in-house.

What tools or technologies you use is irrelevant if the people who must use them hate using them, or if they don’t achieve the outcomes and enable the behaviors we care about. What is important is enabling teams to make changes to their products or services without depending on other teams or systems.

If your teams have no visibility into code deployments—that is, if you ask your teams what software deployments are like and the answer is, “I don’t know . . . I’ve never thought about it!”—that’s another warning that software delivery performance could be low, because if developers or testers aren’t aware of the deployment process, there are probably barriers hiding the work from them.

Statistical analysis also revealed a high correlation between deployment pain and key outcomes: the more painful code deployments are, the poorer the IT performance, organizational performance, and organizational culture

Being a leader doesn’t mean you have people reporting to you on an organizational chart—leadership is about inspiring and motivating those around you.

Continuous delivery is a set of capabilities that enable us to get changes of all kinds—features, configuration changes, bug fixes, experiments—into production or into the hands of users safely, quickly, and sustainably.

investments in technology are also investments in people, and these investments will make our technology process more sustainable