The Anatomy of an Experimental Organization

| October 7, 2015 | 5 Comments

I am a software developer. I see the world from that perspective. In reality though that is only one viewpoint. While it is important that we are effective at delivering software, what really matters is that we are effective at delivering business value.

When I describe Continuous Delivery to people I generally spend a fair amount of time impressing on them that it is not about tools and technicalities. It is not even about the relationship between developers and operations or product owners and testers. Continuous Delivery is about minimizing the gap between having an idea and getting that idea, in the form of working software, into the hands of users and seeing what they make of it. This vital feedback loop is at the core of not just good development, but of good business too.

I have been lucky to work in, or with, several companies that I would describe as Agile companies, not just companies practicing Agile development. These organizations are fundamentally different in approach in almost everything that they do. The principal characteristic of this kind of organization is that they are experimental in their approach to everything. For some more traditional organizations this sounds scary. “Experimenting, that sounds like you don’t know what you are doing”. Well, yes, we don’t know what we are doing, none of us really know!

feynman

 

To quote one of my heroes, Richard Feynman:

It doesn’t matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence – then it is still a guess!

 

 

Research into the performance of successful organizations says that 2/3rd of their ideas that are implemented in software produce zero or negative value:

“Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve”

“Avinash Kaushik wrote in his Experimentation and Testing primer that “80% of the time you/we are wrong about what a customer wants”

“Mike Moran wrote that Netflix considers 90% of what they try to be wrong”

(Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, Nils Pohlmann 2013).

(http://ai.stanford.edu/~ronnyk/2013%20controlledExperimentsAtScale.pdf)

 

Two thirds, or more, of the ideas of GOOD companies are waste.

Further, the experience of these companies, and the data, says that nobody can tell which third of ideas are the good ones before they are delivered. Clearly there is a place for guessing and predicting customer demand for innovative ideas, but it is important to remember that for every iPhone that you invent, you will have to go through three or four Newtons first.

If we are generally so bad at guessing, then the only sane strategy is to embrace the uncertainty. Optimize to have lots of ideas and to evaluate them quickly so that you can discard the poor ones as fast, and cheaply, as possible. This is what really effective organizations do. Watch Henrik Kniberg’s wonderful descriptions of the Spotify Culture. (See how Netflix works)

These companies know that their ability to guess is poor, because they are becoming more scientific in their approach. Instead of guessing what their customers want, and then assuming that their customers like what they are getting, they are measuring. These companies are designing experiments. They define metrics, which will identify what their customers think, and then carry out the experiment and reflect on the results in order to learn and improve.

So how do these experimental companies differ from others?

Value Innovation over Prediction

Innovation is the thing that differentiates the great companies from the rest. Great companies create new markets, provide new products or services that change how people do things.

More traditional companies value predictability. The trouble is that you can’t be both predictable and innovative at the same time. They are different ends of a spectrum. The only really predictable thing is status-quo. So if you want your company to be great, you need to value innovation and discard your reliance on predictability – at least for some of your products. One aspect of this is launch-dates. Apple don’t pre-announce their products, they are very secretive. Only when the products are ready do they announce. This allows them to strive for “Insanely great” as Steve Jobs so memorably put it.

State Your Hypothesis

This shared vision is kind of fractal. It operates at all levels in effective organisztions. Being hypothesis driven helps to establish this clarity, this shared purpose. Useful hypotheses can range from, stating what you expect the outcome of a test to be before you execute it, to stating how you think your global business strategy will work out. Some people even talk about “Hypothesis Driven Development.”

Seek the Right Experiment

As soon as you have a new idea. Whatever its nature, the next question should be “How can we test this idea?”. If it is an idea about how to improve your process, figure out an experiment to see if it works: “Ok, let’s try doing without the estimation meeting for a while and see if we save time”. If it is an idea about improving your product figure out an experiment to test that too: “Let’s release this feature and A/B test it against the existing service to see which one makes more money”.

Establish Feedback Loops

Experimentation means nothing without the closure of the feedback loop. After each experiment there should be a way to evaluate the results and figure out what to do next. The outcome from each experiment should be a real action of some kind. Depending on the results of the experiment you should either: Drop the change; Adopt the change; Run through the cycle again with a new experiment to learn more.

The establishment of effective feedback loops is a vital attribute of experimental organizations. Continuous Integration, Continuous Delivery, Experiments in Production, Test Driven Development, Retrospective meetings and Incident Reviews are all mechanisms for establishing effective feedback.

Assume the Fallibility of Experts

All of the really high-performance organizations that I have seen are notable in their attempts to minimize hierarchy. Decisions based on the highest paid person in the room are always only guesses! Seek evidence, allow individuals and teams the freedom to experiment and make decisions based on what they find. Treat people’s opinions with respect, but recognize that whomever it is that holds them, they are still only opinions. Test the hypothesis!

Question Everything

It is not just important to remember that experts are fallible, but there should also be no sacred cows. Everything should be up for evaluation, and if it is found wanting should be amenable to change. Technology, practice, process, office-space, team organization, even personnel.

If it is not working, try something else.

Eliminate Waste

Successful organizations avoid doing unnecessary work. The Lean mantra of eliminating waste is a powerful one. We should be rational and objective in assessing everything that we do in the light of our real goals and finding how to make our work more efficient.

The most common response that I get when coaching teams how to do better is: “We don’t have time to improve”. Investing in making your team and their working practices more efficient is almost never a cost!

Leadership is Different to Management

Spotify talk of having “Loosely-coupled, tightly aligned teams” as being one of their goals. They aim for high autonomy and high alignment. Dan Pink talks of the importance of “Autonomy, Mastery and Purpose” in motivating people towards excellence. These ideas require leadership, not management. The goal of leadership should be to establish a common vision, a shared purpose for the organization without telling people how to achieve it. Ideally leadership is about inspiring people.

 

My experience has been that experimental organizations are generally much more successful than their more traditional counterparts. They also tend to appeal to the most talented people, because of the creative freedom that they offer.

People sometimes ask me “how do you know when your organization is working well?”. I think that you know things are on the right track when your first response to any challenge or question is: “How could we try that out?” rather than “I think this is the answer”.

To learn more about adopting DevOps and Continuous Delivery  at the enterprise, listen to the free webinar here.


Dave Farley

About the Author ()

Dave Farley is a thought-leader in the field of Continuous Delivery, DevOps and Software Development in general. He is co-author of the Jolt-award winning book 'Continuous Delivery' a regular conference speaker and blogger and a contributor to the Reactive Manifesto. Dave has been having fun with computers for over 30 years. Over that period he has worked on most types of software, from firmware, through tinkering with operating systems and device drivers, to writing games, and commercial applications of all shapes and sizes. He started working in large scale distributed systems about 25 years ago, doing research into the development of loose-coupled, message-based systems - a forerunner of MicroService architectures. Dave has a wide range of experience leading the development of complex software in teams, both large and small, in the UK and USA. Dave was an early adopter of agile development techniques, employing iterative development, continuous integration and significant levels of automated testing on commercial projects from the early 1990s. Dave is the former Head of Software development at LMAX Ltd, home of the OSS Disruptor, a company that are well known for the excellence of their code and the exemplary nature of their development process. Dave is now an independent software developer and consultant, and founder and director of Continuous Delivery Ltd.

  • amit shah

    Nice post. At the start of the article, you make a point of feedback loops. I understand that they are good when they are fast, but I guess that applies to consumer based applications like shopping sites for e.g. amazon. I have realized that the feedback loops are quite longer in enterprise based solutions. Delivering features at a faster pace (i.e. daily or weekly) may not be even worth. Release cycle of 1-3 month is fast enough. What are your thoughts on this? Would implementing continuous delivery be beneficial in such cases? Do you have case studies in this area?

    • Amit,
      thanks I am pleased that you like it.

      I believe that Continuous Delivery (CD), and in particular faster feedback, is valuable in all software development.

      This isn’t just about A/B testing in big web-sites.

      Being experimental is pervasive, it introduces effective feedback cycles at all levels in your development process and for all kinds of different aspects of development, team and business organisation.

      CD works for lots of different kinds of organisations, banks, car manufacturers, device manufacturers, insurance companies, software product orgs as well as the big Web companies that everyone thinks of. I do have case studies for each of these sectors. In all cases they report significant increases in quality and productivity.

      I believe that simply by focusing on reducing your cycle-time (primary delivery feedback cycle) you will be forced to improve your development process in all sorts of ways. As a secondary effect of this, you will see the benefits that I mention.

      However, I do see ‘Continuous Delivery’ as distinct from ‘Continuous Deployment’.

      In Continuous Deployment your software will be deployed to production every time it passes all of its tests. This is one style of Continuous Delivery, but not the only one. For some businesses this makes no sense. For others, it gives the ideal feedback loop.

      I believe that the decision to deploy is business dependent. If you sell software as products deployed on your customer’s hardware, it is inappropriate to push every change, automatically to your customers.

      Continuous Delivery is about working in ways that keep your software in a releasable state. We prioritise its releasability over new features. This means that we can decide to release whenever it makes sense from a business perspective. For some organisations, this means every time the tests pass, for others it means once per week for others once per day.

      For Enterprise systems I think that 1-3 months is too slow to get the real benefits. I believe that you should be aiming to release your changes at the end of each iteration of development, generally at least once every couple of weeks (as a rough guideline).

      • amit shah

        Thanks for your reply.

        Do you have case studies of enterprises that have a weekly or bi-weekly sprint cycle? Would be keen in going through them.

        The reason I say monthly or quarterly spring cycles are good was because the feedback cycle in case of enterprises is slow. Not every feature that gets released, gets used by software users.

        What would be the right metric(s) to analyze and conclude that the current sprint cycle is too slow?

        Appreciate your time in following up.

        • I have personal experience of such organisations.

          Scrum, the most popular agile approach in large organisations. It generally advises sprints that are 1 or 2 weeks in duration. Each sprint should produce a “potentially shippable product increment”.

          One of the metrics that suggest that your release-cycle is too long is that the users don’t use the features that you release :-/

          One of many reasons for releasing in small, frequent, increments is to learn from our users what they need and to target that more effectively.

          Another symptom of longer release cycles is higher bug rates. Organisations that release more frequently tend to have fewer bugs in production.

          Another useful metric is how often you introduce problems at the point of release and how long it takes to fix them once they are introduced.

          Amazon, for example, saw a 75% reduction in outages at the point of release and a 90% reduction in the time taken to address such outages once they began releasing continuously.

          Just as a reminder, I am an independent software consultant who helps people to solve this kind of problem 😉

          • amit shah

            Great. Thanks Dave for your valuable inputs.
            I will analyze the metrics you mention. Appreciate your help!