Best Practices in Experimenting for Improvement KATA

This week we are going to be talking about experimenting your way to success in Continuous Improvement. This is part 4 of a 4 part series on Improvement Kata.

We’ve talked about understanding your challenges,  grasping your current condition,  setting your target condition, and now we are going to talk about experimenting from your actual to your target condition.

Quick note, if you haven’t had a chance to pick up a copy of Toyota Kata and read it,  I would highly encourage you to do so.

I’d like to start with a story on this topic. A plant manager calls in a mechanic because the plant is having a lot of downtime issues on one of their main machines. Everybody is trying everything and no one can figure it out. They call in the outside mechanic, the mechanic comes in, looks at the situation, pulls out a screwdriver, tightens one bolt, leaves, and on the way out drops off the bill, the bill is $10,000. The plant manager asks the mechanic, what’s the idea here? you were here 10 minutes and tighten one bolt. Why $10,000? 

The mechanic looks at the plant manager and says, hey, coming in here probably it’s $10 – $15 bucks, tightening one bolt $10, knowing exactly which bolt to tighten and exactly how to do it, is $10,000. 

That is the level of proficiency that you want to develop within your organization and within your people. 

With this in mind, we are going to talk about what is experimentation for CI.

Why is it so important? What are some best practices? and last but not least, I’m going to share a couple of tips with you on how to do better experiments for Continuous Improvement.

What is experimenting for Continuous Improvement?

Experimenting is the process of testing your assumptions. We all have assumptions, we all have biases toward what we think will make improvements and what we think doesn’t really matter.

Experimenting is the systematic approach to validating every assumption and then discovering the truth about what’s really happening with your process. 

So here are a few characteristics of effective experimentation. First of all, you want it to be hypothesis led, second of all, you want it to be objective, and third but not least, you want it to be controlled.

So, let’s unpack each one of those right here, right now. 

Why do you want your experiments to be hypothesis led?

By having a clear and well-defined hypothesis you also have a very clear direction for your testing. A good hypothesis has a yes or no response. If I make this adjustment will it increase output by 25%? that is a very clear hypothesis, the experiment will simply be turning up the dial and measuring the output. fairly simple. By doing that and controlling all other variables, which we will talk about more in a second here, I have very clear confirmation of my assumption. Another key point here is that it establishes parameters, you want to be very clear about the variable that you are testing, and very clear about what variables are not tested; in which case you want to make sure that they are not changing from trial to trial as part of your experiment. The next key characteristic of effective experimentation is objectivity.  The results of an experiment should be validated empirically, meaning, there is data, there is evidence, there are other facts to support what you found. You also want to accept the results for what they are, rather or not they confirm your hypothesis.

I know culturally, in a lot of places, we have a pressure to feel like we have to be right, we can’t be wrong, we can’t state a hypothesis that is not right, but in more progressive cultures, in cultures that really practice the scientific method and scientific thinking, being wrong is completely acceptable. The most important thing is that you learn from what you did. Finally, you want your experiments to be controlled, the reason you want to do this is that you want to reduce the noise and confusion. There are always a million things going on, a million variables in play and they are always in flux. What you want to do is try to control those variables to the greatest extent that you can and adjust one variable at a time and practice those iterations changing that one variable to really get a confirmation as to the impact of changing that one variable. It’s one thing to say that changing this affects that, and it is another thing to say changing this by 10 degrees affects that by 35%.

Why is it important to effectively experiment?

There are three incredible reasons, one it accelerates the learning and discovery process, number two, it produces repeatable results, and if it’s not repeatable, it is not science, and number three, it leads you down a path of true operational excellence. Let’s dive down each of those individually here. 

First, it accelerates the learning and development process, effective experimentation in truly understanding what is driving the outcomes you are seeing, and then being able to turn the dial and adjust those outcomes, actually makes you a better decision-maker. A lot of the decisions we make every day are based on some experiences and some facts, but a lot of it is rutted in assumptions as well.

Experimentation is the process of taking off each assumption and transitioning from a point of I think I know what is going on to I know exactly what is going on, we’ve tried it, this is what is driving the results we are seeing. 

It also develops better problem-solving skills, not only for yourself but throughout the organization. The scientific method is the problem-solving method. By systematically transforming your assumptions into facts, actually gives you more power about what happens in the future, more capability for solving the problems that lie before you and your specific area of the business. Number two, produces a repeatable result. And what is science other than the power to predict the future, repeat the future, and affect the future in a positive way, you experiment to produce more repeatable results.

If you really get to the truth of the matter you can make the exact same change, given the exact same circumstances you can reproduce the exact same outcome that gives you a lot of power over producing the outcomes that you desire. The third thing is that it actually leads you to your own individual journey, your own path towards operational excellence.

If you go to the best companies in the world, you realize they have great tools, great processes, fantastic management systems, but what they really nail down is the ability to experiment, the ability to find their own way to excellence. If you go to Toyota, they’ll be glad to share with you all of their tools, methods, and processes. Because the thing they know, what they understand, is that if you are not proficient in problem-solving, if you are not proficient in executing the scientific method, if you are not at experimenting your way to your own truth, to your own success, there is no way you can compete with them in the long run. You’ll always be at best as good, more likely, never quite catching up to their level of operational excellence.

The key here is learning to experiment in your own way to success. In doing so you are creating your custom-fit solutions. In every business, there are innumerable variables at play and your business is also unique in that way, therefore, what works in your business might not work in another business and vice-versa.

Through effective experimentation, you will find what works in your specific business environment. Also, this is a technique that increases your adaptability and competitive advantage. As I said, by copying and pasting tools and processes from other companies, does not give you a competitive advantage. At best you will be as good as them, more likely you’ll always be chasing their tail. So you are probably wondering, what are some of the best practices in experimenting for a Continuous Improvement. I’m glad you asked. The key here is to ask what you are experimenting toward.

Improvement Kata actually provides an excellent framework for setting yourself up for effective experimentation.

Steps to effective experimentation

The first step is gaining clarity on your long term vision, this could be a year or more, or if you are in a more volatile market you might want to plan for a year or less. Gain clarity on your current state.

What is actually going on in your business, then gain clarity on your next target condition within your knowledge threshold where you can start experimenting towards that first target condition. As you start planning actual experiments, PDCAs and excellence framework (Plan, do, check, act) In the planning phase, of course, you are executing the improvement kata up to stage four, and then you start forming hypotheses about what you will do and what you think the outcome will be. In the do phase you are actually conducting those experiments. In the check phase you are validating rather or not your hypothesis was true or false, and in the Act phase, you are either standardizing, documenting what you’ve learned, sharing it with others or you are continuing with that PDCA cycle, continuing to conduct experiments until you can produce the results that you want.

Tips for experimenting toward your target condition

Before we end, here are a few tips for more effectiveness around performing experiments for continuous improvement.

#1 form a hypothesis. Be crystal clear about what variable you are going to change, what variables you are going to control, and then what you expect the outcome to be.

#2 Be objective. Accept the results for what they are. Try not to get into the habit of forcing results to fit your earlier narrative. Only the truth will get you better business results.

#3 change one thing at a time. A lot of us have gotten in the habit in Continuous Improvement to change a lot of things at the same time thinking that is the best way to get great results really fast and sometimes it can give you some good numbers early on but the problem is that you don’t know which thing that you changed actually produced the result you want.

Sure, you can have an idea, but the fact is that you have improved it, and what tends to happen in those situations is that you got 10 people involved, 10 people throw ideas in the hat, 10 ideas get implemented, everybody walks out of the room thinking that it was their idea that made all the difference, and nobody really learned much of anything. So the best practice here is to get into the one-piece flow of idea implementation at a time. That is the scientific method.