Understanding the purpose of measurement

The purpose of measurement is to help us reach our goals, more quickly and with less cost / resources being expended. It’s not to judge our people or teams, it’s to focus on improving our business processes and our effectiveness at executing our strategies.  Performance measurement when done well provides a continuous feedback loop on our strategies effectiveness which we can use to leverage and grow further.

But unfortunately too many strategies are based on assumptions – untested assumptions. The problem with assumptions is of course that they have no evidence supporting them. As a result, they are often wrong, diverting our focus, wasting our precious time and resources.

Ask yourself this: how much money do you think your organisation spent last year on change or improvement initiatives that failed to produce the change or improvement it was supposed to?

At Rubica our research shows that approximately 63% of executives goals require them to initiate strategic change to close the gap between current performance and meet their annual objectives and goals. Yet according to research from the Economist 70% of Executives initiatives fail to deliver what they were originally intended to produce.  More importantly, how often do you think real understanding exists around why the initiatives failed?

These questions will probably make for some uncomfortable reading, because we should have the answers, but usually don’t. That said, if you are an evidence-based leader, you will have the answer’s because your work isn’t based on assumption – it is based on considered experiments. Experimentation is a key attribute of a resilient organisation, ready to adapt and find better ways of working!

Such experiments don’t need to be complex or rigorous. But they do need to be well designed, and have just enough rigidity so you confidently know what works, what doesn’t and by how much so you can move forward with confidence and purpose.

How to devise robust measurement experiments

1. Test your hypothesis.

Experiments work well when there is a very specific change (known as ‘value stories’ at Rubica) to test that focuses on a single factor. Broad hypotheses aren’t testable because they are made up of multiple factors and can’t be pinned down.

For example:

“How can we increase community engagement?” and “Will education in environmental sustainability decrease environmental impact?” are broad questions, so they aren’t good a good case for an experiment.

Instead, they need to be broken down into smaller parts that are a single-factor, measurable and testable. Something more appropriate would be: “Do letterbox pamphlets increase community participation in local events?”

2. Measure the result.

To get the most from experiments, good measures of the things being tested need to exist. The most important thing to measure is the result you are trying to impact.

For example:

Community participation in local events may be measured by the percentage of residents that attend local events, calculated weekly or monthly. But it could be measured more accurately by counting the residents that are the target audience for each event. Local events are usually tailored for segments of the community, like teenagers, the elderly, people with pets, or mothers of young children.

3. A representative sample.

Select a representative sample. The best way to do this is selecting a sample at random, called a ‘simple random sample’. When starting out, this is the most appropriate approach and keeps things simple.

For example:

If trying to increase community engagement, pamphlets would be sent to only a sample of community members for the experiment, and if the experiment worked, you could confidently send those pamphlets to the entire community. But if we do our experiment using a sample that is fundamentally different in some way to our targeted community, the experiment will teach nothing that can be confidently used/learnt from.

4. Apply a baseline.

A baseline may be a group, area or time frame where a treatment isn’t applied – enabling the comparison of a baseline against where a treatment has been applied – highlighting its degree of effectiveness.

For example: 

To test the impact of pamphlets on community engagement, pamphlets would be sent to residents in a sample of streets. Then, at each event the street where each participant lives would be captured. A comparison can then be made i.e. the percentage of residents from the sample streets who attended against the percentage of residents from other streets who attended.

5. An action plan.

Before starting a treatment, the details of an experiment need to be planned, sequenced and resourced. This will include:

1. Collecting baseline data.
2. Selecting the sample.
3. Carrying out the treatment.
4. Collecting data during the treatment.
5. Analysing the data to test the hypothesis.
6. Drawing conclusions.
7. The plan that needs to be followed, without letting any bias or confounding effects to sneak in.

6. Take note of the evidence.

The outcomes of a properly designed experiment shouldn’t be ignored: listen to the data and decide what it says about the impact of a treatment. These learnings should then be shared with others who may be able to build on the learnings for future experiments. From here, the learnings need to be acted upon – this is the only way that experiments will contribute to closing a performance gap.

For example:

If the pamphlets had no effect at all on community engagement in local events, the production and printing and distribution of pamphlets should stop. Creating the opportunity for another experience to be initiated.

To know if something worked or not, the size of the difference made must be known. If a proper business experiment has been designed, and by using the framework above, the effect of change initiatives will be known with confidence.