Evaluating development and improvement interventions

During a recent management workshop, the vexed question of how best to evaluate development initiatives and interventions arose.  This reminded me of the comments made by a spokesman for the UK's Ministry of Defence, during an international conflict that took place a few years back. He stated that the MoD evaluated the effectiveness of their aerial attacks by asking three questions:

• Did the missile hit the intended target?
• Did the damage caused by the attack materially reduce the enemy's functionality?
• Were the combined effects of the various strikes advancing the overall aims and objectives of the war?

These criteria provide a useful metaphor for assessing the validity and effectiveness of targeted initiatives in other areas, not least organisations.

Maintaining a clear line of sight

Applying the MOD’s thinking to the business world provides a useful approach for exploring, validating and evaluating development interventions and/or ‘service’ provision in general. The critical insight here is to recognise the importance of maintaining a clear line of sight between specific actions and the ultimate aims and objectives being pursued.

Too often, evaluation processes begin with the interventions themselves and work backwards, in an attempt to prove the business benefit. In contrast, the MoD story emphasises the need first to understand the context within which the specific actions are taking place. It is the context that validates the intervention, not the other way round.

The A-H of validation and evaluation

I've set out below a series of questions that can be a useful way of generating meaningful conversations around the validity and value of specific development and improvement activities.

First understand the business context:

• What are the overall AIMS and ASPIRATIONS of the organization?
• In the light of these, what critical BUSINESS OBJECTIVES and strategies are needed to realise them?
• Do we have the necessary CAPABILITIES, and an enabling CULTURE and CLIMATE, to meet these objectives in both the short and long term?

Where the above questions reveal significant gaps or vulerabilities:

• What actions do we need to take to DEVELOP the necessary capabilities and/or to help to shift the underlying cultural DYNAMICS?

The results of these actions then need to be validated and evaluated from three angles:

• EFFICACY - Were the interventions delivered effectively and efficiently, in relation to the specific objectives set?

This is the equivalent of the MoD’s "Did they hit the target?" question.

• FUNCTIONALITY – Did the actions result, over time, in relevant improvements in functional and/or business unit capability, and in the capacity to act?

The "Did it affect functionality" question.

• GOAL ATTAINMENT – Have the combined impacts of these interventions led to enhanced organisational performance, or increased the probability of doing so?

The "Is it helping us to 'win the war'?" question.

In considering each of the above, it is particularly important to take account of:

• HAZARDS arising from:
  • the uncertain, socially complex and dynamic nature of organizations; and
  • the effects of shadow-side dynamics and informal coalitional activity on so-called "rational" decision making and action taking.

A matter of judgement

I've included "Hazards" as a caution against the sometimes obsessive attempts that are made to prove that particular initiatives and interventions contribute directly to"the bottom line." Any such 'proof' is illusory. Even where extensive research suggests that a statistically significant correlation exists between a specific action and overall organizational performance, this does not mean that the former caused the latter. The underlying dynamics of organizations are complex. For the most part, these do not allow simple cause-and-effect links to be established between specific inputs and overall outcomes. Validating and evaluating development and improvement activities is therefore always a matter of judgement. The above questions are simply intended to facilitate meaningful conversations as a useful way of informing this judgement.

Doing the right things

Overall, making sure that you are ‘doing the right things’, that is maintaining a clear line of sight between business performance and development interventions, is usually more critical to the long-term success of the organisation than ‘doing things right’. Doing the wrong things expertly well is a sure-fire way to destroy value and put a business at risk.