The table below provides a simple framework for assessing the benefits of participation, based on the four generic reasons why engagement is carried out.
| Goals / Purpose | Possible indicators (examples) | How to get data (examples) | Important assumptions (examples) |
|---|---|---|---|
| Improved governance | Increased trust in government | Surveys before and after the engagement process | Trust may be affected by a wide range of influences; this process may only be one among many |
| Social capital and social justice | Increased equality of access to decision-making
Developed new contacts / given access to networks | Demographic analysis of participants + feedback from them on the difference made by the exercise
Questionnaires after engagement events; interviews later | Social capital can be a difficult concept, and is not always understood to operate beyond the local level; but the importance of increasing access to different people and new networks does work at a national level |
| Improved quality of services / projects / programmes | Costs saved by people taking more responsibility for service outcomes and making less demand (e.g. healthy living)
Quicker decisions by avoiding conflict | Feedback from doctors and patients through surveys, polls etc.
Collecting costs of dealing with conflict (e.g. complaints, objections, campaigns etc.) | It is difficult to separate the impacts of engagement from other elements of service improvement
The costs of conflict are rarely recorded, so data would have to be collected from scratch |
| Capacity building and learning | Greater awareness and understanding of the issues
More confidence and willingness to get involved in future | Questionnaires with participants after the process; and follow-up interviews later.
| These are relatively straightforward issues to test with participants before, during and after the process |
An alternative model, for use when there are clear targets, has been developed by Vivian Twyford in Australia for use with some of the International Association of Public Participation principles and values (see Table 2).
These two models provide a couple of examples of the types of indicators that can be used to measure the success of engagement exercises, depending on the objectives of the exercise.
It can be very useful to think about the indicators / success factors for engagement while refining the objectives of the engagement exercise overall. There is no point carefully deciding on objectives if there is no effective way of measuring whether they have been achieved.
Success factors answer the question: how will we know if it a success? It is a success if … Success factors are usually quite broad statements (e.g. the engagement reached a ‘broad range’ of people; engagement was effective).
Indicators do the same thing, but are usually more specific and are essentially ‘headlines’ that illustrate the point – they are not comprehensive evidence that something has been achieved, they simply spotlight a specific aspect (e.g. people understood what was going on; people felt they influenced the final policy; policy-makers felt the final policy was improved by the public input).
Indicators need to be meaningful (so those reading them will understand why they are important) and measurable (so data can be collected relatively easily). They may use qualitative data (e.g. from interviews with participants, decision-makers etc) and quantitative data (e.g. surveys, demographic analysis).
Vivian Twyford’s model of analysis (see Table 2 below) separates success measures and indicators, as well as offering targets.
| Goal / Objective | Success measures | Indicators | Targets | How to get data |
|---|---|---|---|---|
| The public contribution will influence the decision | Data is gathered from the public, summarised and circulated, processed into usefully formatted information and given to decision-makers in time for them to use for decision-making
Decision-makers genuinely consider information from the public
Public is provided with feedback on summarised input and how it has been used | Decision-makers sign off on minutes of discussions of input
% of participants who receive / understand / believe summaries of public input;
% of participants who receive feedback on how input has been used | 65% of decision-makers sign off on discussion minutes
75% of participants receive / understand / summaries of public input;
75% of participants receive report on how input used | Decision-maker discussion held and minutes taken
Each decision-maker requested to sign off minutes
Telephone survey of random sample of participants |
| Provide timely, balanced and objective information on the problem, alternatives considered and solution | Improved public image of client organisation | Stakeholder satisfaction with organisation’s performance in meeting its charter | At least 50% of stakeholders believe organisation’s performance is satisfactory or better in the first year, and satisfaction trend rises over 3 years | Survey seeking levels of stakeholder satisfaction in meeting its charter; run every year for 3 years to randomly selected sample of known stakeholders; survey attracts a minimum of 25% response in first year |