Participation is an emerging field, so evaluation and review of practice is very important. Indeed, formal evaluation is emerging as an integral part of good public participation management for on-going projects.
Planning a review process in advance is also important to ensure that the learning is gathered from the work as it happens. This enables those involved (and others) to judge whether or not the process has been successful. Evaluation can be an independent process working alongside the participatory process, or part of the management of the process (see below).
Coverage of the review
Success criteria for the review will need to be developed. These are likely to include:
- Whether the process met its own objectives and originally agreed purpose / aims;
- Whether the process met explicit and implicit demands from participants;
- Whether the process met standards of good practice in participatory working.
In addition, the review may wish to cover:
- Whether the level of participation (e.g. consultation or partnership) was appropriate;
- Whether the methods and techniques were appropriate and worked as expected;
- Whether the level and range of responses from participants legitimised the exercise;
- Whether the costs were as expected and reasonable (staff time, money etc);
- Whether what was produced and organised (outputs e.g. documents, meetings) was appropriate and worked well;
- Whether the ways in which the responses from the process (e.g. recommendations) were dealt with were appropriate and effective;
- What was achieved during and after the process (outcomes).
Since some of the outcomes are likely to be intangible (e.g. improved relationships, a sense of empowerment etc.), it is useful to set benchmarks which these can be measured against. Ideally, both quantitative and qualitative methods will be used: quantitative methods involve collecting numbers for measurement and judgement; qualitative methods involve collecting data from people to allow description and interpretation.
Any review process will include a basic description of what took place, for which the headings for this section can provide a framework (i.e. scope, purpose, context, who was involved, outputs, outcomes, institutional response).
An evaluation only works if it includes the perspectives of all those involved in the process – including whoever is leading the process, decision-makers and the participants. For example, one participative process was initially judged a ‘complete’ success; when questioned on the subject of establishing buy-in to the initiative the organisation commissioning the process responded that “What’s come out of the workshops has been a document… and everyone feels they own the document”. When a key participant was asked, he responded “If there are several stakeholders that feel they’ve got a degree of ownership over the decisions then I think they were the ones manipulated the most”.
The views of the participants can be the most useful, but also the hardest to get if not gathered at the time.
On-going review
For on-going participative initiatives it can be very useful to have a system of management that continually checks whether the process is meeting the purpose agreed at the start. This can happen through the regular design / delivery team meetings. This approach is especially useful if the team undertaking the analysis has a broad knowledge of other methods available so that if the current approach is not working an alternative method can be used.
An iterative approach enables a process to adapt to new and unforeseen circumstances. No matter how much planning is put in, when working with participatory processes the unpredictable is inevitable (be it new political agendas or participant responses). The trick is to have an iterative and flexible approach to managing a process, which helps you respond to the unpredictable.
Risk management
A robust review process can be an effective form of risk management. It helps map out the views and perspectives held at the start of a process and raises awareness of the challenges that the process may face.
Inclusion of a review process can increase costs (if separate from the management process), which can be difficult to justify when cost savings are sought. However, a robust review process should be considered an essential part of the management required in any process operating in an inherently uncertain environment. The costs of not accounting for the risk and being faced with things going wrong, as often happens when there is insufficient time for proper reviews as the process unfolds, may be far higher than review costs will be.