What do we mean by appraisal?

The terms ‘appraisal’ and ‘evaluation’ are often used interchangeably. However, in this guidebook we use them to refer to two different forms of assessment. Appraisal is the ex ante process of deciding how well a scheme or strategy will perform. Evaluation is the specific application of appraisal to the ex post assessment of completed projects. In both cases the question is: “How well does this scheme or strategy meet the objectives which we have set?” Appraisal and evaluation need to be conducted in a consistent way. We consider evaluation in Section 15 .

Why do we need appraisal?

Choices in transport policy are rarely easy. One scheme may offer a greater reduction in congestion, but at the expense of the environment; another may be environmentally more effective but restrict access for key groups of residents. Appraisal is a means of assisting the decision-maker to make effective choices between such options. As shown in the box, appraisal can contribute to a number of the steps in the logical structure (Section 6).

What is an appraisal framework?

It is essential to assess the problems, solution or strategy being considered against the full set of policy objectives (Section 7). Since these objectives represent different commodities and concepts, it is simpler to consider them separately. An appraisal framework enables this; at its simplest it is a table in which each column is a scheme or strategy option, and each row is an objective. One option should be doing nothing, to provide a base for deciding whether doing something is worthwhile. Other options can be variants of a particular scheme (e.g. light rail), different policy instruments (e.g. light rail, bus service increases, fares) or different combined strategies.

How are objectives and problems reflected in the appraisal framework?

All objectives should be represented in the framework, and each should be covered in a similar way. One of the easiest ways of doing this is to use the indicators selected for assessing performance against objectives (Section 7). These also provide a means of assessing the scale of problems, at least in aggregate. However, problems are usually associated with particular places or times, and this requires a more detailed framework. Treatment of equity issues also requires more detail and disaggregation. A choice is therefore needed on whether the framework is to be a relatively simple, aggregate one for the whole city, or one which provides more detail for different locations, times of day and groups of individual.

How can an appraisal framework be used to help make decisions?

Appraisal is a technical process, but it is also a key input to participation and decision-making. Decision-makers can simply choose between options using detailed appraisal framework tables of this kind. However, the choices and trade-offs become increasingly complicated as the framework gets larger. Two techniques are available to help overcome this.

Cost-benefit analysis uses money as the comparator. Changes in amounts of travel, travel time, accidents and the environment are assigned money values, based on observations of the choices which people make. Costs and benefits are each calculated, relative to doing nothing, for each future year. The net benefit is then discounted to the present day and summed over all years in the appraisal period to give, as a single indicator of performance, a net present value of the benefits. The appraisal period will normally be longer than the plan period, to allow for longer term impacts. A scheme with a positive net benefit is worth building; the option with the highest net benefit is the best. The main weaknesses in this approach are the assumptions required to value attributes like noise and accidents, the difficulty of appraising impacts on future generations, and the fact that the final value appears to determine the decision, rather than encouraging discussion.

Multi-criteria appraisal overcomes some of these problems by allowing the decision-maker to assess the weights to be assigned to different indicators, objectives and impact groups. In this way, differing views on the relative importance of, say, noise and accidents can be reflected. At its simplest, this can be used to generate a single weighted score for each option, allowing the user to decide which is the best option, given the weights they have selected. It is also possible to test sensitivity to variations in weights, and thus identify the solution which is the most robust to changes in weights, or which is least likely to be rejected by someone who takes a very different view on the relative importance of the objectives. ECOCITY developed a multi-criteria appraisal method based on a mixture of qualitative and quantitative indicators and benchmarks, which could be generated without recourse to models. Further guidance on appraisal methods is given in the PROSPECTS Methodological Guidebook.

How can appraisal methods deal with uncertainty?

Uncertainty can arise in the scenarios against which strategies are being tested (Section 3), the structure, timing and sequence of the elements of the strategy (Section 11), the predicted impacts of the strategy (Section 12 ), and the weights to be assigned to the individual indicators (see above). One of the simplest ways of treating appraisal is sensitivity and robustness testing. A strategy is tested against variations in a scenario, or with the model assumptions varied, or with differing weights. If the performance of a strategy is very sensitive to these changes, it is less robust, and therefore more risky. If other strategies (or schemes) perform better than it when the assumptions are varied, it may be better to select them. As with other elements of appraisal, there are more complex ways of analysing uncertainty.