Value For Money Concept

c. Differences between Monitoring and Evaluation

Monitoring is the continuous and routine collection, as well as presentation of information in the program implementation. Evaluation, on the other hand, is a periodic assessment, involving the exercise of judgement, of policy or program implementation by reference to predefined performance criteria such as efficiency and effectiveness (VFM Guideline, 2007).

d. Evaluation criteria for VFM Reviews

The evaluation criteria are reflected in the specific questions that the evaluation will have to answer. In the case of a VFM review, the specific questions are set out in the template Terms of Reference and cover the evaluation criteria: rationale, efficiency, effectiveness, impact and continued relevance. The criteria encompass the established Value For Money criteria (economy, efficiency and effectiveness – note that VFM reviews address ‘economy’ under the ‘efficiency’ heading). Other public policy evaluation criteria may also be relevant depending on the nature of the review. Each of the five evaluation criteria (rationale, efficiency, effectiveness, impact and continued relevance) is discussed in the following paragraphs.

1). Rationale

Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a program and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure exists when private individuals or firms do not produce the optimal level of goods or services from a societal Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a program and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure exists when private individuals or firms do not produce the optimal level of goods or services from a societal

2). Efficiency and economy

Efficiency and economy look at performance in terms of outputs, acquisitions, and use of inputs. It is common for economy and efficiency to be studied together in an evaluation report, although in many cases, it is worthwhile to study them separately (VFM Guideline, 2007).

Economy means securing the appropriate quality of inputs at the best price, while on the frame work or logic model describes as inputs. According to Bastian (2001), economy means the relationship among the market and input on the purchasing goods or services with certain quality and quantity at the best price. Meanwhile, efficiency means optimizing the ratios of inputs to outputs, while on the frame work or logic model describes as Inputs, Activities, and Outputs. Efficiency can be viewed in two different ways: a). whether the same level of output and result could be achieved with

fewer inputs, i.e. at a lower cost; or b). whether a higher quality or quantity of outputs might be delivered from

a fixed amount of input.

3). Effectiveness

Bastian (2001) said that the effectiveness is defined as the extent to which the objectives have been achieved and the planned benefits have been delivered. It should be kept in mind that the results can be positive or negative, and intended or unintended. In terms of the program logic model, the study of effectiveness is the relationship between outputs and results.

4). Impact

Impact is concerned with the wider effects of the program for a larger group of persons (e.g. a sector) or for society as a whole (it includes the medium to long term effects on targeted beneficiaries); the successes and failures in achieving the wider objectives; and the main causes of this. In the study of impact, the same basic performance questions used for effectiveness are applied to the medium and long term sector objectives of policy interventions and programs. The impact of a program is the difference it has made. As with effectiveness, it should be kept in mind that impacts can be positive or negative, and intended or unintended (VFM Guideline, 2007).

5). Continued relevance

In a VFM review, the evaluation of the continued relevance of the program covers the following issues as the table below:

Table II.1. Issues on VFM Isues

Performance Questions Indicator Performance Questions Indicator

Rationale Issue

The validity of the program Why is a public policy Reasons for public policy objectives

intervention necessary?

intervention

What market failure is Evidence of market

being addressed?

failure

Are the program objectives compatible

the Documented links to objectives in the statement statement of strategy strategy

with

Efficiency Issues

The throughput of outputs What are the outputs? How Actual number of outputs many are produces? Are produced output targets met?

The timelines of production How long does it take to Output produce per hour, of output

produce an output?

day, week or month

Are time targets met?

Average time to produce a unit of output

The quality of the output How complete and accurate Accuracy level are outputs?

How many complaints are Number of valid

there about outputs?

complaints about outputs

The cost of an output How much did the output Unit cost of output cost – per item and all together

Distinguish

between

program and administration cost

Economy Issues

price Actual cost of input inputs

The cost of the resource Was

Could a cheaper process be

Unit price or cost of input

obtained?

(Table II.1 continued)

Isues

Performance Questions

Indicator Indicator

Effective Issues The

extent that the What did the program Quantitative and objectives were achieved

achieve?

qualitative measures of actual

and expected Can the “performance gap” performance be measures or otherwise Input/output

indicators

defined?

that reflect contributing factors

The way that the What methods are used to achievement of objective is assess effectiveness? assessed

Appropriateness of

Is

achievement indicators used measured against a valid baseline

the

Suitable and accuracy of baseline indicators

Cost effectiveness Could the same results be achieved in a better or a cheaper way?

Benchmark comparisons to other approaches to the

same problem nationally

and internationally

Impact Issue

Socio economic objectives What are the longer term Indicators directly

impacts

on

targeted related to the targeted

beneficiaries?

long term impacts on direct beneficiaries

Causality Is it possible to isolate the Direct contribution of program contribution to program

to strategic

wider impact

objective

Continued relevance

What is the justification for Estimate of program the continued allocation of benefits expressed in public

funds

to

the quantitative, qualitative

program?

and timing terms and taking account of what

How does the program fit would have been

in with other program?

achieved anyway in that absence of the program

Source: Guidelines of VFM (2007).