Value For Money Concept
c. Differences between Monitoring and Evaluation
Monitoring is the continuous and routine collection, as well as presentation of information in the program implementation. Evaluation, on the other hand, is a periodic assessment, involving the exercise of judgement, of policy or program implementation by reference to predefined performance criteria such as efficiency and effectiveness (VFM Guideline, 2007).
d. Evaluation criteria for VFM Reviews
The evaluation criteria are reflected in the specific questions that the evaluation will have to answer. In the case of a VFM review, the specific questions are set out in the template Terms of Reference and cover the evaluation criteria: rationale, efficiency, effectiveness, impact and continued relevance. The criteria encompass the established Value For Money criteria (economy, efficiency and effectiveness – note that VFM reviews address ‘economy’ under the ‘efficiency’ heading). Other public policy evaluation criteria may also be relevant depending on the nature of the review. Each of the five evaluation criteria (rationale, efficiency, effectiveness, impact and continued relevance) is discussed in the following paragraphs.
1). Rationale
Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a program and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure exists when private individuals or firms do not produce the optimal level of goods or services from a societal Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a program and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure exists when private individuals or firms do not produce the optimal level of goods or services from a societal
2). Efficiency and economy
Efficiency and economy look at performance in terms of outputs, acquisitions, and use of inputs. It is common for economy and efficiency to be studied together in an evaluation report, although in many cases, it is worthwhile to study them separately (VFM Guideline, 2007).
Economy means securing the appropriate quality of inputs at the best price, while on the frame work or logic model describes as inputs. According to Bastian (2001), economy means the relationship among the market and input on the purchasing goods or services with certain quality and quantity at the best price. Meanwhile, efficiency means optimizing the ratios of inputs to outputs, while on the frame work or logic model describes as Inputs, Activities, and Outputs. Efficiency can be viewed in two different ways: a). whether the same level of output and result could be achieved with
fewer inputs, i.e. at a lower cost; or b). whether a higher quality or quantity of outputs might be delivered from
a fixed amount of input.
3). Effectiveness
Bastian (2001) said that the effectiveness is defined as the extent to which the objectives have been achieved and the planned benefits have been delivered. It should be kept in mind that the results can be positive or negative, and intended or unintended. In terms of the program logic model, the study of effectiveness is the relationship between outputs and results.
4). Impact
Impact is concerned with the wider effects of the program for a larger group of persons (e.g. a sector) or for society as a whole (it includes the medium to long term effects on targeted beneficiaries); the successes and failures in achieving the wider objectives; and the main causes of this. In the study of impact, the same basic performance questions used for effectiveness are applied to the medium and long term sector objectives of policy interventions and programs. The impact of a program is the difference it has made. As with effectiveness, it should be kept in mind that impacts can be positive or negative, and intended or unintended (VFM Guideline, 2007).
5). Continued relevance
In a VFM review, the evaluation of the continued relevance of the program covers the following issues as the table below:
Table II.1. Issues on VFM Isues
Performance Questions Indicator Performance Questions Indicator
Rationale Issue
The validity of the program Why is a public policy Reasons for public policy objectives
intervention necessary?
intervention
What market failure is Evidence of market
being addressed?
failure
Are the program objectives compatible
the Documented links to objectives in the statement statement of strategy strategy
with
Efficiency Issues
The throughput of outputs What are the outputs? How Actual number of outputs many are produces? Are produced output targets met?
The timelines of production How long does it take to Output produce per hour, of output
produce an output?
day, week or month
Are time targets met?
Average time to produce a unit of output
The quality of the output How complete and accurate Accuracy level are outputs?
How many complaints are Number of valid
there about outputs?
complaints about outputs
The cost of an output How much did the output Unit cost of output cost – per item and all together
Distinguish
between
program and administration cost
Economy Issues
price Actual cost of input inputs
The cost of the resource Was
Could a cheaper process be
Unit price or cost of input
obtained?
(Table II.1 continued)
Isues
Performance Questions
Indicator Indicator
Effective Issues The
extent that the What did the program Quantitative and objectives were achieved
achieve?
qualitative measures of actual
and expected Can the “performance gap” performance be measures or otherwise Input/output
indicators
defined?
that reflect contributing factors
The way that the What methods are used to achievement of objective is assess effectiveness? assessed
Appropriateness of
Is
achievement indicators used measured against a valid baseline
the
Suitable and accuracy of baseline indicators
Cost effectiveness Could the same results be achieved in a better or a cheaper way?
Benchmark comparisons to other approaches to the
same problem nationally
and internationally
Impact Issue
Socio economic objectives What are the longer term Indicators directly
impacts
on
targeted related to the targeted
beneficiaries?
long term impacts on direct beneficiaries
Causality Is it possible to isolate the Direct contribution of program contribution to program
to strategic
wider impact
objective
Continued relevance
What is the justification for Estimate of program the continued allocation of benefits expressed in public
funds
to
the quantitative, qualitative
program?
and timing terms and taking account of what
How does the program fit would have been
in with other program?
achieved anyway in that absence of the program
Source: Guidelines of VFM (2007).