2016 Manajemen Program Sesi 11 AP Monitoring Evaluation
Monitoring & Evalua/on
Ari Probandari
Learning Objec/ves
• Students are able to explain the importance of
monitoring and evalua/on a program
management
• Students are able to apply concepts of
monitoring and evalua/on in a program
management
What is Monitoring and Evalua/on
(M&E)?
Group Work 1
Dimension
Monitoring
Evalua2on
Frequency
Periodic, occurs regularly
Episodic
Func2on
Tracking/oversight
Assessment
Purpose
Improve efficiency, provide
Improve effec/veness, impact, value
informa/on for reprogramming to for money, future programming,
improve outcome
strategy and policy making
Focus
Inputs, outputs, processes, workplans (opera/onal
implementa/on)
Effec/veness, relevance, impact,
cost-effec/veness (popula/on effect)
Methods
Rou/ne review of reports,
registers, administra/ve
databases, field observa/ons
Scien/fic, rigorous research design,
complex and intensive
Informa2on
source
Rou/ne surveillance system, field
observa/on reports, progress
reports, rapid assessment,
program review mee/ng
Same
Popula/on based surveys, vital
registra/on, special studies
Cost
Consistent, recurrent costs spread Episodic, oSen at the midpoint and
across implementa/on period
end of implementa/on period
The Global Fund. (2011). Monitoring and Evalua/on Toolkits.
Monitoring gives informa/on on where a policy,
program, or project is at any given /me (and
over /me) rela/ve to respec/ve targets and
outcomes. It is descrip/ve in intent. Evalua/on
gives evidence of why targets and outcomes are
or are not being achieved. It seeks to address
issues of causality.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
M&E
Monitoring
Evalua2on
• Clarifies program objec/ves
• Links ac/vi/es and their
resources to objec/ves
• Translates objec/ves into
performance indicators and
sets targets
• Rou/nely collects data on
these indicators, compares
actual results with targets
• Reports progress to managers
and alerts them to problems
• Analyzes why intended results
were or were not achieved
• Assesses specific causal
contribu/ons of ac/vi/es to
results
• Examines implementa/on
process
• Explores unintended results
• Provides lessons, high- lights
significant accomplishment or
program poten/al, and offers
recommenda/ons for
improvement
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Why M&E are needed?
• To ensure that resources are used effec/ve
and efficient (input)
• To ensure that the program is implemented as
planned (process)
• To inform about the level of achieved output,
therefore correc/ons may be conducted if
needed (output/outcome)
• To assess impact of program (impact)
How do we design M&E systems in a
program management?
What aspects of M&E systems should
be planned?
Planning of M&E Systems
Indicators,
baseline, target
Data source +
Methods of data
collec/on
Analysis +
Synthesis
Communica/on
+ Use of
informa/on
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Implementa/on
Results
Results based M&E
Goal
Long term, widespread improvement in
society
Outcome
Intermediate effect of outputs on clients
Outputs
Products and services produced
Ac/vi/es
Task perssonel undertake to transform
inputs into outputs
Input
Financial, human, and material resources
Results
Example: Results based M&E
Goal
Outcome
Implementa/on
Outputs
Ac/vi/es
To end TB epidemic in Indonesia
Increase the case no/fica/on rate annualy to achieve
a minimun case detec/on rate of 70%, by 2019, as
compared to 33% in 2014
34 provincial level training
1 na/onal workshop on the guideline
Intensified Case Finding (ICF) is
implemented in all 34 provinces
Guidelines, training, implementa/on of ICF
Input
Trainers, Trainee, DraS of guideline, etc
Na/onal
Context
Other
Relevant
Context
Global
Context
Goal of
Program
Situa/on
analysis
2
Situa/on
analysis
1
Situa/on
analysis
3
Outcome
Indicators, Baseline, Target
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Indikator yang Baik
Clear
Relevant
Adequate
Economic
Monitorable
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
M&E Data Source
Agregate
Annual
Report
Monthly
Report
Daily
Register
Individual
Medical
Records
Detail
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Design an evalua/on
Ovretveit J. 2003. Evalua/ng Health Interven/on. Open University Press
Purpose
Evaluator role:
internal/external
evaluator?
When the
evalua/on is
undertaken: before/
during/aSer?
Methods
Focus of evalua/on
Scope:
limited or
comprehensive?
How oSen
Focus of evalua/on
Needs
Demands
Feasibility
Inputs
Processes
Process/Forma/ve
Outputs
Outcome
Look at the effect
Methods
Quan/ta/ve
Qualita/ve
Let’s learn from a program evalua/on
research
Assessment of DOTS strategy implementa/on in
hospitals in Indonesia: 2005-2007
Adi Utarini, Ari Probandari, Trisasi Lestari, Hary
Sanjoto, Ktut As/, Agus/na, Mohammad Arifin
Provid
er
Pa/ent
The Study
Servic
e
The Perspec/ves
Burden of TB
DOTS strategy
implementa/on
DOTS Centre
Lab
TB services in
hospital
Pa/ent
District wasor
Prescrip/on
Provider
ISTC
32
Focus of evalua/on
Needs
Demands
Inputs
Processes
Process/Forma/ve
Outputs
Outcome
Methods
Quan/ta/ve
Qualita/ve
Process in external linkage
35
Free TB drugs and choices of
facili/es for treatment
36
Sputum collec/on and prac/ces in the
ward
“HIV pa/ents were put in the same room with TB, TBHIV or other pa/ents with infec/ous diseases” (Field
notes)
37
Use of secondline drugs in TB treatment
Common pattern :
-HRE-Cipro (Cat 1)
-HRZE-Cipro (Cat 1)
-Cipro (Cat II)
38
Most Common Source of TB drugs used by TB pa/ents
• Category I, II and Extra-pulmonary TB Cases:
– first line generic drugs (41-64%)
– FDC provided by NTP (21-40%)
• Children:
– first line generic (82%)
– first-line branded (18%)
39
Deficiencies in TB drugs prescrib/on and
dosage in TB Adult Cases
100%
90%
80%
70%
60%
Type
50%
Dosage
Sec ondline drug
40%
30%
20%
10%
0%
1
2
3
4
5
6
7
8
ISTC Standard Criteria
Hosp
(%)
Chest
Clinic
(%)
Standard 1
Cough for 2-3 weeks as TB suspects
88.9
50.0
Standard 2
Sputum microscopy for TB diagnosis:
Sputum test carried out
94.4
100.0
At least 2 specimens for diagnosis
94.4
100.0
Standard 3
Microscopy examina/on for EP-TB
66.7
50.0
Standard 4
Sputum examina/on for sugges/ve of TB
94.4
100.0
Standard 5
Diagnosis of SS (-) TB:
Repeat sputum examina/on
22.2
50.0
Repeat sputum examina/on&X Ray
22.2
0.0
Clinical judgement&X Ray
33.3
0.0
Scoring system for pediatrict TB
11.1
100.0
Standard 6
41
ISTC Standard
Criteria
Hosp
(%)
C.Clinic
(%)
Standard 7
Treatment monitoring
77.8
100.0
Standard 8
TB treatment:
First line drugs
94.4
100.0
Dossage conform to interna/onal
recommenda/on
94.4
100.0
Treatment supporter
72.2
50.0
Referral for poor adherence
83.3
100.0
Standard 10
SS test for monitoring treatment
88.9
100.0
Standard 11
Recording all medica/ons
88.9
100.0
Standard 12
TB-HIV:
HIV risk assessed
50.0
50.0
Referral to VCT
77.8
50.0
Standard 9
Treatment adherence:
42
ISTC Standard Criteria
Hosp
(%)
C.Clinic
(%)
ARV and TB drugs given concurrently
55.6
50.0
Cotrimoxazole profilaxis
50.0
50.0
Standard 14
Monitoring of drug resistance
44.4
50.0
Standard 15
Drug resistance TB:
Consult to MDR expert
72.2
100.0
Effec/ve four drug regimen
44.4
50.0
Standard 16
Contact tracing
50.0
0.0
Standard 17
Repor/ng all TB cases to health
authori/es
94.4
100.0
Standard 13
ARV treatment ini/a/on:
43
ISTC Public Health Responsibili/es
ISTC Standard Criteria
Hosp
(%)
Chest
Clinic
(%)
Standard 16
Contact tracing
50.0
0.0
Standard 17
Repor/ng all TB cases to health
authori/es
94.4
100.0
44
STRUCTURE
1.
2.
3.
4.
5.
6.
7.
8.
Ownership of
hospitals
Quality assurance
system in hospital
TB case load
Trained staffs
DOTS team
DOTS unit
NTP Guidelines
Financial incen/ves
from DHO to
hospitals
PROCESS
1.
2.
Findings
3.
4.
Commitment from
hospital
Adherence to
standard of
diagnosis and
treatment
Conversion rate
Case holding
process
OUTCOME
1.
2.
3.
4.
Treatment success
rate
Treatment
comple/on rate
Default rate
Coverage of DOTS
within a hospital
Probandari A, Utarini A, Hur/g AK. Achieving quality in the DOTS strategy implementa/on process:
a challenge for hospital PPM in Indonesia. Global Health Ac/on 2008. DOI: 10.3402/gha.v1i0.1831 45
Probandari A, Lindholm L, Stenlund H, Utarini A, Hur/g AK. Missed opportunity for standardized treatment among adult TB
pa/ents in hospitals involved in PPM-DOTS in Indonesia. BMC Health Services Research 2010; 10: 113
TB suspects
Outpa/ent Unit
TB suspects perform sputum test
TB suspects do not perform sputum
test
Hospital
Laboratory
Lab register
Outpa/ent Unit
Medical Record
Report ICD X A.
15-A.19
TB cases
Not TB cases
Loss of follow up among SS (+)
TB cases: 8-18%
DOTS
paBent
register
DOTS
unit
Treated
Refer to other
within the
health service
hospital not
facili/es
Referral rate
under DOTS
from Irawa/
unit
et. al. (2007)
20-53%
46
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Repor/ng M&E Results
•
•
•
•
Full wriren report
Execu/ve summary (1-4 pages)
Oral presenta/on
Policy brief
M&E Result U/liza/on
•
•
•
•
•
•
Mass-Media
Internet
E-government
Annual Report
Public Hearing with House of Representa/ve
Public Discussion
Con/nuing M&E
•
•
•
•
•
•
Demand
Clear roles and responsibility
Trustworthy and credible informa/on
Accountability
Capacity
Incen/ves
• What you have learned from this session?
Ari Probandari
Learning Objec/ves
• Students are able to explain the importance of
monitoring and evalua/on a program
management
• Students are able to apply concepts of
monitoring and evalua/on in a program
management
What is Monitoring and Evalua/on
(M&E)?
Group Work 1
Dimension
Monitoring
Evalua2on
Frequency
Periodic, occurs regularly
Episodic
Func2on
Tracking/oversight
Assessment
Purpose
Improve efficiency, provide
Improve effec/veness, impact, value
informa/on for reprogramming to for money, future programming,
improve outcome
strategy and policy making
Focus
Inputs, outputs, processes, workplans (opera/onal
implementa/on)
Effec/veness, relevance, impact,
cost-effec/veness (popula/on effect)
Methods
Rou/ne review of reports,
registers, administra/ve
databases, field observa/ons
Scien/fic, rigorous research design,
complex and intensive
Informa2on
source
Rou/ne surveillance system, field
observa/on reports, progress
reports, rapid assessment,
program review mee/ng
Same
Popula/on based surveys, vital
registra/on, special studies
Cost
Consistent, recurrent costs spread Episodic, oSen at the midpoint and
across implementa/on period
end of implementa/on period
The Global Fund. (2011). Monitoring and Evalua/on Toolkits.
Monitoring gives informa/on on where a policy,
program, or project is at any given /me (and
over /me) rela/ve to respec/ve targets and
outcomes. It is descrip/ve in intent. Evalua/on
gives evidence of why targets and outcomes are
or are not being achieved. It seeks to address
issues of causality.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
M&E
Monitoring
Evalua2on
• Clarifies program objec/ves
• Links ac/vi/es and their
resources to objec/ves
• Translates objec/ves into
performance indicators and
sets targets
• Rou/nely collects data on
these indicators, compares
actual results with targets
• Reports progress to managers
and alerts them to problems
• Analyzes why intended results
were or were not achieved
• Assesses specific causal
contribu/ons of ac/vi/es to
results
• Examines implementa/on
process
• Explores unintended results
• Provides lessons, high- lights
significant accomplishment or
program poten/al, and offers
recommenda/ons for
improvement
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Why M&E are needed?
• To ensure that resources are used effec/ve
and efficient (input)
• To ensure that the program is implemented as
planned (process)
• To inform about the level of achieved output,
therefore correc/ons may be conducted if
needed (output/outcome)
• To assess impact of program (impact)
How do we design M&E systems in a
program management?
What aspects of M&E systems should
be planned?
Planning of M&E Systems
Indicators,
baseline, target
Data source +
Methods of data
collec/on
Analysis +
Synthesis
Communica/on
+ Use of
informa/on
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Implementa/on
Results
Results based M&E
Goal
Long term, widespread improvement in
society
Outcome
Intermediate effect of outputs on clients
Outputs
Products and services produced
Ac/vi/es
Task perssonel undertake to transform
inputs into outputs
Input
Financial, human, and material resources
Results
Example: Results based M&E
Goal
Outcome
Implementa/on
Outputs
Ac/vi/es
To end TB epidemic in Indonesia
Increase the case no/fica/on rate annualy to achieve
a minimun case detec/on rate of 70%, by 2019, as
compared to 33% in 2014
34 provincial level training
1 na/onal workshop on the guideline
Intensified Case Finding (ICF) is
implemented in all 34 provinces
Guidelines, training, implementa/on of ICF
Input
Trainers, Trainee, DraS of guideline, etc
Na/onal
Context
Other
Relevant
Context
Global
Context
Goal of
Program
Situa/on
analysis
2
Situa/on
analysis
1
Situa/on
analysis
3
Outcome
Indicators, Baseline, Target
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Indikator yang Baik
Clear
Relevant
Adequate
Economic
Monitorable
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
M&E Data Source
Agregate
Annual
Report
Monthly
Report
Daily
Register
Individual
Medical
Records
Detail
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System. World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Design an evalua/on
Ovretveit J. 2003. Evalua/ng Health Interven/on. Open University Press
Purpose
Evaluator role:
internal/external
evaluator?
When the
evalua/on is
undertaken: before/
during/aSer?
Methods
Focus of evalua/on
Scope:
limited or
comprehensive?
How oSen
Focus of evalua/on
Needs
Demands
Feasibility
Inputs
Processes
Process/Forma/ve
Outputs
Outcome
Look at the effect
Methods
Quan/ta/ve
Qualita/ve
Let’s learn from a program evalua/on
research
Assessment of DOTS strategy implementa/on in
hospitals in Indonesia: 2005-2007
Adi Utarini, Ari Probandari, Trisasi Lestari, Hary
Sanjoto, Ktut As/, Agus/na, Mohammad Arifin
Provid
er
Pa/ent
The Study
Servic
e
The Perspec/ves
Burden of TB
DOTS strategy
implementa/on
DOTS Centre
Lab
TB services in
hospital
Pa/ent
District wasor
Prescrip/on
Provider
ISTC
32
Focus of evalua/on
Needs
Demands
Inputs
Processes
Process/Forma/ve
Outputs
Outcome
Methods
Quan/ta/ve
Qualita/ve
Process in external linkage
35
Free TB drugs and choices of
facili/es for treatment
36
Sputum collec/on and prac/ces in the
ward
“HIV pa/ents were put in the same room with TB, TBHIV or other pa/ents with infec/ous diseases” (Field
notes)
37
Use of secondline drugs in TB treatment
Common pattern :
-HRE-Cipro (Cat 1)
-HRZE-Cipro (Cat 1)
-Cipro (Cat II)
38
Most Common Source of TB drugs used by TB pa/ents
• Category I, II and Extra-pulmonary TB Cases:
– first line generic drugs (41-64%)
– FDC provided by NTP (21-40%)
• Children:
– first line generic (82%)
– first-line branded (18%)
39
Deficiencies in TB drugs prescrib/on and
dosage in TB Adult Cases
100%
90%
80%
70%
60%
Type
50%
Dosage
Sec ondline drug
40%
30%
20%
10%
0%
1
2
3
4
5
6
7
8
ISTC Standard Criteria
Hosp
(%)
Chest
Clinic
(%)
Standard 1
Cough for 2-3 weeks as TB suspects
88.9
50.0
Standard 2
Sputum microscopy for TB diagnosis:
Sputum test carried out
94.4
100.0
At least 2 specimens for diagnosis
94.4
100.0
Standard 3
Microscopy examina/on for EP-TB
66.7
50.0
Standard 4
Sputum examina/on for sugges/ve of TB
94.4
100.0
Standard 5
Diagnosis of SS (-) TB:
Repeat sputum examina/on
22.2
50.0
Repeat sputum examina/on&X Ray
22.2
0.0
Clinical judgement&X Ray
33.3
0.0
Scoring system for pediatrict TB
11.1
100.0
Standard 6
41
ISTC Standard
Criteria
Hosp
(%)
C.Clinic
(%)
Standard 7
Treatment monitoring
77.8
100.0
Standard 8
TB treatment:
First line drugs
94.4
100.0
Dossage conform to interna/onal
recommenda/on
94.4
100.0
Treatment supporter
72.2
50.0
Referral for poor adherence
83.3
100.0
Standard 10
SS test for monitoring treatment
88.9
100.0
Standard 11
Recording all medica/ons
88.9
100.0
Standard 12
TB-HIV:
HIV risk assessed
50.0
50.0
Referral to VCT
77.8
50.0
Standard 9
Treatment adherence:
42
ISTC Standard Criteria
Hosp
(%)
C.Clinic
(%)
ARV and TB drugs given concurrently
55.6
50.0
Cotrimoxazole profilaxis
50.0
50.0
Standard 14
Monitoring of drug resistance
44.4
50.0
Standard 15
Drug resistance TB:
Consult to MDR expert
72.2
100.0
Effec/ve four drug regimen
44.4
50.0
Standard 16
Contact tracing
50.0
0.0
Standard 17
Repor/ng all TB cases to health
authori/es
94.4
100.0
Standard 13
ARV treatment ini/a/on:
43
ISTC Public Health Responsibili/es
ISTC Standard Criteria
Hosp
(%)
Chest
Clinic
(%)
Standard 16
Contact tracing
50.0
0.0
Standard 17
Repor/ng all TB cases to health
authori/es
94.4
100.0
44
STRUCTURE
1.
2.
3.
4.
5.
6.
7.
8.
Ownership of
hospitals
Quality assurance
system in hospital
TB case load
Trained staffs
DOTS team
DOTS unit
NTP Guidelines
Financial incen/ves
from DHO to
hospitals
PROCESS
1.
2.
Findings
3.
4.
Commitment from
hospital
Adherence to
standard of
diagnosis and
treatment
Conversion rate
Case holding
process
OUTCOME
1.
2.
3.
4.
Treatment success
rate
Treatment
comple/on rate
Default rate
Coverage of DOTS
within a hospital
Probandari A, Utarini A, Hur/g AK. Achieving quality in the DOTS strategy implementa/on process:
a challenge for hospital PPM in Indonesia. Global Health Ac/on 2008. DOI: 10.3402/gha.v1i0.1831 45
Probandari A, Lindholm L, Stenlund H, Utarini A, Hur/g AK. Missed opportunity for standardized treatment among adult TB
pa/ents in hospitals involved in PPM-DOTS in Indonesia. BMC Health Services Research 2010; 10: 113
TB suspects
Outpa/ent Unit
TB suspects perform sputum test
TB suspects do not perform sputum
test
Hospital
Laboratory
Lab register
Outpa/ent Unit
Medical Record
Report ICD X A.
15-A.19
TB cases
Not TB cases
Loss of follow up among SS (+)
TB cases: 8-18%
DOTS
paBent
register
DOTS
unit
Treated
Refer to other
within the
health service
hospital not
facili/es
Referral rate
under DOTS
from Irawa/
unit
et. al. (2007)
20-53%
46
Kusek & Rist. (2004). Ten Steps to Results Based Monitoring & Evalua/on System.
World Bank.
Repor/ng M&E Results
•
•
•
•
Full wriren report
Execu/ve summary (1-4 pages)
Oral presenta/on
Policy brief
M&E Result U/liza/on
•
•
•
•
•
•
Mass-Media
Internet
E-government
Annual Report
Public Hearing with House of Representa/ve
Public Discussion
Con/nuing M&E
•
•
•
•
•
•
Demand
Clear roles and responsibility
Trustworthy and credible informa/on
Accountability
Capacity
Incen/ves
• What you have learned from this session?