4.Panel4 Edhie Rahmat Building Governance of Health Service Delivery in Easternmost INA province1
Building Governance of Health
service delivery in Easternmost
Indonesia province
USAID Indonesia: Edhie S Rahmat, Ketty Kadarwaty, Sigid Sulistyo
Kinerja/RTI: Marcia Soumokil, Elke Rapp
Presented by Edhie S Rahmat in 3rd INAHEA Congress
Yogyakarta, July 29, 2016
• Focus specifically on improving health
governance in Jayapura city & district,
Wamena and Mimika districts
• Three main activities
– Innovation: improving the
governance and management
structures within District Health
Offices (DHO) and puskesmas
– Incentives: engaging citizens to
demand better health services from
district governments and service
providers
– Replication: disseminating improved
practices
Kinerja Papua
2
Program Assessment:
• Three primary data collection: Organizational Capacity
Assessment (OCA), Customer Satisfaction Survey
(CSS), and Qualitative study
• Purpose:
– Identify capacity change among Key partner units 2 year
intervention
– Changes in perception of unit management (has institutional
growth occurred?)
– Changes in perception of service provided by unit (have clients
received better services?)
– Document program performance, Determine efficacy of KP
approach
3
Methodology: Organizational Capacity Assessment (OCA)
• Hypothesis: Supply side intervention improves partner
health units’ organizational capacity
• Key partners focuses on the following four areas of
governance (Accountability, Responsiveness,
Community Participation & Transparency)
• Instrument: Custom Likert Scale Survey
• Sample: Random; 160 Organization Members (10 per
DHO & Puskesmas)
• Timing: Baseline April 2013, Endline March 2015
• Analysis: Descriptive statistics, T tests, and multiple
regressions
Implemented by Social Impact
Methodology: Customer Satisfaction Survey (CSS)
•
•
Hypothesis: Two-pronged approach (supply & demand)
improves customer satisfaction with puskesmas services
Instrument: Customer satisfaction survey developed by
Ministry for State Apparatus and Bureaucratic Reform (KemenPAN)
specified 14 indicators to measure customer satisfaction with service
units
•
•
•
Sample: Random; 1,800 Service users in partner puskesmas
Timing: Baseline February 2013, Endline February – March
2015
Analysis: Descriptive statistics, T tests, and multiple
regressions
Implemented by Social Impact
Methodology: Qualitative
• Triangulate and augment quantitative data
• Instrument: Custom key informant interview (KII) and focus
group discussion (FGD) guides
• Sample: DHO and puskesmas staff, service users, and MSF
members (24 FGDs and 10 KIIs = 134 respondents)
• Timing: March – April 2015
• Analysis: Triangulate quantitative findings; coding of themes
Implemented by Social Impact
Limitations: Quantitative & Qualitative
•
•
•
•
•
•
•
Cannot make statements about causation
Perception rather than direct measurement
Limited sample size for qualitative
Identifying respondents
Data collection method (paper vs. electronic)
Timing
No qualitative baseline
Implemented by Social Impact
OCA
8
Findings: Province & District-Wide
⁺Note: statistical significance is
noted as follows:
*
p < .1, ** p < .05, *** p < .01
Findings: OCA Qualitative
“The MSF reaches out to the community members and gets input from them. We
[puskesmas staff] then receive community feedback via the MSF on how we
deliver services. For example, we received input on why some of the infrastructure
in this puskesmas was broken, how we should use our vehicle, and why some of
our staff were not friendly, etc. This is how the MSF works with this puskesmas.” –
Head of PKM Mapurujaya
“Studying while working is difficult because our workload is unpredictable. For
instance, I remember I could only attend three video conference sessions. It
shows the fact that studying while learning is difficult, isn’t it?” – Head of DHO Kab
Jayapura
“We have participated in the training and we were very happy because we could
finally learn new things. However, the problem is that our boss did not follow-up
[after training]. We are just staff, so we are just waiting for our boss.” –
Puskesmas staff member, Jayawijaya
Social Impact
Jayawijaya district
Mimika district
12
Overall Findings: OCA
• Overall OCA scores increased in all KP districts
• Accountability and Responsiveness scores remain highest
in all districts, and Community Participation and
Transparency remain areas of weakness in KP districts.
However…
• 50% of units had biggest improvement in dimension
‘Community Participation’
• Change in OCA scores is higher in those units that had low
baseline scores, for example the three puskesmas in
Jayawijaya and Puskesmas Limau Asri in Kabupaten
Mimika
• DHOs did not improve as much as puskesmas and saw
decreases in some governance scores
CSS Dimension Score
Dimensions
Ratings
CSS Score
1. Procedural Easiness
Very Good
3.26 – 4.00 or 81.26 –
100
2. Conformity of service requirements
Good
2.51 – 3.25 or 62.51 –
3. Clarity & Certainty
81.25
4. Discipline of staffs
Fair
1.76 – 2.50 or 43.76 –
5. Responsibility of staffs
62.50
6. Capability of staffs
Not Good
1.00 – 1.75 or 25 –
43.75
7. Quickness of Service
8. Fairness of getting service
9. Friendliness of staffs
Based on the Ministry for State
10.Cost Reasonableness
Apparatus and Bureaucratic Reform,
Decree No.25/2004, regarding the
11.Fairness of Price
Guidance of Developing Government
12.Adherence to hours of operations Service Unit Costumer’s Satisfaction
Index.
13.Comfort of Environment
14.Security of Services
SOCIAL IMPACT
Findings: District-wide
Overall CSS Score⁺
Overall
CSS Score
82.5
80.0
77.5
75.0
72.5
70.0
67.5
65.0
62.5
Very
Good
Good
Rankin
g of
Distric
change
t
s
Baseline
Means
⁺Note: statistical significance is noted as follows: p
< .1, ** p < .05, *** p < .01
*
SOCIAL IMPACT
Change
(OLS
Regression)
1
Jayapura
7.71***
2
Jayawija
ya
4.68**
3
Kota
Jayapura
4.51**
4
Mimika
4.46*
Findings: Qualitative - Quotes
“Before, they were indeed lazy to know but now that we have MSF
they care more about discipline, 3S (sapa, senyum, sentuh) – greet,
smile, touch. Those who are late are no longer late. Because they have
made the promises. So, there are improvements. It’s not possible for
me to say 100%...” - Member of MSF Puskesmas Abepantai, Kota
Jayapura
“Some came late, puskesmas already closed, staff only comes at 9am,
goes home at 10am, 11am. Finally with Kinerja they are continuously
being pushed, the community repeatedly complained, so we finally
addressed the issue and now there are very big changes.” – Member
of MSF Puskesmas Dosay, Kab Jayapura
“On physical issues there are changes….Before patients could only
stand while waiting but now there are seats. This bench is new here.” –
Clients of Puskesmas Hom-Hom, Jayawijaya
“Quite a lot of changes. It is cleaner, they pay more attention to these.
Yes, yes (there’s changes) only we ask that equipment for medical
examinations must be here, and the doctor must be here every
evening as well…To say satisfied,
in my opinion maybe not yet,
SOCIAL IMPACT
because to be satisfied, well I think it’s when the doctors and
Findings: Overall
Most Changes: Kinerja Directly Related CSS Indicators*
Changes in
CSS Score
1.20
1.00
KOTA
JAYAPUR
A
0.80
0.60
0.40
0.48
0.4
JAYAWIJAYA
MIMIKA
1.05
KAB
JAYAPU
RA
0.48 0.5
0.85
0.680.71
0.58
0.39
0.4
0.43
0.20
Biggest Change:
#12 (Adherence to hours
of operation): 67%
#3 (Clarity & Certainty):
16.7%
#13 (Comfort)
: 16.7%
0.00
Highest Ranking:
#12 (Adherence): 33%
#1 (Procedural
Easiness): 33%
#9 (Friendliness): 25%
#13 (Comfort): 8.33%
*All changes are significant at the p <
0.01 level (99% confident)
SOCIAL IMPACT
Findings: Overall
• Despite positive changes (and for some units, ranked as
top 3 biggest changes), discipline of staff (#4) is still
relatively weak (in the lower/lowest range)
• Conversely, adherence to hours of operation(#12) saw
quite significant changes where in some units it even
went from lowest to top 3 indicators in the endline
• Weaker Puskesmas often had the biggest changes – Koya
Barat, Depapre, Hom-Hom and Limau Asri
• Fairness of Price(#11) is consistently the indicator with
the biggest changes, often highest in the endline.
However, this could be due to the new BPJS scheme and
the availability of special funding (Dana Otonomi
Khusus) for Papua affecting the number of people
covered by insurance.
SOCIAL IMPACT
Qualitative Findings
Theme 1: Increased
satisfaction with health
services at partner puskesmas
Clients were mostly satisfied
with simple demands
"...and the nurse or paramedic
staff, they did not use to smile
but now the changes are big,
things that they did not change
before, now have changed…" Puskesmas staff in Mimika
“Before there was no afternoon
service. To improve services in
this puskesmas they must have
afternoon service” – Puskesmas
staff in Jayapura district
Theme 2: Increased
knowledge and motivation
among health unit staff, but
challenges observed with
implementation
Health unit staff education
levels affect satisfaction with
training/program
“…there is personal motivation
in the unit staff, …we finally
have internal motivation…” –
Puskesmas staff in Jayapura
city
“…well, the first year was to
prepare…second year was to
familiarize ourselves. Year
three is where we see whether
it works or not. So, it did work
but definitely with challenges.
Year four and five should run
by itself. ” – Head of
Puskesmas in Jayawijaya
Theme 3: Increased public
engagement in provision of
health services
MSFs in Kota and Kab
Jayapura described
themselves as “overseer”
MSFs in Jayawijaya and
Mimika described themselves
as “helper”
“We feel that MSF is intimidating
us, when they should have been
facilitators. This problem actually
happens in]….” – Puskesmas staff
in Jayapura district
“..this forum is not ending
because Kinerja is no longer here,
but it will continue to meet as a
friend that helps them to advocate
and not just as a
overseer/referee…” District-level
MSF member in Mimika
19
Conslusions:
1. Improved capacity of partner units in
4 governance areas (OCA) & level
of satisfaction in Puskesmas (CSS)
2. Challenges in implementing new
knowledge/skills: biggest
improvements are still on basic
needs (operating hours,
infrastructure)
3. Improvements shown in weaker
areas & among lower capacity
districts
4. Improved knowledge but with limited
institutional change
5. Public engagement mechanism is
successful at improving services
Recommendations:
1. Intervention should align with
DHO’s needs, priorities &
schedules
2. Knowledge transfer and
institutional change are more
difficult in units that have many
competing priorities
3. Utilizing OCA/CSS findings to
address Puskesmas weakness
4. Capacitate DHO to monitor
Puskesmas & motivate
improvements
5. Strengthen public engagement via
MSFs on further subject
(advocacy & budget analysis)
20
THANK YOU
service delivery in Easternmost
Indonesia province
USAID Indonesia: Edhie S Rahmat, Ketty Kadarwaty, Sigid Sulistyo
Kinerja/RTI: Marcia Soumokil, Elke Rapp
Presented by Edhie S Rahmat in 3rd INAHEA Congress
Yogyakarta, July 29, 2016
• Focus specifically on improving health
governance in Jayapura city & district,
Wamena and Mimika districts
• Three main activities
– Innovation: improving the
governance and management
structures within District Health
Offices (DHO) and puskesmas
– Incentives: engaging citizens to
demand better health services from
district governments and service
providers
– Replication: disseminating improved
practices
Kinerja Papua
2
Program Assessment:
• Three primary data collection: Organizational Capacity
Assessment (OCA), Customer Satisfaction Survey
(CSS), and Qualitative study
• Purpose:
– Identify capacity change among Key partner units 2 year
intervention
– Changes in perception of unit management (has institutional
growth occurred?)
– Changes in perception of service provided by unit (have clients
received better services?)
– Document program performance, Determine efficacy of KP
approach
3
Methodology: Organizational Capacity Assessment (OCA)
• Hypothesis: Supply side intervention improves partner
health units’ organizational capacity
• Key partners focuses on the following four areas of
governance (Accountability, Responsiveness,
Community Participation & Transparency)
• Instrument: Custom Likert Scale Survey
• Sample: Random; 160 Organization Members (10 per
DHO & Puskesmas)
• Timing: Baseline April 2013, Endline March 2015
• Analysis: Descriptive statistics, T tests, and multiple
regressions
Implemented by Social Impact
Methodology: Customer Satisfaction Survey (CSS)
•
•
Hypothesis: Two-pronged approach (supply & demand)
improves customer satisfaction with puskesmas services
Instrument: Customer satisfaction survey developed by
Ministry for State Apparatus and Bureaucratic Reform (KemenPAN)
specified 14 indicators to measure customer satisfaction with service
units
•
•
•
Sample: Random; 1,800 Service users in partner puskesmas
Timing: Baseline February 2013, Endline February – March
2015
Analysis: Descriptive statistics, T tests, and multiple
regressions
Implemented by Social Impact
Methodology: Qualitative
• Triangulate and augment quantitative data
• Instrument: Custom key informant interview (KII) and focus
group discussion (FGD) guides
• Sample: DHO and puskesmas staff, service users, and MSF
members (24 FGDs and 10 KIIs = 134 respondents)
• Timing: March – April 2015
• Analysis: Triangulate quantitative findings; coding of themes
Implemented by Social Impact
Limitations: Quantitative & Qualitative
•
•
•
•
•
•
•
Cannot make statements about causation
Perception rather than direct measurement
Limited sample size for qualitative
Identifying respondents
Data collection method (paper vs. electronic)
Timing
No qualitative baseline
Implemented by Social Impact
OCA
8
Findings: Province & District-Wide
⁺Note: statistical significance is
noted as follows:
*
p < .1, ** p < .05, *** p < .01
Findings: OCA Qualitative
“The MSF reaches out to the community members and gets input from them. We
[puskesmas staff] then receive community feedback via the MSF on how we
deliver services. For example, we received input on why some of the infrastructure
in this puskesmas was broken, how we should use our vehicle, and why some of
our staff were not friendly, etc. This is how the MSF works with this puskesmas.” –
Head of PKM Mapurujaya
“Studying while working is difficult because our workload is unpredictable. For
instance, I remember I could only attend three video conference sessions. It
shows the fact that studying while learning is difficult, isn’t it?” – Head of DHO Kab
Jayapura
“We have participated in the training and we were very happy because we could
finally learn new things. However, the problem is that our boss did not follow-up
[after training]. We are just staff, so we are just waiting for our boss.” –
Puskesmas staff member, Jayawijaya
Social Impact
Jayawijaya district
Mimika district
12
Overall Findings: OCA
• Overall OCA scores increased in all KP districts
• Accountability and Responsiveness scores remain highest
in all districts, and Community Participation and
Transparency remain areas of weakness in KP districts.
However…
• 50% of units had biggest improvement in dimension
‘Community Participation’
• Change in OCA scores is higher in those units that had low
baseline scores, for example the three puskesmas in
Jayawijaya and Puskesmas Limau Asri in Kabupaten
Mimika
• DHOs did not improve as much as puskesmas and saw
decreases in some governance scores
CSS Dimension Score
Dimensions
Ratings
CSS Score
1. Procedural Easiness
Very Good
3.26 – 4.00 or 81.26 –
100
2. Conformity of service requirements
Good
2.51 – 3.25 or 62.51 –
3. Clarity & Certainty
81.25
4. Discipline of staffs
Fair
1.76 – 2.50 or 43.76 –
5. Responsibility of staffs
62.50
6. Capability of staffs
Not Good
1.00 – 1.75 or 25 –
43.75
7. Quickness of Service
8. Fairness of getting service
9. Friendliness of staffs
Based on the Ministry for State
10.Cost Reasonableness
Apparatus and Bureaucratic Reform,
Decree No.25/2004, regarding the
11.Fairness of Price
Guidance of Developing Government
12.Adherence to hours of operations Service Unit Costumer’s Satisfaction
Index.
13.Comfort of Environment
14.Security of Services
SOCIAL IMPACT
Findings: District-wide
Overall CSS Score⁺
Overall
CSS Score
82.5
80.0
77.5
75.0
72.5
70.0
67.5
65.0
62.5
Very
Good
Good
Rankin
g of
Distric
change
t
s
Baseline
Means
⁺Note: statistical significance is noted as follows: p
< .1, ** p < .05, *** p < .01
*
SOCIAL IMPACT
Change
(OLS
Regression)
1
Jayapura
7.71***
2
Jayawija
ya
4.68**
3
Kota
Jayapura
4.51**
4
Mimika
4.46*
Findings: Qualitative - Quotes
“Before, they were indeed lazy to know but now that we have MSF
they care more about discipline, 3S (sapa, senyum, sentuh) – greet,
smile, touch. Those who are late are no longer late. Because they have
made the promises. So, there are improvements. It’s not possible for
me to say 100%...” - Member of MSF Puskesmas Abepantai, Kota
Jayapura
“Some came late, puskesmas already closed, staff only comes at 9am,
goes home at 10am, 11am. Finally with Kinerja they are continuously
being pushed, the community repeatedly complained, so we finally
addressed the issue and now there are very big changes.” – Member
of MSF Puskesmas Dosay, Kab Jayapura
“On physical issues there are changes….Before patients could only
stand while waiting but now there are seats. This bench is new here.” –
Clients of Puskesmas Hom-Hom, Jayawijaya
“Quite a lot of changes. It is cleaner, they pay more attention to these.
Yes, yes (there’s changes) only we ask that equipment for medical
examinations must be here, and the doctor must be here every
evening as well…To say satisfied,
in my opinion maybe not yet,
SOCIAL IMPACT
because to be satisfied, well I think it’s when the doctors and
Findings: Overall
Most Changes: Kinerja Directly Related CSS Indicators*
Changes in
CSS Score
1.20
1.00
KOTA
JAYAPUR
A
0.80
0.60
0.40
0.48
0.4
JAYAWIJAYA
MIMIKA
1.05
KAB
JAYAPU
RA
0.48 0.5
0.85
0.680.71
0.58
0.39
0.4
0.43
0.20
Biggest Change:
#12 (Adherence to hours
of operation): 67%
#3 (Clarity & Certainty):
16.7%
#13 (Comfort)
: 16.7%
0.00
Highest Ranking:
#12 (Adherence): 33%
#1 (Procedural
Easiness): 33%
#9 (Friendliness): 25%
#13 (Comfort): 8.33%
*All changes are significant at the p <
0.01 level (99% confident)
SOCIAL IMPACT
Findings: Overall
• Despite positive changes (and for some units, ranked as
top 3 biggest changes), discipline of staff (#4) is still
relatively weak (in the lower/lowest range)
• Conversely, adherence to hours of operation(#12) saw
quite significant changes where in some units it even
went from lowest to top 3 indicators in the endline
• Weaker Puskesmas often had the biggest changes – Koya
Barat, Depapre, Hom-Hom and Limau Asri
• Fairness of Price(#11) is consistently the indicator with
the biggest changes, often highest in the endline.
However, this could be due to the new BPJS scheme and
the availability of special funding (Dana Otonomi
Khusus) for Papua affecting the number of people
covered by insurance.
SOCIAL IMPACT
Qualitative Findings
Theme 1: Increased
satisfaction with health
services at partner puskesmas
Clients were mostly satisfied
with simple demands
"...and the nurse or paramedic
staff, they did not use to smile
but now the changes are big,
things that they did not change
before, now have changed…" Puskesmas staff in Mimika
“Before there was no afternoon
service. To improve services in
this puskesmas they must have
afternoon service” – Puskesmas
staff in Jayapura district
Theme 2: Increased
knowledge and motivation
among health unit staff, but
challenges observed with
implementation
Health unit staff education
levels affect satisfaction with
training/program
“…there is personal motivation
in the unit staff, …we finally
have internal motivation…” –
Puskesmas staff in Jayapura
city
“…well, the first year was to
prepare…second year was to
familiarize ourselves. Year
three is where we see whether
it works or not. So, it did work
but definitely with challenges.
Year four and five should run
by itself. ” – Head of
Puskesmas in Jayawijaya
Theme 3: Increased public
engagement in provision of
health services
MSFs in Kota and Kab
Jayapura described
themselves as “overseer”
MSFs in Jayawijaya and
Mimika described themselves
as “helper”
“We feel that MSF is intimidating
us, when they should have been
facilitators. This problem actually
happens in]….” – Puskesmas staff
in Jayapura district
“..this forum is not ending
because Kinerja is no longer here,
but it will continue to meet as a
friend that helps them to advocate
and not just as a
overseer/referee…” District-level
MSF member in Mimika
19
Conslusions:
1. Improved capacity of partner units in
4 governance areas (OCA) & level
of satisfaction in Puskesmas (CSS)
2. Challenges in implementing new
knowledge/skills: biggest
improvements are still on basic
needs (operating hours,
infrastructure)
3. Improvements shown in weaker
areas & among lower capacity
districts
4. Improved knowledge but with limited
institutional change
5. Public engagement mechanism is
successful at improving services
Recommendations:
1. Intervention should align with
DHO’s needs, priorities &
schedules
2. Knowledge transfer and
institutional change are more
difficult in units that have many
competing priorities
3. Utilizing OCA/CSS findings to
address Puskesmas weakness
4. Capacitate DHO to monitor
Puskesmas & motivate
improvements
5. Strengthen public engagement via
MSFs on further subject
(advocacy & budget analysis)
20
THANK YOU