What EGRA Measures Overview of Early Grade Reading Assessment

15 Comprehension Comprehension is the ability to understand, interpret, and use what has been read. Comprehension is dependent on all other components of reading. EGRA subtasks that measure this skill are reading comprehension and listening comprehension. EGRA measures each of the previously mentioned abilitiescomponents to assess the foundational reading skills. The skills are tested in individual subtasks and presented in order of increased level of difficulty. Because the first few subtasks are easier, EGRA can, therefore, measure a range of reading abilities for beginning readers. EGRA, in Indonesia and elsewhere, is not intended to be a high-stakes accountability measure to determine whether a student should move up to the next grade level. Additionally, EGRA should not be used to evaluate individual teachers. The final EGRA instrument for this baseline study included seven subtasks, all of which are summarized in Table 1.3. 16 Table 1.3: EGRA Instrument Subtasks in a Baseline Study Three of the subtasks were timed, namely letter sound identification, non-word reading, and oral passage reading. Each timed subtask was administered over a one-minute period during which the student responded to as many items in the task as possible. For scoring purposes, the assessor noted which letters or words were read correctlyincorrectly, and at the end of a minute, the assessor noted how many items the student attempted in the time available. The score for each timed subtask was calculated and expressed as “correct items per minute.” Therefore, these subtasks can be characterized as “fluency” measures. Non-fluency subtasks included untimed sets of questions. The results were calculated and expressed as “percentage of items correct out of total items attempted.” 1.4 Overview of the Snapshot of School Management Effectiveness The SSME framework is based on research reported by Craig and Heneveld 1996 and Carasco, Munene, Kasente, and Odada 1996. The SSME is an instrument that yields a quick but rigorous and multifaceted figure of school management and pedagogic practice in a country or region. The instrument was designed to capture “best” indicators of effective Subtask Skill Description: the student was asked to... Letter sound identification timed Alphabetic principle: letter- sound correspondence ...say the sound each letter makes, while looking at a printed page of 100 letters of the alphabet in random order and in upper and lower case Nonword reading timed Alphabetic principle: letter- sound correspondence Fluency- automatic decoding ...read a list of 50 nonwords printed on a page. Words were constructed from actual orthography, but were not real words in bahasa Indonesia; such as “kone” Oral passage reading timed Fluency- automatic word reading in context ... read a grade-appropriate short story out loud from a printed page Reading comprehension Comprehension ...orally respond to 5 questions that the assesor asked about the short story Listening comprehension Oral language comprehension and vocabulary ...listen to a story that the assessor read out loud, then orally answer 3 questions about the story Oral vocabulary Basic vocabulary and oral langauge comprehension ...point to body parts or objects in the room as identified by the assesor; place pencil to show understanding of prepositions Dictation Oral comprehension; writing skills; alphabetic process ...write down a sentence spoken aloud by the assessor. The sentence was read a total of three times and answers were scored both for word accuracy and for grammar 17 schools that, as past research has shown, affect student learning. The resulting data is designed to let school, district, provincial, or national administrators and donors learn what is currently occurring in their schools and classrooms and to assess how to make their schools more effective. Based on the framework for analyzing effective schools described in the literature about effective schools, the SSME collects information about basic school inputs such as school infrastructure, pedagogical materials, teacher and head teacher characteristics, student characteristics, as well as parental and community involvement and learning outcome data, via the application of core portions of the EGRA. The SSME is administered during one school day by the assessor team. In this baseline study, it was carried out in conjunction with the EGRA and in the same selected schools. Each of the components of the SSME is designed to supply information from a different perspective. The SSME design aims to balance the need to include a broad mix of variables —to allow potentially impactful characteristics to be identified —with the competing need to create a tool that is as undisruptive to the school day as possible. The combined components of the SSME can produce a comprehensive figure of a school’s learning environment, and when the results from multiple schools in a region are compared, it becomes possible to account for differences in school performance. Table 1.4 lists the SSME components in this baseline study. Table 1.4: SSME Components in a Baseline Study Level Main Variables to be Measured Data Sources School School leadership, teacher characteristics, enrollment, attendance, infrastructure and facilities, school closings Head teacher questionnaires Teacher characteristics and practices, pedagogical oversight Teacher questionnaires Infrastructure and facilities, repairs, safety, availability and use of teaching and learning materials School and classroom inventory Student and Parent Student background, interactions with the teacher, interaction with family members Student and parent questionnaires 18 18 To achieve the research objectives, a triangulation research design was applied. Desk research to collect secondary data, along with qualitative and quantitative research was carried out. The details about this triangulation technique are displayed in Figure 2.1. Figure 2.1: Triangulation Research Design

2.1 Desk Research

Relevant secondary data on education statistics at the local level provincial, district, and sub- district along with results from previous studies, notes, and other information related to the programme design pillars as well as indicators were collected and analyzed to support and explain the findings from qualitative and quantitative research. The BPS Statistics Indonesia, Pusat Data dan Statistik Pendidikan Centre for Education Data and Statistics, and MOEC were other resources of the secondary data. The Centre was also the source of information regarding the data on schools and student populations that was used in the sampling process.

2.2 Quantitative Research

Desk Research: Secondary data collection, especially from MOEC at National, Provincial, and District Level Qualitative Research: In-depth interviews with children, parents, community leaders, school personel and education authorities Quantitative Research: Student reading assessment and structured interviews with parents, teachers and head-teachers 2 METHODOLOGY 19 Surveys were carried out among children, parents households, school teachers, and head teachers. The survey was conducted through face-to-face structured interviews. For children, the focus of measurement was on reading skills, while for schools -through teachers and head teachers- key school indicators such as enrollment, attendance rates, school facilities, number of teachers, etc. were also covered in the survey. For these surveys, research instruments, in particular reading assessment, were developed in collaboration with different expertssources and in consultation with the national and provincial education officials. As the main objective of the Rural and Remote Education Initiative for Papuan Provinces are gains in reading skills in grade 2 and 3. Thus, the instrument was designed to measure basic reading skills among students in early grades. The sample was designed to measure reading skills of grade 2 students. However, in certain cases where the number of grade 2 students was less than the sample size required, then students from grade 3 were also assessed. Meanwhile, the parent survey captured several measurements such as: children ’s reading habits at home, parental and family support at home, socio-economic conditions, etc. In addition to the student reading assessment and parent interviews, the survey also covered snapshots on school management effectiveness. The snapshots covered teacher and head teacher interviews, as well as observations on school and classroom facilities. In this baseline study, research protocols of EGRA and SSME developed by RTI International- USAID were adjusted and adapted to a Papuan context, while at the same time also referred to international and national reading assessments for example, EGRA, Save the Children reading tools and the Indonesian g overnment’s measurement- Calistung. Students from second grade or in some cases they might be from third grade were involved with a relatively equal distribution between girls and boys. The surveys covered schools where the UNICEF Programme is going to be implemented intervention schools and schools without UNICEF ’s intervention control schools. For this baseline study, the data analysis will not be conducted separately between these school groups, as intervention schools have not received any intervention yet. 20

2.3 Sampling

The intervention schools that will be supported through the Rural and Remote Education Initiative for Papuan Provinces were selected in collaboration with the District Education Offices DEO. The selection consisted of 120 intervention schools in 6 districts, and it would be equally distributed across districts. Hence, each district had 20 intervention schools to adapt 2 models of intervention: 1. Model A: Cluster, at 10 schools per district 2. Model B: On the job in school, at 10 schools per district Based on the calculation of the sample sufficiencies, using the below formula, each model of intervention and control group should have at least 1,200 student samples in order to have a 5 margin of error at a 95 confidence level: � = [ � � � ⁄ � � ⁄ ] � As a result, we had 2,400 samples of students from the intervention schools and 1,200 students from the control schools. To achieve 1,200 samples of students per intervention model or control school, we assessed 20 students per school. The samples were taken from second grade students. In the case where the number of second grade students was less than 20, students from third grade were assessed to fill the gap. The following mechanism was applied in selecting the classrooms: 1. If the school had more than one second grade classroom, and the total number of students in each classroom was more than 20, we selected one second grade classroom to achieve 20 students in a random manner. 2. If the school only had one second grade classroom, and the total number of students in the classroom was more than 20, then the students from this classroom were randomly selected.