An analysis of Junior High School English un test items administered in 2012, 2013 and 2014.

(1)

ABSTRACT

Shabrina, F. (2015). An analysis of Junior High School English UN test items administered in 2012, 2013 and 2014. Yogyakarta: English Language Education Study Program Sanata Dharma University.

UN, abbreviated from Ujian Nasional, is a high-stakes test which is administered in Indonesia. That is why the result of UN must be trustworthy. Since validity is one of testing principles which represent the trustworthiness (Brown, 2004), this research investigates the validity of UN test items.

To be more precise, this research analyzes the documents of Junior High School English UN administered in 2012, 2013 and 2014. It describes how the three consecutive years of Junior High School English UN looks like and obtains information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN. Moreover, it uses Brown’s testing theories and Bloom’s taxonomy to analyze the data.

Based on the analysis, the three consecutive years of Junior High School English UN tests reading skills which integrate selective and interactive reading. Consequently, the three consecutive years of Junior High School English UN tests materials such as English vocabulary items, grammar and reading comprehension. Next, based on the Bloom’s taxonomy the three consecutive years of Junior High School English UN employs the lowest level of thinking skills. The three consecutive years of Junior High School English UN leads the test takers to think in the levels of remembering, understanding and applying.

Keywords: A High-stakes testing, English UN, Junior High School, Bloom’s taxonomy, Document analysis

(2)

ABSTRAK

Shabrina, F. (2015). An analysis of Junior High School English UN test items administered in 2012, 2013 and 2014. Yogyakarta: English Language Education Study Program Sanata Dharma University.

UN, singkatan dari Ujian Nasional, adalah tes berkonsekuensi tinggi yang diselenggarakan di Indonesia. Itulah mengapa hasil dari UN harus terpercaya. Karena keabsahan merupakan salah satu prinsip test yang mewakili keterpercayaan (Brown, 2004), penelitian ini mempermasalahkan keabsahan dari soal UN.

Untuk lebih jelasnya, penelitian ini menganalisis dokumen UN Bahasa Inggris Sekolah Menengah Pertama (SMP) yang diselenggarakan pada tahun 2012, 2013 dan 2014. Selebihnya, penelitian ini mendeskripsikan UN Bahasa Inggris SMP selama tiga tahun berturut-turut dan menggali informasi mengenai tingkat kemampuan berpikir yang diterapkan dalam UN tersebut. Oleh karena itu, dalam data analisis, penelitian ini menggunakan teori tes bahasa oleh Brown dan taksonomi Bloom.

Berdasarkan data analisis, UN Bahasa Inggris SMP selama tiga tahun berturut-turut menguji kemampuan membaca yang mengintegrasikan membaca selective dan interpretive. Sebagai hasilnya, UN Bahasa Inggris SMP selama tiga tahun berturut-turut mengujikan materi seperti kosa kata bahasa Inggris, tatanan bahasa Inggris dan pemahaman membaca. Selanjutnya, berdasarkan taksonomi Bloom, UN Bahasa Inggis SMP selama tiga tahun berturut-turut menerapkan tingkat kemampuan berpikir tingkat rendah yang menuntun siswa berpikir dalam tataran mengingat, memahami dan mengimplementasi.

Kata Kunci: A High-stakes testing, English UN, Junior High School, Bloom’s taxonomy, Document analysis

(3)

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

A SARJANA PENDIDIKAN THESIS

Presented as Partial Fulfillment of the Requirements to Obtain the Sarjana Pendidikan Degree

in English Language Education

Faradita Shabrina Student Number: 111214160

ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION FACULTY OF TEACHERS TRAINING AND EDUCATION

SANATA DHARMA UNIVERSITY YOGYAKARTA

(4)

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

A SARJANA PENDIDIKAN THESIS

Presented as Partial Fulfillment of the Requirements to Obtain the Sarjana Pendidikan Degree

in English Language Education

Faradita Shabrina Student Number: 111214160

ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION FACULTY OF TEACHERS TRAINING AND EDUCATION

SANATA DHARMA UNIVERSITY YOGYAKARTA

(5)

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

A SARJANA PENDIDIKAN THESIS

Presented as Partial Fulfillment of the Requirements to Obtain the Sarjana Pendidikan Degree

in English Language Education

Faradita Shabrina Student Number: 111214160

ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION FACULTY OF TEACHERS TRAINING AND EDUCATION

SANATA DHARMA UNIVERSITY YOGYAKARTA

(6)

ANANALYSNoFJTINIORHIGHSCHooLEI\GLISHUN

TEST

ITEMS ADMINISTERED IN

2012' 2013 AN{D 2014

Faradita Shabrina Student Number: 1

ll2l4l

Approved by

2 July 2015 Markus Budiraharjo, S.Pd', M'Ed', Ed'D'

(7)

A Sarjana Pendidikan Thesis on

AN ANALYSTS

JUNIOR HIGTI

SCHOOL ENGLTSH

UIT

TEST

ITEMS ADMINISTERED IN

2012, 2013

AND

2014

FARADITA SHABRINA

Student Number: 1 i 12141 60

Defer-rded before the Board of Examiners on 31 JulY 2015

and Declared AccePtable

Chairperson Secretary Member' Member Member

Board of Examiners

Paulus Kuswandono, Ph.D. Drs. Barli Bram, M.Ed., Ph.D' Dr. Emanuel Sunarto, M'Hum' Henny Herawati, S.Pd., M.Hum'

Markus Budiraharjo, S.Pd., M'Ed', Ed'D'

Yogyakarta, 31 JulY 2015

Faculty of Teachers Training and Education Sanata Dharma UniversitY

Dean i:

(8)

09rvtzttr

surrquqs slrpsJBc

W

reurl& eqJ

S I0Z ,(lnf I g 'eUu{u.(EoA

'plnoqs _Jeded_cgrluorcs_{e se}(secuoJoJeJ eql puu suorlelonb eql w pelrc osoqt _lmdxe'eldoed roqlo _Jo_{lro^\ eql;o}sgud ro

>lJo1( eql ureluoc _louseop 'ue11u,r oleq _Iqclqrn 'srsoql spll lerll eJelcep ,(11sauo{ _I

(9)

uu{e}e,(ueru Suua gI0Z,(lnf ₁g :1eEEue1 eped upu>1e,(Eo1Ip _lunql(I

'e.(ureueqes ueSuep _lunqer(es rur uueler(ured uerlnue6

'srlnued _{ru8uqss e.{es}_uuuu_{ue{run}uuoueur}dule} eruules efes epede4.(qe{o: ue{rJeqrueu undneru e,(es uep utft elururew nped eduel struep?{u ue8urluadal {n1un

ulel

Brpeu ne}e }etuo}ur 1p ue{Isu>lrlqndureru

u€p

'se1eqre1 BJeJas ue>lrsnqrrlsrpuetu'e1ep ue1e18ued _lnlueqruelep u,(uulole8ueru'ure1 erpau _Iruuaq ruelep uu>lqqu8ueu 'ueduriluou >lntun >lIBg ?uueq(J uleues ueelelsnd-re4 epedal uu>lueqtueru u(us usDlrruep ue8ueq '(epe ellq) uu{Ireqlp Eue,( _lulEueredegasag

?IOZ (INY TIOZ,ZTOT NI

(SUSISINIHI(IY

SWSJI

ISf,I

A111 HSITSNS

'IOOH]S

HCIH UOINOf .fO SISATYNV

\}'

:lnpnlreq fluef,ef,es qulrull e,fte>i erur€gq e]eueS sB]ISra.\Iu.l ueu1u1snfue4 upede>1ue>lrJeqruotrr e.(us 'uenqeta8ued nult ue8uequeEued rueq

OgIVIZIII:

Burrqeqs ulrperec :

e1Y\srser{BN loruo\

Etue\

euiluq( uleuus s€lrsJo^ruo €.^(1.srsuq8tu ?,(es 'tut qe.,!\?q rp ue3u4 epuegaq 3ue.1

XIWf,OYXY

NYCNIINf,df,X XNINN

HVIIAI'II YAUYX ISYXI-IS:Id

(10)

vi ABSTRACT

Shabrina, F. (2015). An analysis of Junior High School English UN test items administered in 2012, 2013 and 2014. Yogyakarta: English Language Education Study Program Sanata Dharma University.

Keywords: A High-stakes testing, English UN, Junior High School, Bloom’s taxonomy, Document analysis

(11)

vii ABSTRAK

Shabrina, F. (2015). An analysis of Junior High School English UN test items administered in 2012, 2013 and 2014. Yogyakarta: English Language Education Study Program Sanata Dharma University.

Kata Kunci: A High-stakes testing, English UN, Junior High School, Bloom’s taxonomy, Document analysis

(12)

viii

ACKNOWLEDGEMENTS

First of all, I would like to show my great gratitude for Allah SWT. He always raises me up when I am down. Besides, He always gives His endless love and countless blessings to me.

I would like to address my best gratitude to Bapak Markus Budiraharjo, S.Pd., M. Ed., Ed.D., for his greatest support, guidance, patience, suggestions, and time he gives to supervise my thesis. I thank him for teaching me many life lessons. Next, I would like to thank Ibu Henny Herawati, S.Pd., M.Hum., Bapak Dr. Emanuel Sunarto, M.Hum. and Bapak Barli Bram for their suggestions and guidance so I can finish my thesis. I would like to thank Ibu Yuseva Ariyani Iswandari, S.Pd., M.Ed. for being my mother in this home (PBI).

Moreover, I would like to thank Bapak Heri Sumanto, S.Pd., the vice headmaster of SMP Negeri 15 Yogyakarta, who provides the English UN items for me. It is not easy to get the English UN items because they will be used by schools to give practices to the next ninth graders. I was so lucky at that time.

I would like to show my special gratitude for three precious people in my life. I would like to thank my parents, Kusmoro Endidik and Asih Murdani, for their countless love. I thank them for always praying to Allah SWT for my happiness and my success. I also thank them for the encouragement and financial support so I can finish my study. Next, I would like to thank my sister Setya Bella for always entertaining me whenever I feel gloomy or sad. I love you all.

(13)

I would like to express my best gratitude to the best people around me who never stop encouraging me during my study. I would like to thank Mas Adityo Prawinanto for always supporting me. I would like to thank my best friends, BRE and Vivin, for believing me that I can finish my thesis. They always make me believe that a hero lays on me. Next, I would like to thank my partners in crime, Ria and Cemplis, for understanding me and giving me chances to improve my character. I am lucky to have them. I would also like to thank my friends Jum, Dheta, and BTR for giving me inspiration to always be happy no matter what.

Best regards,

(14)

TABLE OF CONTENTS

Page

TITLE PAGE ... i

APPROVAL PAGES ... ii

STATEMENT OF WORK’S ORIGINALITY ... iv

PERNYATAAN PERSETUJUAN PUBLIKASI ... v

ABSTRACT ... vi

ABSTRAK ... vii

ACKNOWLEDGEMENTS ... viii

TABLE OF CONTENTS ... x

LIST OF TABLES ... xii

LIST OF FIGURES ... xiii

LIST OF APPENDICES ... xiv

CHAPTER I. INTRODUCTION A. Research Background ... 1

B. Research Problems ... 3

C. Problem Limitation ... 4

D. Research Objectives ... 4

E. Research Benefits ... 4

F. Definition of Terms ... 6

CHAPTER II. REVIEW OF RELATED LITERATURE A. Theoretical Description ... 8

1. A High-stakes Testing ... 8

2. Ujian nasional (UN) ... 9

3. English Language Tests ... 10

a. Perceptive Reading ... 11

(15)

c. Interactive Reading ... 13

d. Extensive Reading ... 14

4. The Levels of Thinking Skills ... 14

a. Remembering ... 16

b. Understanding ... 17

c. Applying ... 18

d. Analyzing ... 19

e. Evaluating ... 19

f. Creating ... 19

B. Theoretical Framework ... 20

CHAPTER III. RESEARCH METHODOLOGY A. Research Method ... 24

B. Research Setting ... 25

C. Research Objects ... 25

D. Instruments and Data gathering Techniques ... 27

E. Data Analysis Techniques ... 28

F. Research Procedures ... 32

CHAPTER IV. RESEARCH RESULTS AND DISCUSSION A. The Description of the Three Consecutive Years of Junior High School English UN ... 37

1. Testing Selective Reading ... 39

2. Testing Interactive Reading ... 42

B. The Levels of Thinking Skills Employed in the Three-year English UN for Junior High School ... 50

1. Remembering Level of Thinking Skill ... 51

a. Recognizing ... 51

b. Recalling ... 54

2. Understanding Level of Thinking Skill ... 56

3. Applying Level of Thinking Skill ... 58

CHAPTER V. CONCLUSION AND RECOMMENDATIONS A. CONCLUSION ... 62

B. RECOMMENDATIONS ... 64

(16)

xii

LIST OF TABLES

Table Page

Table 3.1 The Analysis of Reading Types Tested in the Test Items ... 29 Table 3.2 The Analysis of the Materials which Were Tested in

the Filling-gap Texts... 30 Table 3.5 The Analysis of Questions which Represented

Reading Comprehension Tests ... 31

Table 3.6 The Analysis of Questions which Employed

Six Levels of Thinking Skills Based on Bloom’s Taxonomy ... 32 Table 4.1 The Data Presentation of Reading Test Items

Based on Brown’s Typology of Reading Tests ... 38 Table 4.2 The Materials which Were Tested in Filling-Gap Texts ... 39

Table 4.3 The Materials which Were Tested in Reading

Comprehension ... 42 Table 4.4 The Classification of the Levels of Thinking Skills

(17)

xiii

LIST OF FIGURES

Figure Page

Figure 2.1 The Changes of Bloom’s Taxonomy in Cognitive Domain ... 15

Figure 4.1 A Filling-gap Text taken from English UN 2012 ... 40

Figure 4.2 A Greeting Card taken from English 2012 ... 43

Figure 4.3 A Letter Taken from English UN 2013 ... 45

Figure 4.4 A Greeting Card taken from English UN 2014 ... 47

Figure 4.5 A Question Taken from the Letter in English UN 2014 ... 49

Figure 4.6 A Letter Taken from English UN 2014 ... 52

Figure 4.7 A Greeting Card Taken from English UN 2014 ... 53

Figure 4.8 A Greeting Card Taken from English UN 2012 ... 55

Figure 4.9 A Question Taken from Greeting Card in English UN 2014 ... 56

(18)

xiv

LIST OF APPENDICES

Appendix Page

Appendix 1 The Coding Process of the Levels of Thinking Skills

Employed in the Test Items ... 70 Appendix 2 The Coding Process to Describe How the Test Items

(19)

1 CHAPTER I INTRODUCTION

This chapter presents information and rationale why the researcher conducts the research. There will be six sections in this chapter. First is the research background which provides information about the background of the research. Second is the research problem which the researcher formulates the problems based on the background of the research. Third is the problem limitation which provides the focus of the research. Fourth is the research objectives which explain the aims of the research related to the research problems. Fifth is the research benefits which contribute benefits of the research for education field. Last is the definition of terms which defines specific words used in the research to avoid misconception.

A. Research Background

High-stakes tests describe tests which have serious consequences for students, teachers, schools and school systems (Hargrove, Jones & Jones, 2003). Madaus (1988) argues that high-stakes tests are to make decisions which affect parties include students, teachers, schools, committees and districts (as cited in Au, 2007). That is why the results of high-stakes tests must be trustworthy. In addition, validity is one of testing principles which represent the trustworthiness (Brown, 2004). Gronlund (1998) notes, “validity is the extent to which inferences

(20)

made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assessment” (as cited in Brown, 2004). Moreover, the advances in the study of cognition and measurement in the 1980s and 1990s require tests to be cognitively rich in order to gain appropriate and meaningful results (Lane, 2004). It means that tests should apply complex cognitive thinking (Lane, 2004). Briefly, tests have to prompt various levels of thinking skills in order to reach the validity.

Regarding the definition of high-stakes tests, Ujian Nasional (UN) constitutes as a high-stakes test. As stated in Permendiknas No. 5 tahun 2005, the results of UN would be used to these considerations: deciding students’ graduation, becoming one of the requirements for students to go to the next level of education, mapping the education quality of schools, determining schools’ accreditation and helping schools to improve the education quality. Since 2015, UN no longer decides the graduation (POS Penyelenggaraan UN 2015). However, the results of UN are used as the requirement to enroll at higher level schools. It means that UN still gives serious impacts to several parties such as students, schools and teachers.

Regarding UN constitutes a high-stakes test, it must be trustworthy in terms of the validity. It means that the results of UN must be appropriate, meaningful and useful in terms of the purpose (Gronlund, 1998 as cited in Brown, 2004). The validity requires UN to prompt cognitively rich assessments which means it employs various levels of thinking skills (Lane, 2004). Responding to that case, the researcher is curious to investigate UN as a high-stakes test which is

(21)

administered in Indonesia. The researcher is going to analyze UN items particularly the English subject. The researcher chooses the English subject because she is now studying in English Language Education Study Program in Sanata Dharma University which obliges her to conduct research about English language or English language teaching.

Moreover, the researcher analyzes English UN items which were administered in 2012, 2013 and 2014. The researcher chose these three consecutive years of English UN because they are the most recent examinations which are held in Indonesia. Moreover, the researcher is going to analyze English UN items for Junior High School students. It is because she has her teacher candidate practice in SMP Negeri 15 Yogyakarta. She also has experiences in teaching ninth grade students as their private teacher.

B. Research Problems

Based on the research background, there are two questions appear.

1. How are Junior High School English UN test items administered in 2012, 2013 and 2014 best described?

2. In the aforementioned Junior High School English UN test items administered in 2012, 2013 and 2014, what are the levels of thinking skills employed?

(22)

C. Problem Limitation

There are two focuses of this research. First is the description of the three consecutive years of Junior High School English UN items. Second is the level of thinking skills which are employed in the three consecutive years of Junior High School English UN items. There will be an analysis of English UN questions. The analysis will be based on the Bloom’s taxonomy theory. According to Bloom (1987), there are six levels of thinking skills namely remembering, understanding, applying, analyzing, evaluating and synthetizing.

D. Research Objectives

This research analyzes the three consecutive years of Junior High School English UN items. Therefore, this research has two objectives as follows.

1. To describe how the three consecutive years of Junior High School English UN looks like.

2. To obtain information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN.

E. Research Benefits

There are six benefits of this research especially for English teachers, schools, UN test designers and future researchers. The benefits are worthy to be contributed in Education.

(23)

1. For Schools

This research describes how English UN items look like and obtains information about the level of thinking skills which are employed in the three consecutive years of Junior High School English UN. It helps schools to prepare materials for UN from the beginning of the new academic year. Schools can predict what the materials which will be tested in next UN are. Therefore, schools can design test items for the try-outs from the beginning of the new academic year. In addition, schools can give appropriate time for students to have extension classes so students will not get tired, stressed and confused because of the extension classes. It is because when students work under pressure, they cannot study the materials effectively (Goleman, 1997, p.117 as cited in Puspitasari, Abidin & Sawitri, 2010, p.4).

2. For English Teachers

By knowing how the English UN items look like and the level of thinking skills, English teachers must not worry about the short-preparation time. They can start teach English UN materials since their students are in the seventh grade. It does not mean that they must adjust the materials, but they can embody the materials which usually are tested in English UN in the learning activities. Furthermore, they can give exercises which promote students to have higher levels of thinking skills in order to get their students ready facing UN.

(24)

3. For English UN Designers

The results and discussion of this research help English UN designers to evaluate English UN items. English UN designers can improve the quality of English UN items. They can update the topics of the reading texts implemented in English UN. Moreover, they can vary the questions by considering to the levels of thinking skills according to Bloom’s taxonomy in order to improve test takers’ thinking skills.

4. For Future Researchers

This research analyzes the levels of thinking skill employed in the three consecutive years of Junior High School English UN based on Bloom’s taxonomy. It is expected to inspire the next researchers to analyze the levels of thinking skills using other perspectives. For example, the next researchers may see the perspective of text genres which lead the test takers not only think about information written in the texts, but also think more critically.

F. Definition of Terms

In this section, the researcher is going to explain the keywords of this research. The explanation will help readers to avoid misconception.

1. Ujian Nasional (UN)

Ujian Nasional (UN) is a high-stakes test which is administered by National Education Standards Board (BSNP). Based on Permendikbud no.66 year 2013, UN has two main functions. First is to measure students’ competences on particular subjects such as Science, Social Science, Mathematics, Bahasa

(25)

Indonesia, and English. Second is to be an achievement indicator of National Education Standard. Furthermore, UN is administered in the second semester of an academic year. It is usually administered on April up to May. Moreover, UN involves students who are in their last grade such as sixth, ninth and twelfth graders. In conclusion, UN is a test which evaluates either students’ competences or education system in Indonesia.

2. English UN Test Items

English UN consists of many test items. The test items of Junior High School English UN are texts and their following questions. For example, a descriptive text and its three questions constitute one test items. Moreover, the scenario of text constitutes one test item. It is because the scenario gives instruction for students about the tasks.

3. Bloom’s Taxonomy

The term taxonomy means a system for naming or organizing things into groups which share similar qualities (“Taxonomy”). Bloom’s taxonomy is an organization of levels of intellectual behavior important in learning (Overbaugh & Schultz, 2012). It consists of a classification of thinking skills include remembering, understanding, applying, analyzing, evaluating and creating (Overbaugh & Schultz, 2012). Therefore, the researcher uses Bloom’s taxonomy to answer the second research question. The researcher uses it to obtain information about the levels of thinking skills employed in the three consecutive years of Junior High School English UN.

(26)

8 CHAPTER II

REVIEW OF RELATED LITERATURE

This chapter presents the review of theoretical writings related to English language testing and levels of thinking skills. This chapter has two main points namely the theoretical description and the theoretical framework.

A. Theoretical Description

This point reviews the theories directly relevant to the topic. This point also reviews other similar research studies relevant to the topic. Regarding the topic of this research, this point consists of four sub-points as follows.

1. A High-stakes Testing

High-stakes tests describe tests which have serious consequences for students, teachers, schools and school systems (Hargrove, Jones & Jones, 2003). Madaus (1988) argues that high-stakes tests are to make decisions which affects parties include students, teachers, schools, committees and districts (as cited in Au, 2007). That is why the results of high-stakes tests must be trustworthy. In addition, validity is one of testing principles which represent the trustworthiness (Brown, 2004). Gronlund (1998) notes,“ validity is the extent to which inferences made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assessment” (as cited in Brown, 2004). Moreover, the advances in the study of cognition and measurement in the 1980s and 1990s require tests to be cognitively rich in order to gain appropriate and meaningful results (Lane,

(27)

2004). It means that tests should elicit complex cognitive thinking (Lane, 2004). Briefly, tests have to prompt various levels of thinking skills in order to reach the validity.

2. Ujian Nasional (UN)

UN, abbreviated from Ujian Nasional, is a high-stakes test which is administered in Indonesia. Stated in Permendikbud No. 97 tahun 2013, UN is an instrument to measure and assess the standard competencies nationally in particular subjects. Moreover, UN is administered mainly by National Education Standards Board (BNSP). Based on POS Penyelenggaraan UN 2015, UN is annual program which is administered on the second semester every year; it is about April to May. Considering UN is administered nationwide, it is exactly directed to students in Indonesia including Elementary School, Junior High School, Senior High School, and Vocational School. The subjects which are tested in each level are different. For Junior High School students, there are four subjects which are tested. They are Indonesian, Mathematics, Science, and English (Bonawati, personal communication, October 21st, 2014). Thus, UN is an Education Department’s program which is administered nationally to test students’ proficiencies of particular subjects they have taken in schools.

Regarding UN is a high-stakes test, it must be trustworthy in term of the validity. It means that the results of UN must be appropriate, meaningful and useful in terms of the purpose (Gronlund, 1998 as cited in Brown, 2004). The validity requires UN to prompt cognitively rich assessments which employs various levels of thinking skills (Lane, 2004).

(28)

After talking about UN, the next sub-point reviews the theories about English language testing. In reviewing the theories, this research uses language testing theories by Brown (2004). Besides, this research involves other experts in order to support Brown’s theories. To be more precise, the review is in the next sub-point.

3. English Language Testing

According to Brown (2004, pp. 3-4), a test is a method of measuring an individual’s ability, knowledge or performance in a given domain. It means that a test represents how good someone’s ability, knowledge or performance in a particular subject. For example, English is one of the subjects which are tested. English language tests measure test takers’ abilities to perform English language skills, which are to read, listen, write or speak (Brown, 2004, p. 3). Besides, an English language tests measure test takers’ English language knowledge about vocabulary items, grammatical rules, and rhetorical features in written discourse (Brown, 2004, p. 3). Thus, English language tests measures test takers’ English language abilities, knowledge or performances.

According to Brown (2004, p. 185), reading is a process of conveying meanings from the written words in order to gain information. Brown states, “The written word continues in its function to convey information, to assume and entertain us, to codify our social, economic and legal conventions…” (2004, p. 185). Moreover, there are four types of reading based on the tasks given namely perceptive, selective, interactive and extensive reading (Brown, 2004, p. 189).

(29)

a. Perceptive Reading

According to Brown (2004, p. 189), perceptive reading is an assessment for students who are at the beginning level of reading a second language. The tasks of perceptive reading imply that the students are in the early stages of becoming “literate.” Therefore, perceptive reading is one of the reading test levels which focus on the components of larger stretches of discourse: letters, words, punctuation, and other graphemic symbols (Brown, 2004, p. 189). The assessment which represents perceptive reading are reading aloud, picture-cued items and written response (Brown, 2004, p. 189).

b. Selective Reading

According to Brown (2004, p. 189), selective reading is one of reading test levels which focuses on the form of a language. It ascertains the test takers’ recognition of lexicons and grammatical rules of language within a very short stretch of language (Brown, 2004, p. 189). For examples, the preposition during has two meanings. First, the preposition during represents the meaning of a period time as in the sentence They work during the night and sleep by day. Second, it represents the meaning of several times in a period as in the sentence I woke up several times during the night (“During”). Another example is the English vocabulary room. It means a part of the inside of a building that is separated from other parts by walls, floor and ceiling (“Room”).

Other English vocabulary items which may be tested are forward, support, nice and careful. The word forward means coming to the front (“Forward”). The word Support means giving encouragement to someone or something because you

(30)

want them to succeed (“Support”). Next, the word nice means enjoyable (“Nice”). Last, the word careful means giving a lot of attention to what you are doing so you do not have an accident, make a mistake, or damage something (“Careful”)

The next examples are about the grammatical rules such as the use of simple past tenses, an adverb there, a pronoun it and simple present tenses. The simple past tense expresses events which began and end in the past (Azar, 1999, p. 27). According to Allen (1995, p. 86), “The great majority of English verbs form the past simple tense and past participle by adding the sound [d], [t] or [id] to the stem…” Allen (1995, p. 86) states that there are irregular verbs used in the simple past tense. Besides, the verb came is one of the examples of the irregular verb (“Come”).

The other examples of the grammatical rules are the use of an adverb there and a pronoun it. The adverb there is an adverb which indicates places (“There”). Next, a pronoun is used in place of a noun (Azar, 1999, p. 132). For example, I read a book; it was good. The pronoun it refers to the noun book (Azar, 1999, p. 132). Next, the simple present is used to tell the truth (Azar, 1999, p. 13). Krohn (1986, p. 11) states the –s form of verbs is used in the simple present tense with third person singular objects; otherwise the simple form is used. For example, He works every morning.

Besides testing vocabulary items and grammatical rules, English language tests may test about English expression. One of English expression which may be tested is compliments. Compliments express appreciation and admiration at someone’s appearance, belongings or works (Bagus, 2012). There are examples of

(31)

compliment such as That’s nice, That’s neat, Pretty good and I really like your skirt (Bagus, 2012).

Moreover, there are two characteristics of selective reading. First, selective reading tasks typically are constructed by using stimuli such as sentences, brief paragraphs or simple charts and graphs (Brown, 2004, p.189). Second, brief responses are intended in selective reading (Brown, 2004, p.189). Briefly, one example of selective reading tasks is filling-gap task (Brown, 2004, p. 189). c. Interactive Reading

According to Brown (2004, p. 201), interactive reading is similar to selective reading. Interactive reading focuses on the form of a language and meanings, but it focuses more on the meanings (Brown, 2004, p. 201). It focuses on how the form of a language makes meanings. Therefore, there are two characteristics of interactive reading. First, it involves the test takers to interact with longer texts (Brown, 2004, p. 189). The test takers have to negotiate with the meanings of texts in order to get intake to the texts (Brown, 2004, p. 189). Second, interactive reading typically uses text genres such as narrative texts, descriptions, memos, recipes, announcements and so forth (Brown, 2004, p. 189). Moreover, Brown (2004, pp. 201-210) states that interactive reading uses tasks such as cloze tasks, a reading comprehension, short-answer-tasks, editing, scanning, ordering tasks, and information transfer (from charts, maps, diagrams and graphs).

Regarding a reading comprehension is an example of interactive reading tasks, it is commonly used in English language tests (Brown, 2004, p. 204).

(32)

According to Brown (2004, p. 3), a reading comprehension consists of a set of texts with the comprehension questions. There are eight features which cover the comprehension. They are main ideas (topic), expressions/idioms/phrases in context, inferences, grammatical features, detailed information, unstated details, supporting ideas and vocabulary in context (Brown, 2004, p. 206). Briefly, a reading comprehension usually involves reading strategies such as skimming, scanning and guessing word meaning from context (Brown, 2004, p. 206). To be more precise, skimming is a reading method of rapidly moving the eyes through the text with the purpose to gain main ideas and general overview of the content of the text (Campbell, 2005). Next, scanning is a reading method of rapidly covering a great breadth of the display in order to locate icons, symbols, or words in the text (Campbell, 2005).

d. Extensive Reading

Expensive reading is a type of reading which involves more than a page text including professional articles, essays, technical reports, short stories and books (Brown, 2004, p. 189). The tasks which represent extensive reading are note-taking and outlining (Brown, 2004, p. 215).

After talking about English testing theories according to Brown (2004), the next part talks about the levels of thinking skills derive from Bloom’s taxonomy. 4. The Levels of Thinking Skills

According to Bloom (1987), here are six levels of thinking skills which are well-known as Bloom’s taxonomy in cognitive domain. Bloom’s Taxonomy is a classification of a set behavior in order to define the goals of learning (Bloom,

(33)

Create

Evaluate

Analyze

Apply

Understand

Remember

1987, pp. 1-2). Athanassiou, McNett and Harvey (2003) also state that Bloom’s taxonomy is a six-classification system which is to observe students’ behavior and infer students’ cognitive achievement. The classification consists of such categories: knowledge, comprehension, application, analysis, synthesis and evaluation (Bloom, 1987). As time goes by, Anderson and Krathwohl (2001) made a revision of Bloom’s taxonomy in cognitive domain. To be more precise, the following graphs will present the revision of Bloom’s taxonomy in cognitive domain. Churches (2008) states these classifications present Bloom’s taxonomy in cognitive domain from the lowest level of thinking skills (remembering) to the highest level of thinking skills (creating).

The Original Bloom’s Taxonomy The Revision of Bloom’s Taxonomy Figure 2.1 The Changes of Bloom’s Taxonomy in Cognitive Domain

These six levels of thinking skill are divided into two categories. The categories are “lowest level of thinking skill” and “highest level of thinking skill” (Anderson, 1990, p. 425). There are two levels which are considered as the lowest

Evaluation Synthesis

Analysis Application Comprehension

Knowledge

HIGH

(34)

levels of thinking skill. They are remembering, understanding and applying (Forehand, 2010). It is supported by Churches (2008) that students cannot understand the concept if they do not remember it first, similarly they cannot apply if they do not remember information and understand concepts. It means that the basic level of thinking start from remembering, understanding then applying. Moreover, there are three levels of thinking skill which are considered as the highest levels (Forehand, 2010). They are analyzing, evaluating and creating (Forehand, 2010). Mayer and Wittrock (1966) note these levels are considered as the higher levels because they leads students to transfer information they have before to solve new problems or new questions (as cited in Anderson & Krathwohl, 2001, p. 63). To be more precise, here is the explanation.

a. Remembering

According to Anderson and Krathwohl (2001, p. 63), remembering promotes retention which means that it leads students to memorize information at some later time and the same way they are given. Churches (2008) states remembering level is a process of retrieving relevant information. In assessing students’ learning, remembering employs two behaviors such as recognizing and recalling (Anderson & Krathwohl, 2001, p. 66). First, recognizing (identifying) leads students to retrieve relevant knowledge from their long-term memory in order to be compared or match with presented information (Anderson & Krathwohl, 2001, p. 69). Recognizing employs tasks such as verification, matching and multiple choices (Anderson & Krathwohl, 2001, p.69). It is convinced by Shepard (1993, 2000) that a multiple choice test mostly focuses on

(35)

the memorization in which it should concern on complex cognitive process (as cited in Haladyna, 2012, p. ix).

Second, recalling (retrieving) leads students to retrieve relevant knowledge from their long-term memory when given stimuli to do so (Anderson & Krathwohl, 2001, p. 69). The stimuli are usually questions (Anderson & Krathwohl, 2001, p.69). An example given by Haladyna (2012, p. 31) that students are given a task to mention one primer number. It needs students’ long term-memory in order to answer the question (Haladyna, 2012, p. 31). Thus, remembering level mostly employs memorization skill in assessing students’ competences.

b. Understanding

According to Anderson and Krathwohl (2001, p. 70), understanding occurs when students are able to construct meaning from instructional messages including oral, written or graphs. It means understanding leads students to build connection between the new knowledge and their prior knowledge (Anderson & Krathwohl, 2001, p. 70). The new knowledge here does not mean really “new”, but it still has connection to students’ existing knowledge (Anderson & Krathwohl, 2001, p.70).

Moreover, interpreting and summarizing are the behaviors employed in understanding level (Anderson & Krathwohl, 2001, pp. 70-73; Curches, 2008). First, interpreting occurs when students are able to convert information from one representational form to another (Anderson & Krathwohl, 2001, p. 70). For example, students are able to paraphrase existing sentences without changing

(36)

meanings (Anderson & Krathwohl, 2001, p. 70). Second, summarizing occurs when students are able to suggest a single statement that represents presented information (Anderson & Krathwohl, 2001, p. 73). It means that summarizing level leads students to construct a representative meaning of the information (Anderson & Krathwohl, 2001, p. 73). An example of summarizing task is to determine a theme or main points of a paragraph or texts (Anderson & Krathwohl, 2001, p. 73). Thus, understanding level mostly focuses on the meaning construction.

c. Applying

According to Anderson and Krathwohl (2001, p. 77), applying occurs when students are able to solve problem by considering the procedural knowledge they already have in minds. It means that students are able to use knowledge and concepts they already have in minds (Forehand, 2010). Therefore, applying is considered as higher level than remembering and understanding (Forehand, 2010). One of applying level behaviors is executing (Anderson & Krathwohl, 2001, p. 77; Churches, 2008). Executing leads students to use procedural knowledge to perform a familiar task (Anderson & Krathwohl, 2001, p. 77). Bransford, Brown and Cocking (1999) note procedural knowledge is knowledge which has rules so that students know when and where to use it (as cited in Anderson & Krathwohl, 2001, p. 52). Thus, applying level employs students’ thinking skill to carry out their prior knowledge and concepts to the new situation (Forehand, 2010).

(37)

d. Analyzing

According to Anderson and Krathwohl (2001, p. 79), analyzing occurs when students are able to categorize information into its constituent parts and determine how the parts are related to one another and to overall structure. Forehand (2010) note that analyzing involves students to be able to distinguish information by creating charts, diagrams, surveys or spreadsheets. Additionally, information here means the pieces of message which students have to organize and underlying the purposes (Anderson & Krathwohl, 2001, p. 79). Analyzing level has behaviors which are employed in leaning such as differentiating, organizing and attributing (Anderson & Krathwohl, 2001, pp. 80-84).

e. Evaluating

According to Anderson and Krathwohl (2001, p. 83), evaluating occurs when students are able to make judgments on quality, effectiveness, efficiency and consistency based on criteria they already perceive. Driscoll and Driscoll (2005, p. 348) state evaluating is when students make judgments about the extent to which methods or materials satisfy existing criteria. Briefly, there are the behaviors which are employed in evaluating skill such as checking and critiquing (Anderson & Krathwohl, 2001, pp. 83-84); Driscoll & Driscoll, 2005, p. 348; Forehand, 2010).

f. Creating

According to Anderson and Krathwohl (2001, pp. 64-65), creating is the highest level of thinking skill which involves students to be able to generate new products using their background knowledge. Driscoll and Driscoll (2005, p. 348)

(38)

and Forehand (2010) argue that creating occurs when students are able to put together parts, elements or materials to form new structures or patterns. Creating does not mean writing which represents the remembering of ideas or the interpretation of materials (Anderson & Krathwohl, 2001, p. 85). Students must synthetize their own ideas or, in other words, take a position of particular cases (Anderson & Krathwohl, 2001, p.85). In brief, there are several behaviors which are employed in creating skill such as generating, planning, producing, inventing, designing and making (Anderson & Krathwohl, 2001, pp. 86-87; Churches, 2008).

B. Theoretical Framework

This research uses several theories such as high-stakes tests, Ujian Nasional (UN), Englisng language testing and Bloom’s taxonomy. First, the theories about high-stakes tests and UN are used to shape the researcher’s thinking framework in writing the research. To be more specific, high-stakes tests describe tests which have serious consequences for students, teachers, schools and school systems (Hargrove, Jones & Jones, 2003). Therefore, UN is considered as a high-stakes testing in Indonesia since it is to make decisions which affect testees include students, teachers, committees, schools and districts (Madaus, 1988 as cited in Au, 2007).

Second, the theories about English language tests are used to answer the second research question. They are used to describe the three consecutive years of Junior High School English UN items. Moreover, the research uses Brown’s

(39)

English language test theories. According to Brown (2004), language tests measure individual’s performance of a language like reading. There are four types of reading; two of them are selective and interactive reading (Brown, 2004). First, selective reading focuses on integrating the form of a language such as lexicons, grammar and discourse features (Brown, 2004). An example of selective reading tasks is fill-in-the-blank in form of multiple choices (Brown, 2004). Second, interactive reading also focuses on the form of a language, but it emphasizes more on the meaning of messages given (Brown, 2004). Therefore, it leads test takers to interact with longer texts in order to gain meanings (Brown, 2004). An example of interactive reading tasks is a reading comprehension (Brown, 2004). It is because a reading comprehension consists of numerous texts which lead test takers to gain meanings (Brown, 2004). Additionally, a reading comprehension consists of questions following the texts (Brown, 2004). These questions are main ideas (topic), expressions/idioms/phrases in context, inference (implied detail), grammatical features, detailed information, unstated details, supporting ideas and vocabulary in context (Brown, 2004, p. 206).

Third, the research uses Bloom’s taxonomy to answer the second research question. Bloom’s taxonomy is used to obtain information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN. Therefore, there are six levels of thinking skills according to Bloom (2004). They are knowledge, comprehension, application, analysis, synthesize and evaluation (Anderson & Krathwohl, 2001). As time goes by, Anderson and Krathwohl revised Bloom’s taxonomy. Therefore, the changes of

(40)

Bloom’s taxonomy in cognitive domain are remembering, understanding, applying, analyzing, evaluating and creating (Anderson & Krathwohl, 2001). These levels work from the lowest, remembering, to the highest, creating (Anderson & Krathwohl, 2001; Churches, 2008).

Accordingly, these six levels are divided into two categories (Anderson, 1990). First are the lowest levels of thinking: remembering, understanding and applying (Forehand, 2010). These levels promote retention in which leads students to work on their memorization (Anderson & Krathwohl, 2001). Second are the highest levels of thinking: analyzing, evaluating and creating (Forehand, 2010). Mayer and Wittrock (1966) note these levels promote transfers in which students are able to transfer their prior information or knowledge to solve new problems. Thus, each level of thinking skills has its complexities which go higher and higher (as cited in Anderson & Krathwohl, 2001).

Regarding the complexities of each level, there are several behaviors which are employed as the goals of learning (Anderson & Krathwohl, 2001; Athanassiou, 2003). Remembering leads students to recognize and recall information they have before (Anderson & Krathwohl, 2001; Churches, 2008; Haladyna, 2012; Shepard, 1993). Understanding leads students to interpret and summarize messages they get (Anderson & Krathwohl, 2001; Churches, 2008). Applying leads students to execute their knowledge prior knowledge in new situations (Aderson & Krathwohl, 2001; Forehand, 2010; Churches, 2008; Bransford, Brown & Cocking, 1999).

(41)

Next, analyzing leads students to differentiate categorize and attribute information into parts and know how they correlated to one another (Anderson & Krathwohl, 2001; Forehand, 2010). Evaluating leads students to check and critique on effectiveness, quality and efficiency based on criteria (Anderson & Krathwohl, 200; Driscoll & Driscoll, 2005; Forehand, 2010). Last, Creating leads students to generate, make, design, plan and invent new ideas by themselves (Anderson & Krathwohl, 2001; Churches, 2008). As a result, these six levels of thinking lead students to perform their competences and abilities better.

(42)

CHAPTER III

RESEARCH METHODOLOGY

This chapter presents the methodology of the research and the analysis. There are six points in this chapter namely research method, research setting, research objects, research instruments and data gathering technique, data analysis technique, and research procedure.

A. Research Method

This research constituted qualitative research. It was because the researcher used her own experiences as the basic background of this research. Merriam (2009, p.5) shares that qualitative research focuses on how the researchers interpret their experiences, how they construct their knowledge and how they attribute meaning to their experiences. Since there were several methods in the qualitative research, this research used a content analysis method. There were two reasons why this research employed the content analysis method.

First, this research analyzed the test items which were the result of human communication. Fraenkel and Wallen (2006, p. 483) states a content analysis was a method to investigate human behavior indirectly through an analysis of their communication. The human communication does not mean as it is, but it relates to many things which are produced by human beings (Fraenkel & Wallen, 2006, p. 483).

(43)

Second, this research had two objectives which were to describe the high-stakes test in Indonesia and to obtain the levels of thinking skills employed in that kind of test. Fraenkel and Wallen (2006, 485) notes the content analysis was to obtain descriptive information about the topic and information useful to education world. Thus, this research would investigate problems which related to education.

B. Research Setting

This research was conducted in ongoing process. The researcher started collecting the data from November 2014. The researcher, then, analyzed the data in January to May 2015. The researcher took times in conducting this research because there were several obstacles she faced. These obstacles actually help the researcher to learn deeply about the topic she investigated. Moreover, there were no particular places such as schools or classes for researcher to conduct this research. It was because this research did not use observation or interviews.

C. Research Objects

As stated before, the content analysis used human communication as the data. Fraenkel and Wallen (2006, p. 483) state the human communication which is investigated in the content analysis was many things produced by human beings. Therefore, the objects of this research were the test items of Junior High School English UN as designed by a team of test designers. Furthermore, this research investigated Junior High School English UN administered in 2012, 2013 and

(44)

2014. The test items of the three consecutive years of Junior High School English UN constituted the population of this research. Fraenkel and Wallen (2006, p. 92) argue that population is the larger group to which the researcher hopes to apply the results. Fraenkel and Wallen (2006, p. 92) also argue that the smaller number of groups taken from the population was called as a sample. Hence, the sample of this research was 10 greeting cards, 19 letters, 32 filling-gap texts along with 183 following questions. Briefly, the researcher used a purposive sampling method in the instrumentation.

A purposive sampling was a method which was used by researchers to select a sample by using their personal judgment on the population (Fraenkel & Wallen, 2006, p. 100). In conducting this research, the researcher had several reasons for defining 10 greeting cards, 19 letters, 31 filling-gap texts and 183 following questions as the sample. First, the sample revealed two different reading types which were tested in the three consecutive years of Junior High School English UN. Second, the sample gave information about the materials which were tested in the three consecutive years of Junior High School English UN. Third, the sample covered the levels of thinking skills which were employed in the three consecutive years of Junior High School English UN. Thus, the researcher used a purposive sampling to judge the sample of this research whether it would represent the topic investigated or not.

(45)

D. Instruments and Data Gathering Techniques

An instrument was a device which the researchers used to gather data, and it involved processes called instrumentation (Fraenkel & Wallen, 2006, pp. 112-113). The instrument of this research was personal construct which used the researcher herself as the major instrument of the research. It was supported by Cohen, Manion and Marrison (2011, p. 498) that the personal construct is a research instrument which uses the researchers’ constructs to interpret what they perceived in terms of their existing mind and the way they behave towards it. Merriam (2009, p. 150) convinces,” Since investigator is the primary instrument for gathering data, he or she relies on skills and intuition to find and interpret data from documents”. Thus, the researcher used her knowledge about Brown’s testing

theories and Bloom’s taxonomy to collect and interpret the data which she found

from the three consecutive years of Junior High School English UN documents. The researcher used coding categories as the technique to gather the data since formulating coding categories was the technique applied in the content analysis (Fraenkel & Wallen, 2006, p. 487). Moreover, the researcher determined the categories before the analysis began. Fraenkel and Wallen (2006, p. 485) argue that the researcher can determine the categories before any analysis began.

The researcher used Brown’s testing reading theories and Bloom’s taxonomy theory to formulate the coding categories. Brown (2004) states that there are four types of reading, both of them are selective and interactive reading. It helped the researcher to answer the first research question about the description of the three consecutive years of Junior High School. Moreover, the researcher

(46)

used Bloom’s taxonomy to answer the second research question about the levels

of thinking skills which were employed in the three consecutive years of Junior High School English UN. Anderson and Krathwohl (2001) states there are six levels of thinking skills which were discovered by Bloom (1987) namely remembering, understanding, applying, evaluating and creating.

Accordingly, the six levels of thinking skills employed certain behaviors which have been the characteristics of each level. For example, remembering employed recognizing and recalling behaviors. Understanding employed interpreting and summarizing behaviors. Next, applying employed executing behavior. Analyzing employed differentiating, organizing and attributing behaviors. Evaluating employed checking and critiquing. Last, creating employed generating, planning, producing, inventing, designing and making behaviors (Anderson & Krathwohl, 2001, pp. 66-88; Churches, 2008).

E. Data Analysis Techniques

Counting is a characteristic of some content analysis methods (Fraenkel & Wallen, 2006, p. 490). The researcher, hence, used counting as the technique to analyze the data which had been categorized. The researcher counted the frequency of several aspects which would be the research results. Fraenkel and Wallen (2006, p. 490) state the content analysis obviously is dealing with counting frequencies to gather the data.

(47)

1. Counting the Frequency of Reading Types

One of the objectives of this research was to describe the three consecutive years of Junior High School English UN test items. In describing the test items, the researcher made a cross tabulation of reading types which were tested in the test items as the continuation of the coding process. The cross tabulation was presented as follows.

Table 3.1 The Analysis of Reading Types Tested in the Test Items

Brown’s Reading Typology English UN 2012 English UN 2013 English UN

2014 Frequency

Percentage (%) Perceptive Selective Interactive Extensive

Table 3.1 was the data presentation of reading types which are based on Brown’s typology of reading tests. According to Brown (2004, p. 189), there are four reading types namely perceptive, selective, interactive and extensive reading. These reading types were used to classify the data which has been analyzed. Moreover, the researcher counted the frequency of the test items which included in each reading type and changed it to the percentage. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

2. Counting the Materials which Were Tested in the Filling-gap Texts Since filling-gap texts represented tasks of selective reading (Brown, 2004, p. 189), they contribute information related to how the three consecutive years of

(48)

Junior High School English UN looked like. The researcher found an aspect which could be the results of the topic she investigated. The aspect was about the materials which were tested in the filling-gap texts. Additionally, the researcher used the following table to help her analyze the data.

Table 3.2 The Analysis of the Materials which Were Tested in the Filling-gap Texts

Materials

Material 1 Material 2

Frequency Percentage (%)

Based on the coding process, the researcher listed the materials which were tested in the filling-gap texts. The researcher, then, counted the frequency of how many times certain materials appeared. The researcher also counted the frequency in form of percentage. Finally, the researcher used the counting result as the findings which would be discussed in the nest chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

3. Counting the Materials which Were Assessed in Reading Comprehension After the coding process, the researcher counted the questions of the three consecutive years of Junior High School English UN which represented the reading comprehension. The researcher used the following table to analyze the data.

(49)

Table 3.3 The Analysis of Questions which Represented Reading Comprehension Tests

The materials

Material 1 Material 2 Material 3 Material 4 Material 5 Material 6

Frequency Percentage

(%)

The researcher counted the frequency of how many times the material appeared. The researcher then counted the frequency in form of a percentage. Briefly, the researcher used the counting result as the data to be discussed in the next chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

4. Counting the Questions which Employed Six Levels of Thinking Skills Based on Bloom’s Taxonomy

Based on the coding process, the researcher counted the questions of the three consecutive years of Junior High School English UN in order to obtain information about the levels of thinking skills employed in that English UN. The researcher used the following table to help her analyze the data.

(50)

Table 3.4 The Analysis of Questions which Employed Six Levels of Thinking Skills Based on Bloom’s Taxonomy

Six Levels

of Thinking Skills The Behaviors 2012 2013 2014 Frequency

Percentage (%)

Low

igh

Creating Generating

Evaluating Checking

Analyzing Organizing

Applying Executing

Understanding Summarizing

Remembering Recalling

Recognizing

The researcher counted the frequency of the questions which represented each behavior of the levels of thinking skills. The researcher then changed the frequency to the percentage. Finally, the counting result would be the finding which discussed in the next chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

F. Research Procedure

There were eight procedures which the researcher passed when conducting this research. The procedures were presented as follows.

1. Determining the Objectives

Since this research analyzed the three consecutive years of Junior High School English UN, it had an objective to describe the English UN itself. The researcher then specified the objectives of this research based on the research problems she formulated. Therefore, this research had two objectives. First was to describe how the three consecutive years of Junior High School English UN

(51)

looked like. Second, was to obtain information about the levels of thinking skill employed in the three consecutive years of Junior High School English UN. 2. Defining the Terms

The researcher defined the important terms of this research to avoid misconception. As Fraenkel and Wallen (2006, pp. 485 – 486) state that researchers as in all research might incur misunderstanding, the important terms must be defined. Consequently, there are two important terms in this research to define. First is UN abbreviated from Ujian Nasional. The term UN was important to be defined because it was one of many examinations which were held in Indonesia. The definition of the term UN helped the readers avoid misunderstanding about what kind of test UN was. Second, “English test items” was another important term in this research. It was important to define because it would frequently appear in each chapter of this research. It helped the reader also

not to assume that the term “items” was the test questions. Briefly, the researcher

defined the term “English test items” as the English reading passages along with

their following questions since Junior High School English UN consisted of reading passages and questions.

3. Locating the Relevant Data

The researcher located the relevant data based on the objective of this research. As Fraenkel and Wallen (2006, p. 486) argue, researchers located the data after they defined the research objectives clearly. Therefore, the researcher used the three consecutive years of Junior High School English UN items as the

(52)

located data. The researcher used that data since the objective of this research was to describe Junior High School English UN.

4. Developing a Sampling

The researcher developed a purposive sampling in this research. The researcher then chose 10 greeting card, 19 letter and 31 filling-gap texts along with 183 following questions. The researcher intentionally determined the sample because it had specific reasons which supported the topic she investigated. As Fraenkel and Wallen (2006, p. 100) state, researchers could use a purposive sampling to determine the sample which specifically supported the purposes of the research.

5. Formulating Coding Categories

The research conducted two coding processes in this research. It was because the researcher applied two techniques when determining the coding categories. The researcher determined the coding categories to gather the data about the levels of thinking employed in the three consecutive years of Junior High School English UN before any analysis began. Fraenkel and Wallen (2006, p. 485) argue, researchers could determine the categories before any analysis began.

6. Reliability and Validity

Reliability and validity were process to measure the consistency, correctness and meaningfulness of the data collecting process which had been conducted (Fraenkel & Wallen, 2006, p. 150). This research used triangulation techniques to check how reliable and valid the data was. Campbell and Fiske

(53)

(1959) argue, “Triangulation is a powerful way of demonstrating concurrent

validity, particularly in qualitative research” (as cited in Cohen, Manion, and Morrison, 2011, p. 195). Moreover, this research applied three types of triangulation in order to measure the reliability and validity of the collected data.

First is time triangulation. Cohen, Manion, and Morrison (2011, p. 196) state, “This type attempts to take into consideration the factors of change and process by utilizing cross-sectional and longitudinal designs.” Therefore, the process which allowed the researcher to make revisions ensured the researcher’s ways of thinking so that it made the collected data valid. Second is theoretical triangulation. Cohen, Manion, and Morrison (2011, p. 196) state, “This type draws upon alternative or competing theories in preference to utilizing one

viewpoint only.” The researcher used theories about testing by Brown (2004) and

about levels of thinking skill by Anderson and Krathwohl (2001) to shape her point of view. That was why these theories made the consistency and correctness of the data collecting process.

Last is investigator triangulation. Silverman (1993) states that researchers can measure the research reliability and validity by engaging more than one observer (as cited in Cohen, Manion, and Morrison, 2011, p. 196). Therefore, the

researcher engaged experts’ judgments to measure the reliability and validity of the collected data. Those experts were the research supervisor and the examiners who examined this research on the thesis defense.

(54)

7. Analyzing the Data

The researcher used counting technique in analyzing the data. It was supported by Fraenkel and Wallen (2006, p. 490) that counting is a characteristic of some content analysis methods. Moreover, the researcher counted the frequency of several aspects which would support the topic she investigated. The researcher counted the frequency in the form of a percentage as well.

8. Reporting the Research Results

Since this research constituted a qualitative research, the researcher reported the research results in terms of descriptions. It was supported by Merriam (2009) that qualitative research focuses on the meaning so it would be dealing with interpretation. Additionally, the researcher used numbers in reporting the data since this research used a content analysis method. As Fraenkel and Wallen (2006, p. 490) state, the end product of the coding process must be numbers. Thus, the researcher used both description and numbers in reporting the research results.

(55)

37 CHAPTER IV

RESEARCH RESULTS AND DISCUSSION

This chapter presents research results which had been analyzed in ways justified in the research methodology. In addition, it discussed the results in light of the research questions. The first research question is about how the three consecutive years of Junior High School English UN is best described. Therefore, the first point of this chapter fully describes how the three consecutive years of Junior High School English UN looks like. The second research question is about the levels of thinking skills employed in the three consecutive years of Junior High School English UN for Junior High School are. The second point of this research obtains information about how high the three consecutive years of Junior High School English UN employs levels of thinking skills. Thus, the research results and discussion are presented in the two main points as follows.

A. The Description of the Three Consecutive Years of Junior High School English UN

Regarding the first research question, this research was going to describe how the three consecutive years of Junior High School English UN. This part reported how the three consecutive years of Junior High School English UN looked like. Moreover, this part described a language skill and materials which are integrated in the three consecutive years of Junior High School English UN. For

(56)

that reason, this research used Brown’s theory of reading test to analyze the data. Here is the data presentation.

Based on the data analysis, the three consecutive years of Junior High School English UN tested reading skills. Brown (2004, p. 185) argues that reading is a process of conveying meanings from the written words in order to gain information. The data showed that the three consecutive years of Junior High School English UN consisted of reading passages which allowed the test takers to convey meanings in order to get information. Brown (2004, p. 3) also argues that

English language tests measure test takers’ abilities to perform English language skills, which are to read, listen, write or speak. To be more precise, the types of reading which are tested in the three consecutive years of Junior High School English UN presented as follows.

Table 4.1 The Data Presentation of Reading Test Items Based on Brown’s Typology of Reading Tests Brown’s Reading

Typology

English UN 2012

English UN 2013

English UN

2014 Frequency

Percentage (%)

Perceptive - - - - -

Selective 24 48 30 102 54.8%

Interactive 18 24 42 84 45.2%

Extensive - - - - -

Based on the data, the three consecutive years of Junior High School English UN tested two types of reading namely selective and interactive reading (Brown, 2004, p. 189). There were 54.8% of the questions which tested selective reading and 45.2% of the questions which tested interactive reading. In addition,

(57)

there were no questions which tested perceptive and extensive reading. To be more precise, the discussion is presented as follows.

1. Testing Selective Reading

The three consecutive years of Junior High School English UN tested reading skill through selective reading. It was proved by filling-gap texts which delivered in the three consecutive years of Junior High School English UN (Brown, p. 189). These texts constituted a selective reading because they assessed the test takers’ knowledge about English grammar and vocabulary items. It was supported by Brown (2004, p. 189) that selective reading focused on the recognition of lexicons and grammar (Brown, 2004, p. 189). Moreover, here is a table which represented the materials which were tested in the three consecutive years of Junior High School English UN through selective reading.

Table 4.2 The Materials which Were Tested in Filling-Gap Texts Materials

English Vocabulary items English tenses

Frequency 88 5

Percentage (%) 94.6% 5.4%

As selective reading focused on lexicons and grammar of a language (Brown, 2004, p.189), 94.6% of questions in filling-gap texts tested the test takers’ knowledge about English vocabulary items. For the rest, 5.4% of questions in filling-gap texts tested the test takers’ knowledge about English tenses. To go further to the discussion how filling-gap texts constituted a selective reading and how the reading tasks assessed the materials, here is an example of feeling-gap text.

(58)

43. … A. for B. from C. since D. during 44. … A. come B. came C. comes D. coming 45. … A. park B. store C. job D. room

I went to Germany with my family … (43) a school holiday a few years ago. It was getting dark, so we were looking for a hotel. Then we … (44) upon two hotels next

each other with … (45) opening. We just randomly chose one and settled of the night.

Figure 4.1 A Filling-Gap Text Taken from English UN 2012

Figure 4.1 corresponded to the two characteristics of selective reading. First, the task was constructed by using a brief paragraph as the stimulus (Brown, 2004, p. 189). The paragraph in Figure 4.1 was considered as a brief because it consisted of only 4 sentences. Moreover, the paragraph was considered as the stimulus because it gave instructions to the test takers about the tasks to do. In other words, it stimulated the test takers to be able to answer the following questions. Second, Figure 4.1 constituted a task which needed brief responses. It is supported by Brown (2004, p. 189) that brief responses were intended in selective reading. To answer the questions, the test takers only needed to choose one correct word from four choices. They did not need to write a phrase which

(59)

consisted of three words or even sentences. Thus, the filling-gap text represented two characteristics of selective reading as Brown (2004) states.

Moreover, Figure 4.1 represented the materials which were assessed in filling-gap texts. First, there were two questions which assessed English vocabulary items. They were questions number 43 and 45. Question number 43 assessed the use of preposition during since it could be used to show period of time (“During”). Furthermore, the preposition during in the reading passage was to show the period of time when the author stared her/his holiday until the end of her/his holiday. Next, question number 45 assessed the vocabulary item in the context. It assessed the meaning of an English word room since the correct answer was a noun which represented something opened in the hotels. Room means a part of the inside of a building that is separated from other parts by walls, floor and

ceiling (“Room”).

Second, question number 44 assessed the test takers’ knowledge about English tenses. It assessed the use of the simple past tense since the reading

passage indicated the author’s experience which happened in the past. As Azar states (1999, p. 27), the simple past tense expresses events which began and end in the past. Because it assessed the simple past tense, the test takers were expected to know the past form of the verb come which the answer was came (“Come”). The verb came is one of the examples of the irregular verb used in the simple past

tense (“Come”). As a result, filling-gap texts integrated the test takers’ knowledge about the forms of English language.

(1)

114 Q L (13), 6613 Detail information Reading comprehension

115 Q L (4), N14 Main idea Reading comprehension

116 Q L (5), N14 Detail information Reading comprehension

117 Q L (6), N14 Detail information Reading comprehension

118 Q L (7), N14 English grammatical

features

Reading comprehension

119 Q FG (40), N14 English vocabulary Focusing on English language forms 120 Q FG (41), N14 English vocabulary Focusing on English

language forms 121 Q FG (42), N14 English vocabulary Focusing on English

language forms

122 Q GC (2), I14 Inference Reading comprehension

121 Q GC (3), I14 English expressions Reading comprehension

122 Q L (4), T14 Main idea Reading comprehension

123 Q L (5), T14 Detail information Reading comprehension

(2)

125 Q L (7), T14 English grammatical

features

Reading comprehension

126 Q FG (40), T14 English vocabulary Focusing on English language forms 127 Q FG (41), T14 English vocabulary Focusing on English

language forms 128 Q FG (42), T14 English vocabulary Focusing on English

language forms

129 Q GC (2), D14 Inference Reading comprehension

130 Q GC (3), D14 English expressions Reading comprehension

131 Q GC (2), Q14 Inference Reading comprehension

132 Q GC (3), Q14 English expressions Reading comprehension

133 Q GC (2), S14 Inference Reading comprehension

134 Q GC (3), S14 English Expressions Reading comprehension

135 Q L (4), J14 Main idea Reading comprehension

136 Q L (5), J14 Detail information Reading comprehension

(3)

138 Q L (7), J14 English grammatical

features

Reading comprehension

139 Q FG (40), J14 English vocabulary Focusing on English language forms 140 Q FG (41), J14 English vocabulary Focusing on English

language forms 141 Q FG (42), J14 English vocabulary Focusing on English

language forms 142 Q FG (43), J14 English vocabulary Focusing on English

language forms 143 Q FG (44), J14 English vocabulary Focusing on English

language forms 144 Q FG (45), J14 English vocabulary Focusing on English

language forms

145 Q L (4), A14 Main idea Reading comprehension

146 Q L (5), A14 Detail information Reading comprehension

147 Q L (6), A14 Detail information Reading comprehension

148 Q L (7), A14 English grammatical

features

Reading comprehension

(4)

150 Q L (5), O14 Detail information Reading comprehension

151 Q L (6), O14 Detail information Reading comprehension

152 Q L (7), O14 English grammatical

features

Reading comprehension

153 Q FG (40), O14 English vocabulary Focusing on English language forms 154 Q FG (41), O14 English vocabulary Focusing on English

language forms 155 Q FG (42), O14 English vocabulary Focusing on English

language forms

156 Q GC (2), F14 Inference Reading comprehension

157 Q GC (3), F14 English expressions Reading comprehension

158 Q GC (2), M14 Inference Reading comprehension

159 Q GC (3), M14 English expressions Reading comprehension

160 Q L (4), M14 Main idea Reading comprehension

161 Q L (5), M14 Detail information Reading comprehension

(5)

163 Q L (7), M14 English grammatical

features

Reading comprehension

164 Q FG (40), M14 English vocabulary Focusing on English language forms 165 Q FG (41), M14 English vocabulary Focusing on English

language forms 166 Q FG (42), M14 English vocabulary Focusing on English

language forms

167 Q L (4), K14 Main idea Reading comprehension

168 Q L (5), K14 Detail information Reading comprehension

169 Q L (6), K14 Detail information Reading comprehension

170 Q L (7), K14 English grammatical

features

Reading comprehension

171 Q FG (40), F14 English vocabulary Focusing on English language forms 172 Q FG (41), F14 English vocabulary Focusing on English

language forms 172 Q FG (42), F14 English vocabulary Focusing on English

language forms

(6)

174 Q GC (2), K14 English expressions Reading comprehension

175 Q FG (40), P14 English vocabulary Focusing on English language forms 176 Q FG (41), P14 English vocabulary Focusing on English

language forms 177 Q FG (42), P14 English vocabulary Focusing on English

language forms 178 Q FG (40), E14 English vocabulary Focusing on English

language forms 179 Q FG (41), E14 English vocabulary Focusing on English

language forms 180 Q FG (42), E14 English vocabulary Focusing on English

language forms 181 Q FG (40), G14 English vocabulary Focusing on English

language forms 182 Q FG (41), G14 English vocabulary Focusing on English

language forms 183 Q FG (42), G14 English vocabulary Focusing on English

An analysis of Junior High School English un test items administered in 2012, 2013 and 2014.

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

ANANALYSNoFJTINIORHIGHSCHooLEI\GLISHUN

TEST

ITEMS ADMINISTERED IN

ll2l4l

AN ANALYSTS

JUNIOR HIGTI

SCHOOL ENGLTSH

TEST

ITEMS ADMINISTERED IN

AND

09rvtzttr

W

ulel

u€p

(SUSISINIHI(IY

ISf,I

'IOOH]S

\}'

OgIVIZIII:

NYCNIINf,df,X XNINN

Create

Evaluate

Analyze

Apply

Understand

Remember

Parts

Dokumen yang terkait

An Error Analysis In Using Tenses Made By The Third Year Students Of SMK 7 Medan

An Error Analysis In Using Tenses Made By The Third Year Students Of SMK 7 Medan

An Analysis On Primary School Students’ Ability To Use Personal Pronouns. A Case Study On The Sixth Year Students Of Sdn No. 101878 Tg. Morawa.

An Analysis On High School Students’ Ability To Master Passive Voice A Study Case : The Second Year Students At SMK Negeri 2 Pematangsiantar

An Analysis On The Content Validity Of English Summative Test Items At The Even Semester Of The Second Grade Of Junior High School

An analysis on the content validity of english summative test items at the even semester of the second grade of Junior High School

The effectiveness of using jigsaw technique in teaching speaking

A Content Analysis of English Textbook "When English Rings a Bell" Used in First Grade Junior High School Base on Curriculum 2013

AN ANALYSIS OF LESSON PLAN DESIGNED BY JUNIOR HIGH SCHOOL ENGLISH TEACHERS IN BOYOLALI An Analysis Of Lesson Plan Designed By Junior High School English Teachers In Boyolali.

AN ANALYSIS OF LESSON PLAN DESIGNED BY JUNIOR HIGH SCHOOL ENGLISH TEACHERS IN BOYOLALI An Analysis Of Lesson Plan Designed By Junior High School English Teachers In Boyolali.

Dokumen yang Anda mencari sudah siap untuk unduhkan