A study on content validity of the assessments prepared by practice teaching students.

i

A STUDY ON CONTENT VALIDITY OF THE ASSESSMENTS

PREPARED BY PRACTICE TEACHING STUDENTS

A SARJANA PENDIDIKAN THESIS

Presented as Partial Fulfillment of the Requirements to Obtain the Sarjana Pendidikan Degree

in English Language Education

By

Matius Teguh Prasetyo Student Number: 061214104

ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION FACULTY OF TEACHERS TRAINING AND EDUCATION

SANATA DHARMA UNIVERSITY YOGYAKARTA

ii

A Sarjana Pendidikan Thesis on

A STUDY ON CONTENT VALIDITY OF THE ASSESSMENTS

PREPARED BY PRACTICE TEACHING STUDENTS

By

Matius Teguh Prasetyo Student Number: 061214104

Approved by

Advisor

Date

iii A Thesis on

A STUDY ON CONTENT VALIDITY OF THE ASSESSMENTS PREPARED BY PRACTICE TEACHING STUDENTS

By

Matius Teguh Prasetyo Student Number: 061214104

Defended before the Board of Examiners on February 1st, 2012

and Declared Acceptable

Board of Examiners

Chairperson : Caecilia Tutyandari, S.Pd., M.Pd. __________________ Secretary : Drs. Barli Bram, M.Ed. __________________ Member : Made Frida Yulia, S.Pd., M.Pd. __________________ Member : Agustinus Hardi Prasetyo, S.Pd., M.A. __________________ Member : Caecilia Tutyandari, S.Pd., M.Pd. __________________

Yogyakarta, Faculty of Teachers Training and Education

Sanata Dharma University

Dean

iv

STATEMENT OF WORK’S ORIGINALITY

I honestly declare that this thesis, which I have written, does not contain the work or parts of the work of other people, except those cited in the quotations and the references, as a scientific paper should.

Yogyakarta, February 1st, 2012

The Writer

Matius Teguh Prasetyo 061214104

v

DEDICATION PAGE

This thesis is dedicated to Bapak, Ibu, Tyas, Andre, Sr. Natalia, Op., the big Family of Bu Irni, Oyo, and the future before me.

vi

LEMBAR PERNYATAAN PERSETUJUAN

PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS

Yang bertanda tangan dibawah ini, saya mahasiswa Universitas Sanata Dharma: Nama : Matius Teguh Prasetyo

Nomor Mahasiswa : 061214104

Demi pengembangan ilmu pengetahuan, saya memberikan kepada perpustakaan Universitas Sanata Dharma karya ilmiah saya yang berjudul:

A Study on Content Validity of the Assessments Prepared by Practice Teaching Students

beserta perangkat yang diperlukan (bila ada). Dengan demikian saya memberikan kepada perpustakaan Universitas Sanata Dharma hak untuk menyimpan, mengalihkan dalam bentuk media lain, mengelolanya dalam bentuk pangkalan data, mendistribusikannya secara terbatas, dan mempublikasikannya di internet atau media lain untuk kepentingan akademis tanpa perlu meminta ijin dari saya maupun memberikan royalti kepada saya selama tetap mencantumkan nama saya sebagai penulis.

Demikian pernyataan ini saya buat dengan sebenarnya

Dibuat di Yogyakarta

Pada tanggal : 20 Januari 2012

Yang menyatakan,

vii

ABSTRACT

Prasetyo, Matius Teguh. (2012). A Study on Content Validity of the Assessment Prepared by Practice Teaching Students. Yogyakarta: Sanata Dharma University.

An assessment in an instruction is purposed to obtain the information about the students’ progress and mastery during or at the end of the instruction. The information should be valid and should represent the true condition of the students. Therefore, the assessment applied by the teachers should be valid and able to represent the intended outcomes the students should perform or master.

Based on the concept above, the writer was interested to conduct a study about the validity of the assessment, especially the content validity. The writer formulated two research questions, namely 1) how is the content validity established by Practice Teaching students in their assessments? and 2) what are the problems which influence the degree of content validity of the Practice Teaching Students’ assessments?

In order to answer those questions, the writer conducted a qualitative study, namely document analysis. The documents were the lesson plans along with some assessments prepared for the lesson plan by the Practice Teaching students. The document analysis aimed at showing how the Practice Teaching students established content validity in their assessments, the process was called content validation. The respondents were five randomly chosen Practice Teaching students of ELESP Sanata Dharma University in academic year of 2010/2011. The analysis was done by comparing the instructional content which was specified to some specific objectives with the intended result or the content of assessments. The analysis was also done to obtain the information about the problems which influence the content validity degree of the respondents’ assessments.

From the analysis, the respondents established some assessments which were valid based on the principle of content validity of assessment but they differed in their degree of content validity. The findings showed that the respondents established high, moderate, and low content validity. The problems which influenced the content validity degree of the respondents’ assessments were 1) the objective formulations were not clear and ambiguous, 2), there were too many objectives in an instruction for a single content, 3) the assessments did not measure all the objectives stated, 4) the assessments’ tasks or procedures did not represent the intended skills, and 5) the intended result of the assessments had little correlation with the content of the instruction. In order to overcome the problems, the writer proposed some possible recommendations, namely 1) producing well planned lesson plan and assessments, 2) stating clear and unambiguous content and objectives, 3) not formulating too many objectives for a lesson, 4) considering the four basic English skills in the assessments, and 5) improving and updating the knowledge about assessment by reading books or any other sources.

viii

ABSTRAK

Prasetyo, Matius Teguh. (2012). A Study on Content Validity of the Assessment Prepared by Practice Teaching Students. Yogyakarta: Universitas Sanata Dharma.

Sebuah asesmen dalam proses pembelajaran dimaksudkan untuk mendapatkan informasi tentang kemajuan dan pemahaman siswa selama atau pada akhir proses pembelajaran. Informasi yang didapat dari asesmen tersebut haruslah valid dan dapat mewakili kondisi nyata siswa. Oleh karena itu, asesmen yang diaplikasikan oleh guru harus valid dan dapat merepresentasikan hasil yang seharusnya dicapai atau ditunjukkan oleh siswa.

Berdasarkan konsep di atas, penulis tertarik untuk melaksanakan sebuah penelitian tentang validitas dari sebuah asesmen, terutama pada validitas isi. Penulis merumuskan dua pertanyaan dalam penelitian ini, yaitu 1) bagaimana validitas isi ditampilkan oleh para mahasiswa yang sedang menjalani Praktik Pengajaran Lapangan (PPL) dalam asesmen mereka? dan 2) apakah masalah yang mempengaruhi tingkat validitas isi dari asesmen para mahasiswa PPL tersebut?

Untuk menjawab pertanyaan-pertanyaan tersebut, penulis melaksanakan sebuah penelitian kualitatif, yaitu analisis dokumen. Dokumen yang dimaksud adalah beberapa Rencana Pelaksanaan Pembelajaran (RPP) bersama dengan asesmen yang disiapkan untuk RPP tersebut yang dibuat oeh mahasiswa PPL. Analisis dokumen dimaksudkan untuk menunjukkan bagaimana para mahasiswa PPL menampilkan validitas isi dalam asesmen mereka, proses ini disebut dengan validasi isi. Responden dalam penelitian ini adalah lima orang mahasiswa PPL Universitas Sanata Dharma yang dipilih secara acak pada tahun ajaran 2010/2011. Analisis dilakukan dengan cara membandingkan isi pembelajaran yang dibagi dalam beberapa tujuan pembelajaran dengan hasil yang diharapkan atau isi dari asesmen. Analisis juga dilakukan untuk mengetahui tentang masalah-masalah yang mempengaruhi tingkat validitas isi dari asesmen para responden.

Dari analisis yang dilakukan, para responden membuat beberapa asesmen yang valid berdasarkan prinsip validitas isi dari sebuah asesmen tetapi mereka berbeda dalam tingkatannya. Hasil penelitian menunjukkan bahwa para responden menampilkan asesmen yang mempunyai validitas isi tinggi, sedang, dan rendah. Masalah-masalah yang mempengaruhi tingkat validitas isi dari asesmen para responden adalah 1) formulasi tujuan pembelajaran tidak jelas dan ambigu, 2) ada terlalu banyak tujuan pembelajaran untuk sebuah isi dalam sebuah proses pembelajaran, 3) asesmen yang dibuat tidak mengukur semua tujuan pembelajaran, 4) prosedur dari asesmen tidak mewakili keterampilan yang diharapkan, 5) isi dari asesmen hanya memiliki sedikit korelasi dengan isi dari pembelajaran. Untuk mengatasi masalah tersebut, penulis menawarkan beberapa solusi, yaitu 1) membuat RPP dan asesmen yang terencana dengan baik, 2) menuliskan tujuan pembelajaran yang jelas dan tidak ambigu, 3) tidak menyebutkan terlalu banyak tujuan pembelajaran dalam satu proses pembelajaran, 4) menyertakan empat keterampilan dasar Bahasa Inggris, dan 5) meningkatkan

ix

dan selalu memperbaharui pengetahuan tentang asesmen melaui buku atau sumber-sumber yang lain.

x

ACKNOWLEDGEMENTS

My biggest gratefulness and never-ending gratitude go to my faithful companions, Lord Jesus Christ and Mother Mary, for endowing me with splendid blessings and love.

I would like to express my deepest and sincere appreciation to my sponsor, Made Frida Yulia, S.Pd., M.Pd. Her enduring guidance and valuable suggestions have given influential contributions to this thesis. I am greatly indebted to the lecturers of Language Learning Assessment subject, Agustinus Hardi Prasetyo, S.Pd., M.A., and Veronica Triprihatmini, S.Pd., M.Hum., M.A., who have helped me evaluate my analysis and to all lecturers who have shared their knowledge and advice which were beneficial for me in finishing this thesis. I would like to thank Jatmiko Yuwono, S.Pd, who has read and evaluated my thesis during his busy time.

Sincere thanks are also expressed to my respondents, five Practice Teaching students of 2010/2011 academic year, who have given me permission to copy and analyze the Lesson Plan and Assessments they made during their Practice Teaching period.

Profound thankfulness is addressed to my beloved parents, Bapak Florentinus Heriana Sukardi and Ibu Yohana Elisa Suryanti Purwaningsih, for their magnificent love, support, and care; to my sister, Bernadeta Tyas Puji Utami, whose hard work always seems admirable to me; to my little brother, Andreas Sulistyo Nugroho, for always inspire me to be a good example for him; and to all families for their kindness, support, and prayers. My deepest gratitude goes

xi

toward Sr. Natalia, OP and the big family of Pak Anwar and Bu Irni; without their help in affording my study, I would never have the chance to study in university. My special indebtedness goes to the big family of F. Aldhika Deinza Saputra, who have given me a place to stay and finish my thesis.

I would like to express my thankfulness to all teachers in SMA and SMK Dominikus Wonosari and SD N Sedono for the beautiful moment and experiences I could get during my work there. The knowledge I get from my study is nothing without some places to apply. I know more about being a teacher from my experiences there. I would like to thank them for their encouragement and motivation for me to finish my thesis.

I would also like to thank Vincensia Retno Woro Purwaningsih for being my number-one supporter. Her love, patience, support and prayers have poured inspirations upon my days. At last, I would like to thank all friends and people whose names cannot be mentioned one by one. I thank them all for lending me a hand in finishing this thesis.

xii

TABLE OF CONTENTS

TITLE PAGE…… ... i

APPROVAL PAGES ... ii

STATEMENT OF WORK’S ORIGINALITY ... iv

DEDICATION PAGE ... v

LEMBAR PERNYATAAN PERSETUJUAN PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS ... vi

ABSTRACT …….. ... vii

ABSTRAK ... viii

ACKNOWLEDGEMENTS ... x

TABLE OF CONTENTS ... xi

LIST OF TABLE ... xv

LIST OF FIGURES ... xvi

LIST OF APPENDICES ... xvii

CHAPTER I. INTRODUCTION A. Research Background ... 1

B. Problem Formulation ... 5

C. Problem Limitation ... 5

D. Research Objectives ... 7

E. Research Benefits... 8

F. Definition of Terms... 10

CHAPTER II. REVIEW OF RELATED LITERATURE A. Theoretical Description ... 12

1. Assessment ... 12

a. The Definition of Assessment ... 12

xiii

2. Methods of Assessments ... 19

3. Principles of Language Assessment ... 20

a. Practicality ... 20

b. Reliability ... 21

c. Validity ... 21

d. Authenticity ... 26

e. Washback ... 26

4. Content Validation ... 26

a. Definition and Process ... 27

b. Instructional Goals and Objectives ... 28

B. Theoretical Framework ... 31

CHAPTER III. METHODOLOGY ... 34

A. Research Method ... 34

B. Research Participants ... 35

C. Research Instruments ... 36

1. The Researcher as the Research Instrument ... 36

2. Documents ... 36

D. Data Gathering Technique ... 37

E. Data Analysis Technique ... 37

F. Research Procedure ... 39

CHAPTER IV. RESEARCH RESULTS AND DISCUSION ... 42

A. Establishing Content Validity in Assessments ... 42

1. Determining the Instructional Content ... 44

a. Text-based Content ... 47

b. Expression-based Content ... 48

c. Tense-based Content ... 49

2. Content Validity of the Respondents’ Documents ... 50

a. High Content Validity Assessment ... 50

xiv

c. Low Content Validity Assessment ... 58

B. Problems in Producing Content Valid Assessment ... 64

1. Problems Influencing the Content Validity Degree of the Assessment ... 65

2. Possible Recommendation to Overcome the Problems ... 68

CHAPTER V. CONCLUSIONS AND SUGGESTIONS ... 71

A. Conclusions ... 71

B. Suggestions ... 74

REFERENCES ... 77

xv

LIST OF TABLE

Table Page

xvi

LIST OF FIGURES

Figure Page

Figure 2.1 Test, Assessment, and Teaching ... 15 Figure 2.2 The Purpose of Instructional Goals and Objectives ... 30

xvii

LIST OF APPENDICES

Appendix A

The Sample of Respondents’ Documents ... 80 Appendix B

The Complete Analysis of the Content Validity of the Assessments

Prepared by Practice Teaching Students ... 105 Appendix C

H.D. Brown’s Lists of Microskills and Macroskills of the Four Basic

Language Skills ... 135 Appendix D

1

CHAPTER I

INTRODUCTION

This chapter consists of research background, problem formulation, problem limitation, research objectives, research benefits, and definition of terms.

A. Research Background

Assessment is a part of teaching and learning activity. Teaching and learning activity includes not only lecturing, discussing, or explaining, but also it includes the ability to assess students’ progress and achievement in learning activity. According to Wiggins (1998), the main purpose of assessment is to “educate and improve the students’ performance, not merely to audit it.” Therefore, the assessment is prepared to help the teachers educate and improve students’ performance (p. 7). The notion is supported by Siddiek (2010) that “assessment is the responsibility of teachers and administrators not as mere routine of giving marks, but making real evaluation of learners’ achievements (p. 133).” Cliff (1981, p. 17) as cited by Siddiek (2010) also believed that assessment must be used to help students. The assessment should be operated in two ways, namely first, the teachers know that the assessment will let the students know when the students will study, what they will study, and the effort they put into their work, second, the students can correct or improve their performance based on the result of the assessment given to them.

Assessment is also used as an aid for the teachers to evaluate their progress in teaching and learning activity. The assessment result can show how far the students can understand the material given. The result becomes the indicator about the success or failure of the students and it helps teachers revise or improve the next teaching and learning activity.

Nevertheless, producing valid and reliable assessment which can serve those purposes is not easy. Teachers need to have a good ability in assessing students’ performance by making good assessment tools based on the principles of good assessment. Since the assessment is used to obtain the accurate and valid information of students’ proficiency in particular topic or material, the assessment has to be valid and accurate, too.

Producing valid assessment requires the teachers to pay attention to so many aspects of the assessment. Accuracy and validity are only two aspects from a lot of aspects of the assessment but they become the basic of how a sound assessment should be, since one of the purposes of the assessment is to obtain accurate and valid information about the students’ progress and mastery. Teachers have to plan and prepare their assessments carefully to make their assessment valid and accurate, yet there are some evident that teachers took very little time in preparing their test (Siddiek, 2010). Therefore, the teachers were not able to make simple analysis to provide reports of their students’ achievement and they were not able to interpret the meaning of figures of the test scores because they lacked the basic statistical knowledge. It can be concluded that valid assessment requires

the teachers or administrators ability in carefully planning and giving sufficient time in the process of its production.

The students of English Language Education Study Program (ELESP) Sanata Dharma University are prospective teachers who are prepared to be good quality teachers. They learn about English and how to teach English in their study in university. They are expected to be able to master all teaching skill, which include the mastery of assessing, the knowledge of how to make good assessment, the ability to apply the principles in making assessment as an integral part of teaching and learning activity, and many more. Since producing good assessment is not an easy work and it needs time in learning and producing, they learn from the concept of assessment to the assessment production. Producing good assessment will support their teaching later.

The students of ELESP learn about the principles of assessment especially in Language Learning Assessment (LLA) subject. The assessment they produce is in connection with language learning since they will teach English language. In LLA, they learn the basic knowledge about assessment and its principles. The students also learn the great importance of assessment in LLA. Here, they have the opportunity to practice producing language assessment, too. Further, they have the opportunity to practice their teaching knowledge in a real school when they take Practice Teaching subject. In this period, the students achieve great opportunities to learn about being a real teacher, in a real school. They obtain the chance to perform their teaching knowledge for real students. They have the chance to learn from practicing in real condition which requires

their all understanding about teaching, classroom management, assessment application, and many more. At this point, they practice to produce sound assessments, which appropriately apply the principles of assessment. Nevertheless, since they are learning, some of the principles of assessment might be ignored or might not be appropriately obeyed.

Based on the description above, the writer is interested in conducting a research about one of the principles of assessment, namely the assessment validity. The writer is eager to conduct a research on the validity of the assessment made by the students of English Language Education Study Program (ELESP) Sanata Dharma University, especially the students who had taken Practice Teaching Subject. The writer chose Practice Teaching students as the sample of the research because, as stated above, they are supposed to master teaching and learning skills, which also include the competence in producing good assessment. Before taking the Practice Teaching subject, the students have to pass the previous subjects learning about the theory of English Language and English Language teaching. After three or four semesters obtaining the theory of English language, the students then are prepared to teach English starting from understanding about teacher and teaching, preparing the instructions, learning about approach, method, and technique in teaching, making assessment, and many more.

The study focuses its attention on the validity of assessment, specifically the content-related evidence or content validity. It has been stated above that the purpose of assessment is to educate and improve the students’ performance by obtaining the information of the students’ proficiency in particular topic, skill, or

material through the interpretation of the assessment result. Therefore, the assessment tools and procedures have to be valid. The validation aimed at obtaining valid information about the students’ performance in particular topic, skill, or material. In other words, if teachers want to achieve valid information of the students’ performance or progress, the assessment they made should be valid, too.

B. Problem Formulation

Through this study, the writer formulates the problems which are presented into two questions.

1. How is the content validity established by the Practice Teaching students in their assessment?

2. What are the problems which influence the degree of content validity of the Practice Teaching students’ assessment?

C. Problem Limitation

The writer limits the problems by focusing on the content validity of the assessment produced by Practice Teaching students of the English Language Education Study Program Sanata Dharma University during their Practice Teaching period. The reason why the writer chose the Practice Teaching students as the respondents of this research is because those students have passed Language Learning Assessment subject in which they have learnt about the concept of assessment and the production of assessment. It is expected that they

are able to produce good and valid assessment. Another reason is that the students have to make assessment for the real students during their Practice Teaching period. The LLA students are not chosen because the assessment they make is not purposed for the real students and they are still practicing. Although in Practice Teaching period the students are also still practicing, they practice for the real students. Therefore, the assessment they produce has to be valid.

The writer limits the validity by focusing on the content validity of assessment. Although there are more than three evidences in validating an assessment, the writer focuses only on content-related evidence or content validity because this evidence is considered the most important evidence in validating an assessment. It is important for an assessment to be valid in its content because the content is the aspect which becomes the main base for teaching in particular topic. Teachers teach materials and behaviors or skills, called as content or the intended skills or performances that the students should know or master. Therefore, the assessment should also assess the content by applying appropriate test specifications or items to be considered valid based on the principles of content validity of an assessment. An example will be good to give more understanding why the writer chooses content validity as the subject of the research. Imagine that there is a teacher who states in his lesson plan that the objectives of the lesson is that at the end of the course, the students are able to produce good and correct recount text using appropriate verb in past form. Nevertheless, in his assessment, he gives the students a recount text completed with ten comprehensive questions based on the text. The assessment has low content validity based on the objectives

stated before because it does not measure what it intends to measure, which is good recount text production. Therefore, content validity in an assessment is very important and needs more attention from those who produce the assessment.

The assessment of this study is the assessment made by the Practice Teaching students during their practice teaching period. The assessment chosen is the assessment which has been prepared before they conduct their teaching and learning activity in class. The spontaneous assessment is not included. It means that the assessment chosen is planned and prepared in written form so the students can read the instructions and do what they are supposed to do in that assessment.

D. Research Objectives

Dealing with the questions mentioned, the study is conducted to achieve two objectives, namely:

1. Figuring out how the Practice Teaching students established the content validity in their assessment. It is expected that the analysis can present the way of the students in preparing and producing the assessment so that it can be considered valid based on the concept of content validity.

2. Figuring out the problems which influence the content validity degree of Practice Teaching students’ assessments. It is done in order to find possible recommendations to overcome the problems.

E. Research Benefits

It is expected that this study will be beneficial for the students and lecturers who are involved in ELESP, and future researchers who want to study about assessment.

For students of ELESP, especially the students who are taking Practice Teaching and preparing sets of instruction in their learning, this study is expected to help them analyze their own assessment based on the theory of content validity. It is stated that it is only one source of making good assessment because there are other criteria to produce good assessment. Yet, validity, especially content validity, takes a really important role in an assessment. By understanding the concept of valid assessment, the students are expected to be more prominent in producing an assessment in their practice teaching period as well as in their real teaching later. It is also expected that the students will always keep in mind that their assessment must be valid in order to obtain valid measurement and result from their students. Valid results will serve as a very good aid for the teachers to improve their teaching.

It is expected also that after reading the result of this study, the students could be more aware of the importance of careful and appropriate planning for their instruction before they conduct teaching and learning activities in the classroom. Formulating appropriate instructional content and its specification in the appropriate objectives formulation will help them conduct teaching and also assessing. Since the study is limited in its content, the students are expected to more actively improve and develop their understanding and critical thinking

toward assessment as an integral part of teaching. It is expected that this study could help the students set their thinking that an assessment is not only used for scoring but also it is used for wider purpose of educating the students and develop the teachers.

For lecturers of ELESP, especially who are involved in teaching evaluation and assessment, this study is expected to serve as a means to obtain depiction of the students’ mastery in producing language assessment. Therefore, the lecturers will be able to plan possible instructions which will enable the students to have a sound assessment which involved some principles of good assessment production, especially the content validity. The lecturers could help their students produce assessment which possesses high validity since the assessment is used to obtain valid information about the students’ progress and mastery on a specific material or content domain.

For future researchers, this study is expected to be a good source for them who want to learn more about the assessment especially in its content validity. Language assessment, like other parts of teaching and learning, offers the really wide area to be explored. A thousand studies will not be enough to explore this wide area because of the time changing which bring the changes in every aspect in human life. It is expected that there always be the next researchers who will explore the world of assessment.

F. Definition of Terms

There are some terms mentioned in this study that need to be defined in order to avoid misunderstanding and to lead the readers to a better understanding on the topic being discussed.

1. Assessment

As stated before, the assessment is an integral part of teaching and learning activity. Assessment can be defined as one or more activities aimed at obtaining the information about the students’ progress or students’ achievement in learning activities. It can be done in several ways namely giving quizzes and tests, paying attention to the students’ performance in the classroom, listening to the student response in teaching and learning activities, and many more (Badder, 2000; Chatterji, 2003; and Brown, 2004).

In this study, the assessments were prepared by the Practice Teaching students during their Practice Teaching period. In conducting Practice Teaching, the students had to prepare and design the instructional process. The instruction involved the process of planning the lesson and assessing the students’ progress. In this study, the assessment analyzed were any various activities conducted by the Practice Teaching students to obtain the information about the students’ performance and progress during the instructional process in their Practice Teaching period. Specifically, the assessments were any assessment documents prepared by the Practice Teaching students purposed to assess the students’ performance and progress during or at the end of the instructional process.

2. Validity

Validity in this study directly points to the validity of the assessment. In an assessment, validity is one of the criteria for a good test. Brown (2001) stated in his book that validity is “the degree to which the test actually measures what it is intended to measure (p. 378).” Gronlund (1998, p. 226) as cited by Brown (2004) stated that validity is “the extent to which inferences made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assessment (p. 22).” This study grounds the validity of assessment to the definitions above. The assessment is valid if the results are appropriate, meaningful, and useful based on the purpose of the assessment or based on what is intended to be measured by the assessment.

In this study, the writer focused on the content validity of assessment. Content validity is one of the assessment principles in which the intended results or the content of an assessment represent what is intended to measure or the instructional content. The content validity of an assessment is presented in the form of degree, namely high, moderate, and poor content validity. An assessment is said having high content validity if the intended result from the assessment can clearly represent what the content of the task or material learnt or what the students should master from teaching and learning activities. In the other hand, if the content of the assessment cannot clearly represent the content of the study, the assessment has poor content validity. The conclusion of the validity is derived from the analysis by comparing the content or the intended result of the assessment with the instructional content.

12

CHAPTER II

REVIEW OF RELATED LITERATURE

In this chapter, the writer discusses the related literature which serves as the basis to answer the research questions. There are two major parts in this chapter, namely theoretical description and theoretical framework.

A. Theoretical Description

In this part, the writer provides some descriptions about the theories used to analyze the content validity of Practice Teaching Students’ assessments. The theories become the key for the writer to be able to analyze the assessments.

1. Assessment

Before going deeper into the discussion of assessment and the principles of language assessment, it is better for the writer to provide a clear definition about the assessment.

a. The Definition of Assessment

Wiggins (1993), as cited by Earl (2003), stated that the word “assessment” is derived from the Latin assidere, which means “to sit beside or with.” This statement can be understood that the teachers sit together with their students to understand what is happening to them. Yet, assessment actually offers more activities than just sitting down with the students. Assessment can be done in various activities. Badder (2000) stated that assessment is a continuous process which involves examining and observing the students’ behaviors, listening to their

ideas, and promoting conceptual understanding by developing questions. Assessment is aimed at receiving information about the students’ progress and the instructional process. Airasian (1991) stated that assessment is a process of gathering, interpreting, and synthesizing information purposed to aid decision making in the classroom. Teachers collect information through assessment to help them make decisions about their students’ learning and the success of their instruction. Again, the information can be concluded from several assessment procedures or methods.

Dealing with the variety of assessment procedures or methods, Brown (2004) stated the definition of assessment in comparison with the term test. Test, in simple terms, is a “method of measuring a person’s ability, knowledge, or performance in a given domain,” while an assessment is defined as an ongoing process that encompasses a much wider domain (p. 3-4). The notion implies that when the students answer the questions from the teacher spontaneously, offer a comment, or try to give opinion, the teacher subconsciously makes an assessment of the students’ performance. The Glossary of the 1999 Standards for Educational and Psychological Testing, as cited by Chatterji (2003, p. 130), stated that assessment is “any systematic method of obtaining information (from test and other sources used) to draw inferences about characteristics of people, objects, or programs.” Miller, Linn, and Norman (2009) and Salvia, Ysseldyke, and Bolt (2001) added that assessment is a systematic process that plays important role in effective teaching. It begins with the identification of learning goals, monitor the progress students make toward those goals, and ends with a judgment concerning

the extent to which those goals have been attained (Miller et al., 2009). Teachers use assessment information to make decision about what the students have learned what and where they should be taught, and the kinds of related services (Salvia et al., 2001).

Assessment, therefore, can be defined as some processes, activities, or systematic methods which are done to obtain information about the students’ progress or performance. The process, activities, or methods are various. Teachers can choose one or more activities to assess the students. Those activities are conducted to obtain information about the students’ progress during teaching and learning process, the students’ mastery of the teaching and learning materials, and the instructional process. It is also important to mention that, according to Siddiek (2010, p. 135), assessment is “an integral part of any effective teaching program, thus it should be subject to planning, designing, modifying, and frequent revision of its validity as tools of quality measurement.” The statement makes the assessment clearer that as a means to obtain valid information about the students and instructional progress, an assessment should be validly planned, designed, modified, and frequently revised.

The most common activity of an assessment is a test. Test is a part of assessment. It means that it is not the only form of assessment made by the teacher. Tests can be useful devices, but they are only one among many procedures and tasks that teachers can ultimately use to assess students (Brown, 2004, p. 4). Having the same idea that test is only one of several assessment procedures, Salvia et al. (2001, p. 13) stated that “during the assessment, data

from observations, recollections, test, and professional judgments all come into play.” It can be concluded that the assessment information can be obtained not only from the test but also from other procedures, namely observation, asking spontaneous questions, listening the students’ responses, and many more. Nevertheless, it should be noticed that test is often the most visible evidence of student learning for parents to see and make judgment about (Walker et al., 2004).

The position of a test in an assessment as a part of teaching activities is illustrated in Figure 2.1.

Figure 2.1 shows that test is a part of assessment while an assessment is a part of teaching. Therefore, test is not the same as assessment but it is only one of the assessment procedures or tasks. Meanwhile, an assessment is under the wide area of teaching. It means that teaching involves assessing. Assessment becomes a very important and crucial part in teaching since it helps teachers understand the progress or achievement the students have made and it can also be a useful device to monitor or evaluate the instruction prepared by the teachers, and many more.

TEST

ASSESSMENT TEACHING

Figure 2.1

Test, Assessment, and Teaching (Brown, 2004, p. 5)

b. Types of Assessment

There are some types of assessment. Brown (2004, p. 5-7) divided the assessment into three divisions namely the informal and formal assessment, formative and summative assessment, and norm-referenced and criterion-referenced tests.

1) Informal and Formal Assessment

The first distinction of the types of assessment is informal and formal assessment. Informal assessment can be done in many forms, starting with incidental, unplanned comments and responses, along with coaching and other impromptu feedback to students (Brown, 2004, p. 5). In contrast, he stated that formal assessment is exercises or procedures specifically designed to tap into a storehouse of skills and knowledge. They are systematic, planned sampling techniques constructed to give teacher and students an appraisal of student achievement. In addition, McAlpine (2002, p. 7) stated that “formal assessments are where the students are aware that the task that they are doing is for assessment purposes.” Tests are the example of formal assessment but not all formal assessments are in the form of tests.

2) Formative and Summative Assessment

Centre for Learning and Professional Development (CLPD) of The University of Adelaide (2011) stated that formative assessment is where the assessment task provides possible development activities they require to improve

their level of understanding on their learning in the current course. Formative assessment is used to evaluate students in the process of “forming” their competencies and skills with the goal of helping them to continue that growth process. Practically, the informal assessments are (or should be) formative (Brown, 2004, p. 6).

Summative assessment aims to measure, or summarize, what a student has grasped, and typically occurs at the end of a course or unit of instruction. CLPD of The University of Adelaide (2011) stated that summative assessment is “where assessment task responses are designed to grade and judge a learner's level of understanding and skill development for progression or certification.” Final exam in a course and general proficiency exam are examples of summative assessment.

3) Norm-Referenced and Criterion-Referenced Tests

Brown (2006) stated that norm-referenced test (NRT) is designed to assess global language abilities, namely academic listening ability, reading comprehension, and many more. It is aimed at spreading the students out along a continuum of general abilities or proficiency. Brown (2004, p. 7) stated that NRT test is the test “in which the test takers’ score is interpreted in relation to a mean (average score), median (middle score), standard deviation (extent of variance in scores), and/or percentile rank.” He also stated that the purpose of NRT is to place the test-taker along a mathematical continuum in rank order.

In contrast, as stated by Brown (2006), criterion-referenced test (CRT) is created to measure well-defined and fairly specific instructional objectives. Furthermore, he stated that the objectives are specific to particular course, program, school district, or state. In addition, Brown (2004, p. 7) stated that CRT is designed to give test-takers feedback, usually in the form of grades, on specific course or lesson objectives. Table 2.1 shows the differences between NRT and CRT.

Table 2.1 Norm-referenced and Criterion-referenced Differences

Characteristic Norm-Referenced Criterion-Referenced

Type of Interpretation

Relative (A student’s performance is compared to those of all other students in percentile terms.)

Absolute (A student’s performance is compared only to the amount, or percentage, of materials learned.)

Type of Measurement To measure general language abilities or proficiencies.

To measure specific objectives-based language points.

Purpose of Testing

Spread the students out along a continuum of general abilities or proficiencies.

Assess the amount of

material known or learned by each student.

Distribution of Score Normal distribution of scores around the mean.

Varies; often non-normal. Students who know the material should score 100%. Test Structure

A few relatively long subtest with a variety of item contents.

A series of short, well-defined subtests with similar items contents.

Knowledge of Question

Students have little or no idea of what content to expect in test item.

Students know exactly what content to expect in test items.

2. Methods of Assessments

According to Chatterji (2003), there are five different assessment methods that focus on the nature of the responses made by test-takers namely written assessment, behavior-based assessment, product-based assessment, interview-based assessment, and portfolio-based assessment.

a. Written Assessments

Written assessment includes all assessments to which test-takers respond using a paper and pencil format or, as is common today, on a computer using a word processor. The response of this assessment can be in the form of structured or open-ended response.

b. Behavior-Based Assessments

This method requires the students to perform behaviors or processes that must be observed directly. The special feature of assessment in this category is that actual behaviors, performances, and demonstrations must be assessed as they are occurring.

c. Product-based Assessments

In this method, the students have to create or construct a product. This product serves as the basis for measurement. The examples of product-based assessments are journals, terms papers, laboratory reports, books, or artwork.

d. Interview-based Assessments

In the interview-based assessment, the students will make spoken responses in an interview situation. The assessment is dependent on the oral responses of the respondents. It is usually open-ended. Interviews provide a means

for the respondents to explain responses given in written or other formats. In validity perspective, it is advantageous because reasoning and explanation skills are in the domain.

e. Portfolio-based Assessments

Portfolio-based assessment provides a comprehensive picture of proficiencies the students made. It is often promoted as the best means for comprehensive documentation of evolving skills and knowledge in a particular area. According to Arter and Spandel (1992, p. 36),

a portfolio is a purposeful collection of student work that tells the story of a student’s efforts, progress, or achievement in (a) given area(s). This collection must include student participation in selection of portfolio contents; the guidelines for selection; the criteria for judging merit; and evidence of student self-reflection.

3. Principles of Language Assessment

Brown (2004, p. 19) stated five principles of language assessment namely practicality, reliability, validity, authenticity, and washback. Since this study places its attention specifically on validity of assessment, the writer will explain the validity in bigger portion than other principles. The principles explained bellow are mainly taken from Brown’s book, Language Assessment: Principles and Classroom Practices (2004, p. 19-30).

a. Practicality

The first principle is practicality. An effective test is practical. This means that it is not excessively expensive, it stays within appropriate time

constraints, it is relatively easy to administer, and it has scoring/evaluation procedure that is specific and time-efficient.

b. Reliability

A reliable test is consistent and dependable. It means that the same test should yield similar result when it will be given to the same students or matched students on two different occasions.

c. Validity

The next principle is validity. This principle becomes the basis of this study. Therefore, as stated above, the writer will give bigger portion to explain about this assessment principle.

The most important quality to consider when producing or selecting an assessment procedure is validity, which refers to the meaningfulness and appropriateness of the uses and interpretations to be made of assessment result (Miller et al., 2009). Bachman (1990) also stated that “validity has been identified as the most important quality of test use, which concerns the extent to which meaningful inferences can be drawn from test scores” (as cited in Liao, 2004, p. 1). According to Gronlund (1998) validity is the most complex criterion of an effective test and arguably the most important principle (as cited in Brown, 2004, p. 22). Validity is defined as “the extent to which inferences made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assessment” (Brown, 2004, p. 22).

Brown (2006) stated that validity is important especially when it is involved in the decisions that teachers regularly make about their students. Further he stated that teachers want to make the decisions of their admission, placement, achievement, and diagnostic based on the test or assessment which actually testing what they claim to measure. Therefore, teachers should consider their assessment validity. The assessment will be considered valid if it measures the objective of the course (Siddiek, 2010) because “aimless is the most single cause of ineffectiveness in teaching and of frustration of education efforts” (Cliff, 1981, p. 27, as cited by Siddiek, 2010, p. 135).

“Validity is the quality of the interpretation of the result rather than of the assessment itself, that its presence is a matter of degree, that it is always specific to some particular interpretation or use, and that it is a unitary concept” (Miller et al., 2009, p. 102). What validated is not merely the assessment procedures but the appropriateness of the interpretation and use of the result of an assessment procedure. Validity is best considered in terms of categories that specify the degree, namely high validity, moderate validity, and low validity (Miller et al., 2009). It is better to avoid thinking of assessment result as valid or invalid.

There is no final, absolute measure of validity, but several different kinds of evidence may be evoked in support. They are content-related evidence, criterion-related evidence, construct-related evidence, consequential validity, and face validity. The research focuses on content validity of an assessment, so the explanation about content validity will be in bigger portion than other validities.

1) Content-Related Evidence

Content is simply defined as something what to teach or what the students need to learn. Price and Nelson (2011) stated that planning for instruction begins with thinking about content. Before making decision about how to teach, teachers should decide what to teach or what students need to learn (p. 3). Furthermore, Price et al. (2011) stated that it is impossible to achieve the intended result from teaching when the content, what to teach or what the students need to learn, is not exactly stated or determined. The content of language learning cannot be separated from the four basic language skills, namely reading, writing, speaking, and listening. Those four skills are integrated in language learning (Brown, 2004).

Musafi (2002) and Hughes (2003) stated that content-related evidence or content validity means that the test samples the subject matter about which conclusions are to be drawn and it requires the test-taker to perform the behavior that is being measured (as cited in Brown, 2004, p. 22). It means that if the test can clearly define the achievement being measured, it fulfills the content-related evidence or content validity.

In this process, content validation becomes the main point. The goal, as stated by Brown (2006), will always be to establish an argument that the assessment provides a representative sample of the content the test claims to measure. Further, the first step in assuring that the test or assessment is valid in its content is by deciding what assessment should be designed to measure. Brown (2006) stated again that thinking about validity may initially involve in defining

what it is that the testers wanted to measure in the first place. The important point in content validation is “to determine the extent to which a set of assessment tasks provides a relevant and representative sample of the domain of tasks about which interpretations of assessment results are made” (Miller et al., 2009, p. 75). Since language learning also involves the four basic language skills, the microskills and macroskills of each skill are also involved in the analysis (the list of microskills and macroskills of each content can be found in Appendix D).

2) Criterion-Related Evidence

Criterion-related evidence is the extent to which the “criterion” of the test has actually been reached. It is best demonstrated through a comparison of results of an assessment with results of some other measure of the same criterion. For example, in a course unit whose objective is for students to be able to orally produce voiced and voiceless stops in all possible phonetic environments, the result of one teacher’s unit test might be compared with an independent assessment – possibly a commercially produced test in a textbook – of the same phonemic proficiency.

3) Construct-Related Evidence

The next principle of an assessment is construct-related evidence or constructs validity. A construct is any theory, hypothesis, or model that attempts to explain observed phenomena in our universe of perceptions. “Proficiency” and “communicative competence” are linguistic constructs while “self-esteem” and

“motivation” are psychological constructs. In the field of assessment construct validity asks, “Does this test actually tap into the theoretical construct as it has been defined? (Brown, 2004, p. 25)”

4) Consequential Validity

Consequential validity encompasses all the consequences of a test, including such considerations as its accuracy in measuring intended criteria, its impact on the preparation of test-takers, its effect on the learner, and the (intended and unintended) social consequences of a test’s interpretation and use (Brown, 2004, p. 26).

5) Face Validity

Gronlund (1998) stated that face validity is the extent to which students view the assessment as fair, relevant, and useful for improving learning (as cited by Brown, 2004, p. 26). Brown also cited Mousavi’s (2002) definition about face validity.

“Face validity refers to the degree to which a test looks right, and appears to measure the knowledge of abilities it claims to measure, based on the subjective judgment of the examinees who take it, the administrative personnel who decide on its use, and other psychometrically unsophisticated observers” (p. 244).

d. Authenticity

The fourth principle of language assessment is authenticity. Brown (2004) cited Bachman and Palmer’s (1996, p. 23) definition about authenticity. It

is defined as “the degree of correspondence of the characteristics of a given language test task to the features of a target language task.” The authenticity may be present in some ways. First, the language in the test or assessment is as natural as possible. Second, the test or assessment’s items are contextualized rather than isolated. Third, the topics are meaningful, relevant, and interesting for the learner. Fourth, the thematic organization to items is provided, such as through a story line or episode. Fifth, tasks represent, or closely approximate, real-world tasks (Brown, 2004, p. 28).

e. Washback

According to Hughes (2003, p. 1), washback is “the effect of testing on teaching and learning (as cited in Brown, 2004, p. 28).” Brown (2004) stated again that in large-scale assessment, washback generally refers to the effects the tests have on instruction in terms of how students prepare for the test.

4. Content Validation

Content validation of an assessment becomes the basis of this study. The process of content validation was complex. The process involved the comparison of instructional content with the assessment content.

a. Definition and the Process

After determining about the content the Practice Teaching students had in their teaching, the writer analyzed the appropriateness of the assessments’ content or the intended result of the assessment with the formulation of the content as specified in the objectives. As stated by Walker and Schmidt (2004), in developing the assessment tasks teachers or the authors of assessment had to match the assessment tasks to the purpose and context of instruction. Tasks should relate directly to the goals of instruction and incorporate content and activities that have been part of the classroom instruction. Walker et al. (2004) added that the teachers or the authors had to ensure that the tasks allow students to clearly demonstrate their knowledge. Further they stated that building sound assessment tasks requires careful thinking about the content the students should demonstrate during the assessment progress. The questions must relate to skills and concept in the unit of study and provide students with the opportunities to demonstrate what they know. Miller et al., (2009) stated that good assessment requires relating the assessment procedures as directly as possible to intended learning outcomes.

Alderson, Clapham, and Wall (1995) stated that content validation process involves gathering the judgment of “experts”, the people whose judgment one is prepared to trust, even if it disagrees with one’s own. A common way for them is to analyze the content of a test and to compare it with a statement of what the content ought to be. Such a content statement may be the test’s specifications, a formal teaching syllabus or curriculum, or a domain specification.

According to Brown (2004, p. 32-33), there are two steps in evaluating the content validity. Those two steps are formulated in two questions, namely 1) Are classroom objectives identified and appropriately framed? 2) Are lesson objectives represented in the form of test specifications? The first step requires the writer to analyze whether the content of the study is specified appropriately in some objectives. This first step is similar with the idea presented by Alderson et al. (1995), analyzing the content.

The next step is comparing between the content which is already specified in some appropriate objectives with the content of the assessment. The process does not compare the content with the assessment procedure but compare the content with the intended results of the assessment to know whether they match and appropriate with the content of the study.

b. Instructional Goals and Objectives

The content validation process requires the classroom objectives in validating the content of an assessment. The classroom objectives are stated in the lesson plan made by the teachers before they apply the whole instruction processes which include the preparation and evaluation.

Before explaining about objectives or classroom objectives, the writer needs to give explanation about goals. Goals and objectives are considered the same but actually they are different. Graves (2000) stated that goals are a way of putting into words the main purposes and intended outcomes of the course.

Further, Graves stated that stating the goals helps teachers focus their vision and priorities for the course.

Goals are general statements, but they are not vague. For example, the goal “Students will improve their writing” is vague. In contrast, the goal “By the end of the course students will have become more aware of their writing in general and be able to identify the specific areas in which improvements is needed” is not vague (2000, p. 75). Finally, Graves (2000, p. 75) stated that “goals are benchmarks of success for a course.”

While the goals are the main purposes and intended outcomes of the course, the objectives are statements about how the goals will be achieved (Graves, 2000). The goals are divided or broken into learnable and teachable units through the objectives. Objectives or expected outcomes transform the general goals into specific students’ performance and behaviors that show the development of student learning and skill (TLC University of California, 2011). Graves (2000) added that “goals and objectives are in cause and effect relationship” (p. 77). It means that if the objectives can be achieved by the students, the goals of the course will be achieved, too. Graves also stated that objectives are in hierarchical relationship to the goals. Goals are more general and objectives are more specific.

Goals and objectives, which are the specification of the content of teaching, play a key role in both instructional process and the assessment process (Miller et al., 2009). The goals and objectives serve as guides for both teaching and learning, communicate the intent of the instruction to others, and provide

guidelines for assessing student learning (Miller et al., 2009). The major purposes are illustrated in Figure 2.2.

Figure 2.2

The Purpose of Instructional Goals and Objectives

(Miller et al., 2009, p. 47)

Objectives are statements that describe the behaviors students can perform after instruction (Airsian, 1991). Objectives describe where teachers want the students to go and they will know if the students got to the destination (Price et al., 2011). Objectives pinpoint the destination, not the journey. Well written objectives help teachers clarify precisely what they want the students to learn, help provide lesson focus and direction, and guide the selection of appropriate practice. Using objectives, teachers can evaluate whether their students have learned and whether their own teaching has worked (Price et al., 2011, p. 13).

Instructional Goals and Objectives

Convey instructional intent to others (students, parents, other

school personnel, the public)

Provide a basis for assessing student learning

(by describing the performance to be

measured) Provide direction for

the instructional process (by clarifying the intended learning

In formulating goals or objectives, according to Graves (2000), the teachers should pay attention to some aspects in teaching. The first, when the teachers want to write the goals or objectives, the teachers should keep in mind about the audience of the goals. Knowing the audience will help teachers consider whether the language they use is accessible to the audiences. The next aspect to be considered is that the goals or objectives formulated should be transparent enough for others to understand. It can be done by unpacking the language to simplify and clarify the goals or objectives (Graves, 2000).

Objectives should be S.M.A.R.T. It means that the learning or classroom objectives should fulfill the principles of simple, measurable, action-oriented, reasonable, and time-specific.

B. Theoretical Framework

In this part, the writer would like to synthesize the relevant theories which become the grounds to analyze the data. Assessment which is defined as some processes, activities, or systematic methods which are done to obtain information about the students’ progress or performance, is the first key word in this study. The writer will analyze the assessment made by Practice Teaching students during their practice teaching period. The analysis will focus on the principle of content validity of assessment which is done by comparing the content of teaching which is specified in the instructional goals or classroom objectives with the appropriateness of the assessment result from its specifications, items, or procedures.

The assessments chosen will be in criterion-referenced test (CRT) type which is created to measure well-defined and fairly specific instructional objectives. This type of assessment should be able to clearly present the objectives the students should achieve. The formal assessment is chosen because the assessment requires Practice Teaching students to plan and prepare their assessment. The planned and prepared assessment can show how well the students establish content validity in their assessment. Nevertheless, the assessment can be formative or summative.

The study places its attention on the content validity of an assessment. Content validity is one of the principles of a test or assessment. Content validity means that the test or assessment samples the subject matter about which conclusions are to be drawn and it requires the test-taker to perform the behavior being measured. If the assessment can clearly define and sample the intended achievements or results, the assessment is considered fulfilling the content validity. The validity of this study concerns with the validity of the intended outcome or result from the assessment and not from the assessment specifications or procedures. Validity is also stated in the matter of degree, namely high validity, moderate validity, and low validity.

The process to analyze the content validity of an assessment is called content validation. The way which is commonly done by the “expert” in content validation is to analyze the content of a test and to compare it with a statement of what the content ought to be. The process is divided into two main steps based on the process of content validation proposed by Brown (2004) with two main

questions, namely 1) Are classroom objectives identified and appropriately framed? 2) Are lesson objectives represented in the form of test specifications? The first step deals with the formulation of classroom or learning objectives. The writer analyzes the Practice Teaching students’ classroom or learning objectives formulation based on the S.M.A.R.T principle of objective formulation. The analysis is used to determine what content the Practice Teaching students want their students to learn. Although the process requires the writer to analyze the Practice Teaching students’ ability in making good learning or classroom objectives, the process focuses more in determining the intended result or content. Since the content statement involved the four basic skills in language learning, the writer also involved the microskills and macroskills analysis of each skill in the assessment tasks.

The second step deals with the comparison or the appropriateness of the teaching and learning content which is specified in the classroom or learning objectives with the intended result of the assessment specifications or procedures. The content of the course which specified in classroom or learning goals and objectives are then compared with the intended result from the assessment planned and prepared by Practice Teaching students. Therefore, if the intended result of the assessment can clearly and appropriately define and sample the intended achievements or results which are stated in the goals or classroom objectives, the assessment has high content validity. In contrast, if the assessment cannot define and sample the intended achievements or results, the assessment has a low content validity.

34

CHAPTER III

METHODOLOGY

In this part, the writer explains about the how the study was conducted. The chapter contains the research method, research participants, research instrument, data gathering technique, data analysis technique, and research procedure.

A. Research Method

The writer applied qualitative research which means that “the writer did not deal with numerical data” (Brown and Rodgers, 2002, p. 12). Ary, Jacob, and Razavieh (2002) defined that qualitative research is aimed at “obtaining a holistic depiction and in-depth understanding, rather than to obtain numerical analysis of data” (p. 25). Specifically, the writer applied document analysis method. Further, Ary et al. (2002) stated that document analysis places its attention at the analysis and interpretation of recorded materials. The documents which were analyzed in this study were the assessments produced by Practice Teaching students of ELESP Sanata Dharma University. The focus would be placed into the validity of the assessment, particularly in its content validity. As it was a qualitative study, the analysis focused on how the content validity was established in Practice Teaching student assessment and figured out the problems which influence the degree of content validity of the respondents’ assessments, rather than showing the numbers of mistakes they made during the production of assessment.

B. Research Participants

The research participants were the students of ELESP Sanata Dharma University who conducted their Practice Teaching during July to December 2010. The participants will be called as respondents in this study. All of them were in the seventh semester in 2010/2011 academic year. Assumedly, they had taken the Language Learning Assessment class, in which they had learnt about the basic concept and principles to produce good assessment. The data of the study were their assessments produced during the Practice Teaching period. The writer chose the students as the participants because throughout the Practice Teaching program, they were given the opportunity to practice teaching in the real school with real students in which they would apply all teacher competences they have obtained. They applied their knowledge and skill in teaching, namely how to handle the class, how to create good teaching materials, how to prepare the appropriate lesson plan, how to assess their students, and many more. Therefore, they also made real assessments for real students which must be good, reliable, and valid.

From those participants, the writer chose five random students who had finished Practice Teaching and asked for the documents which are the lesson plans along with some assessments they prepared during Practice Teaching period. From every student, the writer asked two lesson plans along with the assessments prepared for the lesson plan.

C. Research Instruments

In order to obtain dependable data to answer the research questions, the writer made use of two different types of research instruments.

1. The Writer as the Research Instrument

Poggenpoel and Myburgh (2003) stated that researcher as research instrument means that the writer is the key in obtaining the data from the respondents. The writer asked the data from the participants and also analyzed the data based on the theory of content validity of assessment. Furthermore, the writer reported the result of the study also.

2. Documents

Since this study applied document analysis, the existence of the document was the main requirement. In this study, the documents were the assessments and the lesson plans of Practice Teaching students which were produced during their Practice Teaching period. The writer chose two lesson plans from each student together with the assessments they prepared for the lesson plans. Those documents became the data to answer the research questions. In Practice Teaching, the students did not only practice teaching, which means transferring the knowledge they had learnt, but also they learnt to be real teachers who must be able to prepare the instruction, conduct teaching and learning, know the students’ characteristic, assess their progress, report the progress and marks, and many more. Therefore, they had to make the lesson plans to plan their

teaching. The lesson plan became the first document in this study. The second document was the assessments which should be in accordance with the lesson plans prepared especially in the content being taught.

D. Data Gathering Technique

The writer gathered the data from five students of Practice Teaching subject. The writer collected the data by compiling the lesson plans along with some assessments for each lesson plan they produced during Practice Teaching period. From every student, the writer asked for two lesson plans along with some assessments prepared for the lesson plan. It was done to have more accurate analysis about the students’ ability in producing content valid assessments. A single lesson plan with some assessment items was not considered sufficient to represent the respondents’ ability. It was better for the writer to have two documents to be analyzed to support the findings about how the respondents established content validity in their assessments.

E. Data Analysis Technique

The writer analyzed how the content validity is established from the data that had been collected. The analysis was done by adapting the question offered by Brown (2004, p. 32), “Does the procedure demonstrate content validity?” The question is divided into two questions, namely 1) Are classroom objectives identified and appropriately framed? 2) Are lesson objectives represented in the form of test specifications? The first question did not merely analyze about the

respondents’ ability in formulating the objectives but it was aimed at knowing or determining the content of the instruction planned by the respondents in their lesson plan. The second question dealt with the comparison of the instructional content with the intended result or the content of the assessment. The interpretation of the intended result of the assessment was analyzed based on the instructional content, not merely the assessment tasks or procedures.

The first thing that should be considered was the objectives of the classroom activities or instructions because the instructional content was specified in some objectives. The objectives should be clearly stated and identified in the lesson plan in which the assessment was based. The objective, as stated by Price at al. (2011), pinpoints the destination, not the journey. The objective stated the outcomes the students should achieve after the course or instruction. Nevertheless, the writer found that some respondents formulated learning objectives which did not state the outcomes of the instruction but they only stated the learning activities the students would be involved. Therefore, in conducting the analysis, the writer would first examine the objectives of the course to determine the outcomes which were intended by the respondents. The outcomes could be found in Learning Objectives, “Tujuan Pembelajaran” or they could be found in indicator, “indicator” statements. In the analysis, the writer used the word “objective” which referred to the learning outcomes whether they found in the Learning Objectives or Indicators part. The analysis of the objective formulation used the principle of S.M.A.R.T (Simple, Measurable, Action-oriented, Reasonable, and Time-specified).

Macroskills

1. Recognize the rhetorical forms of written discourse and their significance for interpretation.

2. Recognize the communicative functions of written text, according to form and purpose.

3. Infer context that is not explicit by using background knowledge.

4. From described events, ideas, etc., infer links and connections between events, deduce causes and effects, and detect such relations as main idea, supporting idea, new information, given information, generalization, and exemplification.

5. Distinguish between literal and implied meanings.

6. Detect culturally specific references and interpret then in a context of the appropriate cultural schemata.

7. Develop and use battery of reading strategies, such as scanning and skimming, detecting discourse makers, guessing the meaning of words from context, and activating schemata for the interpretation of texts.

4. Writing

Microskills

1. Produce graphemes and orthographic patterns of English. 2. Produce writing at an efficient rate of speed to suit the purpose.

3. Produce an acceptable core of words and use appropriate word order patterns. 4. Use acceptable grammatical systems (e.g., tense, agreement, pluralization),

patterns, and rules.

5. Express a particular meaning in different grammatical forms. 6. Use cohesive devices in written discourse.

Macroskills

1. Use the rhetorical forms and conventions of written discourse.

2. Appropriately accomplish the communicative functions of written texts according to form and purpose.

3. Convey links and connections between events, and communicate such relations as main idea, supporting idea, new information, given information, generalization, and exemplification.

4. Distinguish between literal and implied meaning when writing.

5. Correctly convey culturally specific references in the context of the written text.

6. Develop and use a battery of writing strategies, such as accurately assessing the audience’s interpretation, using prewriting strategies, writing with fluency in the first drafts, using paraphrases and synonyms, soliciting peer and instructor feedback, and using feedback for revising and editing.

APPENDIX D

The Categorization of the Degree of Content Validity of Assessment

No The

Assessments High Content Validity Moderate Content Validity Low Content Validity

1

Content statement and objectives formulation

The instructional content was stated clearly and it was specified in some objectives which were formulated S.M.A.R.T and clear. The objectives formulation clearly stated the learning outcomes the students should achieved or master.

The instructional content was stated clearly and it was specified in some objectives which formulated S.M.A.R.T and clear. Yet, the objective formulations were not clearly stated the learning outcomes the students should achieved or master.

The instructional content was stated clearly and it was specified in some objectives which formulated S.M.A.R.T and clear. Nevertheless, the objectives formulation did not clearly stated the learning outcomes the students should achieved or master, in other words, the objectives intended the different outcomes seen from the instructional content statement.

2 Procedures _{and skill} content

The assessments appropriately presented various procedures which gave the students the opportunity to perform the intended performances of the skills content involved.

The assessments fairly presented some procedures which gave the students the opportunity to perform the intended performances of the skills content involved.

The assessments presented limited procedures which did not gave the students the opportunity to perform the intended performances or the skills content involved.

3

Intended result or

interpretation of the result

The intended result or the content of the assessments clearly and appropriately matched with the

instructional content. The content of the assessments described the outcomes the students should perform and master.

The intended result or the content of the assessments fairly matched with the instructional content. The content of the assessments described the outcomes the students should perform and master.

The intended result of the assessments or the content of the assessments did not match with the instructional content. The content of the

assessments described the outcomes which were not intended by the instructional content.

vii ABSTRACT

Prasetyo, Matius Teguh. (2012). A Study on Content Validity of the Assessment Prepared by Practice Teaching Students. Yogyakarta: Sanata Dharma University.

An assessment in an instruction is purposed to obtain the information about the students’ progress and mastery during or at the end of the instruction. The information should be valid and should represent the true condition of the students. Therefore, the assessment applied by the teachers should be valid and able to represent the intended outcomes the students should perform or master.

Based on the concept above, the writer was interested to conduct a study about the validity of the assessment, especially the content validity. The writer formulated two research questions, namely 1) how is the content validity established by Practice Teaching students in their assessments? and 2) what are the problems which influence the degree of content validity of the Practice Teaching Students’ assessments?

In order to answer those questions, the writer conducted a qualitative study, namely document analysis. The documents were the lesson plans along with some assessments prepared for the lesson plan by the Practice Teaching students. The document analysis aimed at showing how the Practice Teaching students established content validity in their assessments, the process was called content validation. The respondents were five randomly chosen Practice Teaching students of ELESP Sanata Dharma University in academic year of 2010/2011. The analysis was done by comparing the instructional content which was specified to some specific objectives with the intended result or the content of assessments. The analysis was also done to obtain the information about the problems which influence the content validity degree of the respondents’ assessments.

From the analysis, the respondents established some assessments which were valid based on the principle of content validity of assessment but they differed in their degree of content validity. The findings showed that the respondents established high, moderate, and low content validity. The problems which influenced the content validity degree of the respondents’ assessments were 1) the objective formulations were not clear and ambiguous, 2), there were too many objectives in an instruction for a single content, 3) the assessments did not measure all the objectives stated, 4) the assessments’ tasks or procedures did not represent the intended skills, and 5) the intended result of the assessments had little correlation with the content of the instruction. In order to overcome the problems, the writer proposed some possible recommendations, namely 1) producing well planned lesson plan and assessments, 2) stating clear and unambiguous content and objectives, 3) not formulating too many objectives for a lesson, 4) considering the four basic English skills in the assessments, and 5) improving and updating the knowledge about assessment by reading books or any other sources.

viii ABSTRAK

Prasetyo, Matius Teguh. (2012). A Study on Content Validity of the Assessment Prepared by Practice Teaching Students. Yogyakarta: Universitas Sanata Dharma.

Sebuah asesmen dalam proses pembelajaran dimaksudkan untuk mendapatkan informasi tentang kemajuan dan pemahaman siswa selama atau pada akhir proses pembelajaran. Informasi yang didapat dari asesmen tersebut haruslah valid dan dapat mewakili kondisi nyata siswa. Oleh karena itu, asesmen yang diaplikasikan oleh guru harus valid dan dapat merepresentasikan hasil yang seharusnya dicapai atau ditunjukkan oleh siswa.

Berdasarkan konsep di atas, penulis tertarik untuk melaksanakan sebuah penelitian tentang validitas dari sebuah asesmen, terutama pada validitas isi. Penulis merumuskan dua pertanyaan dalam penelitian ini, yaitu 1) bagaimana validitas isi ditampilkan oleh para mahasiswa yang sedang menjalani Praktik Pengajaran Lapangan (PPL) dalam asesmen mereka? dan 2) apakah masalah yang mempengaruhi tingkat validitas isi dari asesmen para mahasiswa PPL tersebut?

Untuk menjawab pertanyaan-pertanyaan tersebut, penulis melaksanakan sebuah penelitian kualitatif, yaitu analisis dokumen. Dokumen yang dimaksud adalah beberapa Rencana Pelaksanaan Pembelajaran (RPP) bersama dengan asesmen yang disiapkan untuk RPP tersebut yang dibuat oeh mahasiswa PPL. Analisis dokumen dimaksudkan untuk menunjukkan bagaimana para mahasiswa PPL menampilkan validitas isi dalam asesmen mereka, proses ini disebut dengan validasi isi. Responden dalam penelitian ini adalah lima orang mahasiswa PPL Universitas Sanata Dharma yang dipilih secara acak pada tahun ajaran 2010/2011. Analisis dilakukan dengan cara membandingkan isi pembelajaran yang dibagi dalam beberapa tujuan pembelajaran dengan hasil yang diharapkan atau isi dari asesmen. Analisis juga dilakukan untuk mengetahui tentang masalah-masalah yang mempengaruhi tingkat validitas isi dari asesmen para responden.

Dari analisis yang dilakukan, para responden membuat beberapa asesmen yang valid berdasarkan prinsip validitas isi dari sebuah asesmen tetapi mereka berbeda dalam tingkatannya. Hasil penelitian menunjukkan bahwa para responden menampilkan asesmen yang mempunyai validitas isi tinggi, sedang, dan rendah. Masalah-masalah yang mempengaruhi tingkat validitas isi dari asesmen para responden adalah 1) formulasi tujuan pembelajaran tidak jelas dan ambigu, 2) ada terlalu banyak tujuan pembelajaran untuk sebuah isi dalam sebuah proses pembelajaran, 3) asesmen yang dibuat tidak mengukur semua tujuan pembelajaran, 4) prosedur dari asesmen tidak mewakili keterampilan yang diharapkan, 5) isi dari asesmen hanya memiliki sedikit korelasi dengan isi dari pembelajaran. Untuk mengatasi masalah tersebut, penulis menawarkan beberapa solusi, yaitu 1) membuat RPP dan asesmen yang terencana dengan baik, 2) menuliskan tujuan pembelajaran yang jelas dan tidak ambigu, 3) tidak menyebutkan terlalu banyak tujuan pembelajaran dalam satu proses pembelajaran, 4) menyertakan empat keterampilan dasar Bahasa Inggris, dan 5) meningkatkan

ix

dan selalu memperbaharui pengetahuan tentang asesmen melaui buku atau sumber-sumber yang lain.