The evaluation of the seventh grade English textbook When English Rings the Bell a survey research

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

WHEN ENGLISH RINGS THE BELL: A SURVEY RESEARCH

A THESIS A Thesis Presented to

The Graduate Program in English Language Studies in Partial Fulfillment of the Requirements for the Degree of Magister Humaniora (M.Hum.)

in English Language Studies

by Teguh Ariebowo

116332027

THE GRADUATE PROGRAM OF ENGLISH LANGUAGE STUDIES SANATA DHARMA UNIVERSITY

YOGYAKARTA February, 2017

Member

---ATHESIS - '

---~

---

--Presented hy,

--~~

_; =

1

-=- -::: l

~- - =-1

-- -j

=~~

- :_:

-"

·-

--- ~

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

ACKNOWLEDGEMENTS

In the name of Allah The Most Gracious and The Most Merciful. Praise and thanks are due to Allah, who has given me strength to complete this thesis. I also wish to express my gratitude to the following for their contribution throughout the completion of this study.

I owe an enormous debt of gratitude to F.X. Mukarto, Ph.D., my supervisor, for all his patience, continuous supervision, invaluable guidance, encouragement, help, and feedback all the way through the processes of writing this thesis. My gratitude also goes to Dr. J. Bismoko and Dr. B.B. Dwijatmoko for the inspiration, feedback, and suggestion so that I am able to revise this work and make it better. I do not forget to say thanks to all the lecturers and staffs in the Graduate Program of English Studies, Sanata Dharma University, Yogyakarta for the knowledge and skills that I have got throughout the undertaking of my graduate study.

I am also sincerely grateful for having wonderful experience with all English teachers who enthusiastically participated as respondents of this research. Without their contribution and participation, this research would have not be completed. Those supportive and cooperative teachers are from SMP N 1 Sleman, SMP N 1 Galur, and SMP N 1 Gunung Kidul.

I should also thank to my first English teacher in an English course, Ms. Erni Dhukun and Ms. Pipin for introducing English to me in an interesting and fun way. The gratitude is also addressed to Ibu Henny Herawati who gave support and involved me in fruitful discussion about teaching which was able to enlighten me in finishing this work.

Finally my special thanks are due to my beloved parents, my lovely wife and my siblings. Thank you for giving me unstoppable support to accomplish this mission. Last but not least, great thanks also go to people who I impossibly mention those people one by one. Thank you for invaluable source of support and inspiration for my over years studying here.

TABLE OF CONTENT

STATEMENT OF THE ORIGINALITY ... iv

LEMBAR PERNYATAAN PERSETUJUAN PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS ... v

ACKNOWLEDGEMENTS ... vi

TABLE OF CONTENT ... vii

LIST OF FIGURES ... ix

LIST OF TABLES ... x

LIST OF ABBREVIATIONS ... xi

LIST OF APPENDICES ... xii

ABSTRACT ...xiii

ABSTRAK ... xiv

CHAPTER I INTRODUCTION ... 1

A. B. C. D. E. Background ... 1

Problem Identification and Limitation ... 4

Problem Formulation ... 5

Purpose of Study ... 5

Benefits of the Study ... 5

CHAPTER II REVIEW OF LITERATURE... 7

A. Theoretical Review ... 7

1. 2. 3. 4. Textbooks ... 7

Evaluation... 16

Review on Indonesia National Curriculum 2013 ... 23

Review of Previous Studies... 26

B. Theoretical Framework ... 28

CHAPTER III RESEARCH METHOD ... 30

A. Research Method ... 30

B. Research Respondents ... 31

C. Research Instrument ... 32

D. Data Gathering Techniques ... 32

E. Data Analysis ... 33

CHAPTER IV RESEARCH FINDINGS AND DICUSSION... 35

A. General Findings ... 35

B. Detail Findings... 37

1. 2. 3. 4. 5. 6. The Aim of the Textbook ... 37

The Approach of the Textbook ... 39

Design and Organization of the Textbook ... 46

Content of the Textbook ... 51

Language Skill Presented in the Textbook ... 54

Topic Presented in the textbook ... 56

C. Discussion ... 59

1. 2. 3. 4. 5. 6. The Aim of the Textbook ... 59

The Approach of the Textbook ... 61

The Design and Organization of the Textbook ... 62

Language Content of the Textbook ... 66

Language Skill Presented in the Textbook ... 66

The Topic Presented in the Textbook... 67

CHAPTER V CONCLUSION ... 70

A. CONCLUSION ... 70

B. Suggestions ... 73

Bibliography... 75

Appendices ... 78

LIST OF FIGURES

Figure 2.1 The Theoretical Framework ... 29

Figure 4.1 Sample of activity found in textbook: Activity 5 ... 41

Figure 4.2 Sample of activity found in textbook: Activity 6 ... 42

Figure 4.3 Sample of activity found in textbook: Activity 8 ... 43

Figure 4.4 Aim of Chapter 1 ... 48

Figure 4.5 Sample of activity found in textbook ... 53

Figure 4.6 Sample of a simple lyric and a very simple dialog ... 56

Figure 4.7 The new version of Bloom's taxonomy ... 63

LIST OF TABLES

Table 2.1 Criteria of an Effective Textbook ... 15

Table 2.2 Checklist Suggested by Rea-Dickins and Germain ... 22

Table 4.1 Overall Survey Result ... 36

Table 4.2 Survey Result on the Aims of the Textbook ... 38

Table 4.3 Survey Result on the Approach of the Textbook ... 40

Table 4.4 Survey Result on the Design and Organization of the Textbook... 48

Table 4.5 Complexity Level of the Activity ... 51

Table 4.6 Complexity Level and Number of Activity ... 52

Table 4.7 Survey Result on the Content of the Textbook ... 53

Table 4.8 Survey Result on the Skills Presented in the Textbook ... 56

Table 4.9 Survey Result on the Topic Presented in the Textbook ... 58

LIST OF ABBREVIATIONS

C13 TEFL EFL WERB

Curriculum 2013

Teaching English as Foreign Language English as a Foreign Language

When English Rings the Bell

LIST OF APPENDICES

Appendix 1: Questionnaire Draft 1 Appendix 2: Questionnaire Draft 2

Appendix 3: The Blueprint of the Questionnaire Appendix 4: The Respondents’ on Statement

Appendix 5: The Respondents’ Answer on Open Question Questionnaire Appendix 6: Translated transcript of the interview

79 90 98 106 110 115

ABSTRACT

Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When English Rings the Bell”: A Survey Research. Yogyakarta: English Language Studies. Graduate Program Sanata Dharma University.

As the process of curriculum development, Indonesia developed a new national curriculum in 2013 which is called Curriculum 2013 (C13). This new curriculum raised a great discussion and debate among people of Indonesia. The textbooks of any subject which were made based on the C13 then became problematic since its basic competence has become tremendous concern of textbooks especially the English textbook. Therefore, the researcher focuses on the grade textbook of secondary school to be evaluated. In this evaluation study, the researcher try to find out to what extend textbook meets the criteria.

This study belongs to a descriptive quantitative research. This research took the data from the survey among the teachers of English using this textbook in their classroom. To be able to conduct the survey, the researcher developed a questionnaire distributed to 17 teachers as the respondents in the Special Province of Yogyakarta. When this research was conducted, there were only few schools appointed by the government to implement the C13. The questionnaire administered the Likert using scale 1 – 4. The statements were developed from some criteria of effective textbook based on the experts for instance Brian Tomlinson (2008), David Nunan and Cunningsworth (1995). The criteria include a) the aim, b) the approach, c) the design and organization, d) the language content, e) the skill and f) topic presented in the textbook.

The findings showed that some criteria were fulfilled in the textbook but some other criteria were not. According to the survey result, the aims of textbook met perfectly and suited with the aim of the C13. The approach implemented in the textbook met the criteria quite well although some respondents had no idea of what approach they should adopt in this curriculum. A different finding happened in the design and organization of the textbook. In some aspect of this criterion, the respondents saw that textbook did not meet the criteria very well. Different attitude found in responding language content of the textbook. All respondents did not appreciate the language content. Then respondents saw that the topic presented in the textbook met the criteria so much well.

In conclusion, the researcher sees that the textbook has some strengths and weaknesses. The presented topic, for instance, is one of the strengths since it is suitable with the learners’ level and social condition in Indonesia. The illustration found in the textbook represents the gender, ethnic, and religion diversity in Indonesia. However, the textbook is like other books which always have pitfalls. One of the most obvious pitfalls in textbook is the language content since this book does not have grammar reference and its level of difficulty way too simple for grade students. Finally, the researcher suggests material developers to revise this textbook and to English teachers to always conduct evaluation toward books or new materials before utilizing them in classroom. Hence teachers are able to choose the most suitable textbook for their learners.

Keywords: Effective textbook, Textbook evaluation, Material Evaluation. 7th

7th

ABSTRAK

Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When English Rings the Bell”: A Survey Research. Yogyakarta: English Language Studies. Graduate Program Sanata Dharma University.

Sebagai bentuk proses perkembangan kurikulum, Indonesia mengembangkan sebuah kurikulum nasional yang baru pada tahun 2013 yang disebut Kurikulum 2013 (K13). Kurikulum baru ini memunculkan diskusi dan perdebatan besar di kalangan masyarakat Indonesia. Buku-buku teks pelajaran yang disusun berdasarkan K13 kemudian menjadi masalah disebabkan kompetensi dasar yang menjadi kekhawatiran besar dari buku-buku teks pelajaran khususnya Bahasa Inggris. Dengan demikian, peneliti berfokus pada buku teks pelajaran Bahasa Inggris untuk siswa kelas 7 SMP untuk dievaluasi. Dalam kajian evaluasi ini, peneliti mencoba untuk mencari tahu seberapa jauh buku teks bahasa Inggris tersebut memenuhi kriteria.

Kajian ini digolongkan dalam sebuah penelitian kuantitatif diskriptif. Penelitian ini mengambil data dari survei yang dilakukan di kalangan guru-guru Bahasa Inggris yang menggunakan buku teks pelajaran ini di kelas. Untuk melaksanakan survei ini, peneliti mengembangkan sebuah kuisioner atau angket yang dibagikan kepada 17 guru di DIY sebagai respondennya. Ketika penelitian ini dilaksanakan, hanya terdapat beberapa sekolah yang ditunjuk oleh pemerintah untuk mengimplementasikan K13. Angket tersebut menggunakan skala Likert dari 1 – 4. Pernyataan-pernyataan dalam angket dikembangkan dari kriteria buku teks yang efektif yang berdasarkan dari beberapa ahli seperti Brian Tomlinson (2008), David Nunan (1989) and Cunningsworth (1995). Kriteria tersebut meliputi a) tujuan, b) pendekatan, c) desain dan organisasi, d) bahasa isi, e) keterampilan dan f) topik yang disajikan di dalam buku teks.

Hasil dari survei menunjukkan bahwa buku teks tersebut memenuhi beberapa kritera dan juga beberapa hal lain tidak memenuhi kriteria. Berdasarkan hasil survei, tujuan buku teks berkesinambungan dengan tujuan K13. Pendekatan yang diimplementasikan di dalam buku teks cukup memenuhi kriteria walaupun beberapa guru yang mengisi angket tidak mengerti dan memahami pendekatan apa yang seharusnya mereka gunakan di dalam kurikulum ini. Berbeda dengan hasil jajak pendapat mengenai desain dan organisasi buku teks bahwa di beberapa kriteria buku ini tidak dapat memenuhinya dengan baik. Sangat jauh berbeda dengan hasil jajak pendapat mengenai bahasa isi dari buku teks tersebut di mana semua guru yang menjadi responden bahwa mereka bersikap kurang setuju dengan bahasa isi dengan

buku teks tersebut. Kemudian, di bagian topik, para guru berpendapat bahwa topik yang disajikan sudah memenuhi kriteria buku dengan baik.

Pada kesimpulannya, peneliti melihat bahwa buku teks terebut memunyai kelebihan dan kekurangan. Topik yang disajikan, misalnya, merupakan kekuatan dari buku teks ini mengingat bahwa topik yang disajikan cocok dengan tingkatan pembelajar dan kondisi sosial masyarakat Indonesia. Ilustrasi yang ditampilkan dalam buku teks ini memperihatkan keanekaragaman gender, etnis dan agama di Indonesia. Namun demikian, buku teks ini tidak luput dari kekurangan seperti halnya pada buku teks lainnya. Salah satu kekurangan buku teks ini yang sangat terlihat jelas adalah tidak adanya referensi tata bahasa yang bisa dijadikan rujukan oleh pemelajar dan tingkat kesulitan bahasa yang digunakan dalam buku ini terlalu mudah untuk pemelajar tingkat 7 SMP. Pada akhirnya, peneliti menyarankan kepada pengembang materi untuk merevisi buku teks ini. Selain itu paling penting adalah menyarankan kepada setiap guru Bahasa Inggris untuk melakukan evaluasi terhadap buku-buku setiap sebelum menggunakan buku-buku tersebut di dalam kelas. Dengan demikian, para guru dapat memilih buku teks yang cocok bagi para pemelajar di kelas mereka.

Kata kunci: Evaluasi materi pembelajaran, evaluasi bahan ajar, evaluasi.

CHAPTER I INTRODUCTION

This chapter comprises the elaboration of the background of the study, the problem identification and limitation, the problem formulation, the purposes of the research, and the research benefits.

A. Background

It has been declared by experts that material evaluation is significant in education field. First, Rod Ellis (1998) underscored that it is mandatory to evaluate every single book to teach a language. Second, Ellis (1998) illuminated that the evaluation of a textbook is important because it aims to see how far it has worked and is a partial procedure of a macro evaluation of curriculum development. Third, Sheldon (1988) said that a textbook evaluation needs to be administered since a textbook which is imposed on parties, both learner and teacher, by a higher authority (in this case the Indonesia Ministry of Education) and no room for change and modification will result a discontent. From what experts say, it can be concluded that an evaluation process of a textbook is fundamental because it affects effectiveness of teaching and learning process. If it happens that textbook is discontent for students and teachers and it is kept to be used, there will be a hole in the teaching and learning process. So, textbook evaluation is needed to be a patch on the hole through inputs and feedback resulted from the process for its revision.

Another reason why this research is about material evaluation is because textbook is an important resource for (1) presenting language components, (2)

activities, (3) stimulating ideas, (4) self-directed learning, and in some contexts, (5) used as a syllabus and a support for less experienced teachers (Cunningsworth, 1995). Agreeing to Cunningsworth (1995), Tomlinson (2008) stated that textbook evaluation is a crucial process that can have a direct effect on the teaching procedure. Textbook evaluation can be carried out to help a publisher or an institution to make decisions, to help ourselves in developing and selecting textbook, to find out if the book being used meets the needs of the students and to try carry out a research project (2003). Without having good materials to teach, the program will not run very well. Therefore, the writer sees evaluating material is absolutely important and useful for teachers, stakeholders and the program.

There are two other reasons why teachers, experts and material developers should do a material evaluation besides determining whether the material “works” or not. Empirical evaluation of teaching materials can be found in accounts of trialing of new materials (Bernard and Randall, 1995). In this case, the evaluation of teaching materials is taken when there is a new material used in the teaching and learning process. Hence, the main purpose of the teaching materials evaluation is to determine whether the materials are suitable for the teaching and learning process. Another case is when the teaching materials are evaluated to find out whether it is worthwhile using the materials again. In such case, the evaluation is taken when teachers, experts and practitioners plan to evaluate the existing materials. Since it has been explained above that in Indonesia there is a new teaching material for the C13, this evaluation research is conducted to see the teachers’ perspective toward textbook. Last but not

least, according to the need analysis survey in TEFL/TESL teachers and researchers considered textbook evaluation to be one of the most important components of a teacher training course (Hutchinson & Waters, 1991).

The Indonesia Minister of Education decided to load the new in 2013. Hence in 2013, Indonesia government imposed the new curriculum called Curriculum 2013 or C13. The C13 was implemented for some rationales mentioned by the Minister of Education (2012). Those are current global challenges, required competencies, and current negative phenomena especially among young people and discouraging perceptions among Indonesians regarding education (Hamied, 2014). However, Karnodi (2014) in Hamied (2014) stated that the first implementation of C13 in 2013 seemed to raise different reactions among teachers in Indonesia. Some with defiance, as many of them still argue that C13 is a type of curriculum developed in a top-down and centralistic fashion. Therefore, the researcher points that a textbook evaluation of C13 is in the urge to be conducted.

In conclusion, this present study is evaluating the new English textbook used in seventh grade of junior high school in Indonesia mainly concerning that material evaluation is very important today to determine to what extent a material has worked and broader is to facilitate the whole process of curriculum development (Ellis, 1998). When we talk about evaluating teaching material, or textbook, it means that we talk about textbook as a whole package. However, this present study limits the dimension of evaluation. The limitation of the evaluation will be explained in the next subtopic.

B. Problem Identification and Limitation

As mentioned earlier that the C13 raised different reactions among teachers toward its administrative matters to pedagogical matters. Administrative matters include dissemination of the curriculum, budget, distribution of textbooks and teacher training (Weir & Roberts, 1994). Some questions in the evaluation may be raised such as on how the dissemination of the curriculum will be arranged, how much money that the authority should be put in the budget for the whole program, and many other critical questions towards the program. Classroom teacher behaviors, teaching method or the learning process, classroom learner activities and the teaching materials take part in the discussion of pedagogical matters. This explains what Ellis (1998) suggests in his journal that program evaluation covers administrative matters and curricular matters where materials evaluation, teacher evaluation and learner evaluation employ in it. These evaluations later will be called as macro-evaluation. More specific than macro-evaluation is the micro-evaluation. This will involve more specific topic for each aspect mentioned above. For instance, in material evaluation, task evaluation can be done in micro-evaluation.

Too many aspects must be evaluated in macro-evaluation. As a matter of fact, there are only small numbers of a complete holistic evaluation that involves various kinds of information relating to one or both administrative matters (i.e. the logistical and financial underpinnings of the program) and curriculum matters which later can be broken down into materials, teachers, and learners. Hence, this study evaluates a more specific subject. A micro-evaluation will be undergone to examine the English textbook of junior high school in Indonesia. In this case, this research will focus only

on textbook published by the Indonesia Ministry of Education entitled When English

Ring the Bell (WERB) for seven grade level. Then, to evaluate the textbook, the

researcher determined the criteria that a textbook should meet. The textbook criteria will be elaborated in detailed in chapter 2.

C. Problem Formulation

To evaluate the tasks in the English textbook of 7th grade of junior high school using 2013 curriculum WERB research questions are raised as follows:

1. To what extent does textbook meet the criteria?

2. What are the strengths and weaknesses of textbook according to the survey?

D. Purpose of Study

The purposes of this study are, first, to find out to what extent the textbook meets the criteria and to find the strengths of textbook and the weaknesses of textbook according the users. Second, the researcher then can make any justification based on the empirical data and the theory whether the textbook is suitable to be used as the main material, needs additional material, is suitable as the additional material or needs to be revised. Then the researcher then can make suggestion to teachers who use this book, to material developers and to future researchers related to textbook evaluation.

E. Benefits of the Study

7th

This evaluation research on English textbook of grade of junior high school is absolutely beneficial for material developers (in this case the authority), teachers, learners, and stakeholders.

For materials developers, this research evaluation is beneficial as reference in developing textbook to design better and more effective textbook. The material developer in this case the authority may refer to what has been discussed in this evaluation research.

For teachers, this research evaluation will give detailed explanation on how the textbook meets the criteria. Consequently, teachers may respond towards the findings. Then teachers can take any anticipated action to take a positive response to do in relation to the findings.

This research will also be beneficial for stakeholders since they can use result as a reference to develop the whole English program in junior high school. Besides, this particular evaluation research will be valuable for other evaluators especially program evaluators to help them evaluate English program for junior high school.

CHAPTER II

REVIEW OF LITERATURE

This chapter is mainly presenting the literature reviews related to the topic of the research. First, theoretical review is elaborated briefly to build the foundation of this research. Second, related studies are presented and explained in this chapter to point out what have been discussed so far by other researchers and evaluators. Last, a framework is presented in the end of this chapter to give a clear construct of this research.

A. Theoretical Review

This section reviews relevant constructs and critically analyzes their relations in building up theoretical framework of the research. Based on the research problems, it is recognized into three major constructs, namely English textbook, Evaluation in English Language Teaching, and Indonesian curriculum of 2013.

1. Textbooks

Commonly, textbooks are a book about a particular subject that is used in the study of that subject especially in a school. Those are commonly used by the learners as well as the teachers in a school that provides structured syllabus of the course program (Ur, 1991). Usually, the books are accompanied by teacher’s book as the manual in teaching the subject and workbook to support learners in practicing the subject.

a. Definition of Textbooks

Before going further talking about course books evaluation, this section initially talks about the subject of the study which is a textbook. As a matter of fact textbooks are taken for granted in some places while in some other places textbooks are not used at all. In such case, textbooks are only used as the supplementary material. However, some teachers select textbooks and use them in their classes (Ur, 1991). In other words, in some situation, textbook is used selectively and it is used to as supplementary materials in the English course program. Therefore, it is important to clarify the definition on what it is called as a textbook.

It seems that Ur’s (1991) definition about textbook is the easiest to understand. She gives a clear position of the term textbook when the discussion comes to the necessity of using a textbook. It is explained that textbook is book of which the teacher and each student has a copy, and which later on in the learning process it is followed systematically as the basis of a language course.

Textbooks now become a must-have element in EFL/ESL teaching. Those are widely accepted to be the heart of any English language situation (Sheldon, 1988; Hutchinson and Torres, 1994). It is also argued that textbooks have become the common element in the teaching of English as the second language or foreign language (Tsiplakides, 2011). This happens because it seems that it is too difficult for teachers to create their own teaching materials (Sheldon, 1988). Another reason by Sheldon (1988) to justify the argument above is that textbooks lessen the preparation time by providing ready-made teaching texts and learning tasks.

b. Roles of Textbooks

In our discussion about textbooks, a role play of a textbook is inseparable from the discussion. Ur (1991) says that a textbook is a framework for the teachers and learners making them know where they are going and what is coming next. Therefore, there is a sense of structure and progress. A textbook can be also a syllabus of an EFL/ESL course since a textbook should provide a syllabus in it. Besides providing a syllabus, a textbook even provides texts and learning tasks. Agreeing to Ur’s opinion (1991), a textbook is a source for learner practice and communicative interaction (Cunningsworth, 1995). A textbook is a convenience in teaching and learning EFL/ESL since in a textbook the components stick together in order and structured (Ur, 1991). According to Ur (1991), a textbook also roles as a guidance for inexperience teacher or for teachers who occasionally unsure of their knowledge of the language. In other meaning, it can become a support for less experienced teachers (Cunningsworth, 1995). Finally, a textbook may become a reference and source where learners can use textbook to learn new material, review and monitor progress with some degree of autonomy (Ur, 1991; Cunningsworth, 1995).

In conclusion, textbook plays a very important role in teaching and learning process since from the above explanation of the experts; textbook is inseparable from the teacher and student. Three important roles of a textbook in English course program dominate in teaching and learning English in EFL context. They are as a guide book for teachers and learners, a reference book for teachers and learners and a ready-made syllabus for teachers.

c. Advantages and Disadvantages of Textbooks

In regard idea or concept, the discussion definitely dragged to search the advantages and disadvantages of it. In the same time, advantages and disadvantages of textbook will be described based on some experts of language materials developers.

Richard (2001) states in his book entitled Curriculum Development in

Language Teaching that the advantages and disadvantages of the use of textbook

depend on how it is used and the context for its use. He concludes that textbooks provide structure and a syllabus for a program, helps standardize instruction, maintains quality, provides a variety of earning resources, and is efficient. Besides, textbooks also provide effective language models and input, can train teachers and are visually appealing.

However, textbooks may still potentially have negative effects (Richards, 2001) such as textbook may contain inauthentic language, distort content, not reflect

student’s, and can deskill teachers. Lastly, textbook is usually expensive.

Furthermore, textbooks fail to help the learners to make full use of the language experience available to them outside the classroom (Tomlinson, 2008).

In conclusion, textbooks have important roles in teaching and learning EFL/ESL. The relationship between teacher and textbooks is an important consideration and is at best when it becomes partnerships when both share common goals to which each party contributes to the teaching and learning process (Cunningsworth, 1995).

d. Textbook Criteria

It is such an uneasy task to judge whether a textbook or teaching material, in this case is ELT material, is good or bad (Tomlinson, 2008) since every textbook or ELT material has its own strengths and weaknesses and there is no perfect textbook exists. Instead of using the word good and bad, Cunningsworth (1995) used the word

effective and ineffective. Consequently, many experts and researchers have been

doing evaluation on textbook as part of the curriculum development process to find the best possible one that will fit and be appropriate to a particular learner group. Hence in this section of the study, the researcher is explaining what aspects of textbook that are usually evaluated by the experts and researchers as well as teachers to seek the best and most appropriate textbook for a particular group.

Cunningsworth (1995) suggested that it is significant to ensure that careful selection is made and that materials selected closely reflect the need of the learners, the aim, methods and values of the teaching program. To carefully select the materials or textbooks, a predictive evaluation can be administered (Ellis, 1997) and some criteria of a textbook should be considered (Allwright, 1981, Cunningsoworth, 1995, Richard, 2001 and Sheldon 1988).

Goal of a textbook seems to be an important criterion to be evaluated. In many evaluation checklists from the experts such as Allwright (1981), Sheldon (1988), Cunningsworth 1988 and Richards (2001) evaluative question regarding the goal of a textbook is always put at the beginning of the checklist. As also suggested by Nunan (1989), in developing teaching materials the objective, aim or goal of a textbook should be generated in the first hand. In addition, the aims of a textbook should

correspond with the aim of the syllabus and curriculum (Cunningsworth, 1995; Nunan, 1989 and McDonough and Shaw 1993:64). However, conflict of goals cannot be avoided since teachers, learners, materials developers and sponsors involve in teaching and learning management (Allwright, 1981). There is always a conflict of teachers, leaners, material developers and sponsor goals found in textbook. Shortly, Allwright (1981) argued that a textbook may contribute to some ways but it cannot determine goals.

The approach of a textbook is also considered to be evaluated in a textbook evaluation process. Cunningsworth (1995) and (Brown, 2001) suggested some questions to evaluate this criterion whether a textbook is flexible to any teaching and learning situation and style. In a slight different opinion to this approach issue by Allwright (1981) was that in materials evaluation assessors or teachers should be able to measure to what extent the management of the activities are maximally profitable. However, he pinned important clause that textbook can and do contribute to the teaching and learning process, including the teaching approach, but cannot possibly cope with many important decision made by the teachers regarding their various situation (Allwrigth, 1981). In conclusion to the discussion about the approach, the researcher agrees to Allwright’s opinion that detailed local decision is clearly beyond the scope of publication (1981). The materials or textbook then can only impose suggestion to the teachers.

Human being loves everything which is beautiful. It is also not too much to say that aesthetically positive experience through the use of attractive illustration and its design can facilitate learners in language acquisition and development (Tomlinson,

2008) since it can arouse the learners’ interest (Rowntree 1997,92). A good material must provide a pleasurable learning experience which can be achieved by practicing learning items in an interesting and novel way (Richards, 2001). This can be done by using attractive illustration and design (Tomlinson, 2008). Besides the design and illustration, a good textbook should help the learners feel at ease (Rowntree 1997). Therefore the sequence and the grading of textbook should be able to make the learners at ease and suitable for the learners (Cunningsworth, 1995). In the same time, Cunningsworth (1995) also believes that a textbook should allow the teachers to undergo various teaching styles. In a simple way, a textbook should be flexible for the teachers.

The next criterion of a textbook to be reviewed is the language content. In this aspect of textbook, three main elements are described. Those are grammar, vocabulary and phonology. Cunningsworth (1995) sees that those three elements are important in term of learning the language. Therefore, in his textbook checklist, he tried to find out whether a textbook provides adequate grammar item for the students to learn autonomously and teachers as the source of grammar knowledge. Besides the grammar item that should be presented in textbook, vocabulary in terms of quantity and range must be adequately provided in a textbook. Another important aspect that should be presented in textbook related to content is the pronunciation. A textbook has to give teachers and students the pronunciation work in terms of individual sounds, word stress and intonation.

Related to the content of textbook, language skills presented in textbook should be carefully paid attention of since referring to what Tomlinson (2008)

suggests in his critical review book that an effective textbook must provide the learners with the experience of extensive listening and/ or extensive reading. His opinion definitely supports the input Hypothesis by Krashen (1981).

The topic presented in a textbook ideally should have enough variety and sufficiently fulfill the learners’ interest. The topic presented in a textbook also should be content-wise enough for the learners’ language level. The social cultural context presented in a textbook is also being considered to the use of English in the real life. A textbook must also consider the social structure such as gender group, ethnic group, occupational group and other group such as disability (Cunningsworth, 1995; Brown, 2001; and Tomlinson, 2008).

Many experts in textbook evaluation field share ideas regarding the criteria of a textbook. They seem explaining and putting the name for each criterion in different way. However, whatever the term they are using for each criterion, actually they are referring to the same ideas. Therefore, in this study the researcher respectively adapted Cunningsworth’s (1995) evaluation checklist which has covered all criteria used in this study. The six aspects of textbook will be evaluated are its aims, approach, design and organization, language content, skills and topic. In this study the researcher came with six aspects a textbook should have. Each aspect should at least meet with one criterion. The criteria used in this study are as follow:

Table 2.1: Criteria of an Effective Textbook

(Tomlinson, 2008)

source of grammar knowledge (Cunningsworth, 1995)

skills needed

Aim The aims of a textbook should correspond with the aim

of the syllabus and curriculum (Cunningsworth, 1995; Nunan, 1989; and McDonough and Shaw 1993:64).

Approach

a. A textbook should be flexible to any teaching and learning situation and style (Cunningsworth, 1995 and Brown, 2001).

b. Material should take into account that learners differ in learning styles (Tomlinson, 2008)

Design and organization

a. attractive illustration and its design can facilitate learners in language acquisition and development b. Therefore the sequence and the grading of textbook

should be able to make the learners at ease and suitable for the learners (Cunningsworth, 1995).

c. a good textbook should help the learners feel at ease (Rowntree,1997 and Tomlinson, 2008).

Language content

a. a textbook should provide adequate grammar item for the students to learn autonomously and teachers as the b. vocabulary in terms of quantity and range must be

adequately provided in a textbook (Cunningsworth, 1995)

c. A textbook has to give teacher and student the pronunciation work in terms of individual sounds, word stress and intonation (Cunningsworth, 1995)

Language

Skills a. Integrated language materials should provide language b. Materials should facilitate learners with adequate and

rich reading text (Tomlinson, 2008)

Topic a. To achieve impact, a textbook should present appealing topics of interest to the target learners. (Tomlinson, 2008)

b. A textbook should present topics which offer the possibility of learning something new. (Tomlinson, 2008)

2. Evaluation

In the second language education, the main purpose of evaluation is ultimately for decision-making based on the informed judgment. That is why enough data gathering is highly important in the evaluation process (Richards, 1996). The information can be gained from the programmer, administrators, teachers, learners or the stakeholders as part of the implementation of a program and can be used as the program development.

a. Definition of Evaluation

As it is mentioned above, the ultimate purpose of evaluation is to collect information systematically in order to indicate the worth or merit of a program or a project then to make a decision. Therefore, Brown (1989: 223) suggests a good working definition to be the systematic collection and analysis of all relevant information necessary to promote the improvement of the curriculum and assess its effectiveness and efficiency, as well as the participants’ attitudes within a context of particular institutions involved. Similar definition proposed as follow, evaluation is a natural activity as intrinsic part of teaching and learning that enables evaluators to make judgments towards the program or curriculum that is done systematically and in principled way (Rea-Dickinds & Germaine, 1992). Obviously, both definitions agree that evaluation is a systematic information gathering to make a judgment. They also mention a program and curriculum in the context of educational evaluation. In the case of educational evaluation, it can be done before the program commences or before. It of course can also be done during the program is being commenced (Brown, 1994). This is supported by Hewings (1996). The scope of evaluation has

moved from a concern with test result to the need to collect information and make judgements about all aspects of the curriculum, from planning to implementation.

In relation with language program, in this context is English program, evaluation may focus on many different aspects. First, it may focus on curriculum design which is to provide insights about the quality of program planning and its organization. Second, the concern can be on syllabus and program content in which it will deal with the relevancy and engagement of the syllabus and program content and the successfulness of the assessment and test procedures. Third, that is classroom process which aims to provide insights about the extent to which a program is being implemented appropriately. Fourth, material of instructions where evaluators may focus on materials of instruction in which provides insights about specific materials are aiding student learning. Fifth, that is the teachers which need to consider main questions such as who conducted the teaching, what was the teachers’ perception of the program, and what did the teachers teach. Sixth, it can be about teacher training which aims to assess whether training teachers have received is adequate. Seventh, the point is about the students in which the research focuses on investigating what the students learn from the program, what their perception toward the program is, and how the students participate in the program. Eighth, the concern is on monitoring of the students’ progress.

One of evaluation process which is categorized based on time frame it is conducted is formative evaluation or in-progress evaluation of student learning. Based on Sanders (1992) and Weir and Roberts (1994), the focus aims of conducting this type of evaluation are on five matters. First, that is learner motivation where

evaluators try to seek insights about the effectiveness of teachers in aiding students to achieve goals and objectives of the school. Second, that is learner motivation where the evaluators try to seek what drives learners to learn. Third, it is about the institution where the evaluators try to what administrative support was provided, what resources were used, what communication networks were employed. Fourth, learning environment is to provide insights about the extent to which students are provided with a responsive environment in terms of their educational needs. Fifth, staff development aims at providing insights about the extent to which the school system provides the staff opportunities to increase their effectiveness. Last but not least, that is about decision making which is to provide insights about how well the school staff- principals, teachers, and others – make decisions that result in learner benefits.

According to the above explanation, it can be concluded that evaluation is a long process conducted to provide information. The information is to improve curriculum, teacher development, students, the effectiveness of teaching and learning process and materials used. All things mentioned are to achieve educational and instructional goals.

b. Type of Evaluation

In general, evaluation in ELT mainly occupies two kinds of evaluation namely macro-evaluation and micro-evaluation. The distinction between two evaluations is that macro-evaluation deals with the administrative and curriculum matters that undergo evaluation for the whole programs. Besides, in micro-evaluation, it only

deals with the break-down of the curricular matters such as teachers, materials and learners (Ellis, 1998).

Weir and Roberts (1994) distinguish the types of evaluation for language program based on two major purposes, program accountability and program development. Accountability refers to the quality of the work of those involved in a program (Richards, 2001). He also makes a point that accountability evaluation usually examines the effect of a program and its significance towards educational system is usually conducted by an independent institution as the external audience of the education system or by the decision maker. In contrast, development-oriented evaluation aims to improve the educational quality of a program or project (Weir and Roberts, 1994). This kind of evaluation is normally done while the program is in progress.

Richards (2001) gives different opinions about purposes of evaluation to what Weir and Roberts suggested. He distinguishes it into three different kinds of evaluation. The first type is formative evaluation. Formative evaluation refers to an evaluation that is undergone as part of the process of the program to seek what is working and what is not and what problems need to be addressed. Since this kind evaluation focuses on ongoing development and improvement of the program, it is usually conducted during the program. An evaluative question related to textbook that typically raised in formative evaluation is how well textbook is being received.

Another type of evaluation is Iluminative. In this kind of evaluation, evaluators try to find out the different aspects in the program and their implementation. This kind of evaluation focuses on the teaching and learning process

that occurs in the program. Therefore, it is rarely to ask about the materials in this kind of evaluation.

The last type suggested by Richard (2001) is summative evaluation. This third type of evaluation is the type with which most teachers and program administrators are familiar. Summative evaluation concerns on whether a program is effective and efficient and to some extent acceptable. Richard points that this kind of research is conducted after the program has been implemented. This kind evaluation is also asking whether the materials work well. Brown (1994), related to the material evaluation, makes an important point that any evaluation can be carried out before the program commences or after and or during the program.

By relying on the above explanation, this present study agrees on what Brown pointed that any evaluation can be carried out before, after or during the program. In this study, the evaluation is conducted during the program. Another point to reconfirm is that this present study undergoes a material evaluation in form of textbook evaluation.

c. Check List of Textbook Evaluation

Related to textbook evaluation, McDonough, Shaw & Masuhara (2013) give two different stages instead of the type of textbook evaluation. The first stage is a brief external evaluation. This includes criteria which gives an overview of the organizational foundation of the book. Cunningsworth (1995) give similar ideas with different term, impressionistic evaluation. In this stage an assessor only does a brief look to textbook to give a brief description about textbook. Following this is an in-

depth internal investigation of textbook, ‘to see how far the materials in question match up to what the author claims as well as to the aims and objectives of a given teaching program’ (McDonough, Shaw, & Masuhara, 2013). In other words, in-depth internal investigation must be done to analyze to which extent result of external investigation match up with internal investigation.

Another criterion is illuminated by McDonough, Shaw & Masuhara (2013). They comprise nine criteria before conducting in-depth internal evaluation. These criteria are usually found at the cover or in the introduction of the book. The criteria or checklist are in a form of questions that will be answered by evaluators.

In parallel with Shaw (2013) and Rea-Dickins (1992) suggest a checklist in a form of preliminary questions. Then, evaluators answer the questions before analyzing which teaching and learning materials are suitable. The questions are as follows.

Rea-Dickins (1992) provide these preliminary questions above. Though in the end of the procedure, they point important consideration that since teaching and learning materials are sensitive to the language process, evaluation criteria should relate not only to the aims and contents of language learning but also to the procedures of working with texts and performing tasks in the classroom. That is why they give credits to Cunningsworth’s work since as it can be seen in appendix, the criteria cover Germain have pointed. To compare to the criteria suggested by McDonough, Shaw & Masuhara (2013), what Cunningsworth marked gives more

detailed and comprehensive questions related to the methodology in which Rea- Dickins (1992) suggested.

Table 2.2 Checklist Suggested by Rea-Dickins & Germaine

A more recent study held by J. Mukundan, (2011) in developing English language textbook evaluation checklist was done in a focus group discussion. Six participants joined the developers to improve the items of checklist generated by Cunningsworth’s evaluation checklist (1995) in reference to their clarity and inclusiveness. In this study, researchers successfully developed an evaluation checklist for ELT textbook. They added, omitted, and revised the Cunningsworth’s evaluation checklist.

In relation to Indonesian context and considering the above explanation, this present study sees that Cunningsworth’s evaluation check list will work well to Criteria 1: What do materials mean for you?

1. Do you refer exclusively to textbooks, or do you include teachers’ guides, teaching manuals, supplementary units, readers, audio and visual materials, etc?

2. Do you make a distinction between materials designed specifically for first and second language teaching, and also between those targeted specifically for use in school and materials that are non-pedagogic but authentic?

3. Do you include materials produced by the teachers and the learners? Criteria 2: What are the roles of material within your teaching and learning context?

1. What role(s) are they expected to play? 2. What role(s) are they expected to achieve? Criteria 3: How are the materials to be used?

1. Are they to be used as the sole source and resource for teaching? 2. Are they one of several available resources?

evaluate the examined book. Therefore, this present study will be spelt out the evaluation criteria by Cunningsworth in form of questionnaire to be given to the teachers to evaluate the English textbook of seventh grade junior high school in Indonesia.

3. Review on Indonesia National Curriculum 2013

A new curriculum has been implemented in Indonesia since June 2013. This new curriculum is called curriculum 2013 (C13) which is based on the Regulation of Ministry of Education of Indonesia number 22, 2006. C13 has been implemented in all around Indonesia for all levels of education from Elementary School (grade 1-6), Junior High School (grade 7-9), and High School (grade 10-12). In the first step of the implementation, not all grades were using the new curriculum. Only grade 1, 4, 7 and 9 run the C13. This new curriculum was also not implemented in all school in Indonesia. For the first stage of the implementation, Ministry of Education only pointed some schools to become the pioneering schools for this curriculum implementation.

To give an illustration on the C13, the following explanation is a brief description on the core competence and base competence. There are four core competences for English subject in K 13. All of them are related to religious and moral conduct and conceptualized in the base competence.

To make the explanation crystal clear, samples of a base competence stated in C13 (Core and Basic Competence of Curriculum 2013) in the following six points. First, it is composing oral text to express and respond greeting, leave taking,

thanking, and apologizing correct language features in context. Second, it is composing oral and written text to express and respond introducing oneself in short and simple way by paying attention to social function, text structure, and language features correctly and contextually. Third, it is composing oral and written text to mention names of days, month, and time in days, time in numeric form, dates, and year by using the language features correctly and contextually. Fourth, it is composing oral and written text to mention names of animals, things, and public places found in the real life by using the language features correctly and contextually. Fifth, it is composing oral and written text to make names in label and list down things by using language features correctly and contextually. Sixth, it is composing oral and written text to describe personal characteristics, animal and things by using the language features correctly and contextually.

As stated above, it is very obvious that current curriculum deploy three main aspects reflected in the core competence. Those are attitude, skills and knowledge. In addition to the explained aspects, C13 performs the thematic and integrated concept of syllabus which can be seen in the official website of education and cultures ministry.

Taken from the preface of the teacher’s book of English textbook, it is said that C13 gives a significant change in terms of teaching and learning process. Previous curriculum enables teacher to deliver the subject or knowledge. In contrary to the prior curriculum indeed, C13 enables the students to share knowledge among them by the guidance of the teacher. In pedagogical term, the teaching and learning is a two-way teaching and learning process. Though, in the real implementation some

teachers in Indonesia have implemented a two-way teaching process instead of one way. However, such approach, had not been realized until the C13 was issued. As a consequence, by applying this curriculum, students are intended to be more active in doing observation, thinking, questioning, and communicating as well as presenting what they have earned from the knowledge receiving process (http://kemdikbud.go.id/kemdikbud/uji-publik-kurikulum-2013-1).

Four standards that happen to be changed in curriculum 2013 from curriculum 2010 covers graduate competence, process content, and standard of learning evaluation. Related to learner evaluation/assessment, there are four criteria of learner assessment: 1) level of achievement of the use of social function text; 2) level of completion and arrangement of text structure; 3) accuracy of language features such as grammar, vocabulary, pronunciation, word stressing, intonation, and spelling; and 4) level of appropriateness of writing and delivery format. Four methods of learner assessment are provided in this curriculum to meet with the core and base competence. The four methods of learner assessment are assessing student performance, student observation, student portfolio, reflection and pair assessment.

In conclusion to the explanation above, it is highly required to evaluate textbook of new curriculum 2013. It is since many new constructs applied in the curriculum that become valuable source to adjust the materials used. In addition, significant shifted paradigm such as the teaching methodology suggested in the curriculum.

4. Review of Previous Studies

In this section of the study, two previous studies will be reviewed briefly to give depiction and broader knowledge in doing the research. The review of previous studies focus on the methodology of each research to compare one to another in order to get depiction of what and how the researcher conduct an evaluation especially textbook evaluation.

A thesis by Alamri (2008) entitled An Evaluation of Sixth Grade English

Language Textbook for Saudi Boy’s School was conducted to evaluate a new

textbook. Conducted in King Saud University, Riyadh, this research was a survey research which used a set of questionnaire to elicit the perspective of 93 English language teachers and 11 supervisors in Riyadh Educational Zone about textbook in question. The questionnaire comprised 64 groups under 12 main categories. For the questionnaire design, he adopted four different evaluation checklists from four different experts and researchers. As the result of the adoption of those four checklists, (a) general appearance, (b) design and illustration, (c) accompanying materials, (d) objectives, (e) topic contents, (f) language contents, (g) social and culture contexts, (h) language skills, (i) teachability, (j) flexibility, (k) teaching methods, and (l) practice and testing are used as the categories in the questionnaire. After getting the data, the researcher analyzed them through descriptive statistic. The findings showed out of 64 items in the questionnaire, only 13 items had means less than 2.50. The category that had the highest mean was the one on learning components. Meanwhile, the category that had the lowest mean was the one on the teaching methods. Based on the findings, it was also found that there were no

significant differences between the means of two populations of the study, teachers and supervisors, except on the flexibility of textbook. The different natures of their jobs could be a possibly reason for this difference.

Another example of impressionistic evaluation using an empirical qualitative approach is the one done by Nemati (2009) in India. This research involved 26 Pre- University teachers. Because this research also focused on vocabulary factors, besides a questionnaire, the data were also taken from vocabulary response. The questionnaire that was distributed to the respondents then followed by an interview for each respondent to triangulate the data, it was arranged based on textbook evaluation checklist. She adopted several checklists from the authors such as Cunningsworth, (1995) and Ur (1996). After studying the checklists from the authors, she decided to select features which are in common. The criteria used in the questionnaire are as follow (1) logistical factors, two questions about the availability and price of the book, (2) pedagogical factors, or methodology, with two subparts of skills and contents, (3) human factors, including teacher friendly and learner friendly factors, (4) vocabulary factors, entail questions regarding vocabulary learning strategies. From the research result, teachers’ belief toward textbook was revealed based on the criteria. In addition, suggestions to improve textbook were also obtained.

The other example of textbook evaluation research discussed in this section is conducted by Zamanian (2015). Aiming at evaluating ‘Learning to Read English for Pre-University Students’ used in Iranian high schools, the researchers distributed a questionnaire to 75 high school English as a Foreign Language teachers. In designing

the questionnaire, Litz’s checklist (2000) was adopted. After the data were collected, those are analyzed based on a Likert scale. Based on the research results, it was known that textbook was quite far from teachers’ standard. It happened because textbook ignored listening and writing skills. Moreover, it did not follow a communicative approach in teaching different skills and language components.

Similar to those three research, this study is conducted empirically. After being used to teach junior high school students in Indonesia, textbook entitled ‘When English Rings the Bells’ is evaluated. Data are going to be collected by distributing questionnaire to evaluate whether the material is in line with what teachers expect. Similarly, this research also captured textbook evaluation process done by Indonesian teachers after they used textbook in a classroom context. Framework used for this research was Cunningsworth’s (1995). Using the framework, the research focuses on the aim, the approach applied, the language skills presented in textbook, the design and organization, the content of the language, and the topic.

B. Theoretical Framework

From the literature reviewed above, the theoretical framework can be drawn as follows. In conducting this evaluation research, the criteria of a book evaluation are the most important and significant aspects to be paid attention of in resulting a reliable research. The criteria established in this research are selectively adapted from

the Tomlinson’s, Bworn’s, Cunningsworth’s and Nunan’s book evaluation criteria.

Furtherly, the criteria are conceptualized into a research instrument in form of questionnaire. The instrument is used to measure textbook by the practitioners in this

case are English secondary school teachers. Each item of questionnaire is also considering what is currently formulated in C13. Therefore, the concept of textbook evaluation by Tomlinson (2008), Brown (2001), Cunningsworth (1995) and Nunan (1989) contributes largely in drawing the concept of the criteria applied by the researcher in this evaluation study.

The framework is developed from some underlying theories about evaluation in English language teaching (ELT) and textbook. Those encompass the framework on textbook evaluation and textbook development. From the literature reviewed above, the framework can be theoretically drawn as follows.

Figure 2.1 The Theoretical Framework

Cunningsworth (1995)

Nunan (1989)

Textbook Evaluation

Curriculum 2013 (C13)

Checklist

Aim Approac Language Design and

Organization

Language

Content Topic Brown (2001)

CHAPTER III RESEARCH METHOD

This chapter discusses the research methodology which was employed in an attempt to answer the research question empirically. In the relation to the path of data collection, this chapter discuss research method applied. After that, research setting and participants, research instrument, data gathering and analysis technique are scrutinized in this chapter.

A. Research Method

This research was under an umbrella of applied research that enabled the researcher to evaluate a program, in this case, it was to evaluate a textbook. It was adopted in this research since it embraces a set of process in evaluating the quality of textbook. Evaluation research study was designed to find out to what extent a new program, policy, textbook, a new way of doing something and so forth, has met the criteria mentioned in the second chapter of this report.

Evaluation research can be undergone quantitatively or qualitatively. In favor to conduct quantitative research, researcher may use a survey research. To conduct evaluation research using qualitative data, the researcher must use the field research (Neuman 2006). In survey research Neuman suggests to researchers to collect data from a sample or could be smaller group of selected people. He suggests around 150 respondents to generalize result to a larger group of 5,000 people.

B. Research Respondents

This research targeted junior high school teachers who implemented curriculum 2013 as the respondents. As it has been described in the previous chapter, the implementation of curriculum 2013 did not cover all junior high schools throughout Indonesia. Each region and province in Indonesia has pointed selected school to implement and use textbook of curriculum 2013 in odd semester. In odd semester 2014, more schools implemented C 13.

Since it was survey research, it was conducted in 10 junior high schools in Special Province of Yogyakarta. These schools were pointed to implement the C13 in Yogyakarta. Special Province of Yogyakarta has five districts including the city of Yogyakarta. The researcher picked up two schools for each district as sampling. From each school, the researcher took two teachers to be participated in the survey research. The teachers vary in terms of the length of teaching experience. The very least year of experience was two years. While the most experienced teacher had 35 years of teaching experience.

To sum up, this research was supposed to involve twenty teachers teaching in different junior high schools in the province of Yogyakarta. However, due to some circumstances, there were two teachers who refused to join the survey and one teacher filled out the questionnaire with the same answers for every item. As result, this respondent is considered failed. Therefore, by the end of the data gathering, there were only 13 respondents whose data was considered as valid.

C. Research Instrument

Two research instruments were utilized in order to attain feedback from this study. First, it was an evaluative questionnaire adopted from Cunningsworth’s checklist. Second, to triangulate the data taken, an open ended questionnaire was distributed then.

For the evaluative questionnaire, the items were established by adopting Cunningsworth’s checklist which focused on the aim, approach, language skills, design or organization, language content, and topic used. There were seventy-one items prescribed in the questionnaire which these items were measured using Likert scale. In order to reduce the number of invalidity of the survey result, the researcher only used 4 scales instead of 5 scales namely scale 1 for strongly disagree, 2 for disagree, 3 for agree, and 4 for strongly agree to the item.

Beside closed item which was prescribed in form of statement, the questionnaire also encompassed open questions for the respondents to answer. This was aimed to obtain the triangulation data. Besides using open questions, the researcher also used textbook close reading to get the factual evidence from textbook to crosscheck whether the survey results were confirmed.

D. Data Gathering Techniques

In gathering the data, at first, the researcher made an appointment with twenty teachers. However, like what it has been stated above that due to some circumstances, two teachers refused to join the survey and one teacher filled out the questionnaire with the same answers for every item. As result, only thirteen teachers who kindly agreed to the appointment.

The data gathering was conducted by the end of the even semester 2012/2013 Indonesian academic year in June 2013. The researcher started to meet respondents from school to school. So, it took a month to take the data. From thirteen sets of questionnaire were distributed, amazingly all of them returned. However, there was one questionnaire was returned remain unfilled by a respondent.

E. Data Analysis

The very first step of data gathering was distributing questionnaire to the respondents. The respondents were given a week to fill out the questionnaire completely. The researcher came over to each respondent to explain the procedure in filling out the questionnaire although the researcher had written a clear instruction in the questionnaire. This aims to make sure that the respondents were able to fill out the questionnaire correctly. Then, after a week the questionnaires were collected. After the questionnaire were returned to researcher, the results then were put into the tally system as seen in the table below.

After putting all the result in tables, the researcher calculated the data to get the MEAN for each item of the questionnaire. In addition, the standard deviation was also gained to find out whether each item was valid. In the table presented in this survey research report, it comprises the percentage of all fourth scales (1 = strongly disagree, 2= disagree, 3 = agree, and 4= strongly agree), the Mean, and the standard deviation.

To read the table, it can be inferred from the mean of the item. If the mean more than 2.5, it means that respondents or teachers agree to the statement. In

contrary, if the mean of an item is showing less than 2.5, it means the item is not agreed by the respondents. Therefore, the researcher presented the data for each item in two digits of decimal. It aims to see the difference one item to another more precisely.

After completing the tally sheet, the researcher then created tables based on the

result of the survey. The table helps the researcher to describe the result of the survey easily.

Hence, later on the researcher will be able to analyze then discuss the result based on the

diagram and the answers of the open questions.

After obtaining the result survey, the data inform of score are then presented into table in order to make the interpretation easier to elaborate. The researcher later describes the survey result elaborately in chapter IV section one. And then, in the second section of chapter IV the researcher is discussing the survey results and relating them with the theory in learning and material development. This discussion is aimed to get a comprehensible conclusion and decision toward textbook since the main goal of evaluation is to make decision based on the data gathering.

Although the researcher had time constrain, the researcher successfully interviewed

respondents to triangulate the data. In addition, the researcher also happened to give them

several open questions on the questionnaire. They had to write the open questions on the

questionnaire sheet. While the researcher collecting the questionnaire, the researcher tried to

confirm all the answers written on the sheet by. By doing this, the researcher was able to

understand the answers from the respondents comprehendingly. Then, all the answers of the open questions were written in a table and categorized. Hence, these data are ready to use in

CHAPTER IV

RESEARCH FINDINGS AND DICUSSION

In this chapter, it is described the survey result begin with the general findings. The following elaborative description is explaining the survey result of each aspect. Six aspects of textbook being evaluated are its aim, approach, design and organization, language content, skills and topic.

A. General Findings

In general, the evaluated textbook remained credible as a textbook used in English language classroom for secondary school. Despite the debate regarding the C13 and its textbook, the survey result proved everything regarding the textbook. The result for each aspect of the evaluated textbook shows that teachers believe this textbook is still credible and reliable to be used in the English language classroom for secondary school students.

Later in this section, a table showing the overall result of the survey is displayed. The Mean of each aspect is telling the average of the respondents’ response toward the aspects. Each aspect was evaluated by several statements formulated in the questionnaire. Therefore, the mean represents how each aspect meets the criteria based on the teachers’ or respondents’ perspective. Since the scale employed in the questionnaire is 1 to 4, the Mean which is more than 2.5 means that the aspect meets the criteria. To see to what extent the aspect meet the criteria, it can be seen how far the Mean overly reach the limit (2.5).

Teachers believed that the aims of the textbook were definitely in line with the curriculum and the program objectives. As it is seen from the table, the mean for this aspect shows 2.97 and is ranked as the highest mean among other five aspects to be evaluated. The second highest mean of the aspect is 2.76 for the topic of the textbook. This unavoidably shows that teachers consider the topics of the textbook are appropriate for Indonesia context as well as representing different group and ethnicity. As though aims and topic of the textbook, the approach used in the textbook happens to be one of the aspect proving that this textbook remains reliable. This is shown in the table which 2.61 happens to be the mean for approach of the textbook. The other three aspects namely the design and organization, content, and

language skill have almost the same result for each.

Table 4.1: Overall Survey Result

Aspect Mean

No

2.61 2. Approach

3. Design and organization

2.52 4. Content

5. Language skill

2.76 6. Topic

However, teachers and material developer are obliged to see further and deeper on each aspect. Therefore, addition to the material as well as revision to the existing material may be necessary to be done in favor to the improvement of the textbook. Hence teacher may find the additional material focus on the specific

2.32 2.55 1. Aims 2.97

component of the textbook and the material developer may revise the textbook on that specific component as well. Accordingly, the next subchapter will elaborate deeper on the survey result of each item of the aspects.

B. Detail Findings

In this part, all criteria will be elaborated in detail by giving the real survey data and related theory as well as data taken from textbook to enrich findings. The researcher will also add small part of discussion related to certain item of the criteria. In the next section, the researcher will give general discussion about the evaluated textbook. In this sub-section, the detailed findings will start from the very first criterion.

1. The Aims

The first analysis goes to the first aspect, the aims of textbook. As it is seen in the table, there are four items or statements to measure the agreement of the aims of textbook. The first item is rated at 3.11 by the respondents. This mean shows that it is higher than 2 and even higher than 3. This means that teachers are mostly agree on the statement in item 1. Teachers saw that the aims of this textbook closely correspond with the aim of the curriculum. The respondents rated the second item the

aims of textbook correspond closely with the needs of learner at 3 for the mean. As

though the first item, the second item was agreed by the respondents. In conclusion, almost all respondents saw that the aim of this textbook has no issue and think that the aim of textbook had been in line with the curriculum and learner needs. See the table for the detail.

book is not so give a challenge our area. their teacher. 5 14 Based on the

statement in the coursebook, the approach is text-based learning. Various approaches And communicative approach

Yes. Because a text- based approach provides a very wide space for the teacher and students to vary their class with any creative method.

This coursebook is too simple. In other words, the deep and doesn’t to the students. 6 35 Scientific

approach

Scientific approach

No, too simple. There are no task for exposure.

No text. So the students have no chance to observe. 7 2 Scientific

approach

Scientific approach

- It’s used some

text so student can practice their reading

(pronunciation). But using some pictures can make student explore more information from theirself.

8 20 The curriculum suggests that all subjects are taught based in scientific approach i.e. observing, questioning, exploring, associating, and communication. This time being, I am still trying to find the link between this approach and language learning approach I’ve

So far, based on KTsP I use genre-based approach to language teaching. This approach is used to teach year 8 and 9. For year 7 I am suing the Scientific approach as recommended by the curriculum.

Quite appropriate as far as the teachers are creative enough to create students tasks by themselves, because the

coursebook doesn’t display any clear steps or tasks for students.

There are no clear tasks designed to develop which skills. This book presents mostly pictures and simple expressions or sentences with no clear purposes until the teachers refer themselves to the teacher’s book.

the coursebook

of the not good. known.

9 11 Scientific learning

Both Scientific learning and three phase technique

Not really. Sometimes my students still need to be guided in understanding the material. For example teacher must write the explanation on the board, etc.

It is lack of texts.

10 17 The book takes a text-based learning approach.

Communicative and text-based approach.

Yes it is. The text- based approach provides the teaches to use various methods.

The materials is no so deep and doesn’t give more challenge. 11 10 The coursebook

use Speech approach

I think, we will be better if we use easy approach (like pictures, CD and video)

No, it is not.

Because many students don’t know the

knowledge of English especially the student that have low

understanding in English when they are in SD.

The coursebook doesn’t complete The materials of are limited. The arrangement coursebook is

12 10 Discovery learning, project based learning, communicative learning (CLT), role play, CTL teaching.

Discovery learning for reading and writing.

Communicative learning for speaking.

No, because the approach to language learning in general appropriate of the material.

I think, the coursebook of 2013 curriculum is more colorful picture so it is suitable for speaking. But in teaching reading and writing is not suitable. So my problems are in teaching reading and writing if used this coursebook of the ring bells. So I used the others book to teach

example of enough.

teacher has to resources to material. experimenting, and

networking) the

reading and writing.

13 16 Scientific approach

Scientific approach

Yes, because it is not only transfer

knowledge from teacher to student but student can see , ask and analize what teacher present.

Limited materials There are not conversation

14 19 Scientific approach

Scientific approach

Yes it is but not always. It depends on the materials.

There is no grammar focus. There is no listening and speaking. 15 15 Scientific

approach Communicative approach

I use active learning

approach. I try to make the

students active in learning, and the most important thing that they can apply it in their real life.

Sometimes I also use co-operative and collaborative learning.

I think it is appropriate but sometimes it depends on the topic of learning.

Using 5 steps in learning (observing, questioning,

associating, communicating/ students can learn more active and enjoy the learning.

The book covers minimal

material, so the find other develop the

16 16 Scientific approach

Scientific approach

Yes, sesuai. Membuat anak lebih creative.

Masih ada beberapa kekurangan/ kelemahan diantara: Skill listening tidak ada Skill reading kurang

17 10 Learning based on text. Speech or written by an English

language as a communication language

Contextual language teaching

Yes, it is. Because students can

comprehend easily if in language learning use learning by texts

It is very simple because our students have had the materials from primary school.

Appendix 6: Translated transcript of the interview Interview 1

Researcher: What is your comment towards English textbook for grade 7 entitled When English Rings the Bell?

Respondent 1: Oh my God, this textbook is definitely under standard. I am really sure that this book was made in a rush. A very noticeable part is that many typos in this book. And it is too easy for my students, too many pictures but lack of tasks and texts. The government really didn’t pay attention to it, I think. Really… I can’t believe it. At the end, as usual… teachers must work harder to find more materials which are suitable for the students. There are too many problems in this book AND the curriculum itself.”

Researcher: What is wrong with the curriculum?

Respondent 1: Well, the major problem is the approach. In the curriculum it is clearly stated that teachers must apply the scientific approach. However, I never heard of such approach in language teaching and learning. Neither have other teachers. We are in the big confusion because we are mandated to apply what the curriculum says but in other hand we ARE aware that such approach doesn’t exist in language teaching and learning.

Secondly, one of the scoring aspects for final examination is way too much. How can we measure the attitude through English language learning process? Yes we are equipped with teacher’s guide. However, the implementation is way more difficult than what it is prescribed in the teacher’s manual. Too many aspects that are not correlated to language learning need to be measured. Teachers are overwhelmed.

Researcher: What does this book lack?

Respondent 1: Hmm,, obviously, this book lacks reading texts and listening tasks. Do you agree, mas? Right? If you take a look at it closely, this book is lacked of reading texts. And there is no listening section. So teachers must prepare them all. Researcher: So do you think that this book doesn’t cover all skills?

Respondent 1: Definitely, speaking and writing are well covered in this book. But the sample of the conversation for speaking task is too short and too simple. It is not challenging. This book doesn’t cover listening and reading skill pretty well. Researcher: Do you think this book has covered all the student needs?

Respondent 1: Absolutely no. You know, this book is too easy and students need more texts, need more vocabulary. Look, even there is no grammar in it.

Researcher: Sorry to interrupt, so you think that grammar is important to be taught to the students?

Respondent 1: Well, not directly. But at least students can refer to the book for grammar if they find any difficulties when they study at home by themselves. Researcher: Sorry, so are you saying that this book does not promote auto learning or independent learning?

Respondent 1: Hmm… yes, sort of. Students cannot learn by themselves. They need teachers or instructors when they study using this book.

Researcher: So what do you do responding this situation?

Respondent: I do anything possible to do for my students such as finding more relevant materials. I mostly find more texts to support my teaching. And to make more challenging I change a little bit the tasks in textbook and sometimes I

replace it with other different tasks because, you know, the exercises in the book are too easy for my students.

Interview 2

Researcher: What is your comment towards English textbook for grade 7 entitled When English Rings the Bell?

Respondent 2: [release breath] This book is good. It is colorful. Many pictures support the activity. But sometimes I feel it is too many. Hmm … [opening textbook then look at it briefly] oh ya… there is no listening section I guess and the text is very limited though I add with my own texts. I use previous textbook and other textbook to complete When English Rings the Bell.

Researcher: So what is the strength of this textbook?

Respondent 2: colorful, many pictures and … look at this picture [pointing to a page of textbook] this represents Indonesia. So Indonesia, in my opinion. This is good. The topic is varied.

Researcher: And what do you of the lack of this textbook?

Respondent 2: I am a little bit confused to answer this. Well, this book doesn’t have listening section. It is not good because teachers then must find additional material for listening. It is bother us as teacher [laughing]. And what else mas? [laughing]

Researcher: What about the approach, Sir?

Respondent 2: Ah… that one. It is good. Sometimes the Scientific Approach (SA) can be applied in my class but sometimes cannot. And I think I never heard of scientific approach in English teaching, is it? It’s been years since I graduated from my university. I forgot many things about theory of teaching English language. [laughing]

Researcher: So do you think the SA is suitable for your class?

Respondent 2: [sigh] sometimes [smiling]. I think there isn’t such approach in English teaching. But since the government told us to apply it in our class, so I must apply it.

Researcher: So do you apply this approach completely? Respondent 2: Sometimes. So I modify the approach sometimes. Researcher: So do you use textbook every time you teach?

Respondent 2: Not all the time. But most of the time, I use textbook as the guide. So when I feel I need to add materials, then I will find it because this textbook is too easy for many of my students.

Researcher: Do you have anything else to say about this textbook? Respondent 2: Yes, this book still needs to be revised. [smile]