Book Evaluation Criteria Theoretical Description
17
predictive and retrospective evaluation. Predictive evaluation is an evaluation done in order to help the teachers select a textbook. Whereas, retrospective evaluation
done in order to help the teachers identify the strength and weakness of the textbook already used.
To be more specific, together with Masuhara, McDonough and Shaw examine materials of a good book in three stages, namely external, internal, and
overall evaluation McDonough, Shaw, Masuhara, 2013, p. 53. The external evaluation relates to the overview of the materials from the outside. It evaluates the
cover, introduction, and the table of contents. Meanwhile, internal evaluation performs “in-depth” investigation into the materials. Furthermore, the overall
evaluation inspects the workbook in a general use of the materials relates to the syllabus. Byrd 2001 defines three criteria to evaluate books. The first is the
suitability between the curriculum and the materials. It means that the workbooks should match with the learning objectives or goals of the learning process. The
second is the suitability between the students and the needs. It means that the workbooks should fit with the students’ need. The third is the suitability between
the teachers and the materials. It means that the workbooks can be used effectively by the teachers as cited in Ozdemir, 2007.
Tomlinson 2003, p. 16 proposes fourteen criteria that should be measured in the material evaluation process. Those criteria are described as follows.
18
a. The appeal of the materials This relates to how the materials attract students’ attention. It is about the
way the materials being packed. Since an interesting package will encourage the students to learn Tomlinson, 2003.
b. The credibility of the materials to the students, teachers, and administrators This relates to whether or not the materials can be trusted. The materials will
be used by teachers and students in the learning process, therefore it will give an impact for both of them. If the materials cause a bad impact, the materials cannot
be trusted Tomlinson, 2003. c. The validity of the materials
This relates to whether the materials are valid or not. The materials are acceptable for the learning process or not Tomlinson, 2003.
d. The reliability of the materials This indicates whether the materials are reliable or not. Since the materials
will give an impact to language students and teachers in the learning process Tomlinson, 2003.
e. The ability of the materials to interest the students and the teachers This relates to the capability of the materials to get the attention of both the
teachers and the students in the learning process. If both the teachers and the students are interested in the materials, the learning process will go well
Tomlinson, 2003.
19
f. The ability of the materials to motivate the students This relates to the capability of the materials to stimulate the eagerness of
the students to learn. The materials should be one of the students’ motivations to
learn Tomlinson, 2003. g. The value of the materials in terms of short-term learning
This relates to the effectiveness of the materials in facilitating short-term learning. The materials contribute to the learning process at the time of the materials
being discussed or not Tomlinson, 2003. h. The value of the materials in terms of long-term learning
This relates to the effects caused by the usage of the materials in long-term learning. A good material will last for quite a long time, even when the materials
are no longer being used Tomlinson, 2003. i
. The students’ perceptions of the value of the materials This is about the way the students think and feel about the materials. This is
about whether or not the materials help them in the learning process Tomlinson, 2003.
j . The teachers’ perception of the value of the materials
This is how the teachers think about the value of the materials, whether the materials are useful or not. Furthermore, it indicates whether the materials support
the learning process or not Tomlinson, 2003. k. The assistance given to the teachers in terms of preparation, delivery and
assessment
20
This relates to the supporting aids which are provided that can help the teachers. Adequate assistance will support the teachers in the learning process
Tomlinson, 2003. l. The flexibility of the materials
This relates to how the materials can be used. This examines whether or not the materials can be used for different students with different ability. Flexible
materials can be used in large scope not only in one school Tomlinson, 2003. m. The contribution made by the materials to teacher development
This relates to the effect or consequences of the materials to the teachers. It indicates whether the materials can help the teachers develop their competence or
not Tomlinson, 2003. n. The match with administrative requirements
This relates to the arrangement which is suitable with the existing standard. This examines whether or not the materials match the administrative standard.
These fourteen criteria are the elaboration of language assessment theory Tomlinson, 2003.
Sheldon 1988, as cited in Awasthi, 2006 states that a textbook evaluation is “fundamentally a subjective, rule of thumb activity, and that no neat formula,
grid, or system will ever provide a definitive yardstick p. 245 .” Likewise, as
suggested by Cunningsworth 1984, the evaluation process involves professional judgment in every stage. Therefore, personal and professional judgments are
involved in a material evaluation process. The materials are evaluated based on some principles and guidelines which will be formulated in the form of checklist
21
criteria. Moreover, Robinson 1991 lists a number of tools used to carry out evaluation, such as questionnaires, checklists, rating scales, interviews,
observations, and records. However, one of the common methods to evaluate English language teaching material is using a checklist. As stated by Lodico,
Spaulding, and Voegetle 2006, p. 113, even though there are pre-established and standardized checklist, researchers tend to adjust the checklist to the exact setting
and participants in order to gather the data to answer the research questions. Cunningsworth 1995 as cited in Richards, 2001 suggests the use of
different checklists to cover all aspects of the book. The checklist covers some aspects, such as the claims made by the book, the types of the materials evaluation
and the purposes of the materials evaluation. These three aspects are covered within eight areas, namely aims and approaches, design and organization, language
content, skills, topics, methodology, teacher’s books, and practical considerations. Those areas are examined in the form of a series questions which will cover all of
the aspects. Another criteria propose by Supriadi 2000, he suggests four aspects to evaluate books in Indonesia. Those aspects are: content, language, physical
appearance, and national security. The content aspect is about the suitability of the materials with the curriculum, complete materials --that relates to term, symbol and
notation, exampleillustration--, systematic materials, attractive presentation, understandable materials, and ability to encourage the students to learn. Afterward,
the language aspects are about the paragraph, grammar, diction, terminology, and spelling. The next is physical appearance or graphics aspects that include:
typography, layout, printing quality, binding quality, illustration, cover, book size,
22
and the type of the paper. The last is a national security aspect which has an association with law, social and cultural context, ethics, and government policy.
Littlejohn 1998, as cited in Ozdemir, 2007 offers one of the checklists that is used to examine the purpose of the current study. He makes his checklist
according to the three levels of analysis of the book: a. What is there?
The first level consists of items that pursue information about the physical properties of the book. The psychical properties comprise the publication and
design that relate to the layout, durability, print, availability, and illustration. b. What is required of users?
The second level has a detailed task sheet that enables gathering information about the tasks in the book. This level has an intention to analyze the language
learning activities in the book. c. What is implied?
While the third level consists of the items that enable the researchers gathering information about the approach, philosophy, and aims of the book.
Furthermore, Littlejohn 1998 adds that his levels of analysis move from the objective to a more subjective evaluation. The first level is the objective evaluation
while the third is the subjective evaluation. Another criteria defined by Brown 2001, he offers a checklist that consists
of items about the goals and approach of the book, background, the treatment of the skills, content, quality of practice material, sequencing, vocabulary, sociolinguistic
23
factors, and format –that includes the physical properties, accompanying materials
and the teacher’s guide--. Moreover, Sheldon 1998 affirms that evaluation checklists should include some criteria related to the characteristics of a textbook
such as, layout, organization, methodology, aims, and the degree to which a set of materials is not only teachable, but also fits the needs of the teachers’ approaches.
While, Hedges 2000 proposes a two- stage process for evaluating a book. “The
first stage is to assess the content of a book in relation to its professed aims. The second is to assess the book against the needs and context of the intended learner”
p. 357. She suggests a checklist that is divided into five main categories, namely the view of language, the view of language learning, the learner, the view of
education and the environment of learning. In a similar study, Ali 1983 conducts an evaluation on an English language
textbook that is used for the second grade boy’s intermediate level in Saudi Arabia. The evaluation is done through some checklists and questionnaires for English
language teachers and English language supervisors. The categories of the questionnaire are: the introduction of the course, the course’s subject matter, aids,
exercises and activities, the teacher’s manual, the course book’s layout and the physical make up. He also provides some recommendations and modifications at
the end of his study: a. The importance of using more colorful and attractive teaching aids.
b . The pupil’s book should present more interesting and age-appropriate topics.
c . The teacher’s manual should provide teachers with alternative ways for teaching
every lesson.
24
Tomlinson, Dat and Masuhara 2001 use a list of 133 course evaluation criteria to evaluate eight current adult courses published by three leading publishers
in the United Kingdom as cited in Alamri, 2008, p. 35-36. The textbooks are: Language in Use and True to Life by Cambridge University Press, Cutting Edge
and Wavelength by Pearson Longman, Inside Out and Reward by Macmillan Heinemann Press. The checklist used is divided into two main headings: overall
criteria that relate to publisher’s claims, flexibility, syllabus, pedagogic approach,
topic contents, voice, instructions and teachability and specific criteria that relates to the appearance, design, illustration and reading text.
Hapsari 2011 uses a list of 51 criteria to evaluate three workbooks for seventh grade students. The first workbook is English Supplementary Materials
written by English Team of Musyawarah Guru Mata Pelajaran Yogyakarta and it is used in SMP Negeri 9 Yogyakarta. The second workbook names Global written
by Endang Triningsih, Tunijo, Munawir, and H. Ismanto. The second workbook is used in SMP Negeri 2 Yogyakarta. While the third workbook names Cerah
Workbook by CV Teguh Karya and it is used in SMP Joanness Bosco. She offers a checklist that consists of ten criteria, namely general appearance, design and
illustration, objectives, topic contents, language contents, social and cultural context, language skills, teachability, flexibility, and practice and testing. Her
research shows that in general, none of the three workbooks accomplish the standard of a good workbook. The English Supplementary Materials Workbook is
considered as a fair workbook and Global Workbook is considered as a sufficient workbook. Meanwhile, Cerah Workbook is considered as a fair workbook.
25
In a similar study, Kurniawati 2013 evaluates Excellent Workbook for elementary school grade one. This workbook is published by CV. Adi Nugraha. She
uses a list of 49 criteria. The checklist is divided into four aspects, namely physical appearance
–that includes the general appearance, design, illustration and organization--, content
–that includes the objectives, skills, topics, and tasks--, language content and also instruction. As the result of this research, Excellent
Workbook fulfilled criteria on topics and general appearance. Furthermore, she adds that formative evaluation can be used in order to improve the unfulfilled criteria.