Materials Evaluation Main Field Testing

159 an exercise or created a dialogue. The difficulty of several expressions was because of the infrequent and unfamiliar use of them. It can be seen from the following interview part. [Unit 3] “The dialogue is getting longer and we should think about the answer for our improvisation. Something like that. This maybe needs two meetings to make us understand.” On translation; M Ie 102 – Ie 103; page 291 Secondly, the content that needs to be revised was the input that should be given in unit 4. Unit 4 dealt with wedding brochure, in which in the warming up activity, the students were asked to make a sentence represented in the wedding brochure. They realized that they found difficulty in saying a sentence that represented the wedding brochure. Therefore, they suggested the teacher to provide an example of sentence. The suggestion can be seen from the following interview part. [Unit 4] “You asked us to make a sentence from the content of the wedding brochure, right? What we did was taking a sentence from previous unit. Maybe, you can give us an example on how to say it. but later on, we cannot use the example. So we can make use of it.” On translation; M Ie 117; page 292 c Theme 3: Vocabulary and pronunciation practice should be added more. Vocabulary and pronunciation, which are included in micro skills, should be added more. The designed materials probably dealt more on macro skills that made the learning process in the classroom provided less focus on learning micro skills. This kind of concern was felt by the students which made them show their difficulty when they dealt with vocabulary items as well as the pronunciation practice. Their sharing can be seen from the following interview part. 160 [Unit 2] “For these kinds of meanings, we are still.. This one. Perhaps it should be more than this. I mean, the vocabulary. Also how to pronounce the words . That’s all, mbak” On translation; M Ie 50; page 288 d Theme 4: Related to project-based, there should be more explanation on particular step. The explanation of one step of project-based learning need to be elaborated more. The students shared their experience when they dealt with step 3, which was structuring the project. The purpose of this step was to invite them to make a timeline on what processes they they will have related to the preparation of the project. In the workbook, there has been already a table of timeline for them to write down their progress. However, they still found it difficult to comprehend what should they write and do in the step. The teacher’s explanation, therefore, should clarify their misunderstanding. The students’ confusion can be seen from the following interview part. [Unit 2] “We are confused related to this step. I mean, what is it. What is the order? We are still not sure.” On translation; M Ie 79; page 290 3 Interview Analysis as Summative Evaluation After the units were all implemented, focus group discussion was done to evaluate the overall materials. The following discussion is divided into two parts: task-based learning and project-based learning. In dealing with task-based learning evaluation, by adapting Ontario Adult Literacy Curriculum Framework OALCF 2011 of interview protocol, the topics of the questions were classified into five topics, namely content, quality of content, potential effectiveness as a 161 teaching-learning tool, ease of use, and comments. Meanwhile, in dealing with project-based evaluation, by adapting a list of questions proposed by Edwards 2002 quoted in Railsback, 2002: 21, the questions of the interview protocol cover project’s success, students’ improvement, students’ reflection on relating the project with the learning goals, project’s challenges, students’ best effort, implementation of skills in accomplishing the project, and students’ final evaluation on their project. Firstly, in dealing with task-based learning, after transcribing the focus group discussion, five themes were generated. Those themes were 1 the contents of the materials are already appropriate, 2 the quality of the contents of the materials is appropriate, 3 the materials are effective, efficient, and user- friendly, and 4 the program duration should be longer. a Theme 1: The contents of the materials are already appropriate. The first theme is related to the contents of the materials. All of the students agreed that the designed materials have already contained appropriate contents. The contents have represented their needs in dealing with clients. Communicatively, they have learned from the designed materials on how to use the language properly when dealing with wedding organization. The following interview part shows students’ agreement on the appropriateness of the designed materials. Generally, the materials are already appropriate. Through this course, we have received the learning process up to the procedure how to communicate with clients. Done. On translation; N Ie 1; page 298 PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI 162 b Theme 2: The quality of the contents of the materials is appropriate. As the second theme, the students agreed that the quality of the contents of the designed materials has been appropriate already. The contents given in the materials represented what they have done so far in dealing with clients. It has matched with the needs of Denaya Production . The students’ opinion related to the quality of the contents can be seen from the following interview part. The quality of the contents are already appropriate. They are relevant and practicable in our workfield. Already appropriate. On translation; N Ir 6 – Ie 6; page 298 c Theme 3: The materials are effective, efficient, and user-friendly. The third theme deals with the effectiveness, efficiency, and user friendliness of the designed materials. The students agreed that the materials have been effective and efficient in teaching them how to communicate with clients in English. Furthermore, they also agreed that they might opened it as their guidelines when later on they find difficulty in dealing with clients. The following interview part tells how the students showed their opinions related to this theme. We have already learned a lot from the materials. From the first meeting up to today, we have already had a guide book to learn by ourselves later on. If one day we get foreign clients, we will have used this as our guideline to communicate with them although it needs us to re-open the materials again to remind us. The materials are already effective, efficient, and user-friendly On translation; N Ie 8, Ie 12, Ie 15; page 298 d Theme 4: The program duration should be longer. In giving suggestions to the designed materials, the students agreed that they might need to learn English longer. It means that the course needed to be extended. Extending the course can deal with learning the same topics with longer PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI 163 time or learning some additional new topics. They expected that the course would be as what the students in school get the English lessons that follow longer and particular-patent schedule. The following interview part represents their suggestion. The materials are already good. Our comment will be on the course duration. That’s all. If only this course were longer, this course would have been terkurikulum or well-scheduled so that we can learn better. On translation; N Ie 18; page 299 Secondly, in dealing with project-based learning, the results of the interview as a summative evaluation indicated students’ positive attitude. Their positive attitude was elaborated through seven generated themes. The themes are 1 the presentation project itself was the students’ success, 2 the presentation project can deal with content improvisation, 3 the steps to accomplish the project were well-reached, 4 the presentation project traine d the students’ independent learning, 5 the presentation project showed the students’ best effort on the preparation, 6 the presentation project trained the students’ speaking ability, and 7 the presentation project build the students’ willingness to keep learning. a Theme 1: The presentation project itself was the students’ success. The students claimed that having that kind of presentation was their success. Previously they have not presented a wedding concept or any other things related to wedding organization in English. Therefore, in becoming the success in the learning process, they think that they can make use of the presentation in their workfield, particularly, they will use what they get to be implemented in front of the real foreign clients. It can be seen from the following interview part. 164 The success that we got was, the idea was, the presentation method itself. It made us get inspired to use English in our workfield to our real foreign clients. On translation; O Ie 3; page 300 b Theme 2: The presentation project can deal with improvisation. When they were asked about any improvement they can do to the project, they answered that they can make an improvisation on the content of what was being presented. The content of the presentation can be elaborated more, particularly when they present the wedding concept. The elaboration can be done promptly. As an example, when dealing with Javanese wedding party, there are some parts from international wedding concept that have been recently added to be parts of Javanese wedding party. The following interview part discusses the improvisation the students might have in their presentation. Actually, from the presentation, we can improve the explanation of the wedding party. In recent Javanese wedding party, there are some ceremony in international wedding concept that have been already instered, such as throwing wedding bouquette and cutting wedding cake On translation; O Ie 4; page 300 c Theme 3: The steps to accomplish the project were well-reached. The students have passed several steps to prepare the project of presentation well. They said that those steps can make them prepare the presentation project good enough. Moreover, before they worked by themselves, the teacher has helped them in comprehending every single step. The following interview part represents students’ opinion related to the steps accomplished in presentation project. So far, we think that those steps are already enough. The suggestion is for us only to learn them deeper. I mean, to learn every demand of every step deeper. But overall, we have understood the steps well. On translation; O Ie 5; page 300 165 d Theme 4: The presentation project trained the students’ independent learning Students’ independent learning has been a part of their progress in accomplishing the presentation project. As they acknowledged that they have mispronounced some words. Yet, as they were given this project presentation, they could learn that some words have been mispronounced all this time. By keep consulting with dictionary, they realized their mistakes and corrected them by themselves. The students’ story on that can be seen from the following interview part. What suprised us is the way we read and pronounce some words that we thought we read and pronounced them correctly. Meaning that when there are incorrect pronunciation, we then realized the mistakes by ourselves and then directly look for the correct pronunciation from the dictionary at home. On translation; O Ie 6; page 300 e Theme 5: The presentation project showed the students’ best effort on the preparation Related to the best effort, the students agreed that reading the script, as well as pronouncing every single words correctly, became their best effort. They tried hard not to look at notes. All the best they can do was to present the wedding concept as real as possible. It can be seen from the following interview part. We think that our best effort is on the effort to pronounce the words in the preparation. Although in our presentation, we looked at our notes at some parts, but at least, we can do that. Deeper improvisation totally without text is really needed, of course. On translation; O Ie 7; page 300 PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI 166 f Theme 6: The presentation project trained the students’ speaking ability From the project presentation, the students said that in the end, their speaking ability was trained more than any other English skills. Delivering wedding concept in a presentation need a good speaking ability. It was not about the way the students pronounced words correctly, but also the way they deliver a good body language to the audience, so that they can deliver the messages well. It can be seen from the following interview part. Speaking. It is the main skill. It is indeed the key for us to communicate to clients, particularly dealing with wedding concept. On translation; O Ie 9; page 301 g Theme 7: The presentation project build the students’ willingness to keep learning When the students were asked about overall evaluation related to presentation project, they said that the results were still not good and maximum. The meaning of “not good and maximum” here was that the students should have done better than that if they keept learning on that in particular longer time. The students also said that they were willing to learn how to present better, so that in a real workfield, they can make use of it optimally. It can be seen from the following interview part. In our opinion, they way we performed was still not good. It is because learning a language needs time and any other aspects. We need to keep learning, we need to keep reading in order to be able to perform in such kind of presentation like this better. We want our presentation also to have such kind of improvisation. On translation; O Ie 11; page 301 PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI 167

c. Overview of the Suggestion in Main Field Testing

Through the teacher’s field notes and interview with the students, as the results of both formative and summative evaluation, some suggestions were obtained and outlined. The following table represents the overview of the suggestions in main field testing. Table 4.14. The Overview of the Suggestions in Main Field Testing No. Aspects Suggestions 1. Pictures Some pictures need to be either changed or rearranged, for example, a picture that became the cover of Unit 2, pictures of Javanese wedding structure and pictures of Indonesian food. 2. Contents Some contents need to be added in dealing with wedding brochure and simplified in dealing with contents of expressions of asking and giving opinion and probably contents of the wedding contract 3. Instructions In dealing with project-based instructions, particularly on step 3, the explanation should be clearer. 4. Material implementation The students expected that the materials would be better implemented in a longer time, as one unit of the materials seemed difficult to be comprehended in only one meeting. Furthermore, dealing with vocabulary items and pronunciation practice needs to be paid more attention.

d. Measurement Data

To measure students’ English ability, therefore, some tests were conducted. The tests were pre-test and post-test. Furthermore, as the final goal of the project-based learning was that the students present the wedding concept, therefore, the presentation performance was also assessed. The dates of conducting pre-test, post-test, and project presentation were December 5, 2016 and January 3, 2017. Firstly, pre-test and post-test contain similar types of questions. The test also cover English four skills. After they did the test, the results of their performance were analyzed using SPSS computation. The inferential statistics that 168 was used was T-Test, particularly Paired Samples T-Test whose purpose was to test hypotheses concerning two related samples selected from population. Here, pre-test and post-test were the tests before and after the students joined the program course. The raw data on the results of pre-test and post-test can be seen from the following table. Table 4.15 . Raw Data on Students’ Scores in Pre-Test and Post-Test Student Pre-Test Post-Test 1 58.3 72.9 2 62.5 75.0 3 58.3 70.9 4 56.3 72.9 Since test of normality is one of the requirements of T-Test, therefore, first of all, the scores were checked through the test of normality. It is aimed at knowing whether the data distribution is normal or not. If the sig. value is greater than 0.05, the data distribution is normal. By looking at the following table, the sig. value is greater than 0.05. It means that the scores have normal distribution. Therefore, the scores met the demands of the requirement of T-Test. Table 4.16. Tests of Normality Tests of Normality Kolmogorov-Smirnov a Shapiro-Wilk Statistic df Sig. Statistic df Sig. Pretest ,333 4 . ,889 4 ,377 Posttest ,250 4 . ,945 4 ,683 a. Lilliefors Significance Correction As the scores are normally distributed, as the second stage, the scores were analyzed in paired samples t-test. After computing the scores, the following table 169 obtained the result. Before going to the final table of computation, this table was carefully seen to check whether the data are already correct and valid. It can be done by checking the mean and standard deviation for each group and the number of people in each group. Table 4.17. Paired Samples Statistics Paired Samples Statistics Mean N Std. Deviation Std. Error Mean Pair 1 Pretest 58,85 4 2,610 1,305 Posttest 72,90 4 1,715 ,857 In the following table of paired samples test, the final decision or interpretation was determined. From the table, the confidence value chosen is 95. The value in Sig. 2 tailed is 0.001. It means that the hypothesis H is rejected since the P-value is 0.05. In other words, it can be concluded that the hypothesis alternative H 1 is accepted. It means there is significant difference on the scores between pre-test and post-test. Table 4.18. Paired Samples Test 170 Secondly, in dealing with the final project included in project-based learning, the focus of the English skill being assessed was speaking skill. Moreover, the students presented the wedding concepts in pairs. By referring to CEFR, students’ performances were assessed using oral assessment criteria grid. After looking at their performances, as the average, it can be concluded that their spoken production has already hold A2 CEFR level, in which: Table 4.19. Oral Assessment Criteria Grid of A2 CEFR Range Accuracy Fluency Interaction Coherence Use basic sentence patterns with memorized phrases, groups of a few words and formulae in order to communicate limited information in simple everyday situations. Use some simple structures correctly, but still systematically makes basic mistakes. Can make himherself understand in very short utterances, eventhough pauses, false starts and reformulation are very evident. - Can link groups of words with simple connections like “and”, “but, and “because”.

4. Operational Product Revision

In this operational product revision step, the suggestions that were obtained in main field testing were used as guidelines to revise the designed materials. The aspects of the suggestions covered pictures, contents, instructions, and material implementation. Therefore, the following discussion discuss the detailed revisions in each aspect. Firstly, some inappropriate pictures were changed to the appropriate ones. The picture that became a cover in Unit 2 was a picture that represented international wedding concept. In the new picture used, the picture already 171 represented one of Javanese wedding structure, called siraman. Furthermore, within Unit 2, the pictures which represented panggih and wedding party were not really clear. Therefore, the new pictures used already represented more appropriate pose so that the students would directly recognize that the pictures are pictures in panggih and wedding party. Related to the inappropriate arrangement of pictures, pictures of food in Unit 5 were then rearranged randomly since the students were asked to choose the correct picture of food based on their partner’s food description. Secondly, the suggestion dealt with contents of the input. In Unit 4, the students asked for an example to be provided in the warming up activity. Therefore, an example of sentence that represented the provided wedding brochure was provided, so that the students could make use of it as their guideline. Furthermore, in Unit 3, the students learned about expressions of asking and giving opinions. Some expressions were provided there, however, the students felt difficult in applying several expressions which were considered as unfamiliar. Therefore, the unfamiliar expressions of asking and giving opinions were omitted. Meanwhile, related to the contents in wedding contract in unit 6, some descriptions were simplified. Thirdly, in dealing with instructions in step 3 of project-based learning, in the implementation, the future practitioner or teacher should be maximum in explaining the demand in step 3. Forthly, related to the demand on having a longer materials implementation, this suggestion was placed to the suggestions for the future researcher or practictioner in chapter 5. This would also relate to the PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI