Ethics issues Methods of validation

The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 When the participant is ready, heshe may start talking into the tape or thinking aloud the thoughts entering hisher mind. To make sure that the tape recorder works properly the researcher must test the recording machine before the actual data collection starts.

4.2 Procedures in data collection involving complementary methods

When the method is used in conjunction with other methods, precautions must also be taken against possible effects of reactivity - interference with participants‟ cognitive processes Stratman Hamp-Lyons, 1994. In the light of these necessary precautions, the participants are not told about the following tasks before the think-aloud task is completed. In the case of the writer‟s study on reading comprehension strategies Sugirin, 2002, which complemented think-aloud task with retellings, multiple-choice test and interview , the participants were not informed about the multiple-choice questions before completing retellings. They were not told about the interview, which included discussion on the test answers before completing the test. The researcher planned the instructions and the tasks in such a way that task performance would be spontaneous. This was in line with Ericsson and Simon‟s 1980; 1993 suggestion that participants should not be informed of the subsequent retrospective interview before the completion of the think-aloud task; otherwise the foreknowledge might affect their performance.

5. Ethics issues

As a think-aloud protocol study requires earnest participation of the research participants, issues in regard to ethics requirements should receive due attention. Other than obtaining permits from relevant authorities, the researcher should also obtain agreement from the participants that they will participate voluntarily in the study as proven by signed consent forms. Permits and signed consent forms are necessary to protect the privacy of the participants and to maintain the good research conduct so that the reported results will not raise questions of accountability. To guarantee the participants‟ sincerity in their participation, aspects related to the participants‟ privacy and comfort, such as timetabling, location of the room, and the setting up of the recording The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 equipment need the researcher‟s attention. The researcher should also point out that participants are free to withdraw their consent and end their participation if they feel unhappy with any aspect of the research study, and that it will not, in any way, affect their academic standing as students, if the participants are students.

6. Analysis of the data

In general, the analysis of the think-aloud protocols starts by transcribing tapes of the responses to the think-aloud task. Depending on who the readers will be, if the research participants responded in mixed languages, portions of the responses may require translation. The translated version of the transcripts is then coded. One of the coding models that has received wide acceptance is Strauss Corbin‟s 1990 coding model.

6.1 Coding of the protocols: Grounding of codes on data

Think-aloud protocols or transcripts constitute the main data in a think-aloud study. A grounded theory approach Strauss, 1987 is an approach to data whereby codes are derived from what the participants were doing or were assumed to be doing, and no attempts are made to impose pre-determined codes on the data. However, where it is of help and suits the data, known concepts can be adopted or adapted as codes. In the process of coding, a researcher may employ systematic use of every first letter initial of concepts pertaining to, for example, what a reader was doing. For example, a strategy of “ m aking i nter- s entential r elation” is represented by the code MIR or MISR. Another researcher may use the key word or the initial part of the key word of a strategy as the code. For example, “making inter-sentential relation” is coded as “Inter.” In this regard, the researcher selects codes based on forms that will be easily recognised. In a grounded theory approach, research data undergo three coding stages: open coding, axial coding and selective coding Strauss, 1987. An open coding means unrestricted coding of data by scrutinising the transcripts very closely in order to produce concepts that seem to fit the data in regard to the issues pertaining to conditions and strategies. It also means that in the initial stage of coding, all the phenomena in the data have an equal opportunity to be represented by certain codes despite the change, which may The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 occur after relating them to the two categories mentioned earlier. Thus the open coding verifies and saturates individual codes. Labelling of codes may be changed if better terms are invented later. As Strauss 1987:29 illustrates, the analyst learns to play the game of believing everything and believing nothing - at this point - leaving him or herself as open as the coding itself. In the next step, the results of the open coding are examined axially , by applying axial coding. An axial coding is the intense analysis done around one category at a time in terms of the paradigm items. It may also be said that the analysis revolves around the axis of one category at a time. For example, if the problems in comprehending a text make the core category, all problems such as problems with vocabulary, problems with sentence structure, problems with text structure, etc. will be clustered under the core category of comprehension problems. If the vocabulary problems are considered to be the core category, then problems such as synonyms, antonyms, cognates and non- cognates, and so forth, will be clustered around the core category of vocabulary problems. The final step in coding the data is coding the results of the axial coding selectively . A selective coding means that coding is limited only to codes that relate to the core codes in significant ways as to be used in a parsimonious theory Strauss, 1987:33. Aspects pertaining to conditions and strategies that have little or no relevance to the core categories are dismissed from the coding scheme. An alternative is that these aspects are noted down for the purpose of later discussion. The following is an example of the coding processes of fra ctions of a participant‟s responses to a reading text presented. The investigation was intended to describe the strategies used in comprehending the texts written in English. The reading text is as follows: Text 1 In the beginning of the nineteenth century, the American educational system was desperately in need of reform. Private schools existed, but only for the very rich, and there were very few public schools because of the strong sentiment that children who would grow up to be laborers should not “waste” their time on education but should The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 instead prepare themselves for their life‟s work. It was in the face of this public sentiment that educational reformers set about their task. Horace Mann, probably the most famous of the reformers, felt that there was no excuse in a republic for any citizen to be uneducated. As Superintendent of Education in the state of Massachusetts from 1837 to 1848, he initiated various changes, which were soon matched in other school districts around the country. He extended the school year from five to six months and improved the quality of teachers by instituting teacher education and raising their salaries. Although these changes did not bring about a sudden improvement in the educational system, they at least increased public awareness as to the need for a further strengthening of the system. Quoted from Longman Preparation Course for the TOEFL by Deborah Phillips, 1996, pp. 62-3 In the open coding stage, all the phenomena in the data have equal opportunity to be represented by some codes despite the changes that may occur later. In the early stage of coding the following phenomena were coded as follows: Table 1 Example of the processes in open coding Phenomenatypical exponents Labels Codes I don‟t understand this. What is “desperately”? I don‟t know this. I don‟t know what is “Superintendent of Education” here. “He improved the quality of teachers by instituting teacher education” What is it? Identify a problem Question word meaning Question phrase meaning Question sentence meaning Idprob Qwm Qpm Qsm In the axial coding stage, the three phenomena labelled “questioning word meaning”, “questioning phrase meaning”, and “questioning sentence meaning”, which were coded Qwm, Qpm and Qsm, respectively, were all considered to be exponents of the same strategy, were therefore labelled “questioning meaning”, and coded Qm. In the last stage of coding, selective coding, however, the researcher assumed that behind all the questioning of the meaning of words, phrases and sentences, the reader was identifying a problem. It was for this reason that the label “question meaning” The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 coded Qm was dropped, and the strategies were ultimately labelled as “Identifying a problem” and were coded “Idprob”. The process can be illustrated as follows: Table 2 Example of the processes in axial coding and selective coding Label Code Label Code  Label Code Identify a problem Question word meaning Question phrase meaning Question sentence meaning Idprob Qwm Qpm Qsm Identify a problem Question meaning Idprob Qm   Identify a problem Idprob

6.2 Further analyses

Depending on the objectives of the research, further analyses are needed after coding has been completed. For example, in the case of the study on reading comprehension above, all the strategies used by each participant were counted to determine the frequency of their use. Percentages were used to determine the proportion of each strategy used. This way, inferences could be made about a participants tendency in using particular strategies. An examination was also carried out to see whether the tendency of using particular strategies as reflected in the strategy patterns had any relation to the results of the comprehension measures: retelling and multiple-choice test scores, for instance.

7. Methods of validation

Data collection procedures should follow as closely as possible the expert recommendations by authors whose works have been cited, including Ericsson and Sim on‟s 1993 and Pressley and Afflerbach‟s 1995 recommendations on collecting think- aloud data, and the researcher‟s own judgment based on experience and common The 50 th TEFLIN International Conference: Asian Odyssey 2002: Explorations in TEFL, Unika Widya Mandala Surabaya, 29-31 Oct. 2002 sense, e.g. the sequencing of methods of data collection to avoid the effects of “reactivity” Sommer Sommer, 1990:10; Stratman Hamp-Lyons, 1994 in the on- line think-aloud process. As can be seen in the details of the data collection procedures above, the sequence in which the procedures are used, starts with the least and goes to the most reactive procedures, that is, starting with the one involving the least, and going to the one involving the most, intervention of the researcher In addition, triangulations were made by 1 asking the research participants to examine the summary of the think-aloud data and make necessary adjustments to the summary and 2 asking other raters to verify a the whole or a sample of transcripts, b the whole or a sample of translation, and c the whole or a sample of coded protocols.

8. Possible influences on the study