186 Chapter 8: Implementation and evaluation of the cross-disciplinary intervention 8.1 Introduction Whereas chapters 6 and 7 reported on planning, designing and evaluating a subjectspecific essay-writing intervention, this chapter follows a similar design and procedure for an intervention with a broader disciplinary focus, and compares the results, using both qualitative and quantitative methods of data analysis and evaluation. First, an overview is given of the current debate regarding wider-angled approaches, followed by a description of the research methodology that was followed to measure the effectiveness of the cross-disciplinary intervention, and a presentation and discussion of the results. 8.2 Rationale and approach As stated previously, the common-core versus subject-specific debate in language pedagogy has been going on for more than 20 years. In the context of teaching undergraduates to write academic essays, the most compelling argument from the side of common-core approaches might be that it is imperative for university students to move comfortably between the discourses of a number of academic disciplines. They need to "control a range of genres appropriate to each setting, and to handle the meanings and identities that each evokes" (Hyland 2009: 129). Thus, according to Bruce (2008: 34) there has been a movement away from discipline-based ESP course designs and methodology to a more "discourse and genre-based cross disciplinary approach". This trend, combined with the universal reality of undergraduate students being underprepared to engage in academic discourses (Johns 1995; 2002; Lillis 2001; Hyland 2004), and the researcher's desire to compare the effectiveness of narrow- and wide-angled approaches within a genre framework justified the design and evaluation of a cross-disciplinary academic writing intervention. Following suggestions made by an external review panel that was appointed to evaluate the work of the Unit for Academic literacy, and reinforced by the research in progress, 187 the institution of a second-year module in academic writing was approved by Faculty in 2008, with commencement in 2009. The institutionalization of the module, in turn, increased the relevance of the research. Limitations were that the module was not officially prescribed or recommended by any of the existing academic programmes in or outside the Faculty. A further constraint was that the alpha code UAL was assigned to the module, whereas all other modules offered by the Unit for Academic Literacy bore the code EOT, which may have had an impact on the visibility of the module (in an alphabetically organized prospectus). Finally, there was the added financial burden of R2000 (the cost of the module) added to students' annual programme fees. Although more than 30 students indicated interest, this price was too high for a "nice to have" that was unlikely to be covered by a bursary or student loan. Eventually only 14 students registered, of which 11 completed the module. Another limitation was the researcher's lack of foreknowledge regarding the disciplines that would be represented by the students, and thus the specificity of the syllabus and the materials that could be designed in advance. 8.3 Design and implementation of the intervention 8.3.1 Respondents Despite the constraints outlined above, 14 students registered for the module, of which 11 followed through. The attrition rate of 21% can be accounted for as follows: One of the students was an international exchange student from Germany who only attended seven weeks of the 14 week course; another indicated that she was interested only in political analysis, and that the content of the course was not entirely suited to her needs; and the third discontinued the module in the third week because of work load. The remaining 11 were registered for the following academic programmes: BA (2); BPolSci (5); BA Languages (3); BCom (1). The subjects which they were registered for included Accounting, Criminology, Economics, English, Journalism, History, History of Art, Political Sciences, Philosophy, Psychology, Sociology, and Visual Studies. Their sociodemographic profile could be summarized as follows: 2 white males with Afrikaans as their mother tongue; 1 white male with English as his mother tongue; 2 white females with Afrikaans as their mother tongue; 1 black female with Portuguese as 188 her mother tongue, 4 black females with an African language as their mother tongue, and 1 black male with an African language as his mother tongue. 8.3.2 Syllabus and materials To facilitate comparison of the two interventions, and limit variability, the same broad syllabus structure was used as for the subject-specific module, focusing on the use of rhetorical modes, types of claims and types of support in developing an academic argument, while following the Teaching and Learning Cycle of the Australian genre school. Argumentation was given a more prominent role, and using rhetorical modes became one of the secondary threads. This was necessary because of less pronounced relationship between essay structure and primary rhetorical mode in subject-fields other than history. In addition, more emphasis was placed on stance and engagement, because of the lack of the history students' improvement in this area. Table 8.1 represents the presyllabus for the cross-disciplinary intervention. Table 8.1 Presyllabus for the cross-disciplinary intervention Study unit 1: Academic discourse(s) The generic features of academic discourse are discussed with reference to authentic texts, followed by the study of texts from specific disciplines in the Humanities in order to emphasize the need for mastering the characteristic features of disciplinary discourses. Themes • What is academic discourse? • Is there only one academic discourse? Study unit 2: Modes of writing (text types) The mastery of rhetorical modes is practised during a cycle comprising the exploration of excerpts from authentic academic texts, freewriting, explicit teaching of the lexicogrammatical features that characterize each mode, identification of frequently used modes, and independent writing of paragraphs or short essays. Themes • Chronological writing: narratives, recounts and processes • Description • Comparison and contrast • Cause and effect • Exposition • Analysis 189 Study unit 3: Academic arguments: formulating claims Examples of essays are analyzed to identify the types of claims contained in thesis statements. Students also formulate their own claims on the basis of given topics. Themes • Fact and opinion • What is a claim/thesis? • Positioning of the main thesis of an essay • Types of claims (factual; causal; evaluative; recommendations) Study unit 4: Academic arguments: invoking evidence Examples of essays are analyzed to identify types of support and types of evidence, focusing on lexicogrammatical markers. Themes • Types of support (comparison; definition; well-chosen examples; statistics; appeals to audience needs; appeals to authority; addressing a counterargument) • Using appropriate types of support for different types of claims Each study unit comprised a set of outcomes and a learning component containing theory, model texts and a variety of authentic task types, some of which were done collaboratively in class, and some as homework tasks that had to be submitted for marks. Course materials consisted of a 100 page study guide cum workbook, based on the presyllabus, a reader (hard copy) comprising 4 broadly focused articles on the theme selected for the content of the module, viz. Poverty in Africa, and a partially interactive Blackboard-based website. (This theme was suggested by lecturers from the departments who contributed materials for the writing task survey, because of its relevance across disciplines.) The website contained administrative information about the lecturer, the content and assessment as well as a calendar with important dates. Via a link to the library students had access to a variety of scholarly articles (for which copyright clearance was obtained) and web resources. The Discussion Tool allowed students to interact with one another on matters of common interest, and the Announcement Tool was used to alert students to important dates and events on the calendar. Additional class notes and the list of topics for the final examination were uploaded to folders on the home page for the module. 190 Although the workbook contained a selection of texts from a variety of disciplines in the humanities, it was realized that these might not be relevant to the core foci of the students who would register for the course, and that one would need to substitute some of the examples and exercises with more relevant materials through the course of the module. This procedure was regarded to be completely in line with the postmethod strategy of a basic presyllabus, which is adapted on the basis of feedback and learner needs. Assessment was done in accordance with faculty regulations. The semester mark (progress mark) was based on continuous assessment of written homework tasks submitted and marked throughout the semester, and the exam mark, each contributing a weight of 50%. The 14 week intervention (two contact sessions per week) commenced in February 2009. 8.4 Quantitative evaluation 8.4.1 Method All students enrolled for the module had to write a pretest and a posttest. The pretest assumed the format of a 50 minute in-class essay during the second week of the module. All participants received the reader (containing four articles on general aspects of poverty in Africa) a week in advance, and were requested to prepare for the pretest essay. They were allowed to use the reader as an in-class resource. The pretest did not count towards the students' final marks, as they had not received any tuition on essaywriting at that point. Before writing the pretest consent was obtained to use unattributed extracts from participants' essays as well as the analytic scores awarded by the raters. The posttest comprised the summative evaluation of the module. The students were allowed to choose from a list of topics on various issues relating to poverty in Africa, which had been requested from the relevant academic departments. They were given one month in which to prepare for the essay exam. The planning, literature search, literature review, outlining, writing and reviewing had to be done without assistance from the lecturer in order to determine whether the scaffolding introduced throughout the module had equipped them to independently apply the linguistic and structural 191 principles they had acquired. Table 8.2 shows the topics that were chosen by the students: Table 8.2 Essay topics chosen by students in the cross-disciplinary group Topic (and description) Subjectfield No. of students To what extent was poverty an inevitable by-product of European colonialism in Africa? History 1 Whose obligation is it to do something about poverty in society: the rich or the poor? Philosophy 2 Whose obligation is it to do something about the moral problem of poverty: the poor or the government? Philosophy 1 Analyze the poem "London" by William Blake (in the Norton Anthology of Poetry) OR "An abandoned bundle" by Oswald Mtshali (in the Paperbook of South African Poetry ed. Chapman) paying close attention to the way the poem depicts both physical and spiritual poverty. English literature 1 Discuss how Boesman and Lena are dehumanized by poverty and racial discrimination in Athol Fugard's Boesman and Lena. Refer closely to the text throughout your discussion. English literature 3 The policy gap and poverty. Political Sciences 1 Evaluate the United Nations' Millennium Development Goals (MDGs) as a global strategy to arrest poverty, by referring to the MDGs' normative as well as practical contribution to the plight of the poor. Economics 1 Famine and hunger are often associated with poverty. How can this be combated through policy initiatives? Sociology 1 The exam was taken in the Computer Based Testing Laboratory of the Informatorium on campus. Students were allowed to bring into the venue a sheet of paper with five citations, not exceeding 100 words, which they could integrate in their essays. This concession was made to facilitate the assessment of their ability to engage with other authors. Students had to use the 2003 version of Microsoft Word, since not all of them were familiar with the 2007 version. The spell- and grammar-checker was disabled. The pre- and the posttest essays were scored independently by the course designer, who also presented the generic course (Rater 1), and a part time lecturer with more than 20 years experience in teaching English literature, language and academic literacy, as well as a doctorate in Applied Linguistics from a reputable South African university (Rater 2). The assessment instrument was the same as for the subject-specific intervention (compare Table 7.1 in Chapter 7). 192 Rater 1 scored students higher than Rater 2 on both the pretests and the posttests: on average the pretests were scored 3.4% higher by Rater 1 than by Rater 2, and the posttests were scored 2.4% higher by Rater 1 than by Rater 2. The correlation coefficients of the scores of the two raters are 0.96 for the pretest and 0.97 for the posttest, therefore warranting the use of the average of the two raters' scores as a measure of each student's performance. After the rating process had been completed, the two raters discussed their experience with scoring the essays. The second rater suggested that the formulation of certain items should be adapted with a view to future rating exercises, first because the relative weight of certain items was regarded to be either too high or too low, and second, to explain and clarify the scope of particular items, especially in cases where the rater would not be familiar with the terminology of certain paradigms in applied linguistics: • Items 1 and 2 should be combined into one item, Use of source material, because of difficulty to make a clear distinction between Relevance of source data and Integration of source material. Because of the second rater's uncertainty as to the scope of these items, as well as his relative unfamiliarity with the sources that the students had used, he tended to award an average score of 4 on items 1 and 2. • Item 5 (Development of main argument) should be explained by means of bracketed information such as (coherence and logic). • Item 9 (Concord and tense) may be weighted too heavily, and its scope could be extended to "Use of verbs". • For essays in the humanities item 11 (Technical and subtechnical lexis) might not be entirely relevant, first because certain subject-fields do not have as distinctive a nomenclature as others, and second because the use of subtechnical lexis overlaps with item 12 (Style). (Academic vocabulary may be seen as part of academic writing style.) 8.4.2 Presentation and discussion of results The total score for each of the 11 respondents was converted to a percentage for ease of interpretation (compare Figure 8.1): 193 90 81 79 80 70 85 83 68 65 62 60 81 71 55 60 53 51 48 50 % 65 63 58 55 57 57 51 43 Pretest Posttest 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 Respondents Figure 8.1 Comparison of pre- and posttest results of the cross-disciplinary group per respondent The average improvement of the 11 respondents was 10%. With the exception of a single student – whose posttest score was only 2 percent less than her pretest score – all the students showed progress, with the largest improvement being 21%. Figure 8.2 displays the average results per item after conversion to percentages: 97 92 100 90 80 70 68 62 67 56 60 58 46 % 50 61 51 66 69 62 57 71 65 66 59 69 58 67 60 6463 51 Pretest Posttest 38 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Items Figure 8.2 Comparison of pre- and posttest results of the cross-disciplinary group per item Per item, all the posttest ratings were higher than the pretest ratings, except item 13, which was 1% lower. For item 6, the improvement was 24%, while the improvement 194 was between 10% and 15% on a further six items (items 2, 3, 4, 5, 7 and 11). On the remaining five items the improvement was more than 5%. On the three primary dimensions of the analytic scoring instrument the improvement varied between 7% and 15%: Table 8.3 shows the mean improvement on the four dimensions of the instrument. Table 8.3 Percentage improvement of the cross-disciplinary group per dimension Dimension Mean: pretest Mean: posttest Improvement 1. Use of source materials (Items 1-3) 54% 64% 10% 2. Structure and development (Items 4-7) 49% 64% 15% 3. Academic writing style (Items 8-12) 67% 74% 7% 4. Editing (Item 13) 64% 63% -1% 8.4.3 Statistical analysis The Wilcoxon signed-rank test (SPSS version 17; Williams, Sweeney, & Anderson 2009: 764-770) was again used to assess if the differences between the pre- and posttest ratings on each of the 13 questions comprising the instrument were significant. The Wilcoxon signed-rank test is a non-parametric test that is suitable for the analysis of small samples, as in the present case. The test indicates the probability of a significant difference between pre- and posttest ratings, and is appropriate for comparing data from the same participants – in this case the pre- and posttests written by each of the respondents who participated in the intervention. The results presented in Figure 8.2 should be interpreted against the probability values obtained from the Wilcoxon signed-rank test on each of the 13 items, which are represented in Table 8.5. As in the case of the subject-specific intervention one-sided probability values (p-values) are reported, based on the hypothesis that students' skills would improve as a result of the intervention. P-values lower than 0.05 indicate that there is a significant improvement from the pre- to the posttest ratings at the 5% level of significance. Table 8.4 indicates the p-values for the four dimensions: 195 Table 8.4 One-sided p-values of the pre- and posttest ratings on the four dimensions of the cross-disciplinary intervention Dimension p-value Dimension 1: Use of source materials 0.022 Dimension 2: Structure and development 0.003 Dimension 3: Academic writing style 0.004 Dimension 4: Editing 0.321 Overall 0.001 According to Table 8.4 the improvement between the pre- and posttest ratings is significant at the 5% level for three of the four the main dimensions of the scoring instrument. Only on dimension 4 (Editing) the improvement was not significant (p = 3205). In order to establish whether the p-values of all of the individual items were significant, ratings from the Wilcoxon signed-rank test was obtained for each of the 13 items, as represented by Table 8.5 below. Table 8.5 One-sided p-values of the pre- and posttest ratings on the 13 items in the cross-disciplinary intervention, obtained from the Wilcoxon signed-rank test Item 1 2 3 4 5 6 7 8 9 10 11 12 13 *16 p-value 0.086 0.018 0.016 0.076 0.016 0.003 0.013 0.023 0.011 0.080 0.013 0.065 0.321 0.002 *Item 16 was included as it shows the average improvement on all 13 items. 196 Overall, based on the total difference between the pretest and posttest scores (item 16), the improvement is significant with a p-value much smaller than 0.05. For the remainder of the items, the improvement between the pre- and posttest ratings is significant at the 5% level, with the exception of items 1, 4, 10 and 12, which are significant at the 10% level, and item 13, on which students did not show any significant improvement. This was to be expected, since the average percentage for this item was 63% in the pretest and 64% in the posttest. It should be noted again, however, that in the case of items 1, 4, 10 and 12 a larger sample might have resulted in significant improvement at the 5% level for these two questions as well. Similar to the subject-specific intervention, an analysis of the pre- and the posttest essays was also conducted for the cross-disciplinary intervention, focusing on the following key resources within a Systemic Functional perspective: Logical ideation, representing the ideational/experiential function of language, Appraisal, representing the interpersonal function, and Theme representing the textual meaning. 8.5 Textual analysis of the essays 8.5.1 Method Similar to the subject-specific intervention, the pre- and posttest essays of all the participants were tagged electronically for Logical ideation and Appraisal, using the categories and subcategories expounded in Tables 6.5 and 6.6 in chapter 6. Concordances were also built in the same way, using WordsmithTools version 4.0. Regarding thematic development a case study on the pre- and posttest essays of Respondent 1 was performed, comparable with the subject-specific intervention. Analogous to the subject-specific evaluation the focus was set on waves of known and new information at the clause level, using Butt et al.'s technique (2000: 143ff). 8.5.2 Presentation and discussion of findings 18.104.22.168 Ideational analysis Figure 8.3 represents the summarized results of the ideation analysis: 152 142 97 91 89 Pretest 62 37 30 Addition 23 Comparison Causation Incorrect 1711 Correct Correct Correct Incorrect 1616 9 Posttest 35 Incorrect 44 Incorrect 160 140 120 100 80 60 40 20 0 Correct No. of occurrences 197 Time/tense Figure 8.3 Logical ideation: comparison of pre- and posttest results in the crossdisciplinary intervention According to the graph the cross-disciplinary intervention students used more markers of logical relationships in the posttest than they used in the pretest – in all four main categories. There was an increase of more than 50% in each category, except Comparison, where the improvement was 23%. Analogous to the subject-specific intervention, significantly more tokens of Causation and Addition were used in the posttest than in the pretest. There was also a significant decrease in the number of errors in these categories between the pretest and the posttest (80% decrease in the number of Causation errors and 62% in the number of Addition errors). Upon further scrutiny it transpired that the posttest yielded more variety in the use of Causation resources (thus more variety in the representation of subcategories): Where the pretest yielded 7 correct usages of Condition, 4 of Means and 8 of Purpose (with 31 instances of Cause and 41 instances of Consequence), the posttest yielded 19 of Condition, 21 of Means, and 24 of Purpose (with 44 instances of Cause and 34 of Consequence). Also noteworthy is the decrease of 35% errors in the Time/tense category. This decrease cannot be ascribed to an improved mastery of tense, but to an improvement in the use of other markers of temporal relationships. A possible explanation for the relatively few Tense errors in both the pretest and the posttest may be that the overarching topic for both the pretest and the posttest, Poverty in Africa, demanded less skill in moving back 198 and forth between present and past, than was demanded by the topics of the subjectspecific intervention. 22.214.171.124 Interpersonal analysis Appraisal The Appraisal analysis of the cross-disciplinary intervention used the same three categories of analysis as the subject-specific evaluation, viz. Attitude (with subcategories Emotion, Judgement and Social Valuation), Engagement (divided into Attribute and Proclaim) and Graduation (split into Force and Focus). Figure 8.4 145 99 Pretest 52 50 25 23 Attitude Engagement Correct Incorrect 1 2 Correct 10 Posttest 40 13 Incorrect 55 Incorrect 160 140 120 100 80 60 40 20 0 Correct No. of occurrences represents the summarized results: Graduation Figure 8.4 Appraisal: comparison of pre- and posttest results in the crossdisciplinary intervention Figure 8.4 shows that on all three dimensions separately, there were significant improvements: On the Attitude dimension there was an increase from 10 to 55 (= 450%); the number of Engagement markers increased from 52 to 145 (= 173%), and the number of Graduation markers increased from 40 to 99 (= 148%). The steep increase in the number of Engagement markers in the posttest could possibly be ascribed to the lecturer cum researcher's emphasis on the importance of entering into debate with other authors. Even with 14% error on the posttest, it still proves worthwhile to teach students strategies of Engagement − even at undergraduate level. It is particularly encouraging that only 38 incorrect or inappropriate usages of any of the Appraisal resources occurred in the posttest. 199 126.96.36.199 Textual analysis Theme and New Theme analysis was conducted similar to the subject-specific intervention. In an analogous way Respondent 1's essays were selected for the case study, using the following symbols to categorize thematic bonds: • bold vertical arrows Ï and vertical bracketed arrows to indicate strong thematic bonds with previous Themes; • Oblique arrows and oblique bracketed arrows to indicate thematic bonds with previous News; • non-bold vertical arrows ↑ and broken bracketed arrows to indicate weak thematic bonds with previous Themes; • oblique broken arrows and oblique broken bracketed arrows to indicate weak thematic bonds with previous News; • the symbol Ø to indicate the absence of a thematic bond. The quantified results of the Theme analysis of the pre- and posttest essays of Respondent 1 are given in Tables 8.6 and 8.7 below (compare Appendix G on CD for the full essays): Table 8.6 Cross-disciplinary intervention: Pretest 1 (60%) No. of words: 434 No. of paragraphs: 9 No. of clauses: 43 Strong bonds to directly preceding Theme Strong bonds to earlier Theme (s) Strong bonds to directly preceding New Strong bonds to earlier New(s) Weak bonds to directly preceding Theme Weak bonds to earlier Theme Weak bonds to directly preceding New 5 5 10 3 2 3 1 0 12 12% 12% 23% 7% 5% 7% 2% 0% 26% Strong bonds: 23 (53%) Weak and absent bonds: 18 (42%) Weak bonds to earlier New(s) No bonds 200 Table 8.7 Cross-disciplinary intervention; Posttest 1 (81%) No. of words: 649 No. of paragraphs: 7 No. of clauses: 61 Strong bonds to directly preceding Theme Strong bonds to earlier Theme (s) Strong bonds to directly preceding New 30 4 11 49% 07% 18% Strong bonds to earlier New(s) Weak bonds to directly preceding Theme Weak bonds to earlier Theme Weak bonds to directly precedin g New Weak bonds to earlier New(s) Absent bonds 12 1 0 0 0 4 20% 02% 0% 0% 0% 02% Strong bonds: 57 (93%) Weak and absent bonds: 5 (8%) Although the Theme analysis indicated that the pretest contained more strong than weak bonds, there was still a remarkable improvement if compared to the results of the analysis of the posttest: Strong bonds increased from 53% to 93% and weak and absent bonds decreased from 42% to 8%. In order to further decrease the probability that Respondent 1's improvement between the pretest and the posttest was due to chance, another respondent was randomly selected, viz. Respondent 7 (compare Appendix G). In her case the improvement in handling thematic development was less dramatic than in the case of Respondent 1: The pretest contained 18 (33%) strong bonds and 35 (65%) weak and absent bonds, whereas the posttest contained 25 (41%) strong bonds and 34 (56%) weak and absent bonds. In view of the fact that she improved by only 10% between the pretest and the posttest according to the analytic scoring (48% versus 58%), a more modest improvement in terms of her ability to handle thematic progression could be expected. It is important, nonetheless, to observe that the improvement is again present, and noteworthy. 8.6 Opinion survey Similar to the subject-specific intervention an opinion survey was conducted, using the same questionnaire and the same procedures for recording the data and analyzing the results. 201 Figure 8.5 summarizes the average rating per student on each of the five dimensions of the construct. Number of students 12 10 8 Negative Uncertain 6 Positive Very positive 4 2 0 1 2 3 4 5 Dimensions Figure 8.5 Students' opinions on the 5 dimensions of the construct: crossdisciplinary intervention The students' general lack of commitment, and the fact that the lecturer had to reprimand certain individuals for their relative indifference, predicted a less favourable evaluation of the course. However, although few students chose the extremely positive option on the scales, their opinions regarding the course were still generally positive. The dimension that evoked the most favourable responses was the Needs-driven syllabus (Dimension 3). From the raw data it could be determined that only 1 student "disagreed" that his/her expectations had been fulfilled (Statement 15) and that the most important questions he/she had had about essay-writing (Statement 17) had been fulfilled. He/she was "uncertain" as to whether the lecturer had been interested in addressing his/her personal needs (Statement 17). Dimension 4 evoked the least favourable responses. For two reasons this finding was not surprising. As mentioned in the discussion of the subject-specific opinion survey in Chapter 7, students are generally skeptical about the potential of university courses to teach them critical thinking skills, and this perception was sustained by the outcome of that survey on Dimension 4. It would therefore be surprising if the students who 202 participated in the cross-disciplinary intervention would give a positive rating. Thus, the fact that only two students responded positively to this dimension, eight were uncertain and one was negative, did not come as a surprise. In order to gain insight into the responses to the individual statements comprising this dimension, a graph was generated (Figure 8.6). (Note again that the scales for Statements 23-25 were reversed for the statistical analysis to bring their polarity in line with that of Statements 21 and 22.) Number of students 12 10 8 Negative 6 Uncertain Positive 4 2 0 21 22 23 24 25 Statements Figure 8.6 Responses to the concepts evaluated by statements 21-25 (Dimension 4) The results indicated on the graph can be interpreted and explained as follows: Statement 21: It is empowering to know how to write in the genres valued by academics. Six students responded extremely positive (1), while the other five responded moderately positive (2). Since the statement was phrased positively the responses indicate a generally positive evaluation of empowerment through genre knowledge. Statement 22: If one of my academic lecturers says that it is forbidden to refer to myself ("I") in academic writing, I will take issue with him/her. Seven students were uncertain whether they would challenge a lecturer, two were moderately sure that they would, and two were very certain that they would not. The mixed responses were not surprising, since it became very clear during the intervention that subject-fields hosted divergent views in this regard. Furthermore, some tolerated individual differences, while others insisted on following the conventions laid down by 203 the particular discourse community. For instance, certain subject-fields almost prohibit self reference (History), others tolerate it if used in moderation (for example, Economics), while others encourage it (Philosophy). Thus, in a sense this question did not necessarily measure opinions, but rather knowledge of subject-field conventions. Statement 23: One should accept the content of textbooks and academic articles as true. Since this statement was phrased negatively a "positive" answer would have to be either 4 or 5 on the scale. Since eight of the students ticked either 1 or 2, almost two thirds seem to believe that the authority of prescribed sources should not be questioned or challenged. Statement 24: It is impossible to criticize one's own work. All 11 students ticked either 1 or 2, which means (seeing that the question has a negative polarity) that they are all convinced of the value of self-reflection. Of course, this does not mean that they necessarily always take the time to actively pursue this goal. Statement 25: Empowerment in tertiary education means that students should be allowed to write as they speak. The responses to this question were rather mixed, which might have resulted from the negative polarity. Strictly according to the answers only 45% of the respondents seem to harbour a misconception regarding an important objective of the intervention (6 students ticked 1 or 2), two were uncertain, one indicated moderate disagreement, and one indicated complete agreement. The swing of the pendulum in a negative direction does therefore not seem to be meaningful. These explanations indicate that the slightly negative response to Dimension 4 was not a reason for too much concern. Only statement 23 seem to merit further investigation, but perhaps the first semester of the second year at university is still too early to have developed a critical orientation towards work produced by experts, and hopefully students will develop a more interrogating stance as they move closer to graduation. Because of the importance to prove or disprove the measure in which genre-based interventions facilitated transfer, Dimension 5 was also teased out some more. It 204 transpired that all students responded positively or moderately positively to Statement 26 (The course has contributed towards improving my writing ability in English) and Statement 28 (Much of what I have learned about essays I can also use when writing reports and other text types). Furthermore, the majority (8 and 9 respectively) gave positively evaluating answers to Statement 27 (I find it difficult to apply the principles we have learned in this course to writing tasks in other subjects) and 29 (Since I started this module my marks for written work in other subjects have improved). Those who did not respond positively indicated uncertainty. These results are encouraging, particularly in the light of the less than desirable level of commitment and perseverance demonstrated by the students who registered for the cross-disciplinary module. 8.7 Author's reflection on the cross-disciplinary intervention The varied performance of students in the cross-disciplinary intervention seems to derive, at least partially, from their reasons to register for the course. While some of the students who registered had a genuine desire to learn how to write academically in their chosen disciplines, others enrolled simply to accumulate credits towards their degrees. This reflection attempts to highlight the relationship between students' achievement in the essay-writing module and their perceptions about the instrumental value of their learning. It also emphasizes the dialectic relationship between learning to write and writing to learn. Hyland (2009:124) claims that benefits of courses teaching students how to write in an academic way "are only perceived as such if students value what this literacy allows them to do". The present experience with administering the semi-generic intervention resonates with Hyland's claim, and echoes the findings of Lillis (2001:85), Lin (2000), Canagarajah (1999) and Ivanic (1998), viz. that certain students passively resist the assumptions and values which they are assumed to acquire. The fact that some students do obtain average to above average marks in content disciplines may reinforce this passive resistance. The following example is a case in point: After performing well below average on the third assignment in the cross-disciplinary module, one student remarked: Ma'am, I don't understand; in my other subjects I get good marks for my assignments, but in your class I fail. It's not like I don't like you or so; I enjoy your classes. You can see I always attend. 205 In contrast, students with a genuine desire to acquire the essayist literacy of the academy, and who experienced the benefits of applying this literacy in their content subjects, flourished. Respondent 5, for example, obtained high grades in secondary school, and matriculated with distinctions in Afrikaans and English. However, in philosophy, which is one of his major subjects at university, he obtained a just above average mark, and registered for the essay-writing module to improve his marks. Fortunately, for him, the essay topics contributed by the Philosophy Department were related to the philosophy curriculum. It is thus not surprising that this student's marks in the essay-writing module improved by 20% between the pretest and the posttest. His philosophy lecturer testified that his improvement in the content subject was about equal. Thus, motivation alone is not sufficient for success. Subject-field knowledge is another prerequisite. The following anecdote is offered to further support this claim: Respondent 5, a student who is registered for a degree in Journalism, and who chose the same topic as the philosophy student (Respondent 4), viz. Whose obligation is it to do something about poverty in society: the rich or the poor? did not demonstrate the same improvement as the philosophy student between the pretest and the posttest. In fact, she scored four percentage points lower than he did in the posttest. Although her work was grammatically correct, her style, development of the main argument and selection of evidence remained "generic". The most plausible explanation is that although Respondent 5's proficiency in academic English was superior to that of Respondent 4 (based on her score in the pretest and her participation in class) she departed in her posttest essay from a zero knowledge base, since she had not been initiated into the "discourse of philosophy" through regular interaction with reading matter, lecturers and peers in the domain of philosophy. This anecdote highlights a serious design error by the researcher. Students in the crossdisciplinary group were allowed free choices from the topics provided by the content departments. It was anticipated that they would choose topics relevant to their core university disciplines. However, some made their choices on the basis of familiarity, such as topics related to the literary works they had previously studied in English 206 Literature. To be specific, three students chose the drama Boesman and Lena by Athol Fugard, and one chose the poem An Abandoned Bundle by Oswald Mtshali, although only one student in the group was registered for a language programme. As confirmed by discussions at the conclusion of the module, some students' choices were motivated by convenience, rather than the ideals of the Vygotskyan Zone of Proximal Development. For two of the students (Respondents 8 and 9), unfortunately, convenience became a trap, because they deviated from the main focus, which in both cases was the physical and spiritual effects of poverty in apartheid South Africa, as portrayed by the particular author. When discussing their first drafts, it was difficult to convince these students, who had structured their essays according to literary elements instead of characteristics of physical and spiritual poverty, that their essays were "off topic". Based on the researcher's teaching experience, combined with self-reflection, the following improvements for future interventions are suggested: • Introduce extensive writing earlier in the semester. • Build a corpus of authentic materials. • Facilitate a close fit between students' core disciplines and their focus in an essaywriting intervention. Introduction of extensive writing at an early stage is motivated by the empirical observation that students only became convinced of the implications of their lexicogrammatical, stylistic and structural choices after having written a full essay, which was after the 10th week of the 14 week module. A second reason for suggesting that full essays be written much sooner is the clear lack of engagement observed in the students when writing shorter assignments, such as paragraphs or parts of full essays. The haphazard and untidy way in which some of the shorter homework tasks had been executed supports the hypothesis that authenticity feeds into motivation, and motivation plays a major role in the quality of the output. The solution supported by the majority of students during the post-intervention feedback session was to start writing complete essays very soon after the commencement of the course. This does not mean changing the content of the syllabus, 207 but in the context of a genre approach it suggests that joint exploration and joint construction in respect of each of the discourse skills emphasized in the course (rhetorical modes, making and supporting claims, thematic development, cohesion, and stance and engagement) should immediately be followed by independent writing of a full essay. In the assessment of the essays the primary focus should be on the particular skill or ability that the students had practised during the preceding week or fortnight. Concerning materials design, authenticity has been confirmed as a core principle. The researcher relied heavily on exemplars from writing manuals published in the US and the UK (for example Barnet 2008; Oregon State University 1997; Richlin-Klonsky & Strenski 1994; Rosnow & Rosnow 1998; Schmidt 2005), but although some of the essays were good overall examples, not all of them were exemplary in terms of every aspect of the syllabus. In order to address these deficits future interventions could draw on essays written by local students who have successfully completed the intervention. Good examples could serve as model texts, whereas poorer attempts could be used to practise editing. Finally, the designer of an intervention should ensure a close link between the texts and topics that are selected for writing purposes and the content that students have to learn in their core disciplines. It has been proven that writing helps them to master content, while at the same time content knowledge helps them to develop fluency and accuracy. 8.8 Conclusion The results obtained from the quantitative evaluation indicate that students definitely benefited from the cross-disciplinary intervention. However, unlike the subject-specific intervention the improvement was not equal on the three primary dimensions measured by the analytic pre- and posttest assessment: On the dimension Handling of source materials the average improvement was 10%, on Structure and development they improved by 15% and on Academic writing style there was only 7% improvement. The most plausible explanation for the fact that the improvement in the use of source materials was moderate, or less than expected, may have been the fact that students did not need to study the content of the sources in depth for assessment in their content disciplines, and thus they might have been less motivated to engage with sources on the 208 broad topic selected by the course designer. Also, they did not necessarily consult the same sources for the pretest and the posttest, and thus did not become familiar with core resources. A possible reason for the slight improvement on academic writing style is that the respondents' grammar and vocabulary were on already at a fairly high level when they entered the course (67% on average). Furthermore, the intervention did not pay any specific attention to the improvement of grammar, and neither was style explicitly taught, except for brief pointers on issues of formality. The fact that the most significant improvement occurred on the dimension of structure and development was not a complete surprise, in that discourse structure, comprising thematic development at the level of the whole text (thesis and conclusion) the paragraph (topic sentence and paragraph development) and clause level (manipulation of Theme and New), is one of the dimensions that can be taught via templates and explicit instruction. Moreover, both stronger and weaker students are able to grasp the main principles and apply them. The snapshots taken of students' performance on aspects of the three primary areas of meaning-making according to Systemic Functional Grammar indicates that explicit teaching of the grammatical resources for encoding these meanings does pay off. The fact that the students used significantly more Appraisal markers than the subjectspecific intervention students is particularly meaningful, since the lecturer made a concerted effort at teaching Appraisal resources to the cross-disciplinary group. According to the opinion survey, students were positive to moderately positive about the intervention. Although not all the respondents thought that their personal needs had been addressed, they were generally of the opinion that they had learned valuable skills, which they could apply in other contexts, and which had already stood them in good stead. However, the overall impression gained from the outcomes of the opinion survey and the personal experience of the course researcher was that the success of future interventions of this nature would depend, to a large extent, on the authenticity of the materials used and the ability of the classroom lecturer to engage students and ensure active participation. 209 Chapter 9: Comparison of the subjectspecific and the cross-disciplinary interventions 9.1 Introduction Chapters 6 to 8 reported on the design, development and evaluation of two genre-based writing interventions – one aimed at second-year students of history, and the other aimed at second-year students registered for a variety of subjects in the humanities. This chapter compares and evaluates the findings from the two interventions. First, the statistical results of the two interventions obtained from the analytic scoring of the pre- and posttest essays are juxtaposed, followed by a statistical comparison of the improvement resulting from the two interventions. Subsequently, the findings of the SFL-based textual analyses are compared. Lastly, the results from the questionnaire surveys are statistically compared to give an impression of students' appraisal of the effectiveness of the respective interventions. 9.2 Comparison of the essay ratings The evaluation of the subject-specific as well as the cross-disciplinary intervention pivoted on a comparison of the pre- and posttest essay scores, where a standardized analytic scoring instrument was used. The primary aim was to test the hypothesis that students' essay-writing abilities would improve significantly as a result of a genre-based writing intervention, irrespective of the disciplinary scope. The second aim was to establish the difference (if a difference should be found) between the effectiveness of narrow-angled and wide-angled genre-based interventions. For each intervention descriptive statistics were used to indicate the improvement per candidate, per item, and per cluster (dimension) of items. Thereafter statistical tests were conducted to calculate the probability that the improvement was statistically significant. 210 Table 9.1 below compares the improvement, per intervention group, on each of the four primary dimensions of the analytic scoring instrument (Use of source materials, Structure and development, Language and style, and Editing) as well as overall: Table 9.1 Comparison of the two intervention groups in terms of their improvement on the four dimensions of the scoring instrument Dimension 1. Use of source materials 2. Structure and development 3. Academic writing style 4. Editing Mean: pretest Mean: posttest Improvement S-specific 51% Generic 54% S-specific 69% Generic 64% S-specific 18% Generic 10% 56% 49% 74% 64% 18% 15% 62% 67% 81% 74% 19% 7% 64% 64% 81% 63% 17% 18% -1% 8% According to Table 9.1 there is a 10% "overall" difference between the groups in terms of their improvement as a result of the particular intervention. The table shows that the overall improvement of the subject-specific group was about equal on the three primary dimensions measured by the analytic pre- and posttest assessment (between 17% and 19%), while the overall improvement of the cross-disciplinary group was more moderate (8%), and also more variable: 10% on Use of source materials, 15% on Structure and development, 7% on Academic writing style and -1% on Editing. According to the Wilcoxon signed-rank test (the non-parametrical equivalent to the paired T-test) both groups, individually, showed a significant overall improvement between the pretest and the posttest. Compare Table 9.2 below: Table 9.2 The significance of the difference between the improvement of the two groups in the four dimensions of the scoring instrument Subject-specific Dimension 1: Use of source materials 2: Structure and development 3: Academic writing style 4: Editing Overall One-sided p-value 0.004 0.006 0.003 0.008 0.002 Generic One-sided p-value 0.022 0.003 0.004 0.321 0.001 211 Both interventions proved to be successful in their own right. The p-values for three of the four dimensions – Use of source materials, Structure and development and Academic writing style – were well below 0.05 for each group, and thus the improvement was statistically significant for each. Only on Dimension 4, Editing, did the improvement of the cross-disciplinary group not prove to be significant (p = 0.3205), which was predictable in the light of the fact that the performance of the group as a whole decreased by 1% between the pretest and the posttest. In order to establish whether the difference between the two interventions (subjectspecific and cross-disciplinary) was statistically significant, the Mann-Whitney U-test was applied. The Mann-Whitney U-test is the non-parametric equivalent of the independent samples T-test for assessing whether two independent samples of observations come from the same distribution, which is particularly useful for small samples. In statistical terms it assesses the ranked positions of scores in two different groups. If there are significant differences between the two groups, the p-value associated with the test statistic will be smaller than 0.05. The main finding was that overall, the subject-specific group performed significantly better than the crossdisciplinary group, as predicted by the simple comparison in Table 9.1. A p-value of 0.043 was obtained. Because of the significance of the overall difference found between the subject-specific and the cross-disciplinary interventions, separate Mann-Whitney U-tests were run for each of the four main dimensions of the holistic scoring instrument. Table 9.3 shows the p-values for the four dimensions, as well as the overall value. Two-sided values are reported because one group was not necessarily expected to perform consistently better than the other. Table 9.3 Two-sided p-values of the scores from the Mann-Whitney U-test Dimension DIMENSION 1: Handling of source materials DIMENSION 2: Structure and development DIMENSION 3: Academic writing style DIMENSION 4: Editing Overall p-value per dimension 0.223 0.809 0.020 0.020 0.043 212 According to the separate Mann-Whitney U-tests, the subject-specific group did not perform significantly better than the cross-disciplinary group on every dimension. A significant difference was only found with respect to Dimension 3, Academic writing style, and Dimension 4, Editing (p-value, in each case = 0.02). For both these dimensions significant differences were expected on the basis of the simple comparison in Table 9.1. Since the value of the fourth dimension, Editing, was derived from a single item (item 13) a generalization can not be made. It can only be concluded that the subject-specific group succeeded much better than the cross-disciplinary group in improving their spelling and appropriate use of capital letters. No significant difference was found with regard to Dimension 2, Structure and Development (p-value = 0.809). This was not surprising, because according to the tabulated comparison, the improvement of the two groups differs by a mere 3%: 18% for the subject-specific group and 15% for the cross-disciplinary group. According to my own belief, the basic principles of developing an argument at various levels of the text (the whole essay, paragraph and sentence) are largely subject-neutral, and can be taught and learned successfully through a combination of explicit instruction, model texts and sufficient exercise. Similarly, no significant difference between the two groups in terms of Dimension 1, Use of source materials (p-value = 0.223) was indicated by the Mann-Whitney U-test. This finding might seem to be contrary to the result of the simple comparison in Table 9.1. A larger sample may result in a significant p-value. Figure 9.1 shows the distribution of the data on the dimensions of the instrument comprising more than one ratable item (in other words Editing is excluded): 213 Figure 9.1 Box plots displaying the differences between the subject-specific and the cross-disciplinary intervention groups with regard to the three most important dimensions of the analytic scoring instrument according to the ranks assigned by the Mann-Whitney U-test The box plots representing the data for Structure and development and for Writing style show that the cross-disciplinary intervention data are less spread out than the subjectspecific intervention data. Furthermore, the middle 50% (between the 2nd and 3rd quartiles) overlap for both Use of source materials and Structure and development. As a matter of fact, for Use of source materials the distribution of the generic data constitutes a subset of the distribution of the subject-specific data. In the case of Academic writing style there is much less overlap: The middle 50% of the generic data clearly occupy lower ranks than the subject-specific data. In order to establish whether individual items may have influenced the p-values on the main dimensions of the scoring instrument, Mann-Whitney U-tests were run for all 13 individual items of the holistic scoring instrument (compare Table 9.4). No Bonferroni corrections were made for the multiple testing (to avoid the inflation of the type I error 214 rate) as it had already been established that a significant difference existed between the two groups. Table 9.4 Two-sided p-values of the raters' scores from the Mann-Whitney U-test per item Dimension Item DIMENSION 1: Handling of source materials 1 Relevance 2 Integration 3 Stance and engagement 4 Thesis statement 5 Development of argument 6 Conclusion 7 Paragraph development 8 Syntax 9 Concord and tense 10 Linking devices 11 Lexis 12 Style 13 Spelling and capitalization 16 DIMENSION 2: Structure and development DIMENSION 3: Academic writing style DIMENSION 4: Editing OVERALL p-value per item 0.051 0.349 0.654 0.756 0.863 0.557 0.223 0.005 0.314 0.099 0.114 0.387 0.020 p-value per dimension 0.223 0.809 0.020 0.020 0.043 This more detailed analysis identifies specific items that may have exaggerated or diluted the p-values of the dimensions. In the case of dimension 3, Structure and Development, Syntax (item 8, with a p-value to 0.005, and thus significant at the 5% level) and Linking devices (item 10, with a p-value of 0.099, indicating significance at the 10% level) greatly influenced the p-value for the dimension as a whole. On the other hand, although the difference between the intervention groups regarding dimension 1, Use of source materials, was not significant according to the Mann-Whitney test (p = 0.223) the p-value for one of the three items comprising the dimension (item 1, Relevance of source materials) indicates a statistical difference between the crossdisciplinary and the subject-specific groups at the 10% level (p = 0.051). It should be noted that the findings regarding the individual items were not surprising, and plausible explanations for significant differences (or a lack thereof) were not hard to find. Items 1 and 8 will be explored in more detail. With regard to Item 1 it can be argued that the history students actively engaged, quantitatively and qualitatively (in both their history classes and the academic literacy classes), with scholarly sources on a 215 specific theme, viz. The history of Apartheid in South Africa, with particular emphasis on the Native Land Act of 1913. They also became familiar with the core sources included in their history reader, which was also used for the essay-writing module. In contrast, the mixed group was exposed to fairly generic sources on the topic of focus, Poverty in Africa. They might have been less motivated than the subject-specific group to engage regularly with scholarly sources on this topic, since it was not necessary to internalize the content for assessment in their core modules. Furthermore, the students in this group were allowed to write their final exam essay on any of the topics provided by lecturers in the Faculty; and some of them chose topics that seemed to be interesting, but fell outside the focus of the academic programmes for which they were registered. For instance, one of the students, who was registered for a degree in Journalism, chose the topic Whose obligation is it to do something about poverty in society: the rich or the poor? This topic requires familiarity with philosophical ways of arguing. The student managed to structure her essay well and to invoke evidence from relevant sources, but she failed to exhibit mastery of the discourse of philosophy. Against this backdrop it is not surprising that the subject-specific group improved significantly more than the cross-disciplinary group on Item 1. The p-value of Item 8 can be explained as follows: Although none of the interventions paid specific attention to the improvement of syntactic well-formedness, the subjectspecific group had the advantage of becoming familiar with the historian's ways of formulation through extensive reading and writing in the discipline. During the course of the semester they wrote at least eight full academic essays on topics related to the history of Apartheid in South Africa. The respondents in the mixed group – with the exception of the two students who studied Philosophy – wrote only three full essays on aspects of poverty during the course of the semester-long essay-writing intervention. Although plausible explanations can be found for the p-values of the primary dimensions, with specific reference to the impact of individual items, the findings raise questions regarding the validity of the construct underlying the scoring grid. More specifically, they raise questions about the researcher's (and other researchers') clustering of items in analytic scoring instruments for academic writing. Specific questions include: 216 • Can it be claimed that grammar (which might include syntax and cohesive devices), lexis and style constitute the construct Academic writing style? • Is the ability to handle stance and engagement in any way connected to the ability to integrate facts and ideas from source materials in a composition, and the relevance of those facts and ideas to the topic at hand? We now turn to the discourse analyses of the pre- and posttest essays for possible justification of the statistical data, but more specifically to find evidence that might assist course designers to adapt or refocus syllabi and/or teaching materials for future essay-writing interventions. 9.3 Text analysis of pre- and posttests The discourse analyses that were performed on the essays were focused not so much on an overall impression of students' performance but were actually "enlarged detail" snapshots of students' abilities to handle key aspects of meaning-making in academic texts – as identified and described in the literature on Systemic Functional Grammar. Another aim was to explore the value of theory-supported discourse analysis in justifying rating scores. 9.3.1 Logical ideation On the dimension of logical ideation (logical relationships between intra- as well as extra-textual concepts) the subject-specific group showed a large improvement in handling Causation. There was an overall increase from 74 correct usages in the pretest to 134 correct usages in the posttest (= 81% improvement). A moderate improvement was found in the Addition category (from 62 to 79 = 27%), and a slight improvement in the Time/Tense category. Temporal setting and Temporal sequence were mastered fairly well, already at the time of the pretest, and little improvement was demonstrated in the posttest. However, the number of tense errors decreased dramatically (from 43 in the pretest to 12 in the posttest = 72%). A possible explanation is that the history students had never been explicitly taught how to handle tense in historical writing (personal communication with the lecturer). It is likely that the explicit instruction and continuous feedback during the intervention assisted them in internalizing the system. 217 The cross-disciplinary group showed an increase of more than 50% on three of the Logical Ideation categories: Addition, Causation and Time/tense, and 23% on Comparison. Analogous to the subject-specific intervention, Causation and Addition were handled well, and errors also decreased significantly in this category (an 80% decrease in Causation errors and a 60% decrease in Addition errors). It was also encouraging that more variety occurred in their use of causation resources in the posttest: In addition to the subcategories Cause and Consequence, also Condition, Means and Purpose featured prominently in the posttest. Similar to the subject-specific intervention, Temporal relations and Tense were handled well by the cross-disciplinary intervention students in both the pretest and the posttest, but in contrast to the subjectspecific intervention students (whose pretests contained many tense errors) the crossdisciplinary intervention students committed very few tense errors, even in their pretest essays. Only one tense error was recorded in the pretests and two in the posttests. This might be explained by the fact that in humanities disciplines other than history time does not play such a crucial role. 9.3.2 Appraisal The subject-specific students improved inconsistently in their use of Appraisal resources. They showed the most marked increase (47%) in the Attitude category, which includes the subcategories Emotion, Judgment and Social valuation. This may be ascribed to their increased content knowledge, and thus their confidence in evaluating historical figures, institutions and events. The category of Engagement produced disappointing results, in that there was an overall decline from 73 to 38 correct usages. This was mostly due to a decline in the number of Attribution markers (from 44 to 13). The only plausible explanation is that an increase in students' subject-field knowledge – resulting from attending lectures, reading, studying and intensive writing on the history of segregation in South Africa – made them less dependent on sources when writing the posttest essay. The students participating in the cross-disciplinary intervention, on the other hand, improved significantly in their command of Appraisal resources. In the Attitude category correct usages increased (from 10 to 55), in the Graduation category from 40 to 99, and in the Engagement category from 52 to 145. The steep increase in the use of 218 Engagement markers (Attribution, from 19 to 55 and Proclamation, from 33 to 94) stands in stark contrast to the decrease in the subject-specific intervention. Apparently, the emphasis that the lecturer for the cross-disciplinary intervention had placed on a command of Appraisal resources, and the increased amount of exercise in using these, paid off. It is likely that the cross-disciplinary group's increased use of stance and engagement markers, as opposed to the slight and inconsistent improvement by the subject-specific students, contributed to the fact that no significant difference was measured on item 3 by the Mann-Whitney U-test. 9.3.3 Thematic analysis The pre- and posttest essays of the first respondent in the subject-specific and the crossdisciplinary intervention respectively (henceforth Respondent S1 and Respondent G1) were sampled to analyze and plot thematic progression. In both cases improved capability to handle thematic progression was anticipated on the basis of the sizeable difference between the respondents' analytic scores on the pretest and the posttest: Respondent S1's overall score improved from 37% to 69%, and Respondent G1's score improved from 60% to 81%. Further predictors of improvement were the two respondents' scores on the dimension Structure and development, particularly on item 7 (Paragraph development). Respondent S1 scored 2 for this item on the 7-point scale in the pretest, and 5 in the posttest, whereas Respondent G1 scored 3 on the pretest and 6 on the posttest. Although the overall difference between S1's pre- and posttest scores (28%) was more impressive than the difference between G1's scores (21%), G1's scores fell into a higher bracket than those of S1, and thus it could be expected that the percentage of strong thematic bonds in G1's essays would also fall into a higher bracket than the number of strong bonds in S1's essays. This prediction was borne out by the findings: In S1's essays the number of strong bonds (in relation to the number of weak and absent bonds as percentages of the total number of clauses) increased from 38% to 76%, whereas the number of strong bonds in the G1's essays increased from 53% to 93%. Conversely, the number of weak and 219 absent bonds in S1's essays decreased from 62% to 23% and in G1's essays from 42% to 8%. These findings, which signify an impressive improvement in the case of the subjectspecific as well the cross-disciplinary intervention student, are in line with the statistical finding of no significant difference between the two interventions on the dimension of Structure and development. 9.3.4 What the discourse analysis reveals Although no grand generalizations can be made on the basis of these quasicomparisons, there is a clear indication that both the subject-specific and the crossdisciplinary interventions afforded students tools and mechanisms to improve their academic writing. At least some of these resources must have been internalized to facilitate the improvement that took place between the pretest and the posttest. In general, the students who took part in the subject-specific intervention became less reliant on sources, which might have impacted negatively on their explicit use of stance and engagement markers, but could have contributed to the enhanced relevance of the source materials (facts) they used in their essays. In contrast, the students in the crossdisciplinary group acquired a more marked command of stance and engagement than their subject-specific counterparts. They also demonstrated a more varied repertoire of cohesive devices. However, it is more likely that the intensified focus of the generic module on these lexicogrammatical devices (as a result of what the course designer had learned from the subject-specific intervention) had caused the improvement, and not the contextual focus (subject-specific or generic). Based on the thematic analysis of two sample essays (one per intervention) students in both groups benefited from the intervention in terms of developing an argument systematically. This inference is supported by the statistical finding that there is no significant difference between the subject-specific and the cross-disciplinary group in terms of improvement on the dimension of Structure and development. 220 9.4 Opinion survey In order to compare the results of the post-intervention opinion surveys, a MannWhitney U-test was performed for each of the five theoretical dimensions of the opinion survey, viz. (1) Staged and scaffolded teaching and learning model, (2) Purposeful social apprenticeship, (3) Needs-driven syllabus, (4) Critical orientation and (5) Skills transfer. For each of the two interventions the total score for the items comprising each dimension was obtained. The scales were reversed where necessary to facilitate uniform polarity. The spiderweb plot represented as Figure 9.2 shows the differences between the means of the responses of the two groups (after reversal of the scales with a negative polarity): D1 Scaffolding 4 Cross-disciplinary Subject-specific 3 2 D5 Skills transfer 1 D2 Apprenticeship 0 D4 Critical orientation D3 Needs-driven syllabus Figure 9.2 Spiderweb plot of the means of the subject-specific and the crossdisciplinary groups in the opinion surveys From the graph it can be read that, on average, both groups felt reasonably positive about the way that a staged and scaffolded teaching and learning intervention assisted them in improving their academic writing skills (dimension 1); that both groups were, to a large extent, convinced of the positive effects of learning as a member of a discourse community (dimension 2); and also that the module had addressed their personal needs and goals reasonably well (dimension 3). On the other hand, both groups were uncertain as to the effect that the course might have had on their development of a critical 221 orientation (dimension 4). Although both groups were positive about the transferability of the skills they had learned (dimension 5), a predictably higher rating on this dimension (between positive and extremely positive) was obtained from the subjectspecific group. According to the Mann Whitney U-test (compare Table 9.5) the overall opinion of the two groups about the intervention did not differ significantly. As suggested by the differences in the statistical means for Skills transfer, a significant difference (at the 10% level) was found on this dimension: Table 9.5 Two-sided p-values of the opinion survey findings regarding the 5 theoretical dimensions, obtained from the Mann-Whitney U-test Dimension p-value 1 Staged and scaffolded teaching and learning model 0.209 2 Purposeful social apprenticeship 0.260 3 Needs-driven syllabus 0.568 4 Critical orientation 0.130 5 Skills transfer 0.081 TOTAL 0.860 The subject-specific group was thus more inclined to think that the skills they had learned in the course were indeed transferable to other contexts. Further analysis of the data showed that although some students in the cross-disciplinary group were convinced that they could apply what they had learned to more than one discipline, others were much less positive about the transferability of the skills. 9.5 Conclusion From the multifaceted comparison described in this chapter it can be concluded that both the subject-specific and the cross-disciplinary interventions were effective in their own right. In both cases there was a significant improvement in students' writing abilities between the pretest and the posttest: For the subject-specific intervention a pvalue of 0.002 was obtained, while a p-value of 0.001 was obtained for the crossdisciplinary group. 222 A statistical comparison of the performance of the two groups reveals that the students who took part in the subject-specific intervention improved significantly more than those who took part in the cross-disciplinary intervention. Percentage-wise the subjectspecific group improved by 19% overall, while the cross-disciplinary group improved by 8% − a difference which proves to be significant according to the Mann-Whitney Utest: p = 0.004. The improvement of the subject-specific group was also more consistent across the four dimensions of the scoring instrument than the improvement of the crossdisciplinary group. Although both groups expressed fairly positive opinions about the intervention in general, the subject-specific group was significantly more positive than the crossdisciplinary group about the transferability of the skills they had learned. A p-value of 0.086 was obtained on the Mann-Whitney U-test, which means that the difference is significant at the 10% level. These results indicate that genre-specific writing interventions can be effective, whether narrowly or more broadly focused. However, interventions that are more sharply focused on a particular discipline seem to be more effective, primarily as a result of enhanced motivation and more profound engagement with the subject matter through reading and writing with clearly delineated disciplinary foci. Although language proficiency, especially grammar, might not noticeably improve through explicit teaching of lexicogrammar, the findings of this study indicate that a greater awareness of the lexicogrammatical resources can be facilitated through explicit teaching and tasks that make use of authentic materials. 223 Chapter 10: Conclusion 10.1 Introduction The following research questions were formulated in the first chapter to address the issue of undergraduate students' inadequate academic writing abilities: (1) Can genrebased approaches be justified theoretically? (2) How effective are genre-based academic literacy interventions? (3) Which are more effective: specific or generic approaches? Question 1 relates to the input for and justification of the proposed applied linguistic design, while questions 2 and 3 relate to implementation and evaluation of two variations on a particular language teaching approach. Figure 10.1 shows how the research questions have been accommodated in the research design: IMPLEMENTATION: QUASI EXPERIMENT SUBJECT-SPECIFIC INTERVENTION: ↓ GENERIC INTERVENTION ↓ EVALUATION EVALUATION COMPARISON DESIGN Instructional model Genre-based presyllabus Justification THEORETICAL FOUNDATION TECHNICAL IMAGINATION PROBLEM Figure 10.1 Summary of research strategy to address the research questions This chapter attempts to indicate to what extent the research questions have been answered in order to make evidence-based recommendations for the design of future 224 academic writing interventions aimed at undergraduate students in the humanities. First, the theoretical justification of genre-based writing interventions is summarized. This is followed by an overview of the effectiveness of narrow-angled versus wide-angled interventions, and the significance of the difference between the two, as proven by the empirical research. Finally, some limitations of the study are briefly discussed, and recommendations are made regarding the application of the knowledge gained throughout the research process. 10.2 Theoretical justification Genre approaches to teaching academic literacy have drawn from a diversity of linguistic, applied linguistic and language teaching theories. Figure 10.2, which should be read bottom-up, gives a schematic overview of the theories that underpin genrebased approaches: WRITING PEDAGOGY Skills-based approach Practice-based approach Text-based approach LANGUAGE TEACHING THEORIES Multiliteracies Communicative language teaching Traditional Approaches Critical literacies APPLIED LINGUISTIC THEORIES Postmodernism Constructivism Extended Paradigm Model Linguistic approach LINGUISTIC THEORIES CDA Multimodality New Rhetoric Cognitive Linguistics Systemic Functional Linguistics Sociolinguistics Figure 10.2 Theoretical foundations of genre-based writing pedagogies Among the linguistic theories, Systemic Functional Grammar is the theory that is regarded to have contributed most significantly to the theoretical grounding of genre pedagogies. SFL emphasises the systematic way in which language users make vocabulary and grammar choices in particular cultural and situational contexts. This 225 paradigm has had a profound influence on particularly the Australian (Sydney) genre school. Other linguistic theories that have been referred to for justification of specific features of genre approaches are Cognitive Linguistics and Critical Discourse Analysis. Cognitive Linguistics foregrounds genre knowledge: knowledge of content, communicative purpose, participant roles, discourse structure, and register. Critically oriented theories of language and other semiotic systems, such as Critical Discourse Analysis, add a political dimension to genre knowledge, viz. knowledge of power relations and institutional processes, and also emphasize the dialogic relationship between culture, cognition and semiosis. Among the genre schools it is particularly the New Rhetoric and the Sydney schools that are associated with CDA, because of their emphasis on social and intellectual empowerment through genre knowledge, as well as their encouragement of students and professionals to challenge the hegemonic power of conventional genres. The theory of learning that best supports genre approaches is Constructivism. Genrebased approaches draw strongly upon the work of Vygotsky, in particular his Zone of Proximal Development. The ZPD is supported by two pillars, viz. cognitive and social apprenticeship, and scaffolding. Cognitive and social apprenticeship are linked to the rhetorical notion of learning as a member of a discourse community, while a scaffolded curriculum aims at initially providing strong peer and teacher support, and then gradually removing the support until the learner is knowledgeable and confident to construct full examples of the genre independently. Vygotskyan views feature prominently in all the so-called "post-process paradigms" in academic writing pedagogy. The methodological input that genre-based approaches have received from language teaching theories derives particularly from Communicative Language Teaching. However, regarding the types of activities included in genre-based teaching programmes, genre-based pedagogy also draws from Traditional approaches and Critical Literacies approaches. When narrowing down the focus to theories of academic writing, it is clear that genre approaches combine Text-based and Practice-based approaches: Text-based approaches 226 draw on the resources of linguistic analysis to understand the functional (rhetorical) and discipline-specific nature of writing tasks. Practice-based approaches emphasize the social and discursive practices through which disciplines constitute themselves. The version of genre-based pedagogies adopted for the present research has relied heavily on Systemic Functional Linguistics with regard to drawing on the established conventions and values of academic disciplines, and making meaningful form-function choices. This version is in essence constructivist, in that the role of the learner as an active maker of meaning is emphasized, as well as the role of the teacher and peers as engaging in dialogue with the learner to create new meaning. The approach is overtly post-process, in that it is a considered combination of language teaching principles and techniques as well as classroom activities, with sufficient opportunity for critical reflection. In the next section an overview is given of the design and evaluation of genre-based writing interventions drawing on the above theories. In particular, the question of effectiveness is addressed. 10.3 The effectiveness of genre-based approaches in general Although a large number of empirical and quasi-empirical studies have been conducted to establish affinities between genres, text types and disciplines at tertiary institutions, it is believed that in situ research is a prerequisite for designing effective interventions − in this case genre-based academic writing courses for second-year undergraduate students of the humanities at the University of Pretoria. To chart the target landscape, a survey of writing tasks was conducted. The results showed that the academic essay is the written genre most frequently required by lecturers of humanities disciplines, and that academic essays are made up of a variety of rhetorical modes. Those that feature most prominently are discussion, explanation, description and (critical) analysis. Subject-fields differ with regard to the rhetorical modes they prefer, the labels they use, and the way they combine different modes. On the other hand academic essays are structured in a fairly similar way across disciplines. 227 They typically comprise an introduction, body and conclusion, and develop an extended academic argument, supported by evidence. However, the nature of the evidence is subject-specific. Against this backdrop it seems that both narrow-angled (subject-specific) interventions, with a close fit between the purposes and conventions of disciplinary communities, and more wide-angled (generic) interventions, which focus on one or more genres shared by a cluster of disciplines (such as the academic essay), could be effective. This is probably the reason why these two distinct approaches still exist within the domain of language pedagogy. However, few experimental or quasi-experimental studies have been conducted to prove the desirability (or the feasibility) of either of these intervention types. The design of any genre-based intervention is ideally preceded by thorough contextual research. For the purpose of designing and evaluating a subject-specific intervention, indepth research was conducted on the conventions of historical writing. History was chosen as the discipline of focus for a subject-specific intervention because the academic essay has been found to be the primary vehicle for undergraduate historical writing. The choice of history as the focal subject was also purposive and convenient, since the Department of Historical and Heritage Studies had expressed interest in the project and had been willing to offer its cooperation. Main findings were that the three main purposes of historical writing are (re)telling a story, understanding and explaining why things happened as they did, and evaluating events, structures and the writings of other historians. In historical texts these concepts have been lexicalized and grammaticalized in systematic ways. Concerning time, seven categories, straddling the boundaries of syntax and semantics, were defined to assist course designers and students in constructing and deconstructing time in historical texts, viz. sequencing time, setting in time, temporal process (phasing in time), (text internal) temporal organization, temporal modality, temporal duration and tense. Two primary ways of construing cause and effect were distinguished: sequential (chronological) causal relations between external events, and "simultaneous" mentioning of causes or effects. In terms of judgment or evaluation the most important categories for the historian are Attitude, 228 Graduation and Engagement, as distinguished in the Appraisal framework within Systemic Functional Linguistics. In order to design a cross-disciplinary intervention research was conducted on the relationship between disciplinary purposes and writing conventions in a number of humanities disciplines, including philosophy, sociology, psychology, history of art and political sciences. Summaries were made of the most important conventions, and exemplars of essays and parts of essays were excerpted from these sources. A basic genre-based presyllabus, comprising one or more cycles of exploration, explicit instruction, joint construction, independent construction and critical reflection, was adapted for subject-specific and generic purposes. Emphasis was placed on rhetorical modes, logical development of an argument from the thesis statement to the conclusion, and engagement with the authors of primary and secondary sources. However, the syllabi differed with regard to the specificity of the disciplinary focus, and thus also the themes of the materials and exercises. 10.4 The effectiveness of narrow-angled versus wide-angled interventions The statistical analyses of the essay scores show that both narrow-angled and wideangled genre-based interventions can be effective. The overall improvement of the students in both groups was statistically significant, although the size of the improvement differed across the four dimensions of the scoring instrument. Only on the dimension of Structure and development was the improvement of the two groups roughly similar (18% in the case of the subject-specific group and 15% in the case of the cross-disciplinary group). Thematic analyses of sampled pre- and posttest essays support the finding that both groups benefited from the instruction on structure and development: both students showed a sizeable improvement in their ability to develop an academic theme systematically. Students from both groups were generally positive about the effect of the intervention on their academic writing abilities, and indicated that their personal needs had been more than adequately addressed. On the other hand, both groups were less positive 229 about their acquisition of critical thinking skills, which was not surprising in the light of the responses typically given by undergraduate students to questions about their acquisition of critical thinking skills. The only significant difference between the two groups was their perceptions about skills transfer. The subject-specific group was more positive, which probably went hand in hand with the fact that they engaged more with relevant subject matter and were more motivated. Despite their more modest overall improvement in comparison with the subject-specific students, the cross-disciplinary group exceeded the researcher's expectations in terms of their mastery of Appraisal resources, particularly Attitude, Engagement and Graduation. The steep increase in their use of Engagement markers stands in stark contrast to the decrease in the essays of the subject-specific group. The improvement in the crossdisciplinary group's mastery of Appraisal resources should probably be ascribed to the lecturer's efforts in exposing the generic students more explicitly to these resources, and designing more appropriate classroom materials. 10.5 Limitations of the study The main limitations of the study include (1) the relatively small sample size, (2) using the scores of only one rater for the subject-specific essays, and (3) the fact that the two interventions were not administered simultaneously. The small sample size may be seen to have impacted negatively on generalization. However, the statistical tests that were chosen (the Wilcoxon signed-rank test and the Mann-Whitney U-test) compensated for this limitation, as they had been designed for small samples. The non-parallel presentation of the interventions limits comparability because the syllabus and materials for the cross-disciplinary intervention were designed with some foreknowledge of what had (not) worked well in the subject-specific intervention. This could have influenced the significance of the statistical difference found between the two groups, as well as differences found in students' use of certain lexicogrammatical resources. 10.6 Summative remarks and the way forward Although it would be dangerous to make grand generalizations on the basis of a quasiexperiment with fairly small samples there is a clear indication that genre-based, 230 scaffolded interventions do assist students in mastering the structural, conceptual and linguistic resources for meaning-making in academic discourse. On the basis of the findings it is believed that subject-specific interventions have a greater chance of succeeding than wider-angled interventions. Their greater success is primarily ascribed to the enhanced motivation that accompanies students' prospects of improving their achievement in content subjects. This prediction is underpinned by the finding that the superior performance of the subject-specific group is statistically significant. Furthermore, it seems that transferability of skills – or at least students' perception of transferability − is enhanced by extensive reading and writing with a particular thematic, and by extension, disciplinary focus. Although narrow-angled interventions seem to be more beneficial than wide-angled interventions, such interventions may, however, be less feasible in that few tertiary institutions have the resources for offering dedicated writing modules – one for each discipline. This suggests research on alternative models for subject-specific teaching of academic writing, such as collaboration with content lecturers in a team-teaching or adjunct teaching context. It also points to the exploration of a combination of narrowangled and generic designs in the same course. Despite the less pronounced effects of wide-angled writing interventions, they do have some effect, and are therefore better than no intervention at all. This has been demonstrated by the significant improvement of the students on the generic course. Definite advantages of cross-disciplinary interventions are the opportunities they afford for making students aware of the dimensions along which subject-fields differ, and acquainting students with the conventions that tie in with the content, epistemology and philosophical underpinnings of a range of subject-fields. If capacity is available, a department, unit or centre with responsibility for teaching academic writing should conduct research on the relationships between subject-field purposes and writing conventions in a whole range of disciplines. It should also be considered to develop genre-based training courses for tutors, who might be Masters or PhD students in the disciplines where writing support for undergraduate students is desired. 231 In any event, attention should be paid to gathering, designing and developing authentic materials. It is desirable to compile a database of authentic model essays to demonstrate the successful application of essay-writing principles in specific subjects, at the level of the students, and relevant to the local (at least the South African) context. Published examples of good essays lack authenticity and are often not exemplary in every respect. In addition, students should ideally focus their writing on a particular discipline for the duration of a semester-long writing module, even within the boundaries of wide-angled modules. The greater effectiveness of the subject-specific intervention has shown that immersion into the content and materials of a specific discipline enhances engagement and encourages skills transfer. Finally, students should ideally be engaged in extended writing assignments from the beginning to the end of an intervention: the more text students produce, the more significant their improvement is likely to be.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project