McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Lam, D., Sullivan, S., Kim,
L., Rus, V., Floyd, R., McCarthy, P. M., & Graesser, A. C. (in press). The Writing-Pal: Natural language algorithms to support intelligent tutoring on writing
strategies. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution.
Hershey, PA: IGI Global.
The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on
Writing Strategies
Danielle S. McNamara, Roxanne Raine, Rod Roscoe, Scott Crossley, G. Tanner Jackson, Jianmin Dai, Zhiqiang Cai, Adam Renner, Russell Brandon, Jennifer Weston, Kyle Dempsey, Diana Lam, Susan Sullivan, Loel Kim, Vasile Rus, Randy Floyd, Philip McCarthy, and Art Graesser The University of Memphis The Writing‐Pal (W‐Pal) is an intelligent tutoring system (ITS) that provides writing strategy instruction to high school students and entering college students. One unique quality of W‐Pal is that it provides feedback to students’ natural language input. Thus, much of our focus during the W‐Pal project has been on Applied Natural Language Processing (ANLP). This chapter describes W‐Pal and various NLP projects geared toward providing automated feedback to students’ writing during writing strategy training and practice. Our motivation to develop W‐Pal rests on two underlying assumptions. First, we assume that writing well is important to success academically as well as professionally (Geiser & Studley, 2001; Light, 2001; Sharp, 2007). Writing skills allow individuals to articulate ideas, argue opinions, and synthesize multiple perspectives. Effective writing is essential to communicating persuasively with others, including teachers, peers, colleagues, co‐workers, and the community at large (Connor, 1987; Crowhurst, 1990; National Commission on Writing, 2004). Second, we assume that strategies facilitate performance on tasks, and that teaching students to use strategies can hasten the acquisition of skills (McNamara, 2009). Strategies have been found to facilitate and enhance performance on a variety of learning‐related tasks, which leads to the expectation that the same might be found for writing. Many students lack the skills necessary to successfully communicate in writing. For example, the 2002 NAEP report (Institute of Education Sciences, 2003) indicated that more than two thirds of American students scored below their proficiency levels in writing assignments (4th graders: 72%; 8th graders: 69%; 12th graders: 79% below proficiency appropriate for their grade level). In addition, only 2% of these three sample grades wrote at advanced levels. We believe that the solution lies not in continuing to correct their grammar and spelling (Shaughnessy, 1977), but rather in teaching students powerful writing strategies that scaffold them toward more effective written communication, meeting a wide array of writing needs applicable to many writing genres. The difference between skills and strategies is central to understanding the purpose and intent of W‐Pal. Skills are acquired through deliberate practice over long periods of time (Ericsson, 2006). For example, reading skills are acquired from very early childhood to young adulthood (Ericsson & Kintsch, 1995). Like reading skills, writing skills also take time to acquire. Skilled writers have likely engaged in extensive deliberate practice, received feedback on their work, and have likely written for a wide variety of genres (e.g., essays, letters, summaries, short responses, etc.). Through these experiences, children slowly acquire the necessary skills to be successful writers. For example, the use of correct grammar is gained through extensive instruction, practice, and feedback. Likewise, correct spelling is based on a relatively stable body of knowledge, constructed over time, again with extensive instruction, exposure, practice, and feedback. Correct grammar and spelling cannot come from a simple mnemonic or a couple of hours of practice. Hence, grammar and spelling are considered skills, and are, by consequence, not a focus of W‐Pal. By contrast, freewriting is a strategy that can be taught and practiced relatively quickly (and refined over time through practice). Similarly, there are strategies to help students plan, construct, assess, and revise their writings. These strategies are rules of thumb, short‐cuts, and mnemonics that can help less skilled students to compensate for their weaknesses in the short term and become skilled writers in the long term. An Overview of W‐Pal W‐Pal consists of two principle components: Strategy Training and Essay Writing. The Strategy Training Component includes lessons that correspond to the three phases of the writing process: Prewriting, Drafting, and Revising. Training modules are provided for strategies that facilitate each phase of writing. Prewriting modules include (a) Freewriting, and (b) Planning. Drafting strategies include (a) Introduction Building (b) Body Building, and (c) Conclusion Building. The revising strategies include (a) Paraphrasing, (b) Cohesion Building, and (e) overall Revising of the text. Thus, there are eight strategy training modules, in addition to a Prologue Module that introduces the students to the program. W‐Pal lessons are presented by three pedagogical agents, Dr. Julie, a teacher agent, and Sheila and Mike, two student agents. The student agents learn the strategies from Dr. Julie, engaging in discussions and asking questions about the strategies. For example, in Figure 1, Dr. Julie (on the left) is explaining the importance of writing skills to Sheila and Mike. Each module consists of an introductory lesson and several guided practice activities. The second component of W‐Pal is the Essay Writing module. In this module, students write complete essays and are provided with feedback on the essay and suggestions to use particular strategies to improve the essay. The students can view particular modules and choose to write complete essays at any time. Although there is an implicit, recommended sequence for these modules, the students are not obligated to complete the modules in any particular order. Figure 1. The Prologue discusses how writing skills are beneficial within the classroom and beyond.
Essay Genre W‐Pal targets the needs of high school students, who are often required to write short prompt‐
based essays as a means toward the fulfillment of college and university entrance requirements. The following are examples of prompts used for these types of essays: Example Prompt 1. Many people believe that to move up the ladder of success and achievement, they must forget their past, repress it, and let it go. But others have just the opposite view. They see their old memories as a chance to reckon with their past and integrate past and present. Do personal memories hinder or help people in their effort to learn from their past and succeed in the present? Example Prompt 2. The arts—literature, music, painting, and other creative activities—are considered important by some but not by others. Many people consider the arts unnecessary because they provide us with nothing more than entertainment. Yet, others believe that the arts are extremely valuable because they teach us about the world around us and help people find meaning in life. What is the main value of the arts in society today? Typically, prompt‐based essays are persuasive (argue for one side of an issue), relatively short (they must be written under time constraints of 20‐25 minutes), and source‐independent (students are not allowed access to external references). These essays have an archetypal structure of an introduction paragraph, approximately three body paragraphs, and a conclusion paragraph (Albertson, 2007; Myers, McCarthy, Duran, & McNamara, in press). W‐Pal focuses on this genre of essay for a number of reasons: secondary school educators tend to cover this kind of essay in their curriculum, college entrance exams such as the SAT usually require this type of essay, and professionals benefit from skills with writing these persuasive essays. In addition, strategies for this genre of essay are basic: they are applicable to virtually all genres of essays. Although W‐Pal is intended for high school students, the writing strategies that W‐Pal teaches are sufficiently general to be applicable and effective for a wide range of audiences. Underlying Principles and Structure W‐Pal was founded on the successful implementation of the intelligent tutoring system, iSTART, which was developed to teach reading strategies to high school students (McNamara, Levinstein, & Boonthum, 2004). Like iSTART, W‐Pal focuses on a set of well documented strategies that are taught by pedagogical agents who guide the student through presentation and practice sessions. W‐Pal modules follow the three heuristics that are used in iSTART (McNamara, O’Reilly, Rowe, Boonthum, & Levinstein, 2007). The first is that to‐be‐learned information needs to be presented, modeled, and practiced following a faded scaffolding model (Collins, Brown, & Newman, 1989; McNamara et al., 2007; Rogoff, 1990). The second is that of vicarious learning (Bandura, 1997; Craig, Graesser, Sullins, & Gholson, 2004; Zimmerman & Risemberg, 1997), whereby the learning process is modeled by animated pedagogical agents. The third is that the modules are interactive, eliciting responses from and holding interactive dialogues with the student. As such, computational linguistic algorithms are needed to guide the interactions with the students and to provide adaptive feedback. These heuristics help to ensure that the system induces self‐reflective, generative, and metacognitive learning on the part of the students. Lesson content within W‐Pal is based on English Composition curricula and is designed to provide the users with strategies that will help students to compose a persuasive essay. The scripts for the interactions were developed through a collaborative process between ITS experts and composition experts. The lessons average about 20‐30 minutes in length, and are accompanied by an introductory text and a summary text that compliment the videos in two ways: They allow teachers to have paper copies of the lessons, and they also allow the students to review the lessons without necessarily re‐
watching the interactive videos. The W‐Pal interface also contains a notepad feature that allows students to digitally record information within the program. The lessons are presented in the contexts of a virtual classroom. This classroom features a whiteboard to present information as the students and teacher describe it during discussion. The lessons are created through a combination of Media Semantics Character Builder and Loquendo Text‐to‐Speech Engine. The whiteboard information is created within the Character Builder software or inserted from slides designed in Microsoft PowerPoint. Lessons are interspersed with brief checkpoints. Throughout the lessons, students practice the strategies by completing brief exercises (i.e., checkpoints) and completing practice modules (i.e., challenges). Checkpoints are brief probes, multiple choice questions, tasks, or mini‐games inserted into the lessons that help to maintain student engagement, discourage passive viewing, and provide reinforcement of learning through multiple testing opportunities. Challenges are longer than checkpoints and are intended to be accessed after the corresponding lesson has been completed. To potentially enhance student engagement, all of the challenges are designed as games (McNamara, Jackson, & Graesser, 2010). Many of the challenge practices parallel the structure of one or more of a lesson’s corresponding checkpoints. The following are brief descriptions of the W‐Pal modules. In the section following, we describe our work to develop the NLP algorithms to support these modules. W‐Pal Modules Module 1: Prologue. The prologue module provides the context and purpose of the W‐Pal training. During this module, the student agents express a general discontent with their writing grades and are concerned because they do not understand how to fix their problems. The teacher explains to them that she can help with their essays. She also explains how the ability to write well will help them in their lives in general (see Figure 1). The teacher explains the structure and purpose of a persuasive essay and then presents the strategies that the student will learn throughout the W‐Pal program. These presentations are brief and merely an exposure to the strategies. Because the prologue module is intended to prepare the students for the content‐based modules that follow, it does not contain any checkpoints or challenges. The remaining modules all contain checkpoints and challenges that allow the students to practice using the strategies. Module 2: Freewriting. The Freewriting module provides strategies for establishing ideas before planning the essay. During freewriting, the student writes, without stopping, as much as possible in a given period of time (e.g., 5 minutes). Students are taught the mnemonic FAST PACE to facilitate the freewriting and idea generation process. The letters in FAST PACE stand for techniques that students can use to generate ideas: Find evidence, Ask and answer questions, Spell out your argument, Think about the other side, Re‐read the Prompt, Add details, Connect ideas, and Provide more Examples. Freewriting helps writers generate ideas. Then, with a little organization after the freewrite, writers have a better idea of what they can write about, what they would need to learn more about, and what they should include in their essay. Module 3: Planning. After freewriting, the student needs to plan the essay and choose the ideas that are most relevant to a chosen position. The planning module is designed to help the student visually: (1) establish a position, (2) establish arguments for that position, (3) establish support for the arguments, and (4) integrate these components into an organized structure. Students are exposed to a variety of ways to organize an essay and learn strategies that facilitate that process. Module 4: Introduction Building. The goals of an essay introduction are to clearly state the position on the topic, provide a preview of the remainder of the essay, and to engage the reader’s interest. The Introduction Building module includes the TAG strategy as a mnemonic device for required elements of a good introduction: a clear Thesis statement, a succinct preview of the main Arguments that will be explained in the body, and a stylistic technique that Grabs the reader’s attention. The students also learn more specifically about attention‐grabbing techniques to make their introduction more engaging, including: ask‐a‐question, historical review, set a scene, personal anecdote, and controversy. Module 5: Body Building. Body paragraphs have two main features: A topic sentence and evidence sentences. The Body Building lesson teaches the mnemonic Keep Initial Sentence Short (KISS) and tell. Students should have a short topic sentence (KISS) that is explained further in their evidence sentences (tell). Evidence sentences are crucial to body paragraphs because they support the topic sentence. They are usually longer than topic sentences because it takes more words to explain than to state a claim. During this module, students learn how to build evidence sentences from outlined concepts. In this way, the student is scaffolded towards composing a body paragraph from outlines. Module 6: Conclusion Building. Conclusions summarize and reiterate the author’s position and major arguments, wrap up the essay, and do so in an engaging manner. The RECAP mnemonic helps students to keep track of these requirements: Good conclusions need to Restate the thesis of the essay, Explain how the arguments supported the thesis, Close the essay using a closing phrase (e.g., In conclusion), Avoid adding new arguments and information, and Present the conclusion in an interesting way. This module also describes four stylistic techniques that can be used to hold the reader’s attention and conclude essays in a memorable way (i.e., personal anecdotes, further research, importance, and make a recommendation). These techniques are also intended to reduce writing anxiety, and help students wrap up an essay more easily. Module 7: Paraphrasing. Paraphrasing is revision at the sentence level. Its primary purpose in the context of W‐Pal is to improve writing. The central purpose of the lesson is for students to learn that a sentence can be written in many different ways. The lesson progresses through several paraphrasing techniques (e.g., changing words, changing structure, changing words and structure, and condensing sentences). Checkpoints prompt students to reflect on the many different ways a sentence can be reworded. During this lesson, students generate several paraphrases using different techniques, and afterwards they are shown expert paraphrases to reinforce the notion that there are multiple ways to word a sentence. This feedback also helps to demonstrate how the meaning of a sentence can be preserved despite changes in structure or wording. Module 8: Cohesion Building. This module targets revision at the inter‐sentential level. Coherence refers to the overall sense of unity in a text, which is achieved by clearly expressing the relationships between ideas in the text using cohesive devices. W‐Pal provides instruction on three cohesion strategies for linking ideas across two or more adjacent sentences: Threading, This‐and‐That, and Connectives. Threading is the term used for linking sentences with common words or phrases; This‐and‐
That refers to resolving ambiguity when using demonstratives (words that depend on an external frame of reference); and Connectives refers to the use of words that make explicit the logical relations between two clauses or sentences (e.g., therefore, nevertheless, meanwhile, etc.). Module 9: Revising. The revising module targets revision at the global level. Students often see revision not as a chance to develop and improve a piece of writing, but merely to hastily correct any typos or cosmetic errors. Revision, however, is the very essence of the writing process. The primary purpose of this lesson is for students to learn the importance of getting a “big picture” view of the whole draft and to revise the entire draft to ensure the coherence of the essay. The ARMS mnemonic suggests courses of action students could take when revising an essay, including Adding, Removing, Moving, or Substituting ideas and text. The TETRIS mnemonic focuses on specific elements of the essay to assess: Thesis and conclusion, Evidence, Transitions and organization, Relevance, Interest, and Spelling and grammar. This helps students think critically about particular sections or qualities of the essay, and how each piece contributes to the essay. Essay‐Writing Component. The Essay Writing Component scaffolds the writing of complete compositions by providing feedback on a complete essay and suggestions on strategies that might help to improve the essay. The student is provided with an essay prompt (see Examples 1 and 2 earlier) and instructions to write a persuasive essay. After freewriting and planning, and writing an initial version the essay, the student is provided with feedback coupled with suggestions to use corresponding strategies that they learned during the lessons. Natural Language Processing in W‐Pal The foundation of much of the NLP in W‐Pal comes from Coh‐Metrix. Coh‐Metrix is an automated tool that provides indices on multiple linguistic features of language. This tool is described more fully in several other chapters within this volume (e.g., McNamara & Graesser, in press). It, as well as other similar tools developed in our laboratory, allow us to fully explore the linguistic features of writing. Predicting the Quality of Essays One of our goals in the W‐Pal project has been to explore the extent to which linguistic features are predictive of the quality of essays. In the context of W‐Pal, our overarching goal is to provide feedback to the students on their essays, particularly within the Essay Writing Component. Hence, a good deal of effort has been devoted toward the collection and NLP analyses of sample essays. We have been particularly interested in the role of cohesion in the quality of essays. This interest is partially motivated by Coh‐Metrix, which was developed expressly for that purpose: to provide indices on linguistic features associated with and indicative of text cohesion. Cohesion refers to the presence or absence of cues in the text that help the reader to infer relationships between ideas expressed in the text. These cues include referential overlap (words or ideas that are repeated across sentences or larger sections of text) and connectives (words such as because, however, and therefore, that express relationships between ideas). When there are more explicit cues, the text is considered higher in cohesion and is more easily understood by the reader (McNamara, 2001; McNamara, Louwerse, McCarthy, & Graesser, 2010). Likewise, many have assumed that cohesion plays an important role in writing (e.g., Collins, 1998; Connor, 1990; DeVillez, 2003; Witte & Faigley, 1981). Better writing is assumed to be more cohesive and cohesive devices are assumed to be necessary elements of a text that afford effective communication and facilitate the goal of conveying a writer’s arguments. To explore these assumptions, McNamara, Crossley, and McCarthy (2010) analyzed a corpus of 120 argumentative essays written by college undergraduates and scored by expert raters. The essays were scored on a 1‐6 scaled SAT rubric and then categorized as high or low quality essays. The results indicated that no indices of cohesion (e.g., word overlap, causality, connectives) were associated with differences between higher and lower quality essays. More specifically, the results indicated that high quality essays can be high or low in cohesion, and that low quality essays can be high or low in cohesion. In contrast, the indices that were found to be related to essay quality were those that render comprehension more, rather than less, challenging. These indices included lexical diversity, word frequency, and syntactic complexity. The essays were rated as being higher in quality when they had a greater variety of words, the words were less frequent in language (i.e., less familiar), and the syntax was more complex. These three indices accounted for 22 percent of the variance and within a discriminant analysis correctly classified 67 percent of the texts. The results of the McNamara et al. (2010) study provided initial evidence that text cohesion may not be indicative of essay quality. That is, expert raters in the McNamara et al. study appear to judge texts that are more difficult to process as more proficient. This trend has also been found among second language (L2) writers. Crossley and McNamara (in press) examined the extent to which linguistic features predicted essays scores for 1200 L2 college level essays. The results, similar to those reported for L1 essays by McNamara et al., indicated that higher quality essays were more lexically diverse, composed of less familiar words, and were more syntactically complex. Indeed, indices of cohesion were negatively correlated with essay quality. The results indicated that higher proficiency L2 writers do not produce essays that are more cohesive and readable, but instead produce texts that are more linguistically complex. These results prompted the question of whether providing training in W‐Pal to be more cohesive might actually be detrimental to students’ writing quality. Specifically, will recommendations to use connectives and to increase referential overlap actually decrease scores? We have explored this issue from two angles. The first was to verify that human ratings of essay coherence and continuity were positively related to essay quality. Indeed, Crossley and McNamara (2010) found that expert ratings of coherence and cohesion correlated .80 and .65 respectively with overall ratings of the essay quality. Interestingly, Coh‐Metrix indices of cohesion were negatively related to expert’s ratings of coherence. Thus, the absence of cohesive devices resulted in a more coherent mental representation of the text in the minds of the expert raters. Our second approach was to examine the issue experimentally. The goal of the study (unpublished; see McNamara & CSEP Lab, 2010, project report) was to examine whether essays scores were positively influenced by two factors, cohesion and elaboration. These factors were manipulated experimentally, in contrast to the correlational approach used within the corpus studies. The 35 participants wrote two essays and were then asked to spend 15 minutes elaborating on the ideas in the essay. These four essays (2 original, 2 elaborated) were then modified by an experimenter in terms of cohesion by adding lexical coreference and conceptual overlap. Coh‐Metrix scores confirmed that the cohesion manipulations were associated with increased scores on the indices related to cohesion. This produced a total of eight essays on two topics, with four versions per topic. Twelve trained raters evaluated the essays using a standardized rubric, with three raters assigned to each essay. The results of the study indicated that both manipulations improved essays scores. Having students elaborate their essays resulted in higher scores, and increasing the cohesion of students’ essays resulted in higher essay scores. These results indicate that encouraging students to add to their essays will have positive benefits and also, providing instruction about cohesion and how to increase cohesion should have positive benefits. In sum, Coh‐Metrix cohesion scores are not related to essay quality but increasing cohesion is positively related to essay quality, and expert raters of essays consider higher quality essays to be more coherent and more cohesive. Our analyses indicating that the absence of cohesion potentially resulted in a more coherent representation for the expert raters might be interpreted within the Reverse Cohesion account of reading comprehension (e.g., O’Reilly & McNamara, 2007), according to which readers with more knowledge tend to make more inferences when reading text with cohesion gaps. However, this Rater Inference account could not at the same time explain the results of our experimental study in which manipulations of cohesion in the essays led to higher scores. Thus, it seems more likely that cohesion and coherence in essays emerge from different linguistic and semantic features. Our current focus in the W‐Pal project is in solving this puzzle computationally. We expect that the solution lies in computational assessments of both the structure of the essay and semantic aspects of the essays, in conjunction with the linguistic features. Predicting the Quality of Freewrites While much of our effort has been devoted to computational analyses of essays, another focus has been on collecting and analyzing freewrites. One strategy to prepare to write an essay is to generate ideas using freewriting. Within the Freewriting module (see Figure 2), students practice freewriting in the context of a game called Freewrite Feud. The goal of the Freewrite Feud game is to type as much as possible and to include as many keywords as possible that overlap with the game’s randomly chosen nine keywords. Points are awarded for the number of matching keywords and bonus points are awarded based on an NLP algorithm that assesses the quality of the freewrite. Figure 2. In the final freewriting checkpoint, students generate freewrites for two minutes.
The algorithm that assesses the freewrite within this module is based on a three large corpora of of freewrites assessed by expert raters (using a rubric specifically designed for freewrites). These corpora provide numerous examples of freewrites of differing quality that students can study as part of the lessons and practice activities. Analyses of these freewrites also contribute to indices and algorithms for assessing freewrite quality, length, and content diversity (Weston, Crossley & McNamara, 2010; Weston, Crossley, & McNamara, in press). A chapter in this volume describes one such study (Weston et al., in press). The results showed that many linguistic features positively correlated with the expert ratings of the freewrite, including referential cohesion, syntactic complexity, and lexical difficulty. However, within a regression analysis, the only significant predictors were the number of words and stem overlap. Based on this study, better freewrites seem to be characterized as being more cohesive and containing more words. Thus, whereas linguistic features associated with cohesion have not been found to be related to essay quality, this is not the case with regard to freewrite quality. Our future analyses will examine the relationships between freewrites and essays, and in particular whether certain features or qualities of freewrites are related to subsequent essay quality. Predicting the Parts of Essays A second goal in the W‐Pal project has been to explore the extent to which linguistic features can differentiate between the parts of essays (i.e., introduction, body, conclusion) and are predictive of the quality of these parts. There are two aspects of W‐Pal driving this work. First, the three drafting modules (Introduction Building, Body Building, and Conclusion Building) include challenges in which the students write parts of the essays. Second, during the Essay Writing Component of W‐Pal, the students write complete essays. We need computational algorithms to assess the presence of these essay sections and their quality. The majority of this work is in progress; however, several studies have been completed that explore the ability of linguistic features in distinguishing between the parts of the essay (Dempsey, McCarthy, Myers, Weston, & McNamara, 2009; Crossley, Dempsey, & McNamara, submitted). In the study reported by Crossley et al. (under review), 182 initial, middle, and final paragraphs from student, argumentative essays were analyzed using Coh‐Metrix. The paragraphs were classified by human raters based on paragraph type (introduction, body, conclusion). The eight indices found to significantly discriminate between the parts of the essays included measures of length, cohesion, and word difficulty. The performance of the reported model was well above chance and reported an accuracy of classification that was similar to human judgments of paragraph type (66% accuracy for human versus 65% accuracy for our model). The model’s accuracy increased in analyses restricted to those paragraphs that were longer and provided more linguistic coverage and those paragraphs judged by human raters to be of higher quality. As the quality and length of the paragraph increased, the linguistic differences among paragraph types became more acute. The results reported by Crossley et al. (under review) indicated that introductory paragraphs are shorter, include few cohesion cues, and include words that are more specific, meaningful, and imageable as compared to body and conclusion paragraphs. In comparison to conclusion paragraphs, the words in introductions are also less familiar. This combination of linguistic features results in a rhetorical structure that is less embedded syntactically than other paragraphs allowing for the production of a clear, direct main idea. Because the goal of an introduction paragraph is merely to state the main idea and the themes of the supporting arguments, the introduction does not depend on cohesion to produce a coherent structure. In contrast, body paragraphs were found to be longer and more cohesive compared to introductory and conclusion paragraphs. They also tended to include less imageable (abstract) words compared to introductions, but more imageable (concrete) words compared to conclusions. These linguistic features can be associated with the rhetorical purpose of body paragraphs (Grady, 1971). For instance, more words and higher cohesion support the assumption that body paragraphs feature a tighter coupling of ideas that expand on the supporting arguments introduced within the first paragraph. Unlike an introduction paragraph, body paragraphs elaborate ideas and thus likely do not rely on highly imageable words. The features of conclusion paragraphs also corresponded to the rhetorical purpose of final paragraphs to summarize the information in the essay without presenting new information (Grady, 1971). The process of summarization produces shorter paragraphs containing more connectives. The words used in the summarization are found to be less specific in nature, but also more familiar. Our results thus far support the assumption that paragraph types contain specific linguistic features that allow them to be distinguished from one another. In addition, the corpora also provide stimuli for the modules in which the students distinguish between good and poor introductions and conclusions. Our current work is focused on expanding the linguistic and semantic features that are considered in order to predict the quality of the paragraphs. In sum, the research outlined in this chapter shows promise of being building blocks for the construction of a successful algorithm capable of assessing the quality of essay parts. Anticipated Computational Approach within Essay Writing Component Within the Essay Writing Component of W‐Pal, the student writes a complete essay and receives feedback that scaffolds the use of W‐Pal strategies to improve the essay. We are currently building this system. We anticipate that our approach to this computational challenge will be to use layered algorithms. First, the system will assess whether the essay is long enough. The first step to improving short essays is to write more. The strategies for writing more were covered within the Freewriting module (i.e., FAST PACE). The second step will be to assess whether the essay contains enough paragraphs. First, the archetypical essay contains five paragraphs, and at the least contains an introduction, body, and conclusion. Hence, if the student has not included enough paragraphs, a reminder can be provided to use strategies provided within the Planning Module, which covers the structure of the essay and how to organize the essay. Once the essay is long enough and contains enough paragraphs, then the quality of the entire essay and the parts of the essay can be assessed. This assessment can be done in parallel. If the separate parts are not distinguishable in terms of their features or are low in quality, then the student will be advised to use strategies associated with the drafting modules. For example if the first paragraph does not have features associated with Introductions (as compared to Bodies and Conclusions) and does not have features associated with high quality introductions, the student would be reminded of the TAG strategy. If the algorithms indicated that the final paragraph did not contain features characteristic of good conclusions, then the student would be reminded of the RECAP strategy. At the same time, certain features may indicate that the essay is low in quality overall. If that were the case, the student might be reminded of strategies to revise the essay at the sentence (i.e., paraphrasing), paragraph (i.e., cohesion strategies), and global levels (e.g., the ARMS and TETRIS strategies). Of course, it is important not to overwhelm the student with suggestions and frustrate the student with too much negative feedback. Hence, after a few revisions, the student’s essay would be considered complete and the student will be given an essay score. Across multiple essays, with greater practice in using the strategies, we expect the overall score for students’ essays to increase. In time, no prompting to use the strategies should be necessary because the student would use them on the first attempt. Conclusion Our work on W‐Pal has been primarily devoted to its design and the development of the architecture, the pedagogical content, and the NLP algorithms. The purpose of this chapter was to provide a brief overview of the system and to describe some of our ANLP efforts towards building an interactive ITS. We are currently conducting pilot tests and evaluations of the separate modules, which will prepare us for the overall empirical evaluation of the tutoring system in its entirety. This evaluation will comprise a year‐long study of high‐schools students who will work through the modules. We will observe the students’ performance and compare their performance to students who do not receive writing tutoring. In order to optimize the efficacy and utility of this program, it is necessary to empirically examine a number of factors related to the lessons, interface and pedagogical material; and how those factors influence students’ engagement in the tutoring system and their improvements in writing. Such studies will also allow us to refine our NLP algorithms. We expect to engage in numerous iterations to improve and refine our algorithms as well as how they are implemented in terms of feedback to the student writer. The objectives of W‐Pal are to improve high school students’ writing abilities and reduce demands on teachers. The former objective should provide students with some of the tools necessary to become successful writers, whereas the latter will allow teachers more focused time with students. Both of these objectives contribute to a foundation for a better‐prepared youth, because more adept writers have increased aptitude in career and personal success. It is also worth noting that the development of W‐Pal’s components will certainly have secondary benefits to a number of other professionals, including teachers, human‐computer interaction researchers, gaming designers, education researchers, linguistics scientists, computer scientists, interface designers, as well as researchers working within the general framework of Applied Natural Language Processing. References Albertson, B. R. (2007). Organization and development features of grade 8 and grade 10 writers: A descriptive study of Delaware Student Testing Program (DSTP) essays. Research in Teaching of English, 41, 435‐464. Bandura, A. (1997). Self‐efficacy: The exercise of control. New York, NY: Freeman. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453‐494). Hillsdale, NJ: Lawrence Erlbaum. Collins, J. L. (1998). Strategies for struggling writers. New York, NY: Guilford Publications. Connor, U. (1987). Research frontiers in writing analysis. TESOL Quarterly, 21, 677‐696. Connor, U. (1990). Linguistic/rhetorical measures for international student persuasive writing. Research in the Teaching of English, 24, 67‐87. Craig, S. D., Graesser, A. C., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29, 241‐250. Crossley, S. A., Dempsey, K. B., & McNamara, D. S. (under review). Classifying paragraph types using linguistic features: Is paragraph positioning important? Research in the Teaching of English. Crossley, S. A., & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. Crossley, S. A., & McNamara, D. S. (in press). Predicting second language writing proficiency: The role of cohesion, readability, and lexical difficulty. Journal of Research in Reading. Crowhurst, M. (1990). Reading/writing relationships: An intervention study. Canadian Journal of Education, 15, 155‐172. Dempsey, K. B., McCarthy, P. M., Myers, J. C., Weston, J., & McNamara, D. S. (2009). Determining paragraph type from paragraph position. In C. H. Lane & H. W. Guesgen (Eds.), Proceedings of the 22nd International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 33‐
38). Menlo Park, CA: The AAAI Press. DeVillez, R. (2003). Writing: Step by step. Dubuque, IA: Kendall/Hunt Publishing Company. Ericsson, K. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. Feltovich, and R. R. Hoffman, R. R. (Eds.). Cambridge handbook of expertise and expert performance (pp. 685‐706). Cambridge, UK: Cambridge University Press. Ericsson, K. A., & Kintsch, W. (1995). Long‐term working memory. Psychological Review, 102, 211‐245. Geiser, S., & Studley, R. (2001, October). UC and the SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Paper presented at the Meeting of the Board of Admissions and Relations with Schools of the University of California. Grady, M. (1971). A conceptual rhetoric of the composition. College Composition and Communication, 22, 348‐354. Institute of Education Sciences. (2003). The nation’s report card: Writing 2002, NCES 2003‐529. Retrieved from http://nces.ed.gov/nationsreportcard/pubs/main2002/2003529.asp Light, R. (2001). Making the most of college. Cambridge, MA: Harvard University Press. McNamara, D. S. (2001). Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51‐62. McNamara, D. S. (2009). The importance of teaching reading strategies. Perspectives on Language and Literacy, 34‐40. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). The linguistic features of quality writing. Written Communication, 27, 57‐86. McNamara, D. S., & CSEP Lab (2010). W‐Pal: Writing Pal. Annual project report submitted to the Institute of Education Sciences (IES). McNamara, D.S., Jackson, G.T., & Graesser, A.C. (in press). Intelligent tutoring and games (ITaG). In Y.K. Baek
(Ed.), Gaming for classroom-based learning: Digital role-playing as a motivator of study. IGI Global.
McNamara, D. S., Jackson, G. T., & Graesser, A. C. (2009). Intelligent tutoring and games (iTaG). In H.C. Lane, A. Ogen, & V. Shute (Eds.), Proceedings of the Workshop on Intelligent Educational Games at the 14th Annual Conference on Artificial Intelligence in Education (pp. 1‐10). Brighton, UK: AIED. McNamara, D. S., Levinstein, I. B., & Boonthum, C. (2004). iSTART: Interactive strategy trainer for active reading and thinking. Behavior Research Methods, Instruments, and Computers, 36, 222‐233. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh‐Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292‐330. McNamara, D. S., O’Reilly, T., Rowe, M., Boonthum, C., & Levinstein, I. B. (2007). iSTART: A web‐based tutor that teaches self‐explanation and metacognitive reading strategies. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 397‐
421). Mahwah, NJ: Erlbaum. Myers, J. C., McCarthy, P. M., Duran, N. D., & McNamara, D. S. (in press). The bit in the middle and why it's important: A computational analysis of the linguistic features of body paragraphs. Behavior Research Methods. National Commission on Writing. (2004). Writing: A ticket to work... Or a ticket out, a survey of business leaders. Retrieved from http://www.host‐collegeboard.com/advocacy/writing/publications.html O’Reilly, T., & McNamara, D. S. (2007). Reversing the reverse cohesion effect: Good texts can be better for strategic, high‐knowledge readers. Discourse Processes, 43, 121–152. Rogoff, B. (1990). Apprenticeship in thinking. New York, NY: Oxford University Press. Sharp, D. B. (2007). Learn to write. ISA Career website. http://www.isa.org/Template.cfm?Section=Careers&Template=/ContentManagement/Content
Display.cfm&ContentID=5328 Shaughnessy, M. P. (1977). Errors and expectations: A guide for the teacher of basic writing. New York, NY: Oxford University Press. Weston, J., Crossley, S. A., & McNamara, D. S. (2010). Towards a computational assessment of freewriting quality. In H. W. Guesgen & C. Murray (Eds.), Proceedings of the 23rd International Florida Artificial Intelligence Research Society (FLAIRS). Menlo Park, CA: The AAAI Press. Weston, J., Crossley, S. A., & McNamara, D. S. (in press). Differences in freewriting quality: Perspectives, approaches, and applications. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution. Hershey, PA: IGI Global. Witte, S., & Faigley, L. (1981). Coherence, cohesion, and writing quality. College Composition and Communication , 32, 189‐204. Zimmerman, B. J., & Risemberg, R. (1997). Self‐regulatory dimensions of academic learning and motivation. In G. D. Phye (Ed.), Handbook of academic learning: Construction of knowledge (pp. 105‐125). New York: Academic Press.