ETS Testing Programs. Educational Testing Service CriterionSM Online Writing Evaluation

पीडीएफ

दस्तावेज़

English Language Training

Topic

Reasons For

Attending College

Experience Or Books

Money On Technology

Young People

Teaching Older People

Money And Success

Playing And Winning

Group Member Or Leader

Change In Your School

New Product

Resource Disappearing

Changes In The 21 st

Century

Change Your Hometown

Writing Style

Persuasive

Timing

30 minutes

0 minutes

30 minutes

These topics may be assigned to students; their essays, once submitted, will yield either a holistic score or diagnostic feedback, depending on how the instructor has established the parameters within Criterion. If instructors want their students to write on other topics (related to a unit in social studies or science, for example) those essays will be able to receive diagnostic feedback only.

If you want your students to receive an accurate holistic score, they should write only on the topics that correspond to their grade level. Remember that the holistic score is based on hundreds of essays written by other students at that grade level. For example, if your tenthgrade class were to write on twelfth-grade topic, your students’ writing would be compared to that of twelfth graders, so their scores may not reflect an accurate evaluation of their writing ability.

Often writing instructors are interested in knowing where Criterion’s topics originated. While the topics resemble those in writing textbooks, the topics and the essays used to build holistic scoring models come from a variety of sources.

ETS Testing Programs

Established high-stakes testing programs at ETS released topics and scored essays for use in

Criterion. These programs include:

· Scored essays from established, high-stakes testing programs

· GMAT – graduate level writing

National Samples of Student Writing

Many topics were written by instructors and assessment specialists and administered to a national sample of writers. Trained readers scored these essays, and automated scoring models were built using these data.

· Mode-specific topics: description, narration, persuasion, cause-effect, process analysis, problem-solution, writing for assessment, writing for literature, workplace writing

· Mode-specific rubrics: based on widely accepted holistic writing standards

· Grades 4, 6, 7, 8, 9, 10, 11, 12

· 6 point and 4 point holistic scales

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 53

Topics Developed by Instructors

Just as students need a purpose to write, instructors need a purpose for assigning an essay topic. When instructors want to provide their students with on-going practice in the basic modes of writing, then the topics available in Criterion can be used effectively. There are instances, however, when an instructor may decide that none of the topics is appropriate for an assignment. When this occurs, the Instructor Topic option can be used. The Instructor

Topic feature allows an instructor to write a topic to reflect a content area or subject. Instructors can tailor their writing activities to an assignment or project and capture students’ responses for evaluation.

Remember, students writing on Instructor Topics will not receive holistic score feedback.

Deciding When To Use Criterion Topics Or Your Own Topics

Criterion Topics

Do I want to assign a Criterion topic or write one of my own?

Do I want to see how well my students are writing in comparison to a particular grade level?

Do I want to assign a particular mode of writing or a response to a reading selection or class discussion?

Do I want my students to receive holistic score feedback and diagnostic feedback, or just diagnostic feedback?

· More than 110 topics that reflect the domain of writing.

· Grades 4-12

· College Level: 1 st

Year/Placement, 2 nd

· Upper Division/Graduate School

· English Language Training

· Persuasive

· Expository

· Narrative

· Process Analysis (How To)

· Comparison-Contrast

· Cause And Effect

· Problem And Solution

· Workplace Writing

· Writing For Assessment

Year

· Analysis Of Argument

· Analysis Of Issue

· Holistic score on either a 6-point or a 4-point scale.

· Diagnostic feedback in grammar, usage, mechanics, style, and organization and development.

Instructor Topics

· Can be created for any content area and for any writing purpose.

· Can be created for any grade level, content area, or writing purpose.

· Can be designed to suit any content area or subject matter. A reading passage (e.g., paragraph or poem) can be attached to the topic.

How do we incorporate

Criterion in an established writing curriculum?

· To support writing in a specific mode.

· To support writing on a subject or issue.

· To provide writing practice in a timed setting.

· Diagnostic feedback in grammar, usage, mechanics, style, and organization and development.

· Feedback on organization and development is available for instructor topics, but may not be as good as for Criterion topics.

· Holistic score is not available for instructor topics.

· To support writing in the content areas.

· To support writing across curriculum.

· To support team teaching.

Feedback Messages For Different Categories of Diagnostic Feedback

The diagnostic feedback students receive on their writing identifies the difficulty as cited by

Criterion, and gives students suggestions as to how they might correct the problem. In addition, as noted above, students can access the Writer’s Handbook via a direct link.

For example, if Criterion has located a statement that it identifies as an introduction, the feedback reads “Is this part of your Introduction? An introduction attracts the reader’s interest and provides background information. It also contains your thesis statement. Look in the

Writer’s Handbook for ways to improve your introduction.” If Criterion cannot locate a statement

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 54

that it identifies as an introduction, the feedback reads: “Criterion has not identified an

introduction in your essay. Good writers use an introduction to tell what the essay is about and to set their essays in motion. Look in the Writer’s Handbook for some advice about writing an introduction.”

Other feedback comments are similar; they are tailored to the type of feedback but are, of necessity, general in nature. They frequently direct students to the Writer’s Handbook for additional guidance. Students may also seek additional help from their writing instructor.

Tips for Instructors

The extent to which you will be able to use Criterion in your teaching, and the way in which you do so, will be determined largely by the access both you and your students have to both computers and the Internet. Criterion is web-based, and while students may do much of their writing off-line, and even on a piece of paper, they must, eventually, be able to type their work into Criterion and submit it, via the Internet, for scoring and feedback.

The chart below outlines three different scenarios for computer and Internet access at your school and in your students’ homes. Factors to consider are:

· the number of computers in your classroom, and whether they have an Internet connection.

· your access to a computer lab with Internet connections.

· the extent to which your students can use a computer at home, and whether they have an

Internet connection there.

· whether you have a computer, with an Internet connection, at home.

Evaluate your teaching environment and situation to determine how often you can ask your students to write in Criterion.

Computer And Internet Access

Where does your school have

Internet access?

Scenario 1

· Computers with Internet access are located in the school library or other remote location only.

Scenario 2

· Class has access to a computer lab with Internet access for less than one hour per week.

Scenario 3

How much computer access is there at your school?

How much computer access do your students have at home?

Do you have compute and

Internet access in your classroom, office, or at home?

· Computers with Internet access are available in the library or other remote location.

· Students have access to computers with Internet access in the classroom, but there are not many machines, so students’ time must be scheduled.

· No students have computers or Internet access at home.

· You have no computer or

Internet access at home.

· The class has access to a computer lab with Internet access for less than one hour per week.

· There are a number of computers with Internet access available for student use in the classroom; students must schedule their use, but they do not have to wait for long.

· Some students have computers and Internet access at home.

· You have a computer at home, and Internet access, but it is very slow.

· Class has access to a computer lab with Internet access on a frequent basis.

· Classroom computers have

Internet access.

· Class has access to a computer lab with Internet access on a frequent basis.

· There are many computers with Internet access available for student use in the classroom; students have access when they need it.

· Virtually all students have computers and Internet access at home.

· You have a computer and high-speed Internet access at home.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 55

Now, to guide you in your planning, you could complete the following table for your own instructional situation.

Web Access

Computer Access In School

Student Computer Access At Home

Instructor Computer Access At Home

As instructors have used Criterion with their students, they have offered the following suggestions:

Students can handwrite their essays and then type them into Criterion. Or they can write offline in a word processing program and then cut and paste their essays into Criterion’s text entry box.

Be sure to schedule sufficient time in the computer lab so students have time to write and submit an essay, and view the feedback. While the feedback is immediate, the process of revision takes time.

The key to the effective use of Criterion is in making the assignment, selecting the appropriate types of feedback, and engaging students in revising their work.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 56

C r r i i t t e r r i i o n R e f f e r r e n c e

G u i i d e

Criterion Online Writing Evaluation is not a “black box” or “magic” just because it provides instantaneous feedback. Criterion has distinct capabilities that invoke proven technology. Criterion is an application that teachers can use to support the writing process; it’s a set of tools that students can use to work independently and get immediate feedback. This reference guide is intended to provide additional insight into the Criterion service and its capabilities, and to clarify the language used to describe the technology.

Frequently Asked Questions

Using Criterion In Teaching

Q. How can Criterion help students?

A. Students get a response to their writing while it is fresh in their minds. They find out immediately how their work compares to a standard and what they should do to improve it. Criterion also provides an environment for writing and revision that capable and motivated students can use independently. This environment, coupled with the opportunity for instant feedback, provides the directed writing practice so beneficial for students.

Q. How does Criterion handle an unusual writing style?

A. Criterion looks for specific features of syntax, organization, and vocabulary. If the essay under consideration is not sufficiently similar to those in its database of already scored essays,

Criterion posts a warning saying that it is unable to give an accurate score. Warnings usually result from essays that are too brief or those in which the vocabulary is unusual or the content is off topic.

Q. Will the use of Criterion stifle creative writing among students?

A. Not necessarily. Criterion is designed to be used for evaluating writing done under testing conditions — situations in which even the most creative writers concentrate on “playing it safe” with straightforward and competent writing.

Q. Will Criterion catch cheating or plagiarism?

A. No. Criterion simply evaluates the essay. It is up to the institution to assure that students are working independently and submitting their own work.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 57

Q. What information does Criterion report to teachers?

A. Teachers have easy and secure access to each student’s portfolio of essays, diagnostic reports, and scores as well as summary information on the performance of entire classes.

Q. What information does Criterion report to students?

A. Typically, students get an annotated diagnostic analysis of errors and features of grammar, usage, mechanics, style and organization and development, as well as a holistic evaluation each time they submit an essay. However, schools can block students from seeing their scores and/or any portion of diagnostic feedback — and may choose to do so if they use

Criterion for high-stakes testing. Schools also have the option of allowing students access to their own portfolio of essays, diagnostic reports, and scores.

Q. Does Criterion discriminate against students who may be bright but who may not have mastered standard English — for example minorities and ESL students?

A. No. Criterion is incapable of discriminating on the basis of race, sex, national origin, or student’s previous history because these factors do not exist in its analysis. The program simply measures features in a piece of writing and compares them to features in previously scored essays used to define the rubric.

So if the collection of sample essays includes essays that use non-standard English and also earn high scores, then Criterion will assign a high score to other essays with the same features.

Using Criterion For Testing And Placement

Q. How often does the computer’s score agree with the score of a faculty reader?

A. Almost all the time. ETS Technologies researchers found exact agreement, or a difference of only one point, in as many as 98 percent of the comparisons between the computer’s scores and those of a trained essay-reader using the same scoring guides and scoring system.

Q. How do schools use Criterion for testing?

A. Some schools use Criterion scores for exit testing — combining a Criterion score with the score from a faculty reader in the same way they combined scores from two different faculty readers. If the two scores differ by more than one point, a different faculty reader also evaluates the essay. Some schools use Criterion for benchmark testing, assigning Criterion-scored essays at specified points during the school year.

Q. How do schools use Criterion for placement?

A. Some colleges assign students to composition classes on the basis of their scores on a Criterion-scored essay — or the combination of the Criterion score and other indicators.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 58

Q. How do students feel about being scored by a machine?

A. Most of today’s students have had experience with instant feedback in computer programs and are becoming more comfortable with the idea of computerized scoring.

Q. Can Criterion score essays on other topics?

A. No. Criterion can only evaluate essays on those topics for which it has been “trained.” ETS Technologies maintains an annual schedule of adding new topics at each level. Schools and districts can also work with ETS Technologies to create new customized topics. The training process for a topic requires a minimum of 465 essays that have been scored by faculty readers.

Understanding The Technology

Q. How does Criterion come up with its scores?

A. Criterion is based on a technology called e-rater® that was developed at Educational Testing Service. E-rater compares the new essay to samples of essays previously scored by faculty readers — looking for similarities in sentence structure, organization, and vocabulary. Essays earning high scores are those with characteristics most similar to the high-scoring essays in the sample group; essays earning low scores share characteristics with low-scoring essays in the sample group.

Naturally, the sample essays must be scored very carefully and the collection must include sufficient essays at each score point.

Q. What’s the technology used in e-rater?

A. E-rater is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using

NLP for the past 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information. E-rater uses NLP to identify the features of the faculty-scored essays in its sample collection and store them — with their associated weights — in a database. When e-rater evaluates a new essay, it compares its features to those in the database in order to assign a score.

Because e-rater is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which e-rater’s database is created.

Q. Can students trick Criterion?

A. Yes. Since e-rater cannot really understand English, it will be fooled by an illogical, but well-written, argument. Teachers can stop students from deliberately trying to fool Criterion by announcing that a random sample of essays will be read by independent readers. Additionally, Criterion will display an

“Advisory” along with the e-rater score when an essay displays certain characteristics that warrant attention compared to other essays scored against the same topic.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 59

Q. Must students be connected to the Internet to use Criterion?

A. Not necessarily. Although the scoring of essays requires an

Internet connection, students can compose their essays offline using any word processing application and then cut and paste them into the service after they log on to the Criterion site.

Q. Is the holistic score related to the diagnostic feedback?

A. Holistic score and diagnostic feedback are two separate and independent systems; one does not have an effect or impact on the other. A holistic score of 6 can reveal lots of errors in grammar, usage, mechanics, and style; however, it can also contain all of the elements in organization and development— introduction, thesis, main and supporting ideas, and a conclusion. A holistic score of 2 may reveal few errors and few elements in organization and development.

To receive a holistic score, a topic must have a corresponding erater scoring model.

To receive the most accurate kind of organization and development feedback, a student should write on a Criterion topic instead of an “instructor topic” that does not have a corresponding e-rater model.

Diagnostic feedback in grammar, usage, mechanics, and style is designed to identify certain errors in these categories. It is not evaluative.

Q. What kind of technology is used in Criterion?

A. Criterion incorporates a variety of automated scoring tools to provide immediate feedback. Automated scoring technology is developed using natural language processing, a sub-field of artificial intelligence. These sophisticated scoring techniques produce instant results using statistics, computational linguistics, algorithms. However, none of these automated scoring tools can read essays like humans do. These tools process information; they do not read and analyze it.

Criterion incorporates a variety of automated scoring tools to provide immediate feedback. Automated scoring technology is developed using natural language processing, a sub-field of artificial intelligence. These sophisticated scoring techniques produce instant results using statistics, computational linguistics, algorithms. However, none of these automated scoring tools can read essays like humans do. These tools process information; they do not read and analyze it.

Automated scoring tools are predictable and literal. An e-rater model will, for example, award a score of 3 to all essays that

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 60

resemble “3-ness” in its training set. It will score consistently and endlessly in this way.

Q. How are e-rater scoring models created?

A. E-rater learns to score by analyzing hundreds of essays that have been scored by trained faculty readers who use a scoring rubric that establishes the rules for awarding the individual score points. An e-rater scoring model is accurate only if the essays scored by readers show clear distinctions among the score points.

E-rater scoring models are built using essays written by a specific population of students on topics that are appropriate for that population. These essays are scored by readers (usually teachers) who are familiar with the student population. The readers guide their scoring using a rubric, which is a set of established standards delineated at a range of score points.

When using Criterion, it is important to match the topic with the appropriate level of student writer. For example, while a particular 12 th

grade topic may be appealing from a content point of view, it is not appropriate for use with an 8 th

grade class. The

8 th

graders’ writing would be compared to the 12 th

graders’ writing, so the e-rater score would not be used to evaluate comparable writing.

Glossary Of Terms

Advisory: A message that provides additional information about the essay. Criterion issues an Advisory when the essay is too brief to evaluate, may not be relevant to the topic, or exhibits a unique style not typically found in other essays on the topic.

E-rater®: The automated essay scoring technology that supports

Criterion. E-rater works by comparing new essays to samples of essays previously scored by faculty readers.

Holistic Scoring: A method of evaluating essays that considers the total impression rather than individual aspects. Criterion uses the e-rater technology to provide holistic scores on essays.

Natural Language Processing: The application of computational methods to analyze characteristics of electronic files of text or speech. E-rater, the technology underlying Criterion, uses natural language processing to identify the features of the essay that correspond to scoring guide criteria — syntactic variety, organization of ideas, and vocabulary usage.

Rubric: An established rule. Criterion uses scoring rubrics (also called scoring guide).

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 61

Scoring Guide: A description of the features of the writing at each level on the scoring scale. Criterion makes the scoring guides, as well as a sample essay at each score point, available to students and teachers.

E-rater

: What It Is, How It Works

General Approach

E-rater

is the automated scoring system that has been developed to assist in the evaluation of test takers' responses to open-ended essay questions. The current version of the system is capable of assigning scores (on a 0 to 6 scale) to two kinds of essays -- one that requires examinees to discuss an issue and another that requires them to analyze an argument.

E-rater is empirically based. It is "trained" by being fed samples of essays that have been previously scored by human readers. The samples are randomly selected for each essay prompt so as to represent a complete range of possible scores. Essentially, e-rater uses natural language processing techniques to duplicate the performance of human readers.

In its attempt to model human readers, e-rater uses several subroutines to extract a wide variety of features of the essays that it evaluates. These features are then used in combination to predict the scores that were assigned previously (by human readers) to the essays in e-rater’s training sample. Ordinary (stepwise) least squares linear regression analysis is used to select the set of features that best predicts these scores. A total of 50-60 features are "extractable," but in practice only a subset of the most predictive features, usually about 8-12, are retained and used for each essay prompt. The least squares regression weights for these predictors are applied to each new essay to estimate a score. The score is rounded to the nearest integer from 0 to 6 in order to place it on the scale used by human readers.

E-rater is prompt specific. That is, it is trained for each different essay prompt. Thus, the final set of features used to compute scores for any given prompt is likely to differ somewhat across prompts. Several features are much more likely than others to recur as predictors across different prompts. Some of the same features tend to be predictive of scores for both the Issue and the

Argument prompt types.

Although e-rater is empirically based, it is not "blindly empirical."

E-rater could, of course, take a "brute-force" approach, extracting numerous features indiscriminately. However, not all of the extractable features correspond equally well to the features that

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 62

human readers are instructed to consider when they score essays.

Therefore, e-rater features are required not only to be predictive of human readers' scores, but also to have some logical correspondence to the characteristics that human readers are trained to consider. These characteristics are specified in the scoring rubrics that guide human readers when they score essays.

Specific Techniques

Although certain "surface features" (such as the number of words in an essay) are easily extractable, these purely surface characteristics are not used by e-rater. Instead, the focus is on three general classes of essay features -- structure (or syntax), discourse (organization), and content (or prompt-specific vocabulary).

Structure

Structural characteristics include such notions as "syntactic variety," that is, the use of various structures in the arrangement of phrases, clauses, and sentences. In this category, several specific features have proven to be predictive of human readers' scores.

They are:

· the number of subjunctive modal auxiliary verbs (would, could,

should, might, may)

· the prevalence of infinitive clauses ("To support his expensive

hobby, Phil Atelist would soon need to get a second job at the post office.")

· the proportion of complement clauses ("He felt that it was well

worth the effort.")

· the incidence of subordinate clauses ("Although the argument

points out that trends are part of our society, it does not explain why.").

Organization

How well test takers are able to organize their ideas is a major focus for human readers and for e-rater. E-rater evaluates organization by extracting a variety of rhetorical features, i.e., characteristics that are associated with the orderly presentation of ideas. Of special interest here are "cue" words or terms that signal where an argument begins and how it is being developed. (By argument, we mean, generally, any rational presentation that uses reasons or examples to persuade the reader.) For example, terms such as "in summary" and "in conclusion" are used for, what else, summarizing. Words and phrases like the following are used to express opinions or beliefs:

· certainly, clearly, obviously, plainly, possibly, perhaps, potentially,

probably, fortunately, generally, maybe, presumably, unless, albeit, luckily, normally, apparently, herein, likely, surely, ideally,

undoubtedly, naturally

· for certain, for sure, of course, to some extent, above all, if only,

in order to, in order for, so that, so as to.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 63

The significance of many of the words listed above is that they signal the start of an "argument" (in the sense that we have described), and the number of arguments developed is a feature that is emphasized by e-rater. Arguments sometimes begin simply with I or we, or with "parallel" words or phrases that signify the start of another line of argument. The latter include the following:

firstly, essentially, additionally, first, second, third, another, secondly, thirdly, fourth, next, finally, final, last, lastly, moreover, too, also, likewise, similarly, initially, further, furthermore, first of all, in the first place, for one thing, for a start, second of all, many

times, more importantly, most importantly.

The onset of an argument is also signaled by rhetorical words and phrases such as suppose, supposedly, and what if. The subsequent details are often preceded by words and phrases like:

if, specifically, particularly, when, namely, for example, for instance,

e.g., in this case, in that case, such that, as well as, in that, such

as, about how, in addition, in addition to.

Words or phrases that are used to contrast points of view or to indicate alternative opinions are often found in arguments. These include words and phrases such as these:

otherwise, conversely, however, nonetheless, though, yet, meanwhile, while, but, instead, although, still, notwithstanding,

anyway, unlike, on the contrary, in contrast, by comparison, in any

case, at any rate, in spite of, rather than, on the other hand, even

then, even if, even though, apart from, instead of.

Some words or phrases signal the presentation of evidence to support an argument. These include since, because, actually, in

fact, after all, as a matter of fact, and because of. Still others signify the stage at which inferences are being made:

accordingly, consequently, hence, thus, ultimately, so, thereby, then, therefore, following, after, afterward, afterwards, as a consequence, as a result, if so, if not, as such, according to, in turn,

right after.

Yet other words and phrases signal that an argument is being

reformulated (alternatively, alternately, that is, in other words,

briefly). In short, E-rater looks for linguistic clues (such as logical connections between sentences and clauses) that signify logical or well-ordered thinking.

Content

By means of "topical analyses," e-rater also considers the content or vocabulary of the essays it evaluates. These analyses are predicated on the assumption that well-written essays are more responsive (or relevant) to the topic posed than are poorly written essays. Better essays also tend to use more precise and specialized vocabulary. Therefore, the expectation is that, with

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 64

respect to the words they contain, good essays will bear a greater resemblance to other good essays than to poorer ones. Weak essays, on the other hand, will be similar to other weak essays.

Acting on this assumption, e-rater evaluates each essay and assigns a number based on the similarity of its content to samples of previously-scored essays. This evaluation proceeds as follows.

First, for every essay a determination is made of each word's

"contribution" to the essay. (Variations of a word, such as "walks,"

"walking," or "walked," are considered to be the same word.) A word's relative contribution to an essay is estimated by computing a weight for each word in every essay. These word weights reflect both the frequency of a word in a given essay (relative to the frequency of other words in the essay), and the distribution of the word across other essays. The formula is such that words appearing relatively frequently in an essay will, all other things being equal, receive a higher weight than words appearing less frequently in the same essay. However, words that tend to be used in many essays will get lower weights, all else being equal, than those used in few essays.

In order to assign a numerical value to reflect an essay's content, the word weights for each essay are compared with the weights computed for words in the previously scored training essays. An essay's content feature score is assigned by determining the set of essays that its word weights most closely match. Essays whose word weights correlate most strongly with those computed for highly scored essays will get higher content scores than will essays whose word weights resemble those of weaker essays.

As an example, consider the following (partial) results based on essays analyzing an attempt to convince companies to utilize billboards to increase sales. The two most highly weighted words in essays that received the lowest score (i.e., 1) were "picture" and

"percent." Both of these words received much lower weights in essays that got higher scores. On the other hand, the two most heavily weighted words ("recognition" and "local") in the best essays (i.e., 6s) received much lower weights in the poorer essays.

The weights assigned to these four specific words in essays at each score level were as follows:

Score level

Word 1 2 3 4 5 6

Percent .51 .25 .24 .24 .05 .05

Recognition .00 .04 .06 .19 .41 .69

Local .05 .11 .11 .20 .16 .32

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 65

Besides evaluating an essay's content as a whole, e-rater also considers an essay's content argument-by-argument. The rationale is that additional information can be extracted about an essay's content by examining clusters of word groupings, in this case individual arguments. In a manner similar to that used for essays as a whole, a content score is assigned to each argument on the basis of how well an essay's arguments match the content of essays scored by human readers. The formula used to compute the argument content score assigns higher values to essays with many arguments than to those with few.

E-rater Models

The final step is to use all of the features that e-rater extracts (or rather the values that it assigns to these features) to predict the scores assigned by human readers. A model, i.e., a set of features that is most predictive of human readers' scores, is specified for each essay prompt. As stated earlier, the set is generally somewhat different for each prompt. Some features are used much more frequently than others in the predictive models. The most frequently occurring are:

(1) content by argument

(2) content by essay

(3) the number of subjunctive auxiliary verbs

(4) the ratio of subjunctive auxiliary verbs to total words in the essay

(5) the total number of argument development terms.

The weights assigned to these variables may be either positive or negative. For instance, the number of subjunctive auxiliary verbs typically receives a positive weight, while the ratio of such words is usually weighted negatively. Although the weights assigned to most of the features discussed earlier are usually positive, some features tend to have negative weights. These include certain words used to present evidence (because, since, actually, …) and the use of pronouns (I, we) to begin arguments.

The overview provided here should give the reader some sense of how e-rater functions. More detail is available in a number of reports that can be downloaded from the following Web site: http://www.ets.org./research/erater.html

. If you are unable to download, contact us and we'll send the papers electronically as

Word documents or by regular mail in hard-copy form.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 66

नमस्ते! मैं एक AI चैटबॉट हूँ जिसे विशेष रूप से ETS Testing Programs. Educational Testing Service CriterionSM Online Writing Evaluation के साथ आपकी सहायता करने के लिए प्रशिक्षित किया गया है. मैंने दस्तावेज़ की अच्छी तरह से समीक्षा की है और आपको आवश्यक सटीक जानकारी का पता लगाने या सामग्री को स्पष्ट और सरल शब्दों में समझाने में मदद कर सकता हूँ. चाहे आप विशिष्ट सुविधाओं, समस्या निवारण चरणों या सामान्य उपयोग पर मार्गदर्शन की तलाश कर रहे हों, अपने प्रश्न पूछने के लिए स्वतंत्र महसूस करें. आप अपनी चिंताओं या ज़रूरतों के बारे में जितनी अधिक जानकारी प्रदान करेंगे, मैं उतनी ही सटीकता और प्रभावी ढंग से आपकी सहायता कर सकता हूँ!

सहायक टाइपिंग कर रहा है

सहायक कीबोर्ड से बाहर है. कृपया, बाद में फिर से प्रयास करें.

विषयसूची

ऐसा लगता है कि आप एक ऐसे दस्तावेज़ पर पहुँच गए हैं जो कई भाषाओं में लिखा गया है, जिसमें एक ऐसी भाषा भी शामिल है जो आपकी मूल भाषा नहीं है. आप अपनी पसंद की भाषा के बिना पृष्ठों को अस्थायी रूप से छिपाने के लिए निम्न लिंक का उपयोग कर सकते हैं.