Educational Testing Service Criterion Online Writing Evaluation User Manual

PDF

Document

Advertentie

The Criterion User Manual

Rosedale Road, MS18-E

Princeton, NJ 08541

tel 877.909.6442

fax 609.734.1460

www.etstechnologies.com

Welcome to Criterion

Online Writing Evaluation, a web-based service that evaluates a student’s writing skills and provides instant score reporting and diagnostic feedback to both the instructor and student.

Criterion utilizes automated scoring technologies to provide writers with instant score and writing analysis feedback on writing samples they submit. Criterion’s user interface enables quick access to numerous features, including a holistic score based on nationally recognized standards and detailed diagnostic feedback, provided in an annotated version of the student’s writing sample.

Students can access a password-protected website, submit essays on topics assigned by their teacher, and then receive an overall holistic score from e-rater®, a proprietary automated scoring engine developed by Educational Testing Service. Users also receive online diagnostic feedback from Critique

Writing Analysis Tools, also developed by Educational Testing Service to analyze elements of grammar, usage, mechanics, style, and organization and development.

Potential errors from these categories are identified within the essay itself, and targeted feedback is offered for each identified error.

Criterion captures and displays actual errors and features within the student’s essay, including fragments, agreement errors, misspellings, sentences containing errors of commonly confused words, and sentences missing initial capitalization and final punctuation. Heuristic-based diagnostic feedback helps writers focus on their errors and features as they revise their essays, while a work-in-progress revision capability allows them to make revisions as they review each category of feedback. Students can ask instructors for advice about an error or feature by writing questions and comments and attaching them to a particular essay.

Instructors can also insert their own comments about the essay both within the essay and in a message board. Criterion also provides students with online access to a comprehensive writer’s handbook with reference materials on how to improve their writing. All of this feedback is designed so that the student can utilize it along with the revision capability.

All of the student’s writing, scores, annotated feedback, and instructor comments are saved to a secure virtual portfolio that both the instructor and student can access.

The Criterion library of topics spans middle school, high school and college levels. In addition, instructors can create and assign their own writing tasks utilizing an open topic function.

This User Manual is designed to support the activities of the system administrator and instructors as they incorporate Criterion as a tool to supplement writing instruction. It is divided into three sections. The first section, Using the Criterion System, contains a step-by-step overview of all essential functions in Criterion. Each module is illustrated with screen captures of the system to

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 2

familiarize instructors and help them navigate through the application. This overview is intended to allow instructors to work independently as they begin using Criterion in their classrooms. The second section, Using Criterion to Support Writing Instruction, provides a context for understanding how the different components of Criterion can be used to support the teaching of writing and revising an essay. In addition to explaining Criterion's holistic score and diagnostic feedback capabilities, this section presents a variety of instructional approaches and activities to help instructors plan their use of the system in their classrooms. Refer to "Computer and Internet

Access" to review a series of questions to help you analyze the availability of technology in your classrooms and alternate locations in your schools. The final section, Criterion Reference

Guide, contains articles and resources for all users.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 3

Using The Criterion System

Quick Start

Administrator Quick Start ............................................................................................................. 6

Instructor Quick Start ................................................................................................................... 8

Student Quick Start.................................................................................................................... 10

Criterion Administrators

Getting Started........................................................................................................................... 12

Setting Up Districts, Schools, and Instructors............................................................................. 15

Using District, School, and Instructor Reports............................................................................ 16

Email Correspondence With Your Users.................................................................................... 17

Criterion Instructors

Getting Started........................................................................................................................... 18

Setting Up Your Classes............................................................................................................ 19

Setting Up Your Assignments .................................................................................................... 20

What Your Students See And Do In Criterion............................................................................. 24

Using The Class And Student Reports....................................................................................... 24

Creating And Inserting Pop-Up Notes ........................................................................................ 27

Writing Comments To Your Students......................................................................................... 27

Criterion Students

Getting Started........................................................................................................................... 29

Completing An Assignment........................................................................................................ 32

Accessing The Score Analysis................................................................................................... 34

Accessing The Annotated Feedback Analysis............................................................................ 36

Using The Student Portfolio ....................................................................................................... 38

Using The Writer’s Handbook .................................................................................................... 39

Using Criterion To Support Writing Instruction

Overview Of The Writing Process .............................................................................................. 41

How Criterion Supports The Writing Process ............................................................................. 42

Holistic Scoring .......................................................................................................................... 42

Diagnostic Feedback ................................................................................................................. 43

How To Use Criterion................................................................................................................. 44

Criterion Reference Guide

Frequently Asked Questions ...................................................................................................... 57

Glossary Of Terms..................................................................................................................... 61

E-rater®: What It Is, How It Works ............................................................................................. 62

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 4

Figure 1:

Criterion Online Writing Evaluation:

Students access a passwordprotected website, submit essays on topics assigned by their teacher, and instantly receive a holistic score and comprehensive diagnostic feedback.

U s i i n g T h e C r r i i t t e r r i i o n

S y s t t e m

This section contains a step-by-step overview of all essential functions in using Criterion

. Each module is illustrated with screen captures of the system to familiarize instructors and help them navigate through the application.

This User Manual is designed to support the activities of the system administrator and instructors as they incorporate Criterion as a tool to support writing instruction.

Administrators will utilize Criterion’s management and reporting tools to monitor usage and productivity.

Instructors can work independently as they begin using

Criterion in their classrooms.

If you need assistance with the Criterion Online Writing

Evaluation service, please email your Criterion Account

Manager at ETS Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 5

Figure 2:

Administrators can access the

Criterion service from their browser using an instructor-specific URL.

Figure 3:

From the top navigation panel on any page you can access links to

“Help,” ”Printer-Friendly Version,” and “Log Out.”

Figure 4:

Administrators log in from their own

Criterion Welcome Page. An administrator is designated for your account to coordinate access and set-up.

A d m i i n i i s t t r r a t t o r r Q u i i c k

S t t a r r t t

Use the following abbreviated steps to quickly begin set-up of your Criterion

account. Be sure to refer to

Criterion Administrators on page 12 for

comprehensive instruction.

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

A Criterion Administrator must be identified as a primary contact for your Criterion account. Your Administrator will need the

Administrator Access ID and Password to register. The first time you use Criterion, you will be required to register as the

Administrator. After you have registered, you will not be required to register again.

The Administrator has access to usage reports that the instructors do not. The Administrator may also add additional schools to the account. You have the option to set up your teachers’ ID’s and

Passwords or you may have them self-register using the School

Access ID and Password you create.

Also, if you are the Administrator and an instructor, you will need to register as an instructor too. See Instructor Quick Start.

Register As A User

To register, you will need your School ID and School Password.

Space is provided below to record the School ID and Password given to you by your Criterion Account Manager at ETS

Technologies, Inc.

Access ID:

Access Password:

This is the Criterion ID and Password given to you by ETS Technologies.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 6

Figure 5:

Administrators must register as a firsttime user.

Figure 6:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion.

You will use the following URL to access the Criterion Administrator

Welcome Page:

www.etstechnologies.com/criterion/administrator

The first time you click on this link, you will need to click on

Register. After registering, you will always click on Login and use the User ID and password that you create to login to view usage, add schools, email instructors or create instructor ID’s and

Passwords.

Space is provided below to record the User Name and Password that you create.

User Name:

Password:

Once you register, please go to page 15 of the Criterion

Administrator User Manual for step-by-step instructions on how to use the administrator functions and options.

If you need assistance with the Criterion Online Writing Evaluation service, please email your Criterion Account Manager at ETS

Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 7

Figure 7:

Instructors can access the Criterion service from their browser using an instructor-specific URL.

Figure 8:

From the top navigation panel on any page you can access links to

“Help,” ”Printer-Friendly Version,” and “Log Out.”

Figure 9:

Instructors log in from their own

Criterion Welcome Page. An instructor can create classes, create assignments, view student essays, and create and view class reports.

I n s t t r r u c t t o r Q u i i c k S t t a r r t

Use the following abbreviated steps to quickly begin set-up of your Criterion

assignments. Be sure to refer to

Criterion Instructors on page 18 for comprehensive

instruction.

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

Register As A User

You will need your School Access ID and Password to register.

The first time you use Criterion, you will be required to register.

After you have registered, you will not be required to register again.

Access ID:

Access Password:

This is the Criterion ID and Password given to you by your school administrator.

You will use the following URL to access the Criterion Instructor

Welcome Page:

www.etstechnologies.com/criterion/instructor

The first time you click on this link, you will need to click on

Register. After registering, you will always click on Login and use the User ID and password that you create to login and add classes, change topics, view student reports or view student IDs and passwords:

Space is provided below to record the User Name and Password that you create for yourself.

User Name:

Password:

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 8

Figure 10:

Instructors must register as a first-time user.

Once you register, please go to page 20 of the Criterion Instructor

User Manual for step-by-step instructions on how to use the instructor functions and options.

If you need assistance with the Criterion Online Writing Evaluation service, please email your Criterion Account Manager at ETS

Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

Figure 11:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 9

Figure 12:

Students can access the Criterion service from their browser using an instructor-specific URL.

Figure 13:

From the top navigation panel on any page you can access links to

“Help,” ”Printer-Friendly Version,” and “Log Out.”

Figure 14:

Students log in from their own

Criterion Welcome Page. A student can complete assignments, view assignment scores and feedback, and access an online portfolio.

S t t u d e n t Q u i i c k

S t t a r r t t

Use the following abbreviated steps to quickly begin completing Criterion

assignments. Be sure to refer to

Criterion Students on page 29 for comprehensive

instruction.

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

Students can complete assignments, view assignment scores and annotated diagnostic feedback, and access an online portfolio that provides access to all of their written work and capabilities to comment back to their teacher.

Register As A User

You will need your School Access ID and Password to register.

The first time you use Criterion, you will be required to register.

After you have registered, you will not be required to register again.

Access ID:

Access Password:

This is the Criterion ID and Password given to you by your instructor.

You will use the following URL to access the Criterion Student

Welcome Page:

www.etstechnologies.com/criterion/student

The first time you click on this link, you will need to click on

Register. After registering, you will always click on Login and use the User ID and password that you create to login and complete or revise an assignment, restart a saved draft, or view your online portfolio.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 10

Figure 15:

Students must register as a first-time user.

Space is provided below to record the User Name and Password that you create for yourself.

User Name:

Password:

Once you register, please go to page 32 of the Criterion Student

User Manual for step-by-step instructions on how to use the student functions and options.

Figure 16:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 11

Figure 17:

Administrators can access the

Criterion service from their browser using an instructor-specific URL.

Figure 18:

From the top navigation panel on any page you can access links to

“Help,” ”Printer-Friendly Version,” and “Log Out.”

C r r i i t t e r r i i o n

A d m i i n i i s t t r r a t t o r r s

Each Criterion

client will be required to assign at least one Criterion administrator to manage the account from the state, district, or school level. ETS Technologies will provide that administrator with an ID and Password that will allow access in order to begin building the account.

In order for teachers to access Criterion, the administrator will need to create a second ID and

Password to share with users.

The administrator will have to create district, school, and teacher accounts for each user. Teachers will be required to register the first time they log in. After they have registered, they will not be required to register again. Teachers will simply log in to create their classes and assign topics, add classes, change topics, access their student’s writing portfolios, view student reports, and much more.

Getting Started

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

If you need assistance with the Criterion Online Writing Evaluation service, please email your Criterion Account Manager at ETS

Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 12

Figure 19:

Administrators log in from their own

Criterion Welcome Page. An administrator is designated for your account to coordinate access and set-up.

Select “Register” to create your User

ID and Password as a first-time user.

Select “Log In” each time you return to

Criterion. You will enter the User

Name and Password that you created at registration.

Figure 20:

Administrators must register as a firsttime user.

The URL You Will Need to Access Criterion

To access the Criterion service and log in as the Criterion

Administrator, use the following administrator-specific URL:

www.etstechnologies.com/criterion/administrator

From this Administrator Welcome Page you will be able to register and log in to add districts, schools and instructors, and view usage and performance reports, and email instructors and students using the Criterion service.

Instructors will log in from the Instructor Welcome Page via an instructor-specific URL.

Students will log in from the Student Welcome Page via a studentspecific URL.

Registering to Use Criterion

The first time you use Criterion, you will be required to register.

After you have registered, you will not be required to register again.

To register, you will need your School ID and School Password.

Space is provided below to record the School ID and Password given to you by your Criterion Account Manager at ETS

Technologies, Inc.

Access ID:

Access Password:

Select Register to register as a first-time new user. You will only have to do this the first time you register as an administrator. You will be prompted to create your own User Name and Password to use for each future log in.

The Log In page is the first page that displays when you go to the

Criterion website.

How To Log In

Click the Log In button if you have already registered and know your user name and password. Clicking on the Log In button will take you to the Log In Information page.

Select Log In each time you return to Criterion. You will enter your

User Name and Password to create a school districts, schools, and instructors.

Figure 21:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 13

Figure 22.

Figure 23.

Space is provided below to record the User Name and Password that you create for yourself.

User Name:

Password:

Helpful Hints About Creating Passwords:

· We recommend using lower case letters with no spaces.

· Your Instructor Name must be unique to the system.

How To Edit Administrative Information

To make edits to administrative information, school districts, individual schools, and individual instructors, click on the box at the left of an existing class name and select the edit link from the left menu. Revise the identification and access information within the appropriate fields. See Figure 22.

How To Delete Administrative Information

To delete administrative information, school districts, individual schools, and individual instructors, click on the box at the left of an existing class name and select the edit link from the left menu.

Select and delete the administrative information from your Criterion roster. See Figure 23.

The Administrator Options page is displayed when you select a district on the District Report page and click on the District

Administrator Options link. The dropdown list defaults to the district selected on the Districts Report page. You can change you selection by selecting a different district from the dropdown list.

Select Add New District to display the Add New District page.

To edit information about the districts, click in the checkbox next to the district name and select the Edit District Information link to display the Edit District Information page.

To delete a district, click in the checkbox next to the district name that you want to delete. Then select the Delete District link. You will be prompted to confirm your selection before the website deletes the district.

Clicking on the district name displays the schools listed for that district.

Select one of the Email links to display the Email age.

At the bottom of the right-hand pane are two links: Select All and

Clear. Click Select All to select all of the checkboxes next to the

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 14

Start here if you want to set up a district with multiple schools under the district.

Start here if you want to set up just one school with multiple teachers in the school.

Figure 24. district names. Click Clear to deselect all of the checkboxes next to the district names.

Setting Up Districts, Schools, And Instructors

Setting Up A School District

The Add New District page is displayed when you select the Add

New District link on the District Report page.

Complete the District ID, District Full Name, and District Password fields. Note: The District ID must be unique. The District ID will only be used for registration purposes. The District Full Name will be displayed within the website.

Once you have completed all of the fields, click the Save button to save the new district information. A message will display indicating that the district was added. You will be returned to the Add New

District page so that you can add additional districts, if necessary.

Click the Save and Return button to save all changes and return to the Districts Report page.

Click the Cancel button to ignore all changes and return to the

Districts Report page.

Setting Up A School

The Add New School page is displayed when you select the Add

New School link on the School Reports page. See Figure 24.

Complete the School ID, School Full Name, and School Password fields. Note: The School ID must be unique. The School ID will only be used for registration purposes. The School Full Name will be displayed within the website.

Once you have completed all of the fields, click the Save button to save the new school information. A message will display indicating that the school was added. You will be returned to the Add New

School page so that you can add additional schools, if necessary.

Click the Save and Return button to save all changes and return to the Schools Report page.

Click the Cancel button to ignore all changes and return to the

Schools Report page.

Setting Up An Instructor

The Add New Instructor page is displayed when you select the Add

New Instructor link on the Instructor Open page. See Figure 25.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 15

Figure 25.

Figure 26.

Figure 27.

Figure 28.

To add a new instructor, complete the fields on this screen. Note:

the User ID must be unique. Once you have completed the fields, click the Save button to save this instructor. You will remain on this page so that you can add more instructors. Note: The password

and name fields are not case sensitive.

Click the Save and Return button to save the new instructor information and return to the Instructor Options page.

Click the Cancel button to discard the new instructor information and return to the Instructor Options page.

Using District, School And Instructor Reports

Using School District Reports

The Districts Report page is your Home page and shows the districts that are assigned to you. See Figure 26. If you do not have any districts currently assigned to you, a message will display indicating this.

The left pane includes the following menu options:

· Add New District

· Edit District Information

· Delete District

· Email All Administrators in District

· Email All Instructors in District

· Email All Students In District

· Export Report Data

· Edit My User Information

· District Administrator Options

The right pane includes the districts that are assigned to you.

Select Add New District to display the Add New District page.

In most cases, you will need to select one or more districts by clicking the checkboxes to the left of the district names. Then select a function from the menu on the left. Click a district name to access the list of schools for that district.

To edit district information, select one district. Then select the Edit

District Information link to display the Edit District Information page.

To delete a district or districts, select one or more districts. Then click on Delete District to display the Delete Confirmation dialog.

You will be prompted to confirm your selection before the website deletes the district or districts.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 16

Figure 29.

Figure 30.

Figure 31.

To send an email to all of the administrators in a district, select one or more districts. Then select Email All Administrators in District to display the Email page. See Figure 27.

To send an email to all of the instructors in a district, select one or more districts. Then select Email All Instructors in District to display the Email page. See Figure 28.

To send an email to all of the students in a district, select one or more districts. Then select Email All Students in District to display the Email page.

To export report data, select a district. Then click on Export

Report Data to display a form for exporting the report data.

Select Edit My User Information to display the Edit User

Information page. See Figure 29.

At the bottom of the right-hand pane are two links: Select All and

Clear. Click Select All to select all of the checkboxes next to the district names. Click Clear to deselect all of the checkboxes next to the district names. See Figure 30.

Email Correspondence With Your Users

To send an email to all of the administrators in a district, select one or more districts. Then select Email All Administrators in District to display the Email page. See Figure 31.

To send an email to all of the instructors in a district, select one or more districts. Then select Email All Instructors in District to display the Email page. See Figure 32.

To send an email to all of the students in a district, select one or more districts. Then select Email All Students in District to display the Email page. See Figure 33.

Figure 32.

Figure 33.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 17

Figure 34:

Instructors can access the Criterion service from their browser using an instructor-specific URL.

Figure 35:

From the top navigation panel on any page you can access links to

“Help,” ”Printer-Friendly Version,” and “Log Out.”

C r r i i t t e r r i i o n I

I n s t t r r u c t t o r r s

To access Criterion

, you will need to obtain an ID and

Password from your Criterion account administrator.

The first time you use Criterion, you will be required to register using that ID and Password. After you have registered, you will not be required to register again.

You will then be able to log in to create your classes and assign topics, add classes, change topics, access your student’s writing portfolios, view student reports, and much more.

Getting Started

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

If you need assistance with the Criterion Online Writing Evaluation service, please email your Criterion Account Manager at ETS

Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

The URL You Will Need to Access Criterion

To access the Criterion service and log in from the Instructor

Welcome Page, use the following instructor-specific URL:

www.etstechnologies.com/criterion/instructor

From this site you will be able to register and log in to add classes, to change topics, to view student reports or to view student IDs and passwords.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 18

Figure 36:

Instructors log in from their own

Criterion Welcome Page. An instructor can create classes, create assignments, view student essays, and create and view class reports.

Select “Register” to create your User

Name and Password as a first-time user.

Select “Log In” each time you return to Criterion. You will enter the User

Name and Password that you created at registration.

Figure 37:

Administrators must register as a firsttime user.

Figure 38:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion

Students will log in from the Student Welcome Page via a studentspecific URL.

Registering to Use Criterion

The first time you use Criterion, you will be required to register.

After you have registered, you will not be required to register again.

To register, you will need your School ID and School Password.

Space is provided below to record the School ID and Password given to you by your school administrator or Criterion Account

Manager at ETS Technologies.

Access ID:

Access Password:

Select Register to register as a first-time new user. You will only have to do this the first time you register as an instructor. You will be prompted to create your own User Name and Password to use for each future log in.

The Log In page is the first page that displays when you go to the

Criterion website.

How To Log In

Click the Log In button if you have already registered and know your user name and password. Clicking on the Log In button will take you to the Log In Information page.

Select Log In each time you return to Criterion. You will enter your

User Name and Password to create classes, create assignments, view student essays, and create and view class reports.

Space is provided below to record the User Name and Password that you create for yourself.

User Name:

Password:

Setting Up Your Classes

Setting up a class in Criterion is not difficult, although it will require a little time. This is time invested up front, and then the application is very easy to use. You will need your class roster information in order to begin setting up your classes.

Select Add New Class from the left menu. Create and enter all of the identification and access information into the fields. Space is provided below to record the Class ID, Class Name, and Password

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 19

Figure 40:

Select “Add New Class” to access a screen that will prompt you to enter the necessary data to set up a new class to use Criterion.

Figure 41:

Select “Edit Class Information” to edit the data about the class you have selected from your roster of classes.

Select “Delete Class” to delete the class you have selected from your roster of classes.

Start here if you want to set up an assignment.

Figure 42:

Select “Assignment Options” to access a screen that will allow you to view, create, and access the assignments for a class. that you will use to access your class reports.

Class ID:

Class Full Name:

Password:

Click on Save and Return. Click on Cancel to cancel your work.

Helpful Hints About Creating Passwords:

· We recommend using lower case letters with no spaces.

· Your Instructor Name must be unique to the system.

To make edits to class information, click on the box at the left of an existing class name and select Edit Class Information from the left menu. Revise the identification and access information within the appropriate fields.

To delete a class, click on the box at the left of an existing class name and select Delete Class from the left menu. This will delete the class from your Criterion roster.

Setting Up Your Assignments

The Assignment Options page displays when you select a class and the Assignment Options link on the Class Report page.

To create or manage an assignment, click the box at the left of an existing class name and select Assignment Options from the left menu.

The Assignment Options page shows the assignments for each of your classes. The left menu includes the following options:

· Create Assignment

· Edit Assignments

· Copy Assignment

· Delete Assignment

· Export Report Data

The right window includes the list of assignments for your class. If you have multiple classes assigned to you, the right window will also include a drop-down list for selecting a class.

In most cases, you will need to select one or more assignments by clicking the corresponding check boxes. Next, select an

Assignment function from the menu on the left. Click an assignment's name to view or edit that assignment's options.

To create an assignment, select a class. Next, select the Create

Assignment link to display the Create Assignment page.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 20

Figure 43:

Select “Create Assignment” to access a screen that will prompt you to enter the necessary data to set up a new assignment. Select “Edit Assignment” to edit an assignment. Select “Copy

Assignment” to copy an assignment.

Select “Delete Assignment” to delete an assignment.

Figure 44:

First, select the writing level of your students by selecting a grade level from the first drop down menu. Then select the essay topic from the next drop down menu. The topic you select and the prompt are displayed. Finally, follow the prompts to turn on or off the feedback options for your assignment.

To edit an assignment, select a class. Next, select an assignment and the Edit Assignment link to display the Edit Assignment page.

To copy an assignment, select an assignment. Select only one at a time. Next, click on the Copy Assignment link to display the

Copy Assignment page.

Select one or more assignments and the Delete Assignment link to display the Delete Confirmation dialog. You will be prompted to confirm your selection before the assignment is deleted.

To export report data, select an assignment. Select only one at a time. Next, click on the Export Report Data link to export the report data into a format that can be used in a word processor or spreadsheet.

At the bottom of the right-hand window are two links: Select All and Clear. Click Select All to select all of the checkboxes next to the assignment names. Click Clear to deselect all of the checkboxes next to the class names.

Create Assignment

Select the class from the drop-down menu at the top of the page for which you want to create an assignment. Select Create

Assignment from the left menu.

The Create Assignment page is displayed when you select the

Create Assignment link on the Assignment Options page.

Initially, this screen just has the Assignment Name entry box and

Essay Topics dropdown list. When you select a topic, the rest of the screen will be populated.

To begin the task of creating an assignment, select an Essay Topic

Category and an Essay Topic. If there is more than one question in the selected Essay Topic, an Essay Question dropdown list will be displayed beneath the Essay Topic list. If this is the case, you should also select an Essay Question. Selecting the Essay Topic

Category and Essay Topic will display the options and default values specified by ETS Technologies.

If the selected topic is an instructor topic, the Question Text field will be editable. You will be able to write your own essay question.

For these questions, you will be able to view the Feedback

Analysis Report. A score will not be available.

The Create Assignment page includes the following options, if made available by ETS Technologies:

· Time Limit

· Show Warning When [x] Minutes Remain

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 21

Figure 45:

Select “Copy Assignment” to access a screen that will prompt you to enter the necessary data to copy an assignment.

· Spell Checker Available

· Students Can Save Essay to Complete Later

· Limit Students To [x] Submission(s)

· Show Holistic Score To Students When Advisory Present

· Show Grammar Feedback

· Show Usage Feedback

· Show Mechanics Feedback

· Show Style Feedback

· Show Organization and Development Feedback

· Start Assignment [date and time]

· Stop Assignment [date and time]

Make your selections for the assignment you wish to create.

Click the Add This Assignment button to save the assignment information and remain on this page so you can add another assignment.

Click the View All Topics button to link to a web page containing descriptions of all of the topics.

Click the Save and Return button to save the assignment information and return to the Assignment Options page.

Click the Cancel button to ignore the current assignment information and return to the Assignment Options page.

Edit Assignment

Select the class from the drop-down menu at the top of the page for which you want to edit an assignment. Select Edit Assignment from the left menu.

The Edit Assignment page is displayed when you select an assignment on the Assignment Options page and select the Edit

Assignment link.

Edit the assignment options, as necessary. Then click the Save

Changes button to save your changes and return to the

Assignment Options page.

Click the Cancel button to ignore all changes and return to the

Assignment Options page.

Copy Assignment

Select the class for from the drop-down menu at the top of the page for which you want to copy an assignment. Select Copy

Assignment from the left menu.

The Copy Assignment page is displayed when you select an assignment on the Assignment Options page and select the Copy

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 22

Figure 46.

Assignment link. The option provides a means to easily copy an assignment to your other classes.

Select one or more classes to receive a copy of the selected assignment(s) by clicking the checkboxes to the left of the class names. Then click the Copy button at the bottom of the page.

Clicking the Copy button will copy the assignment, prompt you to rename the assignment if the assignment name already exists in that class, and return you to the Assignment Options page.

Click the Cancel button to return to the Assignment Options page without copying the assignment(s).

Delete Assignment

Select the class from the drop-down menu at the top of the page for which you want to delete an assignment. Select Delete

Assignment from the left menu.

Confirm Deletion

This page serves as a prompt to confirm your deletion selection before the website removes it.

Click Yes to continue with the deletion. You will be returned to the calling page.

Click No to cancel the deletion process and return to the calling page.

Export Report Data

Select the class from the drop-down menu at the top of the page for which you want to export report data. Select Export Report

Data from the left menu.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 23

Figure 47.

What Your Students See And Do In Criterion

Writing diagnostics are provided in the following areas:

Categories Sub-Categories

Grammar

Usage

Mechanics

Style

Organization and

Development

Fragments

Garbled sentences

Subject-verb agreement

Verb-form errors

Pronoun errors

Possessive errors

Wrong or missing words

Article errors

Confused words

Wrong form of word

Faulty comparison

Nonstandard verb or word form

Spelling

Missing capitalization of proper nouns

Missing initial capital letter in a sentence

Missing question mark

Missing final punctuation

Missing apostrophe

Missing comma

Missing hyphen

Fused words

Compound words

Duplication words

Repetition of words

Inappropriate words or phrases

Sentences with passive voice

Long sentences

Short sentences

Sentences beginning with coordinating conjunctions

Introductory Material

Thesis statement

Main ideas

Supporting ideas

Conclusion

Transitional words and phrases

Using The Class And Student Reports

The Classes Report page is your Home page and shows the classes that are assigned to you. If you do not have any classes currently assigned to you, a message will be displayed to indicate this.

The left menu includes the following menu options:

· Add New Class

· Edit Class Information

· Delete Class

· Assignment Options

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 24

Figure 48.

Figure 49.

· Batch Print Reports

· Email All Students in Class

· Import Student Information

· Export Report Data

· Edit My User Information

The classes that are assigned to you are displayed in the right window.

Add Class Information

Select Add New Class to display the Add New Class page.

In most cases, you will need to select one or more classes by clicking the checkboxes to the left of the class names. Next, select a function from the menu on the left. Click a class name to access the list of students for that class.

The Add New Class page is displayed when you select a class and the Add New Class link on the Class Report page.

Complete the Class ID, Class Full Name, and Class Password fields. Note: The Class ID must be unique. The Class ID will be used only for registration purposes. The Class Full Name will be displayed within the website.

Once you have completed all of the fields, click the Save button to save your changes. A message will display, indicating that the class was added. You will be returned to the Add New Class page so that you can add additional classes, if necessary.

Click the Save and Return button to save all changes and return to the Classes Report page.

Click the Cancel button to ignore all changes and return to the

Classes Report page.

Edit Class Information

To edit class information, select one class. Next, select Edit Class

Information to display the Edit Class Information page.

The Edit Class Information page is displayed when you select a class and the Edit Class Information link on the Classes Report page.

Edit the Class ID, Class Full Name, and Class Password fields, as necessary. Note: The Class ID must be unique. The Class ID will only be used for registration purposes. The class full name will be displayed within the website.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 25

Figure 50.

Once you have edited the class information, click the Save

Changes button to save your changes and return to the Classes

Report page.

Click the Cancel button to ignore all changes and return to the

Classes Report page.

Delete A Class

To delete a class or classes, select one or more classes. Next, click on Delete Class to display the Delete Confirmation dialog.

You will be prompted to confirm your selection before the website deletes the class or classes.

Assignment Options

To access the assignment options, select one class. Next, select

Assignment Options to display the Assignment Options page.

Error Reports

To view error reports, click on the Error Reports button. Select a class, then an assignment.

Batch Print Reports

To batch print reports, select a class. Next, select Batch Print

Reports to display the Batch Print Reports page.

Email All Students in Class

To send an email to multiple students, select one or more classes.

Next, select Email All Students in Class to display the Email All

Students in Class page.

Import Student Information

To import student information, select a class. Next, click on Import

Student Information to display a form that allows you to browse for a data file.

Export Report Data

To export report data, select a class. Next, click on Export Report

Data to display a form that allows you to browse for a data file.

Edit My User Information

Select Edit My User Information to display the Edit User

Information page.

At the bottom of the right-hand window are two links: Select All and Clear. Click Select All to select all of the checkboxes next to the class names. Click Clear to deselect all of the checkboxes next to the class names.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 26

Creating And Inserting Pop-Up Notes

The Create and Manage Pop-Up Notes page is displayed when you select the Pop-Up Notes link on the Score Analysis Report.

To insert a pop-up note, click a word in the text. This will place an

symbol to the left of the word that you clicked and display the

Create Pop-Up Note page.

To edit or delete an existing pop-up note, click on an

symbol or the word following the symbol. This will display the Edit Pop-Up

Note page.

Click the Save and Return button to save the pop-up note and display the Score Analysis Report, updated with any new

symbols.

Writing Comments To Your Students

Using The Standard Comments Library

The Standard Comments Library page is displayed when you click on the Comments Library… button on the Create/Respond to

Comment page or the Create Pop-Up Note page. If you reached this page from the Create/Respond to Comment page, the button on the bottom left will be labeled Add Text to Comment. If you reached this page from the Create Pop-Up Note page, the button on the bottom left will be labeled Add Text to Pop-Up. The instructions will also change dynamically, depending on how you reached this page.

The Standard Comments Library is a repository of comments or pop-up text that you can reuse. Selecting one of the standard comments displays its text in the box in the upper right of the display.

To append standard comment text to the end of the comment you are creating in the Create/Respond to Comment page, select a standard comment text and click on the Add Text to Comment button.

Click the Create New Comment Text button to display the Create

New Standard Comment page.

Click the Save Changes button to save the changes to a comment.

Click the Delete Comment button to delete the currently selected comment. You will be prompted to confirm the deletion before the comment is actually deleted from the system.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 27

Click the Save and Return button to return to the Create/Respond to Comment page.

Creating A New Standard Comment

The Create New Standard Comment page is displayed when you click the Create New Comment Text button on the Standard

Comments Library page.

To create a new standard comment, follow these steps:

· Enter a name in the Standard Comment Name field.

· Enter the text for the new comment in the standard comment text field

· Click the Add This Comment button to save the comment and add it to the library. This action will blank out the two entry fields. You will remain on this page so you can create another new comment.

Click the Save and Return button to return to the Standard

Comments Library page and add the new standard comment to the list.

Click the Cancel button to return to the Standard Comments

Library page without adding the new comment.

Creating A Comment

The Create/Respond to Comment page is displayed when you click the Create a comment link or the Respond to this comment link from the Score Analysis Report.

The Create/Respond to Comment page displays the selected student's essay in the pane on the right. The top left pane is prefilled with the name of the student and a text entry box. The bottom left pane includes the Comment History.

If you did not enter a courtesy title at registration, e.g. Mrs., your first name will be displayed in the comments. You can add a title by editing your user information.

To create a comment, type your text in the text entry box provided.

When you are done composing your comment, click the Post

Comment button. This will save your comment, return you to the

Score Analysis Report, and provide an indication to the student that a comment or response exists.

Click the Comments Library… button to display the Standard

Comments Library page.

Click the Cancel button to ignore your comment or response and return to the Score Analysis Report.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 28

Figure 51:

Students can access the Criterion service from their browser using an instructor-specific URL.

Figure 52:

From the top navigation panel on any page you can access links to

“Help,” or ”Printer-Friendly Version,” and the “Writer’s Handbook” or

“Revise Essay.”

C r r i i t t e r r i i o n S t t u d e n t t s

To access Criterion

, you will need to obtain an Access

ID and Password from your instructor. The first time you use Criterion, you will be required to register using that

Access ID and Password. After you have registered, you will not be required to register again. You will then be able to log in to complete your assignments, view your score analysis and annotated diagnostic feedback, access your writing portfolio, and view your reports.

Getting Started

Navigating Through The System

To begin using Criterion, you will need an Internet connection and a web browser. We recommend using Internet Explorer 5.5 or higher or Netscape 6.1 or higher.

From the top navigation panel on any page you can access Help,

Printer-Friendly Version, and Log Out.

Help will open and guide you simultaneously and parallel to the screen and task on which you are working.

You can print a screen or report in a printer-formatted version using the Printer-Friendly Version link.

You can log out and return to the Criterion log-in page using the

Log Out link.

The URL You Will Need to Access Criterion

To access the Criterion service and log in from the Student

Welcome Page, use the following student-specific URL:

www.etstechnologies.com/criterion/student

From this site your students will be able to register and log in to

complete or revise an assignment, restart a saved draft, or view your online portfolio

Students will log in from the Student Welcome Page via a studentspecific URL.

Registering to Use Criterion

The first time you use Criterion, you will be required to register.

After you have registered, you will not be required to register again.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 29

Figure 53:

Students log in from their own

Criterion Welcome Page. A student can complete assignments, view assignment scores and feedback, and access an online portfolio.

Select “Register” to create your User

Name and Password as a first-time user.

Select “Log-In” each time you return to

Criterion. You will enter the User

Name and Password that you created at registration.

Figure 54:

Students must register as a first-time user.

Figure 55:

Enter the User Name and Password that you created at registration to log in each time you return to Criterion.

To register, you will need an Access ID and Access Password.

Space is provided below to record the Access ID and Access

Password given to you by your instructor.

Access ID:

Access Password:

Select Register to register as a first-time new user. You will only have to do this the first time you register as a student. You will be prompted to create your own User Name and Password to use for each future log in.

The Log In page is the first page that displays when you go to the

Criterion website.

How To Register

The New User Registration page is displayed when you enter a valid Access ID and Access Password on the access Information page.

Enter the requested information in the fields provided and click the

Create Registration button. The website will check to see if all of the required fields have been completed, that the two password fields match, and that the fields contain valid characters. A valid user name will contain alphanumeric characters only and must be unique. Spaces and symbols will not be accepted in the user name field. All other fields on this screen can include alphanumeric characters, as well as spaces and symbols.

Note: In order to use the website's email function, you must complete the Email Address field.

If all of the information is validated, the Home Page will display in a new window.

Click the Cancel button to return to the Log In page.

Using Criterion In Multiple Classes

Click the New Class button if you already have a user name and password, but want to register yourself in an additional class.

Clicking on this button will take you to the Log In Information page.

How To Log In

Click the Log In button if you have already registered and know your user name and password. Clicking on the Log In button will take you to the Log In Information page.

Select Log In each time you return to Criterion. You will enter your

User Name and Password to create classes, create assignments, view student essays, and create and view class reports.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 30

Space is provided below to record the User Name and Password that you create for yourself.

User Name:

Password:

The Student Home Page displays when you log on or register. If you are registered in multiple classes, the names of the classes will appear under the Home Page heading in a drop-down list next to

Select a Class. If you are registered in one class only, the name of your class will appear under the Home Page heading.

The Home Page includes links for viewing new comments from your instructor, viewing your portfolio, or selecting an assignment.

Click View a New Comment From Your Instructor to display the

Portfolio page and access your essays and their associated comments.

Click View your Portfolio for this class to display the Portfolio page.

Click on an assignment name to display the Assignment

Instructions Page.

Click the Edit My User Information button to display the Edit User

Information page.

The Edit User Information page is displayed when you click on the

Edit My User Information button on the Home Page.

Modify the displayed information as necessary and click the Save

Changes button. The website will check to see if all of the required fields have been completed, that the two password fields match, and that the fields contain valid characters. A valid user name will contain alphanumeric characters only and must be unique. Spaces and symbols will not be accepted in the user name field. All other fields on this screen can include alphanumeric characters, as well as spaces and symbols.

Note: In order to use the website's email function, you must complete the Email Address field.

If all of the information is validated, the website will return you to the Home Page.

Click the Cancel button to return to the Home Page.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 31

Start here if you want to set up an assignment.

Figure 56.

Figure 57.

Figure 58.

Figure 59.

Completing An Assignment

From the Student Home Page, click on an assignment name to display the Assignment Instructions Page.

The Assignment Instructions Page is displayed when you select an assignment from the Home Page.

This page includes the instructions that you should follow in completing your assignment.

Click on the Home link to return to the Home Page.

After reviewing the instructions, click the Continue button to proceed to the next step in the assignment.

The Essay Entry Page is displayed when you click Continue on the Assignment Instructions Page. This page is where you complete your assignment.

The Essay Entry Page includes the name of the assignment, a description of the topic, and an entry field in which you can write your essay. Depending on the settings that have been set up for your assignment, the Essay Entry Page may also include the following:

· View Supporting Document button

· Time limit for completing the essay

· Time remaining on the clock

· Submit This Essay button

· Save This Essay to Finish Later button

· Check Spelling button

· Warning when approaching the time limit

If your instructor has included a document for you to use in writing your essay, the View Supporting Document button will display between the Topic description and the essay entry box. Click on the View Supporting Document button to access the document.

If the Time Remaining is more than 60 minutes, the page will display the time as hours, minutes and seconds, e.g., 1:39:47.

Click the Submit This Essay button to submit your essay for evaluation. When you click this button, Criterion will evaluate your essay and will either display the Score Summary page or the

Feedback Analysis page in a new browser window.

Click the Save This Essay to Finish Later button to save the essay for completion at a later time. Clicking this button saves the essay, but does not submit it for evaluation. If you log off after clicking this button, the Bookmark page will display the next time

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 32

you log on. You also have the option of clicking this button and continuing to work on the essay. If you choose this option, the time remaining on the clock (if the essay is being timed) will continue where you left off.

Click the Check Spelling button to display the Spell Checker page.

If you are still working on your essay when the total time elapses

(for timed essays), the website will display a dialog indicating that your time has expired. If you have not previously saved your essay, you will be given options to submit your essay (click OK) or to ignore your work (click Cancel). If you have previously saved your essay, you will be given options to submit your essay (click OK) or to ignore changes since your last save (click Cancel).

Using The Spell Checker

The Spell Checker page is displayed when you click the Check

Spelling button on the Essay Entry Page.

Misspelled words or words not found in the dictionary are underlined on the Spell Checker page. Click each underlined word to display the Spell Checker Options page and view suggestions for possible alternatives.

Click the Save and Return button to display the page from which you clicked the Check Spelling button.

The Spell Checker Options page is displayed when you click an underlined word on the Spell Checker page.

The Spell Checker Options page displays the word in question by itself, as well as in the context of the sentence in which it appears in your essay. Select a word from the list of suggested words. Then click the Change button to change the word in the sentence that is displayed. If you want to change all occurrences of this word in your essay, click the Change All button.

Click the Save and Return button to accept all changes and return to the Spell Checker page.

Click the Cancel button to ignore changes on this page and return to the Spell Checker page.

How to Bookmark Your Work

The Bookmark page is displayed when you log onto Criterion after clicking the Save this essay to finish later button on the essay entry page in your previous session. This page is also displayed when you select an assignment that was saved since its last submission. These assignments are noted as In Progress on the

Home Page.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 33

Figure 60.

The Bookmark page displays the date you were last working on the assignment and the name of the assignment.

Click the Continue Where You Left Off button to redisplay the essay entry page. If the essay is being timed, the timer will be set to whatever its value was when you last saved the essay.

Click the Restart This Assignment button if you want to start over from the beginning of the assignment. The website will prompt you with a message indicating that this function will delete the saved copy of your essay. You will need to confirm your selection in order to proceed. Once you have confirmed your selection, the essay entry page will display with the essay field blank.

Click the Proceed to Home Page button to display the Home

Page.

Accessing The Score Analysis

If enabled by your instructor, the Score Summary page is displayed when you click the Submit this Essay button on the Essay Entry

Page. The Score Summary page is also displayed when you click the View Score Summary link from the Score Analysis Report. If you accessed this page from the reports, the "Click the Continue button for more information…" text will be removed and the

Continue button will be replaced with a Return button.

The Score Summary page includes your name, assignment name, and advisories. Depending on the settings that have been set up for your assignment, the Score Summary page may also include the following:

· Time limit for completing the assignment

· Time taken (the amount of time used to complete the assignment

· Score

· Score Summary

Click the Home link to return to the Home Page.

If your instructor has chosen to display this page, clicking the

Continue button will display the Score Analysis Report page.

If your instructor has chosen to display Writing Analysis Tool feedback only, clicking the Continue button will display the

Feedback Analysis Report.

If your instructor has chosen not to display this page or the Writing

Analysis Tool feedback, the website will display a message

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 34

indicating that your essay has been submitted and will return you to your Home Page.

The Score Analysis Report

The Score Analysis Report is displayed when you click the

Continue button from the Score Summary page. It is also displayed when you click the View Score Analysis button from the

Feedback Analysis Report. Clicking on a link in the left pane (Score

Analysis Menu) displays the corresponding information in the right pane.

The Score Analysis Report includes the Score Analysis Menu on the left, from which you can view your Score Summary report, sample essays, and the Scoring Guide for your topic. On the right side of the Score Analysis Report is your essay and comments from your instructor. When you first arrive at this page, your essay will be displayed in the right-hand pane.

Click the View Your Essay link to display your essay. This link is useful if you have recently selected other links, such as sample essay links, and you want to return to your essay.

Click the View Score Summary link to display the Score Summary page. This button will only be displayed if this feature has been enabled in the assignment.

Click the sample essay links (Score of 6, Score of 5, etc.) to view a sample essay that represents a score for the number selected.

Click the View Scoring Guide for My Topic link to display the scoring guide for your topic. The scoring guide will provide information on how your topic was scored.

Click the View Feedback Analysis button to replace the Score

Analysis Menu with the Feedback Analysis Menu. This button provides access to the options on the Feedback Analysis Report.

This button will only be displayed if this feature has been enabled in the assignment.

Click the Close Report button to close the Score Analysis Report browser window.

Click the Revise Essay button to display the essay in edit mode.

This allows you to make changes to your essay so that you can resubmit it. You can switch back and forth between the Feedback

Analysis Report and the essay in edit mode so that you can incorporate feedback from the report in your changes to your essay. If you click the Revise Essay button on the Score Analysis page, the button will be disabled until you close the essay. The

Revise Essay button is active if the current date is within the date

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 35

Figure 61. range for your assignment and if you have not exceeded your number of submissions.

Click the Printer-Friendly Version button to display the current report in a format suitable for printing using the browser's Print function. The printer-friendly version will include the text of each instructor pop-up note after the paragraph in which it appears. It will also include comment text at the end of the report.

Click the Writer's Handbook button to display the Writer’s

Handbook page.

If your scored essay includes any

symbols, roll your cursor over each symbol to display notes from your instructor.

Click the Create a comment link to write a comment to your instructor. Click the Respond to a comment link to respond to your instructor's comment.

Accessing The Annotated Feedback Analysis

The Feedback Analysis Report is displayed when you click the

View Feedback Analysis button on the Score Analysis Report.

The View Feedback Analysis button is only available if your instructor or administrator has chosen to make this option available to you.

The horizontal upper menu (Feedback Analysis Menu) contains the major error categories, including Grammar, Usage, Mechanics,

Style, and Organization and Development. See Figure 62.

Selecting an error category from the horizontal menu displays the error types for the category in the left pane. Selecting an error type displays your essay marked-up according to the category selected.

For example, if you select the Usage error category and the

Confused Words error type, the website will display your essay with all of the confused words marked.

The words or groups of words that fall into the selected error category/type are highlighted. Roll over the highlighted text to display the feedback in a tooltip-style textbox.

The Feedback Analysis page for the Organization and

Development error category is displayed when you select the

Organization and Development link from the Feedback Analysis

Report. Click on an Organization and Development element, such as Thesis or Conclusion, in the left menu pane to highlight the corresponding portion of your essay. The highlight color in your essay will vary for each error type. Organization and Development feedback includes the following:

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 36

Figure 62: Click the “Revise Essay” button to display the essay in edit mode.

· Introductory Material

· Thesis

· Main Ideas

· Supported Ideas

· Conclusion

· Other

· Transitional Words and Phrases

· Repetition of Ideas

Click the Show All Elements link to display all discourse elements in a single report. When you click the Show All Elements link, the link toggles to a Show Individual Elements link. The Show

Individual Elements link allows you to return to a display in which discourse elements are displayed one at a time.

Click the View Score Analysis button to display the Score

Analysis Report.

Click the Close Report button to close the Feedback Analysis

Report and display the next step in the assignment.

Click the Revise Essay button to display the essay in edit mode.

This allows you to make changes to your essay so that you can resubmit it. You can switch back and forth between the Feedback

Analysis report and the essay in edit mode so that you can incorporate feedback from the report in your changes to your essay. If you click the Revise Essay button on the Score Analysis page, the button will be disabled until you close the essay. The

Revise Essay button is active if the current date is within the date range for your assignment and if you have not exceeded your number of submissions.

Click the Printer-Friendly Version button to include the information displayed in the Organization and Development

Feedback Analysis Report, along with all of the pop-up feedback associated with words or phrases with symbols, in a format suitable for printing. You can use your browser's Print button to print the report. The pop-up text will print at the end of the essay.

Click the Writer's Handbook button to display the Writer’s

Handbook page.

Click the View Score Analysis button to replace the Feedback

Analysis Menu with the Score Analysis Menu on the Score Analysis

Report.

Click the Close Report button to close the Feedback Analysis

Report.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 37

See Figure 63.

Figure 64.

Click the Revise Essay button to display the essay in edit mode.

This allows you to make changes to your essay so that you can resubmit it. You can switch back and forth between the Feedback

Revise Essay button is active if the current date is within the date range for your assignment and if you have not exceeded your number of submissions.

Click the Printer-Friendly Version button to include the information displayed in the current Feedback Analysis Report, along with all of the pop-up feedback associated with words or phrases with symbols, in a format suitable for printing. You can use your browser's Print button to print the report. The pop-up text will print at the end of the essay.

Click the Writer's Handbook button to display the Writer’s

Handbook page.

Using The Student Portfolio

The Portfolio report is displayed when you click the Portfolio button on the Home Page. You can filter the entries in the report by clicking on the links above each column. For example, to filter the entries based on the date submitted, click on the Submitted link to sort the entries in ascending or descending order. An upward pointing arrow indicates that the entries are sorted in ascending order. A downward pointing arrow indicates that the entries are sorted in descending order.

To display all entries in the report, click the Show all entries radio button. See Figure 64.

To limit the entries in your report to a specific date range, click the

Show entries with: radio button. Then click the Dates between: checkbox and select the start and end dates for the report. See

Figure 65.

To limit the entries in your report to a specific score range, click the

Show entries with: radio button. Then click the Scores between: checkbox and select the start and end scores for the report.

You can limit entries to just a date range, just a score range, or a combination of the two elements.

Click the Refresh Report button to redisplay the current report display.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 38

See Figure 65.

Click an assignment name link to display the assignment's Detailed

Report (Score Analysis Report).

Create or Respond to An Instructor Comment

The Create/Respond to Comment page is displayed when you click the Create a Comment link or the Respond to this Comment link from the Score Analysis Report. See Figure 66.

The Create/Respond to Comment page displays your essay in the pane on the right. The top left pane is pre-filled with the name of the instructor assigned to your class and a text entry box. The bottom left pane includes the Comment History. If there is more than one instructor assigned to your class and you are writing the first comment, your comment will be assigned To Instructors instead of to a particular instructor's name.

To create a comment, type your text in the text entry box provided.

When you are done composing your comment, click the Post

Comment button. This will save your comment and provide an indication to your instructor that a comment or response exists.

Click the Cancel button to ignore your comment or response and return to the page from which you selected Create a Comment or

Respond to this Comment. Clicking Cancel has no effect on the comment history.

Using The Writer’s Handbook

The Writer's Handbook Contents page is displayed when you click on the Writer's Handbook button from the toolbar on the Score

Analysis Report or the Feedback Analysis Report. The page is also displayed when you click the Contents button from any Writer’s

Handbook page. See Figure 67.

Click the Error Categories and types link to display the Writer's

Handbook page.

Click the Advice to Writers link or one of its sub-links to display writing advice.

Click the Scoring Guides and Holistic Scoring link to display information about how the essays are scored.

Click the Advisories link to display advisories for your essay.

Click the Close Handbook button to return to the page from which you called the Writer's Handbook.

Figure 66.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 39

W s r r i i i i t n t i i g n

C g I

I r r i i n t t e s t t r r r r i i o u n c t t i i

T o o n

S u p p o r r t t

This section provides a context for understanding how the different components of Criterion can be used to support the teaching of writing and revising an essay. In addition to explaining Criterion's holistic score and diagnostic feedback capabilities, this section presents a variety of instructional approaches and activities to help instructors plan their use of the system in their classrooms. The section "Computer and Internet

Access" leads instructors through a series of questions to help them analyze the availability of technology in their classrooms and alternate locations in their schools.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 40

Overview of the Writing Process

Most writing instructors engage their students in what is known as the “writing process,” in which students improve their essays by incorporating these steps:

· Planning and Drafting

· Revising

· Editing

· Proofreading

The steps or stages may be referred to by different names, but the process remains the same: students gradually refine successive drafts of a writing assignment, typically with feedback from an instructor or peers. The different steps in the process involve students in working from the more general to the more specific as they improve the piece of writing. That is, their first revisions address global, structural matters, while later revisions focus on polishing and refining the essay.

Planning and Drafting

In response to an assignment, or other need for a written piece (such as a desire to write a letter to the editor of a newspaper), students prepare a first draft of their ideas. Some students will first prepare an outline; others will just begin writing. The result, however, of the drafting stage of the writing process is an essay in response to an assignment or other need.

Revising

When they revise their written work, students most often focus on the structure and development of their writing. They try to determine whether the essay as a whole “hangs together,” the ideas flow logically from paragraph to paragraph, if the organization permits moves the reader from one point to the next. At this stage, writers pay attention to the order of the paragraphs, deciding, for example, to change their sequence. It is important for writers, in the revising process, to identify all the major parts of the essay: the thesis, introduction, main ideas, supporting ideas, and conclusion. Then, they ensure that each of these elements contributes to the overall coherence of the essay.

At the revising stage, writers deal with the big issues in their work, whether the essay “works” overall. They don’t worry, at this stage, with the details. Instead, they attend to the structural elements of the essay, ensuring that they all contribute to the overall effect.

Editing

In the editing stage, students focus on all the issues of grammar, usage, mechanics, and style that might compromise the effectiveness of their writing. The items that are identified and corrected at this stage include sentence fragments, word form and verb agreement errors, stylistic errors of word repetition, choppy sentences, and poor transitions. These issues are usually best addressed one category at a time.

Targeted feedback is very helpful during both the revising and the editing process, particularly for inexperienced writers. Students, when they are first developing their skills, know what they

intended to say, and frequently don’t realize that their words, as written, don’t convey their intended meaning. Or, the meaning is clear but not eloquent, or the reader’s attention is diverted by stylistic or mechanical errors. Another set of eyes—or tools--can help enormously in helping a student to focus those aspects of an essay that could be improved. Of course, as

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 41

students acquire skill in writing, they learn to review their work with these possible errors in mind. But at first, they need feedback and support.

Proofreading

This is the last stage in the writing process and the most mechanical. It includes spelling errors and mistakes in the use of punctuation and capital letters. At this stage, students are well advised to read their own work with a highly critical eye, on the lookout for errors they may have missed earlier in the process. Of course, spell checkers in most word processing programs can support students as they make a “final pass” through their essay. Other strategies are useful as well, such as reading the essay aloud, and reading it from the last sentence backwards to the first, looking at the spelling of each word.

The writing process is difficult for some students to learn. It requires writers to look at their own work from a reader’s perspective; regard their writing as a work in progress; improve their drafts through an iterative process. For some students this is counterintuitive; once they have produced a piece of writing, they tend to think they are finished. Consequentially, some students resist (at least at first) the notion of rereading and revising their work. However, all good writing depends on adopting a critical attitude to one’s own work, and a willingness to improve it. The writing process is designed to assist in this process, and to cultivate habits of mind that enable students to continue this effort on their own.

How Criterion Supports the Writing Process

Criterion supports all of the stages of the writing process. Instructors, however, need to be clear in their purpose for using Criterion if they want to derive the maximum benefit from the application. When instructors chart a clear course for their use of Criterion, students have the benefit of more feedback on their writing than they could receive from an instructor working without the application.

Holistic Scoring

Instructors can use the holistic scoring capability in Criterion to ascertain their students’ general level of writing, and the students themselves can learn the areas of their writing that can be improved.

The holistic evaluation capability of Criterion provides students with rubric-based feedback about their writing. Keep in mind that the holistic score is available for the topics in the Criterion system. For each essay topic, an automated scoring model is built using essays scored by trained readers using a scoring rubric. Criterion’s holistic scoring works by comparing a student’s writing to hundreds of essays written on the same topic and evaluated by writing instructors. For students to get a holistic score, they need to write on a topic for which a

Criterion model has been built. It is also important o remember that the scored essays used to build models were written by students under timed testing conditions; therefore, the writers had no opportunity to revise, use spell check, or reflect on what they had written. So, when students write on Criterion topics in a regular class, working under more relaxed conditions, instructors and students alike should recognize that their scores may not precisely compare to those of the samples.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 42

Criterion’s library of topics spans grade four to graduate level, and instructors have over one hundred prompts from which to choose. At each level, there are many topics on a number of writing modes, such as description, narration, persuasion, writing for literature, comparisoncontrast, cause-effect, and others. The topics and modes are compatible with state curriculum standards, and may be aligned to a school district’s or university’s writing curriculum.

A word of caution: it is important for both students and instructors to recognize that the technology that enables Criterion to do holistic scoring is not immune from making errors.

Criterion, after all, does not actually “read” the essays – it is not a human being! Instead, it uses a highly sophisticated technology (natural language processing techniques) to analyze student writing and compare it with hundreds of scored samples. This technology is what gives holistic scoring both its power and its limitations. Because of its sophistication, the technology can score student essays very rapidly –in under 10 seconds! But because it is based on technology, the scores provided by Criterion should be used as one piece of evidence about a student’s writing. While the level of agreement between Criterion and trained readers is very high – 98% – it is not perfect.

There are even situations in which Criterion does not know what to “make” of a student’s essay.

For example, when a student writes in a highly imaginative way or takes a novel approach to a topic, the essay may not resemble any of those in the original training set. Or if it is too short a holistic score may not be valuable. Essays that are very repetitive may also raise a flag. In these cases, an “advisory” message will accompany the score. These advisories are a signal to both the student and the instructor that the essay may be problematical in some way and that the score information should be carefully reviewed.

But even with the realities inherent in any scoring application, Criterion is ideal for use in establishing the general level of writing skill in a group of students. It is a system designed to support independent learning and to help students determine their areas of strength and where they need to concentrate their efforts.

Diagnostic Feedback

The diagnostic feedback capability of Criterion closely matches the steps of the writing process.

And because the different feedback categories may be switched on and off (enabled and disabled) instructors can structure the experience of revising, editing, and proofreading for their students. Diagnostic feedback may be provided to students on essays they write on any topic whatsoever. Therefore, while instructors may assign one of the Criterion topics, they need not do so.

Depending on students’ skills in typing and spelling, come “errors” made by Criterion in scoring a student essay may simply reflect the fact that the application could not determine accurately what word the student intended to use.

Instructors can determine which types of diagnostic feedback students will receive on their writing. And the different categories of feedback have been designed to mirror the steps of the writing process. The categories are as follows:

· Organization and Development (introductory material, thesis statement, main ideas, supporting details, transitional words and phrases, and conclusion)

· Style (repetition, passive voice, essays statistics)

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 43

· Grammar (fragments, subject-verb agreement, errors of verb form, subject-verb agreement, pronouns, possessives)

· Usage (confused words, wrong word, wrong form of a word, wrong article)

· Mechanics (spelling, punctuation)

When instructors enable these types of feedback, in this sequence, they help students walk through the steps of the writing process, those of revising, editing, and proofreading. Of course, instructors may activate all the diagnostic feedback categories simultaneously. However, if student writing has lots of errors, the range of feedback may be daunting to the student; the capability of phasing in the different categories, on the same piece of writing, sends a powerful signal to the student regarding the revision and editing process.

It should be noted that while diagnostic feedback may be provided to student writing on any topic (not only the set topics within Criterion), the feedback for organization and development is more highly refined on the Criterion topics than it can be on an open topic. This is due to how the system was developed, with hundreds of essays read by English instructors, and benchmarked for, among other things, their organization. Thus, if students needed particular emphasis on organization and development, an instructor could assign a Criterion topic, and activate only the diagnostic feedback category of “organization and development.”

A signature feature of Criterion is the Writer’s Handbook. Its topics include all those of interest to students seeking to improve their writing, such as organization and development, style, grammar, mechanics, punctuation, etc. Furthermore, when students receive diagnostic feedback in Criterion related to any of these aspects of their writing, a link is provided to that section of the Writer’s Handbook so the student can consult it if they so choose. The Writer’s

Handbook is available from every feedback screen and the student’s portfolio.

How to Use Criterion

Criterion is a powerful technology, and instructors will find many uses for it. These uses include:

· Helping students improve their writing. This is the primary purpose of writing instruction, and the use to which most instructors and students will put Criterion. As has been described above, the capabilities of Criterion mirror the steps of the writing process, an approach that most writing instructors use with their students.

· Placing students in writing classes. When many students are enrolled in a school or a university, instructors may find that their work can be more effective if they cluster students of the same general level of writing together. The holistic scoring capability is most suited to this purpose.

· Supporting writing instruction in the resource or tutorial center. Some schools and universities have writing labs or centers where students may go to work on their writing, with intermittent help from an instructor. For motivated students, Criterion can strengthen those approaches, by enabling students to write and submit essays for feedback, with assistance, when needed, from an instructor.

· Helping students prepare for high-stakes writing assessments. There are situations when students must produce high-quality writing, on demand. This may be a testing environment, by which students qualify to enter, or exit, a program (for example a high school proficiency test.) Alternatively, students, when they are candidates for a job, may be asked to submit a writing sample. Criterion is ideally suited to help students prepare for such situations; they can write an essay on one of the Criterion topics, in an amount of time the instructor

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 44

establishes, and submit it for a score. This lets them see how they are doing in the development of their writing skills, and where they need to improve.

Incorporating Criterion in Your Writing Instruction

Criterion can be applied in a variety of instructional settings and for a variety of uses to support the writing and revision process. Keep these two tips in mind as you plan to use Criterion:

· Schedule the computer lab and assignments so your students have enough time to write and submit an essay and view the diagnostic feedback.

· Remember that students can handwrite their essays and then type them into Criterion, or they can write offline in a word processor and use cut-paste to place their essays in

Criterion’s text entry box.

Evaluating a Benchmark Writing Sample For Your Classroom

Before Students Write

When: Beginning of the school year or semester or with a new class.

Purpose: To collect the first writing sample on an essay topic that is grade-level appropriate and accessible to writers.

Select a Criterion topic at the appropriate grade level.

Review rubric and benchmark essays.

Select feedback categories that are most appropriate to students. Starting with the holistic score and organization and development as diagnostic feedback may be suitable for an initial assignment.

Share rubric with students; discuss features of writing and score range.

While Students Write

Engage in prewriting and brainstorming activities

Observe students writing,

interact and advise as needed.

Ask students to use spell check and then submit essays for scoring.

Ask students to look at immediate feedback for an initial impression.

After Students Write

Create class summary of holistic scores and diagnostic feedback reports. Make instruction plan to address the problems identified for the whole class or a targeted group.

Discuss the process of writing in Criterion and what the feedback means.

Assign a revision based on all or a subset of the feedback categories.

Using Criterion For Mode-Specific Writing Practice: Writing in a Specific Mode of Development

Before Students Write While Students Write After Students Write

When: Introducing a new mode of development, such as persuasive, expository, narrative, etc.

Purpose: To focus on the development and organization of the mode of writing, using

Criterion to capture the drafts and revisions as students work through the development and organization.

Select a Criterion topic that resembles the mode you are about to introduce.

Review rubric and benchmark essays.

Select feedback categories that are most appropriate to students. Starting with the holistic score and grammar, usage, and mechanics feedback may be appropriate as students begin to write in a new mode.

Share rubric with students; discuss features of writing and range of scores.

Engage in prewriting and brainstorming activities

Observe students writing,

interact and advise as needed.

Ask students to use spell check and then submit essays for scoring.

Ask students to look at immediate feedback for an initial impression.

Create class summary of holistic scores and diagnostic feedback reports. Make instruction plan to address the problems identified for the whole class or a targeted group.

Discuss the process of writing in Criterion and what the feedback means.

Assign a revision based on all or a subset of the feedback categories.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 45

Using Criterion For Mode-Specific Writing Practice: Writing Across The Curriculum

Before Students Write While Students Write

When: Language arts and subject-area instructors team teach and create writing assignments.

Purpose: To give instructors access to student portfolios and commenting capabilities.

Diagnostic feedback in

grammar, usage,

mechanics, style, and

organization and

development is available.

No holistic score is available because there is

no e-rater model.

Collaborate on writing assignments, selecting any reading passage or stimulus material to accompany the topic.

Select feedback categories that are most appropriate to students. Starting with the grammar, usage, and mechanics may be suitable as instructors and writers begin to interact.

Introduce the assignment, explaining its dual purpose — content and writing evaluation.

Explain how they will evaluate the final essay, providing clear expectations and/or a scoring rubric to help students understand the process.

Assign a first draft and students to use diagnostic feedback for revision and submit a second draft for their evaluation.

Instructor Topics: Using Your Own Topics in Criterion

Before Students Write

When: Criterion topics are not suitable for a teaching purpose.

Purpose: Instructors develop a topic in any mode or subject area.

Create a topic in any mode or subject area that students will draft, revise, and finalize.

Select feedback categories that are most appropriate to students. Starting with grammar, usage, and mechanics feedback may be appropriate as students begin to draft their essays.

Share your scoring expectations or rubric with students; discuss features of writing you will be evaluating and range of scores or grades.

Remind students that only

diagnostic feedback in

grammar, usage, mechanics, style, and organization and development is available.

No holistic score is available because there is

no e-rater model.

While Students Write

Observe students writing,

interact and advise as needed.

Ask students to use spell check and then submit essays for scoring.

Ask students to look at immediate feedback for an initial impression.

After Students Write

Evaluate the submitted work and write comments to the student, using either comment feature in Criterion.

Encourage students to establish a dialogue with instructors online or in conference to discuss aspects of their writing that they don’t understand.

Review the outcomes of the assignment—student accessibility, quality of responses, use of feedback— and revise accordingly.

After Students Write

Read and comment on at least one draft of the essay.

Offer one or two positive aspects of the essay and one or two suggestions for improvement.

Encourage students to collaborate with one another as peer reviewers to discuss their drafts

Review the outcomes of the assignment—student accessibility, quality of responses, use of feedback— and revise accordingly.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 46

Collecting Essays to Build An Automated Scoring Model: Using Criterion For Collecting Essays on Your Own Topics

Before Students Write After Students Write

Preparing the Essays for Model Building

When: A district, school, or department elects to develop a topic and scoring model that reflects its standards, population, and scoring.

Purpose: To create an automated scoring model to be used by a district, school, or department to evaluate student writing for benchmarking or outcomes or practice.

Notes:

1. When programming the topics in the Instructor

Topic utility, do not click on any diagnostic feedback components.

Set a time that reflects the actual timing requirements of the assignments.

2. Create individual topics for each grade level. Do not ask students at different grade levels to write on the same topic.

3. Consult with content specialists at ETS

Technologies for guidelines for developing your own models. There

is a fee for model

building services.

Create a topic that reflects the requirements of instructors and students. Program this topic in criterion as an

Instructor Topic.

Develop or adopt a rubric that reflects the scoring features and standards of the users.

Schedule a day or week when all students write on the assigned topic.

Note: You will need at least

500 essays that will result in scores across the scoring scale in order to develop a scoring model.

Print a random sampling of essays from the assignment.

Schedule a session with instructors familiar with holistic scoring to select benchmark and training essays from the collected essays. Use the established scoring rubric to reach consensus about the scores for each benchmark and training essay.

Schedule a scoring session with qualified readers.

Score each essay twice; resolve discrepancies.

Enter scores into the Class

Report.

Note: ETS Technologies will provide scoring services. Call for more information and a fee schedule.

Use instructions for the

Criterion Support Services, export essays and scores electronically to ETS

Technologies.

Evaluate scores and data at

ETS Technologies.

Process scoring model

Evaluate results and certify model for use.

Using Criterion For Placement into Writing Classes at Community Colleges, Colleges, and Universities

Before Placement Testing After Placement Testing

When: English Department or

Admissions Office requires a writing sample to determine placement into remedial or college level courses.

Purpose: To alleviate the need for trained faculty readers to score placement essays.

Decide which topic or topics/rubric should be provided for placement testing session.

Agree on an a cut score decision.

Determine who will review a subset of the essays and scores for quality assurance purposes.

Create a summary report of surface-level errors and holistic scores to help inform instructors of the problems students reveal as they test.

Provide students who enter non-college level courses with access to Criterion throughout the semester in the writing lab, tutorial center, or for use with an instructor.

Use Criterion to collect exit-writing samples to evaluate improvement.

Create a summary report and compare with placement summary reports to track improvement.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 47

Using Criterion in the Writing Lab or Tutoring Center

Initial Use

When: Students are referred by instructors for additional writing support, or when students themselves seek additional feedback on their writing.

Purpose: To provide tutors and writing mentors with access to topics, feedback, portfolios, and a way to communicate with instructors about student progress.

Discuss what student perceives as his or her problems with writing.

Ask students to write on an appropriate topic in order to get a writing sample.

Discuss holistic score and diagnostic feedback, explaining any unfamiliar terms.

Offer one or two positive aspects of the essay and one or two suggestions for improvement.

Ongoing Use

Ask students to enter drafts of other writing assignment into

Criterion using the Instructor

Topic option.

Review all drafts and provide feedback using the Comment functions.

Create a summary feedback report to help the student focus on surface-level errors.

Elaborate on the rhetorical elements of an essay, explaining the development of paragraphs and the organization of essays.

Encourage student to submit drafts and write comments to tutors.

Using Criterion to Help Students Prepare for Timed Writing Assessments or Tests

Before Writing Evaluating the Results

When: Prior to writing an essay during a high-stakes testing situation

Purpose: To prepare students for writing on demand in a timed session on topics they have not practiced on.

Note: If the actual assessment is not computer delivered, you may ask students to handwrite their essays in a timed session.

Then they can type them into

Criterion and submit them for evaluation. This strategy combines the reality of the testing situation and immediate feedback.

Select a topic that is similar to the one used in the assessment.

Program the assignment for the amount of time that students have in the actual assessment.

Remind students their essays will receive a holistic score.

Discuss the assessment’s rubric and score range, explaining the features incorporated in the holistic scale and how holistic scores are awarded by readers.

Provide several brainstorming strategies to help students respond to the assessment topic and make good use of their time.

Ask students to look at the immediate holistic score and diagnostic feedback for an initial impression of their performance.

Create class summary of holistic scores and diagnostic feedback reports. Make instruction plan to address the problems identified for the whole class or a targeted group.

Encourage students to articulate the problems and anxieties they faced in writing on an unfamiliar topic in a timed session.

Collaboration

With Instructor

Inform instructor of student’s progress.

Provide access to Criterion so instructor can review the student’s work.

Communicate with the instructor to establish a link between classroom learning and writing lab support.

Encourage student to communicate with tutors and instructor.

Revising

Encourage students to practice writing in a timed session as often as they can to prepare to the assessment.

Help students evaluate their weaknesses and offer suggestions for addressing them during testing.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 48

Using the Student Portfolio for Viewing Writing Performance During Student Progress Conferences

Before the Conference During the Conference Follow-up

When: During studentteacher conference, parentteacher conferences, IEP evaluations, writing across the curriculum discussions.

Purpose: Evaluate writing progress by looking at samples, comments, and reports.

Determine, with students as appropriate, which writing samples will be used for the conference. Help them select their best work.

Create a summary report of the student’s errors to show along with the actual responses.

List several positive aspects of the student’s work as well as areas for improvement.

Review student writing with student, colleagues, or parents, explaining the assignments and the writing process.

Elicit others’ impressions of student writing and progress shown. Ask for positive feedback that could be included in the comments fields.

Show the running commentary that you and your colleagues have provided.

Write progress report, IEP, or other document as appropriate to the situation.

Share some of the comments with the student.

How Students Can Use Criterion on Their Own for Writing Practice

In the Tech Lab

After instructors post assignments, students can work independently to write and revise their drafts.

Students can:

· get immediate score and/or diagnostic feedback.

· have ready access to the

Writer’s Handbook.

· write comments to instructors if any feedback is unclear.

Online at Home

Students can access Criterion form their home computers using their IDs and passwords.

Students can:

· write a draft and save it to submit at another time.

· get immediate score and/or diagnostic feedback.

· have ready access to the

Writer’s Handbook.

· write comments to instructors if any feedback is unclear.

Offline

Students can write a response in any word processing program and then copy and paste it into Criterion.

Students can:

· submit the essay for immediate feedback when they are online.

· handwrite essays and then type them into Criterion.

Student Portfolio

Students can evaluate their own progress by reading the essays in their portfolio.

Students can:

· arrange their work by date, assignment, and scores.

· review holistic or diagnostic feedback in each report.

· read comments from instructors and/or write to them.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 49

Essay Topics Available In Criterion

Criterion’s library of topics contains over 110 prompts, starting at grade four through graduate level writing. Instructors can select topics that are geared to their students’ writing levels and interests. The chart below lists the grade level, abbreviated title, and mode of development for each topic. For a complete listing of the full topic and associated scoring rubric, click on www.etstechnologies.com/criterion/topics/topics.htm

8 th

Grade

Elementary School

4 th

Grade

Middle School

6 th

Grade

7 th

Grade

Topic

Castle

Favorite Object

Invisible Friend

Topic

Best Pet

Start A Hobby

Jobs For The Future

Memorable Day

Special Foods

Change At School

Enjoyable Book

Statues and Monuments

Topic

Weirdest Place

School Issue

School Vacations

Obey Laws

Book Review

Performance/Competition

Make A Movie

Make A Gift

Topic

Lengthening The School Year

Designing A TV Show

Space Visitor

Memorable Trip

Freedom Of Expression

Compare Songs

Favorite Food

School Sports

Impact Of Television

Special Event

Author’s Message

Writing Style

Descriptive

Persuasive

Writing Style

Persuasive

Process Analysis (How To)

Writing For Assessment

Descriptive

Comparison And Contrast

Cause And Effect

Response To Literature

Writing For Assessment

Writing Style

Descriptive

Persuasive

Writing For Assessment

Response To Literature

Cause And Effect

Comparison And Contrast

Process Analysis (How To)

Writing Style

Persuasive

Informative

Narrative

Descriptive

Persuasive

Comparison And Contrast

Process Analysis (How To)

Writing For Assessment

Cause And Effect

Response To Literature

Timing

25 minutes

Timing

30 minutes

Timing

30 minutes

Timing

25 minutes

30 minutes

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 50

High School

9 th

Grade

10 th

Grade

11 th

Grade

12 th

Grade

Topic

TV Programming Choice

Reaction To A Book

Field Trip

Interesting Career

Role Model

Landscape And Climate

Encourage Recycling

Technology

Topic

Local Issue

Longer School Year

Pet Care Letter

Compare Two Products

Global Issues

Environmental Issue

Effective World Leader

Write A Review

Topic

Enforcing Dress Code

Choose A Character

Favorite Character

Compare Two Schools

Violent Weather

Letter Of Complaint

Tax Dollars

Historic Event

Topic

Voter Registration Drive

Writing Mentor

Special Object

Plans After Graduation

Dress Code

Video Games

Community Service

Obstacles In Life

Freedom Of Speech

Memorable Character

Consumer Report

New Invention

Help Your Community

Writing Style

Persuasive

Response To Literature

Writing For Assessment

Workplace Writing

Writing For Assessment

Comparison And Contrast

Problem And Solution

Cause And Effect

Writing Style

Problem/Solution

Writing For Assessment

Workplace Writing

Comparison And Contrast

Persuasive

Cause And Effect

Writing For Assessment

Response To Literature

Writing Style

Persuasive

Writing For Assessment

Response To Literature

Comparison And Contrast

Problem And Solution

Workplace Writing

Writing For Assessment

Cause And Effect

Writing Style

Persuasive

Informative

Narrative

Expository

Persuasive

Problem And Solution

Writing For Assessment

Response To Literature

Comparison And Contrast

Cause And Effect

Workplace Writing

Timing

30 minutes

Timing

25 minutes

30 minutes

Timing

30 minutes

Timing

30 minutes

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 51

College Level

2 nd

Year

Topic

Being An Adult

Want/Should

Games

Acceptable Behavior

Instincts

Fads And Trends

Images Of Beauty

Breaking Traditions

Rebellion

Topic

Community Service

Grades

Upper Division/Graduate School

Topic

Workplace Safety

City L

House Of Beef

Speedee Airlines

Zapped/Zonked

Big Boards

Drivers – Young And Old

Restaurants

Styx River Bridge

Ready-To-Ware Benefit

Terms

Job Satisfaction/Success

Monetary Systems

Location

Academic Background

Short-Term Goals

Essential Traits

Popular Entertainment

Environmental Standards

Children/Internet

Government Support

Of The Arts

Writing Style

Persuasive

Writing Style

Persuasive

Writing Style

Analysis Of Argument

Analysis Of Issue

Timing

45 minutes

Timing

40 minutes

Timing

30 minutes

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 52

English Language Training

Topic

Reasons For

Attending College

Experience Or Books

Money On Technology

Young People

Teaching Older People

Money And Success

Playing And Winning

Group Member Or Leader

Change In Your School

New Product

Resource Disappearing

Changes In The 21 st

Century

Change Your Hometown

Writing Style

Persuasive

Timing

30 minutes

0 minutes

30 minutes

These topics may be assigned to students; their essays, once submitted, will yield either a holistic score or diagnostic feedback, depending on how the instructor has established the parameters within Criterion. If instructors want their students to write on other topics (related to a unit in social studies or science, for example) those essays will be able to receive diagnostic feedback only.

If you want your students to receive an accurate holistic score, they should write only on the topics that correspond to their grade level. Remember that the holistic score is based on hundreds of essays written by other students at that grade level. For example, if your tenthgrade class were to write on twelfth-grade topic, your students’ writing would be compared to that of twelfth graders, so their scores may not reflect an accurate evaluation of their writing ability.

Often writing instructors are interested in knowing where Criterion’s topics originated. While the topics resemble those in writing textbooks, the topics and the essays used to build holistic scoring models come from a variety of sources.

ETS Testing Programs

Established high-stakes testing programs at ETS released topics and scored essays for use in

Criterion. These programs include:

· Scored essays from established, high-stakes testing programs

· GMAT – graduate level writing

National Samples of Student Writing

Many topics were written by instructors and assessment specialists and administered to a national sample of writers. Trained readers scored these essays, and automated scoring models were built using these data.

· Mode-specific topics: description, narration, persuasion, cause-effect, process analysis, problem-solution, writing for assessment, writing for literature, workplace writing

· Mode-specific rubrics: based on widely accepted holistic writing standards

· Grades 4, 6, 7, 8, 9, 10, 11, 12

· 6 point and 4 point holistic scales

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 53

Topics Developed by Instructors

Just as students need a purpose to write, instructors need a purpose for assigning an essay topic. When instructors want to provide their students with on-going practice in the basic modes of writing, then the topics available in Criterion can be used effectively. There are instances, however, when an instructor may decide that none of the topics is appropriate for an assignment. When this occurs, the Instructor Topic option can be used. The Instructor

Topic feature allows an instructor to write a topic to reflect a content area or subject. Instructors can tailor their writing activities to an assignment or project and capture students’ responses for evaluation.

Remember, students writing on Instructor Topics will not receive holistic score feedback.

Deciding When To Use Criterion Topics Or Your Own Topics

Criterion Topics

Do I want to assign a Criterion topic or write one of my own?

Do I want to see how well my students are writing in comparison to a particular grade level?

Do I want to assign a particular mode of writing or a response to a reading selection or class discussion?

Do I want my students to receive holistic score feedback and diagnostic feedback, or just diagnostic feedback?

· More than 110 topics that reflect the domain of writing.

· Grades 4-12

· College Level: 1 st

Year/Placement, 2 nd

· Upper Division/Graduate School

· English Language Training

· Persuasive

· Expository

· Narrative

· Process Analysis (How To)

· Comparison-Contrast

· Cause And Effect

· Problem And Solution

· Workplace Writing

· Writing For Assessment

Year

· Analysis Of Argument

· Analysis Of Issue

· Holistic score on either a 6-point or a 4-point scale.

· Diagnostic feedback in grammar, usage, mechanics, style, and organization and development.

Instructor Topics

· Can be created for any content area and for any writing purpose.

· Can be created for any grade level, content area, or writing purpose.

· Can be designed to suit any content area or subject matter. A reading passage (e.g., paragraph or poem) can be attached to the topic.

How do we incorporate

Criterion in an established writing curriculum?

· To support writing in a specific mode.

· To support writing on a subject or issue.

· To provide writing practice in a timed setting.

· Diagnostic feedback in grammar, usage, mechanics, style, and organization and development.

· Feedback on organization and development is available for instructor topics, but may not be as good as for Criterion topics.

· Holistic score is not available for instructor topics.

· To support writing in the content areas.

· To support writing across curriculum.

· To support team teaching.

Feedback Messages For Different Categories of Diagnostic Feedback

The diagnostic feedback students receive on their writing identifies the difficulty as cited by

Criterion, and gives students suggestions as to how they might correct the problem. In addition, as noted above, students can access the Writer’s Handbook via a direct link.

For example, if Criterion has located a statement that it identifies as an introduction, the feedback reads “Is this part of your Introduction? An introduction attracts the reader’s interest and provides background information. It also contains your thesis statement. Look in the

Writer’s Handbook for ways to improve your introduction.” If Criterion cannot locate a statement

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 54

that it identifies as an introduction, the feedback reads: “Criterion has not identified an

introduction in your essay. Good writers use an introduction to tell what the essay is about and to set their essays in motion. Look in the Writer’s Handbook for some advice about writing an introduction.”

Other feedback comments are similar; they are tailored to the type of feedback but are, of necessity, general in nature. They frequently direct students to the Writer’s Handbook for additional guidance. Students may also seek additional help from their writing instructor.

Tips for Instructors

The extent to which you will be able to use Criterion in your teaching, and the way in which you do so, will be determined largely by the access both you and your students have to both computers and the Internet. Criterion is web-based, and while students may do much of their writing off-line, and even on a piece of paper, they must, eventually, be able to type their work into Criterion and submit it, via the Internet, for scoring and feedback.

The chart below outlines three different scenarios for computer and Internet access at your school and in your students’ homes. Factors to consider are:

· the number of computers in your classroom, and whether they have an Internet connection.

· your access to a computer lab with Internet connections.

· the extent to which your students can use a computer at home, and whether they have an

Internet connection there.

· whether you have a computer, with an Internet connection, at home.

Evaluate your teaching environment and situation to determine how often you can ask your students to write in Criterion.

Computer And Internet Access

Where does your school have

Internet access?

Scenario 1

· Computers with Internet access are located in the school library or other remote location only.

Scenario 2

· Class has access to a computer lab with Internet access for less than one hour per week.

Scenario 3

How much computer access is there at your school?

How much computer access do your students have at home?

Do you have compute and

Internet access in your classroom, office, or at home?

· Computers with Internet access are available in the library or other remote location.

· Students have access to computers with Internet access in the classroom, but there are not many machines, so students’ time must be scheduled.

· No students have computers or Internet access at home.

· You have no computer or

Internet access at home.

· The class has access to a computer lab with Internet access for less than one hour per week.

· There are a number of computers with Internet access available for student use in the classroom; students must schedule their use, but they do not have to wait for long.

· Some students have computers and Internet access at home.

· You have a computer at home, and Internet access, but it is very slow.

· Class has access to a computer lab with Internet access on a frequent basis.

· Classroom computers have

Internet access.

· Class has access to a computer lab with Internet access on a frequent basis.

· There are many computers with Internet access available for student use in the classroom; students have access when they need it.

· Virtually all students have computers and Internet access at home.

· You have a computer and high-speed Internet access at home.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 55

Now, to guide you in your planning, you could complete the following table for your own instructional situation.

Web Access

Computer Access In School

Student Computer Access At Home

Instructor Computer Access At Home

As instructors have used Criterion with their students, they have offered the following suggestions:

Students can handwrite their essays and then type them into Criterion. Or they can write offline in a word processing program and then cut and paste their essays into Criterion’s text entry box.

Be sure to schedule sufficient time in the computer lab so students have time to write and submit an essay, and view the feedback. While the feedback is immediate, the process of revision takes time.

The key to the effective use of Criterion is in making the assignment, selecting the appropriate types of feedback, and engaging students in revising their work.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 56

C r r i i t t e r r i i o n R e f f e r r e n c e

G u i i d e

Criterion Online Writing Evaluation is not a “black box” or “magic” just because it provides instantaneous feedback. Criterion has distinct capabilities that invoke proven technology. Criterion is an application that teachers can use to support the writing process; it’s a set of tools that students can use to work independently and get immediate feedback. This reference guide is intended to provide additional insight into the Criterion service and its capabilities, and to clarify the language used to describe the technology.

Frequently Asked Questions

Using Criterion In Teaching

Q. How can Criterion help students?

A. Students get a response to their writing while it is fresh in their minds. They find out immediately how their work compares to a standard and what they should do to improve it. Criterion also provides an environment for writing and revision that capable and motivated students can use independently. This environment, coupled with the opportunity for instant feedback, provides the directed writing practice so beneficial for students.

Q. How does Criterion handle an unusual writing style?

A. Criterion looks for specific features of syntax, organization, and vocabulary. If the essay under consideration is not sufficiently similar to those in its database of already scored essays,

Criterion posts a warning saying that it is unable to give an accurate score. Warnings usually result from essays that are too brief or those in which the vocabulary is unusual or the content is off topic.

Q. Will the use of Criterion stifle creative writing among students?

A. Not necessarily. Criterion is designed to be used for evaluating writing done under testing conditions — situations in which even the most creative writers concentrate on “playing it safe” with straightforward and competent writing.

Q. Will Criterion catch cheating or plagiarism?

A. No. Criterion simply evaluates the essay. It is up to the institution to assure that students are working independently and submitting their own work.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 57

Q. What information does Criterion report to teachers?

A. Teachers have easy and secure access to each student’s portfolio of essays, diagnostic reports, and scores as well as summary information on the performance of entire classes.

Q. What information does Criterion report to students?

A. Typically, students get an annotated diagnostic analysis of errors and features of grammar, usage, mechanics, style and organization and development, as well as a holistic evaluation each time they submit an essay. However, schools can block students from seeing their scores and/or any portion of diagnostic feedback — and may choose to do so if they use

Criterion for high-stakes testing. Schools also have the option of allowing students access to their own portfolio of essays, diagnostic reports, and scores.

Q. Does Criterion discriminate against students who may be bright but who may not have mastered standard English — for example minorities and ESL students?

A. No. Criterion is incapable of discriminating on the basis of race, sex, national origin, or student’s previous history because these factors do not exist in its analysis. The program simply measures features in a piece of writing and compares them to features in previously scored essays used to define the rubric.

So if the collection of sample essays includes essays that use non-standard English and also earn high scores, then Criterion will assign a high score to other essays with the same features.

Using Criterion For Testing And Placement

Q. How often does the computer’s score agree with the score of a faculty reader?

A. Almost all the time. ETS Technologies researchers found exact agreement, or a difference of only one point, in as many as 98 percent of the comparisons between the computer’s scores and those of a trained essay-reader using the same scoring guides and scoring system.

Q. How do schools use Criterion for testing?

A. Some schools use Criterion scores for exit testing — combining a Criterion score with the score from a faculty reader in the same way they combined scores from two different faculty readers. If the two scores differ by more than one point, a different faculty reader also evaluates the essay. Some schools use Criterion for benchmark testing, assigning Criterion-scored essays at specified points during the school year.

Q. How do schools use Criterion for placement?

A. Some colleges assign students to composition classes on the basis of their scores on a Criterion-scored essay — or the combination of the Criterion score and other indicators.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 58

Q. How do students feel about being scored by a machine?

A. Most of today’s students have had experience with instant feedback in computer programs and are becoming more comfortable with the idea of computerized scoring.

Q. Can Criterion score essays on other topics?

A. No. Criterion can only evaluate essays on those topics for which it has been “trained.” ETS Technologies maintains an annual schedule of adding new topics at each level. Schools and districts can also work with ETS Technologies to create new customized topics. The training process for a topic requires a minimum of 465 essays that have been scored by faculty readers.

Understanding The Technology

Q. How does Criterion come up with its scores?

A. Criterion is based on a technology called e-rater® that was developed at Educational Testing Service. E-rater compares the new essay to samples of essays previously scored by faculty readers — looking for similarities in sentence structure, organization, and vocabulary. Essays earning high scores are those with characteristics most similar to the high-scoring essays in the sample group; essays earning low scores share characteristics with low-scoring essays in the sample group.

Naturally, the sample essays must be scored very carefully and the collection must include sufficient essays at each score point.

Q. What’s the technology used in e-rater?

A. E-rater is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using

NLP for the past 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information. E-rater uses NLP to identify the features of the faculty-scored essays in its sample collection and store them — with their associated weights — in a database. When e-rater evaluates a new essay, it compares its features to those in the database in order to assign a score.

Because e-rater is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which e-rater’s database is created.

Q. Can students trick Criterion?

A. Yes. Since e-rater cannot really understand English, it will be fooled by an illogical, but well-written, argument. Teachers can stop students from deliberately trying to fool Criterion by announcing that a random sample of essays will be read by independent readers. Additionally, Criterion will display an

“Advisory” along with the e-rater score when an essay displays certain characteristics that warrant attention compared to other essays scored against the same topic.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 59

Q. Must students be connected to the Internet to use Criterion?

A. Not necessarily. Although the scoring of essays requires an

Internet connection, students can compose their essays offline using any word processing application and then cut and paste them into the service after they log on to the Criterion site.

Q. Is the holistic score related to the diagnostic feedback?

A. Holistic score and diagnostic feedback are two separate and independent systems; one does not have an effect or impact on the other. A holistic score of 6 can reveal lots of errors in grammar, usage, mechanics, and style; however, it can also contain all of the elements in organization and development— introduction, thesis, main and supporting ideas, and a conclusion. A holistic score of 2 may reveal few errors and few elements in organization and development.

To receive a holistic score, a topic must have a corresponding erater scoring model.

To receive the most accurate kind of organization and development feedback, a student should write on a Criterion topic instead of an “instructor topic” that does not have a corresponding e-rater model.

Diagnostic feedback in grammar, usage, mechanics, and style is designed to identify certain errors in these categories. It is not evaluative.

Q. What kind of technology is used in Criterion?

A. Criterion incorporates a variety of automated scoring tools to provide immediate feedback. Automated scoring technology is developed using natural language processing, a sub-field of artificial intelligence. These sophisticated scoring techniques produce instant results using statistics, computational linguistics, algorithms. However, none of these automated scoring tools can read essays like humans do. These tools process information; they do not read and analyze it.

Criterion incorporates a variety of automated scoring tools to provide immediate feedback. Automated scoring technology is developed using natural language processing, a sub-field of artificial intelligence. These sophisticated scoring techniques produce instant results using statistics, computational linguistics, algorithms. However, none of these automated scoring tools can read essays like humans do. These tools process information; they do not read and analyze it.

Automated scoring tools are predictable and literal. An e-rater model will, for example, award a score of 3 to all essays that

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 60

resemble “3-ness” in its training set. It will score consistently and endlessly in this way.

Q. How are e-rater scoring models created?

A. E-rater learns to score by analyzing hundreds of essays that have been scored by trained faculty readers who use a scoring rubric that establishes the rules for awarding the individual score points. An e-rater scoring model is accurate only if the essays scored by readers show clear distinctions among the score points.

E-rater scoring models are built using essays written by a specific population of students on topics that are appropriate for that population. These essays are scored by readers (usually teachers) who are familiar with the student population. The readers guide their scoring using a rubric, which is a set of established standards delineated at a range of score points.

When using Criterion, it is important to match the topic with the appropriate level of student writer. For example, while a particular 12 th

grade topic may be appealing from a content point of view, it is not appropriate for use with an 8 th

grade class. The

8 th

graders’ writing would be compared to the 12 th

graders’ writing, so the e-rater score would not be used to evaluate comparable writing.

Glossary Of Terms

Advisory: A message that provides additional information about the essay. Criterion issues an Advisory when the essay is too brief to evaluate, may not be relevant to the topic, or exhibits a unique style not typically found in other essays on the topic.

E-rater®: The automated essay scoring technology that supports

Criterion. E-rater works by comparing new essays to samples of essays previously scored by faculty readers.

Holistic Scoring: A method of evaluating essays that considers the total impression rather than individual aspects. Criterion uses the e-rater technology to provide holistic scores on essays.

Natural Language Processing: The application of computational methods to analyze characteristics of electronic files of text or speech. E-rater, the technology underlying Criterion, uses natural language processing to identify the features of the essay that correspond to scoring guide criteria — syntactic variety, organization of ideas, and vocabulary usage.

Rubric: An established rule. Criterion uses scoring rubrics (also called scoring guide).

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 61

Scoring Guide: A description of the features of the writing at each level on the scoring scale. Criterion makes the scoring guides, as well as a sample essay at each score point, available to students and teachers.

E-rater

: What It Is, How It Works

General Approach

E-rater

is the automated scoring system that has been developed to assist in the evaluation of test takers' responses to open-ended essay questions. The current version of the system is capable of assigning scores (on a 0 to 6 scale) to two kinds of essays -- one that requires examinees to discuss an issue and another that requires them to analyze an argument.

E-rater is empirically based. It is "trained" by being fed samples of essays that have been previously scored by human readers. The samples are randomly selected for each essay prompt so as to represent a complete range of possible scores. Essentially, e-rater uses natural language processing techniques to duplicate the performance of human readers.

In its attempt to model human readers, e-rater uses several subroutines to extract a wide variety of features of the essays that it evaluates. These features are then used in combination to predict the scores that were assigned previously (by human readers) to the essays in e-rater’s training sample. Ordinary (stepwise) least squares linear regression analysis is used to select the set of features that best predicts these scores. A total of 50-60 features are "extractable," but in practice only a subset of the most predictive features, usually about 8-12, are retained and used for each essay prompt. The least squares regression weights for these predictors are applied to each new essay to estimate a score. The score is rounded to the nearest integer from 0 to 6 in order to place it on the scale used by human readers.

E-rater is prompt specific. That is, it is trained for each different essay prompt. Thus, the final set of features used to compute scores for any given prompt is likely to differ somewhat across prompts. Several features are much more likely than others to recur as predictors across different prompts. Some of the same features tend to be predictive of scores for both the Issue and the

Argument prompt types.

Although e-rater is empirically based, it is not "blindly empirical."

E-rater could, of course, take a "brute-force" approach, extracting numerous features indiscriminately. However, not all of the extractable features correspond equally well to the features that

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 62

human readers are instructed to consider when they score essays.

Therefore, e-rater features are required not only to be predictive of human readers' scores, but also to have some logical correspondence to the characteristics that human readers are trained to consider. These characteristics are specified in the scoring rubrics that guide human readers when they score essays.

Specific Techniques

Although certain "surface features" (such as the number of words in an essay) are easily extractable, these purely surface characteristics are not used by e-rater. Instead, the focus is on three general classes of essay features -- structure (or syntax), discourse (organization), and content (or prompt-specific vocabulary).

Structure

Structural characteristics include such notions as "syntactic variety," that is, the use of various structures in the arrangement of phrases, clauses, and sentences. In this category, several specific features have proven to be predictive of human readers' scores.

They are:

· the number of subjunctive modal auxiliary verbs (would, could,

should, might, may)

· the prevalence of infinitive clauses ("To support his expensive

hobby, Phil Atelist would soon need to get a second job at the post office.")

· the proportion of complement clauses ("He felt that it was well

worth the effort.")

· the incidence of subordinate clauses ("Although the argument

points out that trends are part of our society, it does not explain why.").

Organization

How well test takers are able to organize their ideas is a major focus for human readers and for e-rater. E-rater evaluates organization by extracting a variety of rhetorical features, i.e., characteristics that are associated with the orderly presentation of ideas. Of special interest here are "cue" words or terms that signal where an argument begins and how it is being developed. (By argument, we mean, generally, any rational presentation that uses reasons or examples to persuade the reader.) For example, terms such as "in summary" and "in conclusion" are used for, what else, summarizing. Words and phrases like the following are used to express opinions or beliefs:

· certainly, clearly, obviously, plainly, possibly, perhaps, potentially,

probably, fortunately, generally, maybe, presumably, unless, albeit, luckily, normally, apparently, herein, likely, surely, ideally,

undoubtedly, naturally

· for certain, for sure, of course, to some extent, above all, if only,

in order to, in order for, so that, so as to.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 63

The significance of many of the words listed above is that they signal the start of an "argument" (in the sense that we have described), and the number of arguments developed is a feature that is emphasized by e-rater. Arguments sometimes begin simply with I or we, or with "parallel" words or phrases that signify the start of another line of argument. The latter include the following:

firstly, essentially, additionally, first, second, third, another, secondly, thirdly, fourth, next, finally, final, last, lastly, moreover, too, also, likewise, similarly, initially, further, furthermore, first of all, in the first place, for one thing, for a start, second of all, many

times, more importantly, most importantly.

The onset of an argument is also signaled by rhetorical words and phrases such as suppose, supposedly, and what if. The subsequent details are often preceded by words and phrases like:

if, specifically, particularly, when, namely, for example, for instance,

e.g., in this case, in that case, such that, as well as, in that, such

as, about how, in addition, in addition to.

Words or phrases that are used to contrast points of view or to indicate alternative opinions are often found in arguments. These include words and phrases such as these:

otherwise, conversely, however, nonetheless, though, yet, meanwhile, while, but, instead, although, still, notwithstanding,

anyway, unlike, on the contrary, in contrast, by comparison, in any

case, at any rate, in spite of, rather than, on the other hand, even

then, even if, even though, apart from, instead of.

Some words or phrases signal the presentation of evidence to support an argument. These include since, because, actually, in

fact, after all, as a matter of fact, and because of. Still others signify the stage at which inferences are being made:

accordingly, consequently, hence, thus, ultimately, so, thereby, then, therefore, following, after, afterward, afterwards, as a consequence, as a result, if so, if not, as such, according to, in turn,

right after.

Yet other words and phrases signal that an argument is being

reformulated (alternatively, alternately, that is, in other words,

briefly). In short, E-rater looks for linguistic clues (such as logical connections between sentences and clauses) that signify logical or well-ordered thinking.

Content

By means of "topical analyses," e-rater also considers the content or vocabulary of the essays it evaluates. These analyses are predicated on the assumption that well-written essays are more responsive (or relevant) to the topic posed than are poorly written essays. Better essays also tend to use more precise and specialized vocabulary. Therefore, the expectation is that, with

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 64

respect to the words they contain, good essays will bear a greater resemblance to other good essays than to poorer ones. Weak essays, on the other hand, will be similar to other weak essays.

Acting on this assumption, e-rater evaluates each essay and assigns a number based on the similarity of its content to samples of previously-scored essays. This evaluation proceeds as follows.

First, for every essay a determination is made of each word's

"contribution" to the essay. (Variations of a word, such as "walks,"

"walking," or "walked," are considered to be the same word.) A word's relative contribution to an essay is estimated by computing a weight for each word in every essay. These word weights reflect both the frequency of a word in a given essay (relative to the frequency of other words in the essay), and the distribution of the word across other essays. The formula is such that words appearing relatively frequently in an essay will, all other things being equal, receive a higher weight than words appearing less frequently in the same essay. However, words that tend to be used in many essays will get lower weights, all else being equal, than those used in few essays.

In order to assign a numerical value to reflect an essay's content, the word weights for each essay are compared with the weights computed for words in the previously scored training essays. An essay's content feature score is assigned by determining the set of essays that its word weights most closely match. Essays whose word weights correlate most strongly with those computed for highly scored essays will get higher content scores than will essays whose word weights resemble those of weaker essays.

As an example, consider the following (partial) results based on essays analyzing an attempt to convince companies to utilize billboards to increase sales. The two most highly weighted words in essays that received the lowest score (i.e., 1) were "picture" and

"percent." Both of these words received much lower weights in essays that got higher scores. On the other hand, the two most heavily weighted words ("recognition" and "local") in the best essays (i.e., 6s) received much lower weights in the poorer essays.

The weights assigned to these four specific words in essays at each score level were as follows:

Score level

Word 1 2 3 4 5 6

Percent .51 .25 .24 .24 .05 .05

Recognition .00 .04 .06 .19 .41 .69

Local .05 .11 .11 .20 .16 .32

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 65

Besides evaluating an essay's content as a whole, e-rater also considers an essay's content argument-by-argument. The rationale is that additional information can be extracted about an essay's content by examining clusters of word groupings, in this case individual arguments. In a manner similar to that used for essays as a whole, a content score is assigned to each argument on the basis of how well an essay's arguments match the content of essays scored by human readers. The formula used to compute the argument content score assigns higher values to essays with many arguments than to those with few.

E-rater Models

The final step is to use all of the features that e-rater extracts (or rather the values that it assigns to these features) to predict the scores assigned by human readers. A model, i.e., a set of features that is most predictive of human readers' scores, is specified for each essay prompt. As stated earlier, the set is generally somewhat different for each prompt. Some features are used much more frequently than others in the predictive models. The most frequently occurring are:

(1) content by argument

(2) content by essay

(3) the number of subjunctive auxiliary verbs

(4) the ratio of subjunctive auxiliary verbs to total words in the essay

(5) the total number of argument development terms.

The weights assigned to these variables may be either positive or negative. For instance, the number of subjunctive auxiliary verbs typically receives a positive weight, while the ratio of such words is usually weighted negatively. Although the weights assigned to most of the features discussed earlier are usually positive, some features tend to have negative weights. These include certain words used to present evidence (because, since, actually, …) and the use of pronouns (I, we) to begin arguments.

The overview provided here should give the reader some sense of how e-rater functions. More detail is available in a number of reports that can be downloaded from the following Web site: http://www.ets.org./research/erater.html

. If you are unable to download, contact us and we'll send the papers electronically as

Word documents or by regular mail in hard-copy form.

Criterion, the Criterion logo, ETS Technologies and the ETS Technologies logo are trademarks of ETS Technologies, Inc. Page 66

Hallo! Ik ben een AI-chatbot die specifiek is getraind om u te helpen met de Educational Testing Service Criterion Online Writing Evaluation User Manual. Ik heb het document grondig doorgenomen en kan u helpen de exacte informatie te vinden die u nodig heeft of de inhoud in duidelijke en eenvoudige bewoordingen uit te leggen. Of u nu op zoek bent naar begeleiding bij specifieke functies, stappen voor probleemoplossing of algemeen gebruik, stel gerust uw vragen. Hoe meer details u verstrekt over uw zorgen of behoeften, hoe nauwkeuriger en effectiever ik u kan helpen!

Assistent aan het typen

Assistent is niet meer aan het toetsenbord. Probeer het later opnieuw.

Inhoudsopgave

Het lijkt erop dat u een document hebt bereikt dat in meerdere talen is geschreven, waaronder een die niet uw moedertaal is. U kunt de volgende links gebruiken om tijdelijk pagina''s te verbergen zonder de taal van uw voorkeur.

Advertentie

The Criterion User Manual

Welcome to Criterion

Online Writing Evaluation, a web-based service that evaluates a student’s writing skills and provides instant score reporting and diagnostic feedback to both the instructor and student.

Using The Criterion System

Quick Start

Criterion Administrators

Criterion Instructors

Criterion Students

Using Criterion To Support Writing Instruction

Criterion Reference Guide

This section contains a step-by-step overview of all essential functions in using Criterion

. Each module is illustrated with screen captures of the system to familiarize instructors and help them navigate through the application.

This User Manual is designed to support the activities of the system administrator and instructors as they incorporate Criterion as a tool to support writing instruction.

Administrators will utilize Criterion’s management and reporting tools to monitor usage and productivity.

Instructors can work independently as they begin using

Criterion in their classrooms.

If you need assistance with the Criterion Online Writing

Evaluation service, please email your Criterion Account

Manager at ETS Technologies at [email protected]

or call

1.877.909.6442. You may also refer to a guided web tour at www.etstechnologies.com/criterion/tour .

Use the following abbreviated steps to quickly begin set-up of your Criterion

account. Be sure to refer to

Criterion Administrators on page 12 for

comprehensive instruction.

Use the following abbreviated steps to quickly begin set-up of your Criterion

assignments. Be sure to refer to

Criterion Instructors on page 18 for comprehensive

instruction.

Use the following abbreviated steps to quickly begin completing Criterion

assignments. Be sure to refer to

Criterion Students on page 29 for comprehensive

instruction.

Each Criterion

client will be required to assign at least one Criterion administrator to manage the account from the state, district, or school level. ETS Technologies will provide that administrator with an ID and Password that will allow access in order to begin building the account.

In order for teachers to access Criterion, the administrator will need to create a second ID and

Password to share with users.

Getting Started

Setting Up Districts, Schools, And Instructors

Using District, School And Instructor Reports

Email Correspondence With Your Users

To access Criterion

, you will need to obtain an ID and

Password from your Criterion account administrator.

The first time you use Criterion, you will be required to register using that ID and Password. After you have registered, you will not be required to register again.

You will then be able to log in to create your classes and assign topics, add classes, change topics, access your student’s writing portfolios, view student reports, and much more.

Getting Started

Setting Up Your Classes

Setting Up Your Assignments

What Your Students See And Do In Criterion

Using The Class And Student Reports

Creating And Inserting Pop-Up Notes

Writing Comments To Your Students

To access Criterion

, you will need to obtain an Access

ID and Password from your instructor. The first time you use Criterion, you will be required to register using that

Access ID and Password. After you have registered, you will not be required to register again. You will then be able to log in to complete your assignments, view your score analysis and annotated diagnostic feedback, access your writing portfolio, and view your reports.

Getting Started

complete or revise an assignment, restart a saved draft, or view your online portfolio

Completing An Assignment

Accessing The Score Analysis

Accessing The Annotated Feedback Analysis

Using The Student Portfolio

Using The Writer’s Handbook

Access" leads instructors through a series of questions to help them analyze the availability of technology in their classrooms and alternate locations in their schools.

Overview of the Writing Process

How Criterion Supports the Writing Process

Holistic Scoring

Diagnostic Feedback

How to Use Criterion

Frequently Asked Questions

Glossary Of Terms

E-rater

: What It Is, How It Works

Gerelateerd

PureSafety LSMS User Instructions | AI Chat & PDF Access

Wyoming History Day Contest Instructions | AI Chat

MyAccess! 学生手册：AI 聊天与 PDF 下载

WriterKEY Writing Tools User Guide | AI Chat & PDF Download

Microsoft Office Student Advantage Manual | AI Chat