Custom Applications for the e-rater Scoring Engine

To make use of the e-rater^® technology, ETS offers writing assessment and scoring expertise to institutions that incorporate ETS scoring technology in their courses and online services. You may use standard writing and instructor-created prompts or choose to develop your own.

If you use the standard prompts that follow ETS guidelines, the essay responses are submitted to the ETS grade-level scoring models and receive a holistic score.

If you choose to develop your own customized writing prompts, you’ll need to collect essay responses to these prompts, score them and transmit them to ETS so a scoring model can be built and certified. ETS will help you develop the prompts and score the responses so that a custom e-rater engine scoring model can be built by following the requirements outlined below.

Developing topics

• ETS experts can assist with the development of customized essay topics that reflect national, state or district standards.
• ETS can provide expert advice as you develop your own customized topics, conduct scoring sessions and analyze the writing score.
• Or ETS development specialists can do the development work and conduct the pretest scoring sessions, according to the organization's specifications.

Scoring essays

• ETS can conduct essay scoring sessions and collect scoring data for customized model building while incorporating scoring methodology and quality assurance procedures.
• In addition to handling reader recruitment, training and monitoring, ETS offers an online scoring system that instructors and trained readers can use to score text-processed essay responses.
• When scoring is complete, the essay files and accompanying reader scores can be transmitted to the scoring engine for automated model building.

Quality assurance

• Before building the scoring model for a topic, the e-rater technology will be trained by analyzing essays written on that topic.
• Essays need to receive two scores from independent readers who have applied consistent and fair scoring standards.
• The pool of scored essays also needs to represent the entire scoring scale (for example, a scale from 1 to 6), so that the e-rater engine can be trained to recognize essays at each of the score points.

The e-rater engine models requirements

• Scoring standards that are clearly defined in a scoring guide
• An essay topic that reflects the writing task represented in the scoring guide
• A student population identical or similar to the population that will be using the topic
• Two independent reader scores for each essay response (readers must be trained to use the scoring guide)
• At least 350 student responses per topic in an electronic file, formatted according to ETS specifications