skip to main content skip to footer

Designing a Prototype Tablet-Based Learning-Oriented Assessment for Middle School English Learners: An Evidence-Centered Design Approach ELL ECD IUA CAF

Author(s):
Chapelle, Carol A.; Schmidgall, Jonathan; Lopez, Alexis A.; Blood, Ian; Wain, Jennifer; Cho, Yeonsuk; Hutchison, Amy; Lee, Hye-Won; Dursun, Ahmet
Publication Year:
2018
Report Number:
RR-18-46
Source:
ETS Research Report
Document Type:
Report
Page Count:
57
Subject/Key Words:
English Language Learners (ELL), Tablet Computers, Tablet-Based Assessments, Formative Assessment, Evidence-Centered Design (ECD), Test Design, Task Design, Validity Argument, Interpretation/Use argument (IUA), Theory of Action, Conceptual Assessment Framework (CAF), Construct Frameworks, Domain Modeling, Mobile Apps, Feedback, Middle School Students, Elementary Education

Abstract

The tablet project was undertaken in 2013 to explore the creation of an assessment to reveal English language learners' (ELLs') language capabilities in content area classes such as science. Such assessments could ideally be useful in multiple phases of the instructional process by providing information to learners, teachers, and other stakeholders because the technologies allow for efficiencies of task presentation, response gathering, scoring, storing data, and returning results. With the goal of understanding the development costs required to achieve such benefits, the project used an evidence‐centered design (ECD) framework to create a prototype assessment for ELL middle school students. In addition to describing the ECD process, the report describes how the project mandate helped to focus the complex processes of ECD and how the activities of ECD came into play in sketching plans for the validity argument. The ECD process of domain analysis uncovered perspectives on tablet use in middle schools as well as theoretical perspectives for conceptualizing the tablet users' abilities and learning. Findings included a range of uses of mobile technologies in middle school classrooms in 2013, with little uniformity in practices. Based on analysis of common elements in tablet‐based communication and learning, a framework for characterizing the relevant strategies and abilities was developed. Findings served in the domain modeling process of defining task design patterns, which in turn provided the basis for planning the conceptual assessment framework (CAF). The CAF also drew upon domain analysis findings, which suggested the concepts required for defining a student model (what is to be inferred about the test taker), an evidence model (the basis for making the inference), and the task model (the features of tasks required for eliciting relevant performance). The assessment implementation process produced a working prototype that served in usability testing and gathering feedback from middle school teachers. The report describes how ECD guides the design of tablet‐based formative assessment tasks that take into account the constructs to be measured and the learning that should result from test taking. The evaluation of the project is undertaken in view of the goal of ECD to design test tasks in a manner that serves in the argument for their use. Success is therefore evaluated in part by an analysis of the quality of content it supplied for an interpretation/use argument for the assessment. Overall, the project highlights include (a) the demonstration of how a mandate delimits the ECD processes; (b) the construct framework encompassing both the linguistic and nonlinguistic resources used to create meaning in a digital environment; (c) the approach used for analysis of a domain with emerging, shifting, and variously used technologies; and (d) the use of an interpretation/use argument in the appraisal of an ECD test design.

Read More