1. Introduction

The paper introduces the concept of assessment modality (AM) and its impact on writing tests, focusing on scoring and consequential validity. It discusses the influence of AM on language teachers' perceptions, preferences, and scorings in English as a foreign language (EFL) writing assessments. The study aims to examine the scoring and consequential validity evidence of computer-based (CB) and paper-based (PB) writing tests based on language teachers' ratings, perceptions of text quality, and preferences. The research questions addressed in the study are related to the impact of AM on language teachers and the validity and fairness of CB and PB writing tests in classroom assessment.

2. Literature Review

This chapter discusses the impact of the assessment modality on rater reliability in L2 assessment, examines raters' perceptions and preferences across computer-based (CB) and paper-based (PB) texts, and identifies the need for research on scoring and consequential validity in L2 classroom writing assessment. The chapter also highlights previous research findings on the impact of assessment modality on raters' reliability and provides evidence of the impact of assessment modality on student writing assessment.

3. A Sociocognitive Framework for Validating Writing Test Performance

This chapter describes a sociocognitive framework for validating writing test performance, focusing on the significance of scoring and consequential validity evidence. Scoring validity is crucial for the reliability of test scores, involving rating conditions, rater characteristics, and rater reliability. The chapter discusses how various factors such as assessment modality, raters' experiences, and reliability affect the scoring process. Consequential validity encompasses washback effects on classroom teaching and learning, impact on institutions and society, and the avoidance of test bias. This research seeks to understand the impact of assessment modality on raters' ratings, perceptions, and preferences in EFL classroom writing assessment.

4. Methods

This chapter describes the research site and participants, the research design, and procedures used in the study. The study was conducted at a teacher education program in foreign languages in Colombia, with 38 second-year university students participating. The research design involved a mixed-methods approach to collect both qualitative and quantitative data. The writing tasks and scoring rubric were designed, piloted, and validated to ensure their validity and reliability. The collection of writing samples and the transcription process were carefully conducted to maintain the integrity of the original texts. The rating process and moderation sessions were used to enhance scoring reliability, and interviews were conducted with the raters to gather their perceptions of text quality and preferences for assessment mode.

5. Results

the lowercase letters, but it was clear that the same mistakes were present in both modalities. In contrast, typed texts seemed to contain more spelling mistakes.Sonia and the teacher-researcher agreed with Julian, seeing more spelling and capitalization errors in the typed texts. They also reported seeing more punctuation errors and paragraphs’ issues in the typed texts.

6. Discussion

This chapter discussed the scoring and consequential validity evidence of CB and PB writing tests. It examined how the assessment mode affected raters’ scorings, perceptions of text quality, and preferences for assessment mode. The results showed that raters awarded higher scores to both original and transcribed versions of handwritten texts and awarded significantly higher scores for Text Organization and Language Use, suggesting that they were more lenient with handwritten texts. Raters’ perceptions of text quality differed across texts, and their preferences for AM differed, too. The evidence indicates that the AM affects raters’ scorings, perceptions, and preferences. The study has implications for the scoring and consequential validity of CB and PB writing tests, suggesting that the AM can affect a writing test’s scoring validity and impact raters’ scores and perceptions of texts.

7. Conclusion

The study concluded that computer- and paper-based writing tests affect raters differently, and the assessment mode may threaten the validity and fairness of classroom writing assessments. To accurately interpret and use test scores, language teachers need to collect and examine validity and fairness evidence of their writing tests through their scorings, perceptions, and preferences. The study also acknowledged some limitations, such as the small sample size and the lack of individual analyses per rater. Further research is needed to investigate intra-rater reliability and raters' cognition in assessment modalities. The sociocognitive framework and mixed methods research of Shaw and Weir (2007) were found helpful in examining the scoring and consequential validity evidence of CB and PB writing tests, and further studies are encouraged to replicate the present investigation to confirm or disconfirm the results and usefulness of this test validation framework.