1. Introduction

The chapter provides an overview of the development of language assessment literacy (LAL) and the shift towards understanding the specific needs of different stakeholder groups, particularly language teachers. The chapter also introduces the focus on LAL development, levels of LAL, and learning trajectories, with a specific emphasis on pre-service teacher education. The study aims to investigate the difficulty of certain aspects of language assessment for pre-service teachers of English in Austria, using multi-facet Rasch analysis to generate empirical difficulty estimates. The chapter also covers the literature review on LAL development, description of the dataset, procedures, main instrument, analysis, findings, and implications for LAL development in teacher education programs.

2. Developing Language Assessment Literacy in Teacher Education

The chapter discusses the importance of language assessment literacy (LAL) for language teachers, as well as the different dimensions and theories of LAL. It also addresses the need to develop LAL in teacher education. The chapter outlines the four aspects of the developmental perspective of LAL, including theoretical LAL levels, empirical LAL progressions, longitudinal LAL development, and effective LAL pedagogies. Additionally, it presents a study that integrates the quest for empirical LAL progressions with pedagogical considerations, aiming to provide an empirical basis for course design.

3. Institutional Context of the Study

This chapter discusses the institutional context of the study, focusing on the one-semester course on language assessment for students in the Bachelor of Education program at the University of Vienna. The course provides both theoretical and practical components, including project-based assignments involving test development and selection. The final grade for the course is based partly on a theory component, evaluated through a paper-and-pencil test.

4. Methods

Participants for the study included 420 Austrian pre-service teachers of English who were undergraduate students enrolled in the BEd program at the Department of English and American Studies in Vienna. The 28 test booklets used in the study comprised a total of 142 different test items. Three types of item formats were used: selected-response, limited-response, and open-ended questions. Content-referencing procedure was used to analyze the content of each item, and 206 AKU statements were generated, which were then grouped into categories. Students' responses relating to the AKU statements were scored as correct or incorrect. Multi-facet Rasch analysis was conducted to scale the AKU statements, and difficulty estimates for each item were generated. The program was run to clean the data, and candidates who behaved markedly different from the model’s expectations were removed from the dataset. The difficulty of assessment-related AKUs for Austrian pre-service teachers of English was approached quantitatively by generating difficulty estimates through FACETS, and the second research question was addressed by qualitatively inspecting the item measurement report, examining content clusters and the progression.

5. Results

The results showed that 49 students had infit values higher than 1.5, but after removing these candidates from the data set, a final sample of n = 371 was obtained. The facets map presented a visual representation of the relationships between the elements of the facets, allowing for a direct comparison of elements. Cluster analysis illustrated the progression from easy to difficult AKUs in five levels, with specific illustrations of content clusters. Progression in classroom-based assessment was also characterized by grouping the items according to constructs and comparing the relative difficulty of the respective AKUs, showing a clear progression in aspects such as formative assessment, alternative assessment, and washback.

6. Discussion

The chapter discusses the results of the study, with a focus on the difficulty progression of language assessment literacy (LAL) skills. It emphasizes that the progressions presented are not generalizable, but can be used as an empirical basis for teaching language testing and assessment. The chapter provides design principles for difficulty-informed LAL programs, and illustrates how a revised syllabus for an assessment course at a university could progress from easier to more difficult AKUs in a cyclical and iterative process. It also discusses the importance of leveraging the results of an achievement test for formative purposes, and how the difficulty-sensitive approach outlined complies with teachers' wish to concentrate on practical assessment matters while still providing theoretical teacher education.

7. Conclusion

This chapter provides empirical difficulty estimates based on a LAL theory test used in an assessment course. The difficulty continuum ranges from basic concepts and terminology to AKUs closely associated with practical classroom activities, pedagogical aspects of assessment, formative assessment, and alternative assessment; on to AKUs relating to test methods, simple test design, scoring, and grading; then from basic aspects of validity and reliability in relation to classroom practice on to test constructs, construct definition, and issues of validation; and finally analytical skills and the application of theoretical models. Seven design principles for difficulty-informed LAL courses in teacher education programs have been suggested and illustrated. The study has limitations, including an imbalance between knowledge, skills, and principles, and the narrow scope of the study. Nonetheless, the findings have practical implications for program design and pedagogical decisions.