مشخصات پژوهش

The present study attempted to explore factors affecting EFL writing assessment scores with generalizability theory (G-theory) to investigate improvement of assessment methods. To this purpose, one hundred and twenty students participated in both independent and integrated writing tests. Then their performances were scored by six different raters namely: self, three peers, and two instructors using the TOFEL scoring rubric. The relative and absolute contributions of different facets such as student(S), rater(R), task (T), method of scoring (M), and background of education (B) were investigated by implementation of G-theory using EduG, a G-theory software package. The results indicated three major sources of variance: (a) the student by task by method of scoring (nested in background of education) interaction (STM: B) with 31.8 % contribution to total variance. (b) The student by rater by task by method of scoring (nested in background of education) interaction (SRTM: B) with 26.5% of contribution to total variance and (c) the student by rater by method of scoring (nested in background of education) interaction (SRM: B) with 17.6% of contribution . With regard to the G-coefficients in G-study (relative G -Coefficient ≥ 0.86), it was also found that the result of assessment was highly valid and reliable. The sources of error variance were detected as student by rater (nested in background of education) (SR: B) and rater by background of education with 99.2% and 0.8 % contribution to error variance respectively. Additionally, ten separate G-studies were conducted to investigate the contribution of different facets across rater, task, and methods of scoring as differentiation facet. These studies suggested that peer rating, analytical scoring method and integrated writing tasks were the most reliable and generalizable designs of the writing assessments. Finally, five decision-making studies (D-study) in optimization level were conducted and it was indicated that at leas