Synopsis

When helping students acquire discipline-specific academic literacy skills, science educators often find it difficult to provide consistent feedback on student writing. The primary goal of this dissertation project was to develop a computer-assisted writing assessment (CAWA) system to augment feedback from university chemistry faculty by identifying errors in student texts and providing summary scores for different dimensions of discipline-specific writing. To create this tool, 34 chemistry writing errors that had previously been identified as important to academic chemists were evaluated to determine which were suitable for inclusion in the CAWA system. In the end, 13 error variables associated with two separate dimensions of writing (Audience and Writing Conventions) were judged to be suitable for use in the CAWA system. These 13 error variables were subsequently expanded into a functioning error tagger by creating a total of 160 search terms and patterns that allowed the computer to automatically identify and label several different permutations of each target error variable.;CAWA system performance was subsequently evaluated by means of separate analyses of error-tagging and scoring accuracy using a corpus of Methods sections (N = 150) written by undergraduate chemistry students. Follow-up analyses were also conducted to determine whether CAWA system performance was affected by several potential sources of bias that have been identified in the literature on rater behavior.;Results of the analysis of error-tagging performance indicated that the CAWA system provided highly accurate feedback. The CAWA system was able to capture 74.2% of all Audience errors and 74.6% of all Writing Conventions errors present in the corpus. Of the errors marked by the CAWA system, 95.9% of the Audience error tags and 95.8% of the Writing Conventions error tags were correct. In comparison, the three chemistry faculty raters who scored the same texts found fewer than half of the errors present in the corpus and applied correct error tags less frequently than the CAWA system.;Results of the analysis of scoring accuracy showed that CAWA system scores were more accurate overall than were scores obtained from the three faculty raters. CAWA scores were more highly correlated with benchmark scores ( Audience r = .88, Writing Conventions r = .93), than were scores from any of the three faculty raters (Audience r max = .68, Writing Conventions rmax = .65);Results of the follow-up analyses indicated that CAWA error-tagging accuracy was lower for responses that were shorter, responses that contained fewer total errors, responses that were written in response to the posttest prompt, and responses that were written by non-native speakers. However, results from just one of the follow-up analyses of scoring accuracy---the analysis of the effect of error frequency---found a significant scoring bias. In comparison, follow-up analyses of scores obtained from the chemistry faculty raters uncovered several significant scoring biases. Thus, although CAWA system feedback was not entirely free from bias, it was found to be far less prone to bias than feedback from the faculty raters.