Michael Knievel

ENGL 5365

Dr. Susanmarie Harrington

2 March 1999
back to contemporary issues in writing assessmentpage

Background

Since completing his dissertation, "The Validity of Holistic Scoring: A Comparison of Talk-Aloud Protocols of Novice and Expert Holistic Raters," and completing his doctorate at Indiana University of Pennsylvania in 1988, Brian Huot has developed into a leading scholar in the field of writing assessment. His career began at Indiana University-Purdue University at Indianapolis, where he was an assistant professor and Coordinator of Writing Placement before moving to the University of Louisville, where he is currently an associate professor and coeditor of Assessing Writing, the only journal devoted solely to issues of writing assessment.

Huot's work deals with a number of issues related to writing assessment, but his primary focus has been on the following: holistic evaluation procedures, placement exams, writing across the curriculum, assessment reliability and validity, and, in more recent years, the intersections between computers and composition, more specifically with computers as a tool for assessing student writing. The theoretical undercarriage of all of Huot's work is the recurring theme of contextualized, local writing assessment techniques. "Toward A New Theory of Writing Assessment" (1996) serves as what is probably Huot's most complete articulation of this theory, sorting through the ways in which traditional means of assessment rely too heavily on positivist notions of epistemology, leading to methods that do not fairly consider the complex and rhetorical nature of writing.

However, "Toward a New Theory" is hardly Huot's first treatment of this idea; indeed, all of his work with both holistic scoring techniques and writing across the curriculum is heavily informed by this theoretical assumption. Early works, such as "Reliability, Validity, and Holistic Scoring: What We Know, What We Need to Know" (1990) and "The Literature of Direct Writing Assessment: Major Concerns and PrevailingTrends," included treatments of validity in holistic scoring and efforts to grapple with the research on holistic scoring and other forms of direct assessment. Emerging soon afterward came an exploratory article, "Finding Out What They are Writing: A Method, Rationale, and Sample for WAC Research," which found its place among other works on holistic assessment and writing placement practices. Since 1997, Huot has again turned his attention, in large part, toward WAC issues ("Beyond Accountability: Reading with Faculty as Partners Across Disciplines" and "Assumptions About Assessing WAC Programs: Some Axioms, Some Observations, Some Context"), particularly the problem of assessing WAC programs and engendering cooperation and understanding among interdisciplinary faculty responsible for WAC administration.

Citations and Annotations (arranged alphabetically)

Huot, Brian. "A Survey of College and University Writing Placement Practices." WPA: Journal of the Council of Writing Program Administrators 17.3 (1993): 49-65.

In this article, Huot reports and interprets the results of a survey he conducted, aimed at all two- and four-year colleges and universities on the MLA list of English chairs. The survey was designed to investigate the types of writing classes and writing placement techniques used at these schools in addition to measuring the English departments' general satisfaction with those techniques. Proposed as a means to "provide the beginning of a much needed conversation on writing placement procedures," Huot's survey reveals various differences-- some large, some small-- between large and small schools, public and private schools, etc. Over 50% of the schools responding reported use of a writing sample for placement. However, few schools appoint administrators with significant composition and, specifically, assessment experience to head up placement programs. Huot concludes by arguing for contextualized placement procedures, which recognize the unique relationship between departmental curriculum and students' writing-- and faculty understanding of this relationship.

Huot's survey provides a useful glimpse at the practices and, to some extent, the attitudes of English departments charged with creating and putting into play writing placement programs. Interestingly, almost all schools reported some form of placement (96.5%), and a strong majority of schools reported at least moderate satisfaction with their placement method, regardless of which one it was. By clarifying who is frequently in charge of the placement proceedings, though (those inexperienced with assessment), Huot makes the important link between lack of knowledge and what appears to be a naive, unsupported satisfaction over methodology and results.

Huot, Brian. "Beyond Accountability: Reading With Faculty as Partners Across the Disciplines." Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Ed. Kathleen Blake Yancey and Brian Huot. Greenwich: Ablex, 1997. 69-78.

In "Beyond Accountability," Huot outlines, first, the importance of assessing writing across the curriculum programs before moving into a discussion of how and why this is done at the University of Louisville. His primary emphasis is on how writing teachers might contribute to the empowerment of teachers in other disciplines who are reluctant to take on the role of a writer or evaluator of writing, which Huot notes is a necessary step in developing a comprehensive system of WAC evaluation. By bringing together faculty from various disciplines and dialogically working through student portfolios, member faculty develop their own voice. The reading process is slowed down to heighten awareness of the rhetorical situation faced by both reader and writer.

"Beyond Accountability" is most valuable in that it offers some concrete suggestions for how to deal with a disparate body of assessors. Writing across the curriculum promises to give us a clearer, more complete sense of students as contextualized, rhetorically-responsive writers. To appreciate and evaluate this, then, a diverse body of faculty from various disciplines must be employed-- and they are oftentimes going to be reluctant based on fears about their own abilities to evaluate writing, which many see as something outside of their expertise. Huot demonstrates here that these faculty can be helpful, confident contributors to assessment with training and patience.

Huot, Brian A. "Computers and Assessment: Understanding Two Technologies." Computers and Composition 13.2 (1996): 231-43.

In this article, Huot examines the connections between computers and writing assessment, citing the importance of recognizing them both as separate technologies with unique applications and different assumptions underlying each. The two do have overlap in their largely positivist histories, which have limited, in some ways, the opportunities to effectively intersect with one another. Huot reviews past literature dealing with various computer applications for assessment, noting that most rely heavily on decontextualized views of writing that fail to recognize the unique constraints and the specific rhetorical situations faced by writers. Most past applications have focused on formal elements. Huot suggests that computer networks might be valuable additions to assessment, not necessarily in their ability to read and fairly judge student writing, but instead in their capacity to store large amounts of student text and make it, along with rubrics, comments, and suggestions available to readers in local and larger settings, making possible more broad-based comparisons of writing in different places.

The best parts of this article are first, Huot's connection between the positivism underlying computer technology and the positivism underlying past writing assessment practices, and second, Huot's suggestions for how computers might benefit assessment. While the literature review of past computer applications for assessment is illustrative, the limitations of computers as evaluators of rhetorical documents is somewhat evident. Hence, seeing just how computers might be put to productive use is important to instructors and program directors looking for ways to make computer technology help make their assessment more complete, rather than incomplete.

Huot, Brian. "Finding Out What They Are Writing: A Method, Rationale and Sample for Writing-Across-the-Curriculum Research." WPA: Journal of the Council of Writing Program Administrators 15.3 (1992): 31-40.

Huot discusses the importance of devising a systematic method of gaining the input of other departments that have a stake in writing-across-the-curriculum programs and provides a model of one such method that might aid in making informed WAC decisions. Citing a significant gap in related scholarly inquiry, Huot argues that we have little knowledge of what writing expectations students face in other disciplines and that we need to have knowledge of these requirements in order to make sound decisions. Context is important because curricula vary from college to college and, more specifically, from course to course. To get needed information, the author recommends a method of "focused dialogue" (33-34) that involves multiple interviewers interacting with a small body of disciplinary representatives and then verifying responses with each other and then the interviewees. Huot demonstrates how this technique worked with representatives of the School of Social Work and concludes by noting, first, the impressive breadth of writing being done in other disciplines and, second, the need to continue to refine information-gathering techniques.

This article is significant because it helps me begin thinking of just how writing in composition works in conjunction with writing in other classes. Huot recognizes a "strong developmental theme" (35) in the School of Social Work's writing curriculum, which is useful and pleasing to know about other departmental approaches to writing. Finally, his example of how to gather information from other faculty in other departments is, I think, essential to gaining the input needed to support a legitimate and effective WAC program.

Huot, Brian. "Reliability, Validity, and Holistic Scoring: What We Know, What We Need to Know." College Composition and Communication 41.2 (1990): 201-13.

Huot addresses the issue of holistic scoring, noting well its widespread acceptance in the field of writing assessment, but criticizing this incorporating on the basis of little theoretical understanding of the method. The author contends that holistic scoring techniques have prioritized reliability while sacrificing validity in many cases. Moreover, holistic scorers frequently remain biased toward syntactic features or content, failing to understand or value the overall rhetorical situation within which documents are constructed. Huot contends that the purpose behind assessment situations dictates how validity is conceived of, which can lead to confusion. But Huot ultimately recommends that holistic scoring techniques be developed to more fully account for contextual elements because no single instrument can or will apply to the various situations in which holistic assessment is used. Moreover, raters need to slow down their reading and carefully consider these contextual elements when offering their ratings and evaluations. Finally, more questions need to be asked and addressed, and Huot challenges his peers to do so.

This article is useful in that it serves to demystify, in some ways, a method of evaluation that is widely perceived to be airtight. Huot illuminates several flaws in holistic assessment and, importantly, offers the suggestion that the time has come to move from an overemphasis on reliability to a greater focus on the validity of such a practice. In sum, it is clear that disciplinary agreement on a single assessment technique is not necessarily based on theoretical understanding or rigorous scrutiny-- momentum plays an important role, too.

Huot, Brian A. "The Influence of Holistic Scoring Procedures on Reading and Rating Student Essays." Validating Holistic Scoring for Writing Assessment: Theoretical and Empirical Foundations. Ed. Michael M. Williamson and Brian A. Huot. Cresskill, NJ:

Hampton Press, 1993. 206-36.

In this chapter, Huot first outlines several of the criticisms aimed at holistic scoring procedures before describing a study he performed on holistic raters that deals specifically with the objection that holistic ratings are "generated by scoring procedures which alter fluent reading processes and impede the ability of raters to make sound judgments about writing quality" (206). Using both novice and experienced raters, Huot's study demonstrates that, rather than stifling what he calls "personal comments"-- reflections of a more personal engagement with the text under evaluation-- holistic rating experience and the use of a scoring guideline can help evaluators read more fluidly and with a greater personal engagement. These findings, then, do not support scholarly skepticism aimed toward holistic rating. Rather than impeding judgment, holistic rating systems and experience with them seem to aid in developing and articulating multi-level engagement with the text.

This chapter, first and foremost, addresses a compelling issue in the field of assessment. Because of the prevalence of holistic methods in placement and program evaluation, the question of whether or not this type of reading and rating provides a legitimate reading of student text is of great importance. Huot's methodology and report are somewhat convoluted at times, which can, no doubt, be largely attributed to the messiness of qualitative research design. However, his results do highlight some of the flaws behind critics' assessment of holistic rating without suggesting that holistic rating is necessarily an airtight system.

Huot, Brian. "The Literature of Direct Writing Assessment: Major Concerns and Prevailing Trends." Review of Educational Research 60.2 (1990): 237-63.

This article takes on the ambitious task of mapping the territory of the available scholarly literature dealing with direct writing assessment practices. Huot defines the three main procedures of direct writing assessment (primary trait, analytic, and holistic) before proceeding on to his broader analysis of the direct writing assessment literature, which he groups into three primary subfields of inquiry: topic development and task selection, text and writing quality, and influences on rater judgment of writing quality. Noting that the literature to date has been largely fragmented in nature and inconclusive in each of the subfields, Huot concludes that direct assessment is largely neglected as a scholarly field of inquiry, especially when one considers its widespread-- yet unexamined-- usage in the college and university writing environment. Little information exists with respect to how raters come to judgments about writing, and this needs to be addressed in future research.

Huot's text is clearly quite dense in its content and presentation. The reader is taken on a whirlwind tour of much of the direct writing assessment literature as of 1990, and Huot's ability not only to identify the central arguments contained within disparate pieces but also to categorize and arrange them thematically is extremely useful to the reader attempting to gain a quick, yet thorough, understanding of the issues at stake in direct writing assessment. Hence, Huot's treatment of the issues, coupled with the article's impressive bibliography, couple to make this a useful source and research tool.

Huot, Brian. "Toward a New Theory of Writing Assessment." College Composition and Communication 47.4 (1996): 549-66.

Huot argues forcefully for a new vision of writing assessment, which prioritizes localized context and specific rhetorical situations. Asserting that writing assessment has traditionally been based upon a positivist epistemology, Huot suggests that assessment needs to turn its focus away from reliability and the unfounded assumption that "good writing" exists in some natural form and toward contextualized writing that recognizes and rewards students for responding effectively to local rhetorical exigencies. Disciplinary experts should help to assess the rhetorical situation and the writing done in response to it, and criteria should be made available to students to deepen their understanding of the expectations at hand. By doing this, students are treated as human beings acting in human situations, and neither they, nor their writing are objectified or dehumanized in the manner that positivist assessment tactics foster.

This article's utility rests in the fact that it is probably Huot's most comprehensive individual elaboration of a contextual theory of assessment in the past three years or so. He argues effectively for abolition of positivist assessment techniques by outlining their flaws and recommending the ways in which a contextualized, rhetorical approach to assessment is more humane, more fair, and more representative of student abilities.

Yancey, Kathleen B., and Brian Huot. "Assumptions About Assessing WAC Programs: Some Axioms, Some Observations, Some Context." Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Ed. Kathleen Blake Yancey and Brian Huot. Greenwich: Ablex, 1997. 7-14

Yancey and Huot collaborate to elaborate several assumptions about assessing writing-across-the-curriculum programs and offer a handful of additional observations as a means of introducing the issues to follow in Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Writing across the curriculum programs rely on a "big picture" perspective on writing and assessment. WAC relies on a deeper understanding of writing that recognizes the epistemological nature of discourse. Because of its complexity, WAC calls for a diverse methodological and interrogative approaches to regular, systematic assessment. Assessment of WAC programs is a reformist enterprise, calling for collaboration and rhetorical interchange among its constituents. The chapter closes with a brief outline of the chapters to follow.

This introductory chapter finds its significance in its ability to map out some of the important issues facing those charged with assessing WAC programs. What Yancey and Huot label as "grounding assumptions" and "some observations" in organizing their ideas in the chapter ultimately serve as a set of guiding principles and implicit questions for the reader approaching this text or any text addressing the concerns associated with WAC program assessment.

Yancey, Kathleen B., and Brian Huot. "Construction, Deconstruction, and (Over)Determination: A Foucaultian Analysis of Grades." The Theory and Practice of Grading Writing: Problems and Possibilities. Ed. Frances Zak and Christopher C. Weaver. Albany: State University of New York Press, 1998. 39-52.

Yancey and Huot employ multiple voices, which are ill-defined in number and in person, to construct a dialogic approach to grading that locates the field's assumptions about grades and outlines some of the possible effects that grading has on both students and teachers. Primarily, grading, as it has traditionally been done, is largely acontextual and non-negotiable. The authors point out several flawed assumptions that teachers have about grading and suggest that if we must assume that grades are a necessary evil, then we must recognize the impact they have on both teachers and students in that they sustain academic hierarchies and serve to limit or propel students in their university career and, later, in their occupation. Yancey and Huot conclude that we must consider grading a rhetorical exercise and take pains to provide students with the context and opportunity for response and negotiation in order to consider the robust circumstances within which writing is constructed.

Much of this chapter's power comes in its ability to display power. The authors do well in taking the critique of grading as power beyond the obvious level of the teacher being the final arbiter in a student's fate, arguing that grades ultimately allow teachers to maintain their position in the academy, to exercise their intellectual superiority, and to gain tenure and promotion. Finally, the importance of context is made evident, even in the case of specific writing assignments. Dialogue, rather than monologue, becomes the model.