I've been thinking quite a bit about this question, particularly how a "crowd" might be harnessed within pedagogical settings, specifically in the service of response to student writing. For most of us who teach English classes, our intellectual mobs are thankfully fairly small, but I've become increasingly convinced that even groups as small as 15 can offer each other a volume of feedback, unmatched by most writing teachers or peer revision groups, that can be facilitated with little effort in blogs and wikis. I've experimented a bit with a "crowd review," or online student review of their peers' work, in both my undergraduate and graduate courses with some mild success that I'd like to share here. And as a matter of fact, I will be sharing these thoughts at an October 5th CIT/Edtech Faculty Forum, so I would greatly appreciate any crowd I can gather here to respond to the following piece.
Pedagogy of the Crowd: Collaborative Critique in Cyberspace
Recently I have been participating in a scholarly experiment called a "crowd review." A journal in my field, Postmedieval: A Journal of Medieval Cultural Studies, has boldly decided to replace the traditional double-blind peer review with an open, transparent, and online review process for a forthcoming issue on new media. In their vision statement for the issue, editors Jen Boyle and Martin Foys defend their use of the term "crowd review" to characterize this process by suggesting, "The concept of the crowd to some might seem particularly inimical to academic institutions or professionalism. Crowds imply mobs. Crowds imply amateur opinion, cajoling, and yelling. But crowds also now connect with activities like 'crowd sourcing,' an open call for collaboration among a large group of informed participants interested in exploring, creating, or solving. And crowds also change things. The crowd potentially embodies an exciting challenge to the isolation and insularity of traditional academic organizations, as an opportunity to experiment with the re-structuring of professional and disciplinary affiliation."[i] Rather than submit to the pejorative connotation of a "mob mentality," the editors embrace the "wisdom of the crowd," inviting everyone to read and comment upon essays that have been posted to a blog.[ii] The only requirement is that every commenter must divulge their identities to keep the process transparent, a value that undermines an aspect of peer review that many of us hold sacrosanct: anonymity.
I certainly sympathize with and understand the logic behind blind review, namely that it often mitigates personal biases and licenses the reviewer to evaluate the work on its merits alone. Yet, I have become increasingly convinced that transparency, particularly within cyberspace, has a number of benefits that may, at times, outweigh the advantages of anonymity. If you have attended one of my previous presentations about online role playing, this statement may come as a mild shock. After all, I have previously argued that blog avatars, or anonymous online identities, provide writers safe and empowering perspectives to express their opinions boldly, and often aggressively. For example, my Arthurian literature students, who contribute to my class blog by writing from the points of view of characters in course texts, consistently produce lively prose, engage in heated debates, and even push the limits of the genres they inhabit.[iii] Even more importantly, the inability to identify fellow bloggers immensely democratizes the rhetorical situation: the expertise or knowledge of the writer can only be determined within the dynamic experience of online dialectic. That is, the anonymous contributor, who may be anyone under the sun, challenges traditional notions of epistemological authority, which rely upon an a priori location of authority in a credentialed specialist. Just like blogs, wikis, especially Wikipedia, value a posteriori expertise – only through collaborative production, verification, and approval does information become authoritative. Wikipedians do not care if you are a distinguished professor or a high school dropout. Authority is produced and sanctioned through popular and extensive compilation and collaboration, not through academic pedigree.[iv]
Despite the democratizing benefits of online anonymity, a transparency problem can arise, particularly when the stakes are raised. For instance, if someone manipulates information online in favor of a particular political program without divulging their allegiances or their identity, it is difficult for subsequent responders to evaluate the core assumptions behind the material.[v] This in turn leads to what can be in some cases a mindless consumption of digitally produced media. As Henry Jenkins has suggested, our students have become increasingly adept at manipulating information online, but surprisingly inept at evaluating it.[vi] Sherry Turkle, one of the most influential media theorists, has attributed this pervasive ineptitude to a lack of transparency, which she claims encourages "people to get used to manipulating a system whose core assumptions they do not see and which may or may not be ‘true.'"[vii] When identities, convictions, and sources are made transparent online, information can be evaluated responsibly and used more effectively. Within the academic structures of peer review, this means that critiques that need clarification or justification can be negotiated between writer and reviewer. Moreover, divulging the identity of the reviewer will also, in most cases, reveal the research programs, particular agendas, and sometimes even certain schools of thought that inform the critiques. This information allows the writer to situate the reviewer's responses within a specific context that may be otherwise unavailable through blind review.
In addition to transparency, crowd review offers a high volume of feedback that is impossible for most editorial teams to match. Whereas most journals in the humanities can offer, at most, three reviews of a standard article, a crowd of reviewers can more than quadruple that number. For instance, as of August 29th, one of the essays I commented upon in the Postmedieval crowd review has already received thirteen responses, many of which contained live links to relevant resources.[viii] And because each commenter is identified through a link to an online profile, visitors to the site can see that the reviewers boast a variety of educational backgrounds and academic specialties, all of which are specifically invested in particular aspects of the work at hand. Perhaps more importantly, the crowd review accelerates peer review from dialogue into hyperdialogue by allowing writers to respond directly and immediately to reviewers' comments in order to clarify or even challenge critiques. This kind of public negotiation requires that writers and reviewers be particularly careful about the relevancy and reliability of their comments – aspects that will undoubtedly improve the review process and the eventual scholarly product.
This kind of improvement has certainly proven to be the case for the genre called fan fiction, which has long held such collective intelligence as integral to its popularity. Before the advent of the World Wide Web, fans of popular media such as Star Trek produced and disseminated their written elaborations of the standard plot lines and relationships between characters through printed "zines."[ix] Once fans began to migrate their communities to online environments, however, the scope and depth of their influence increased dramatically.
The hotbed of activity is located at Fanfiction.net, not only the largest digital archive of fan texts, containing approximately a half-million Harry Potter texts alone, but also a highly multilingual writing space, containing stories in at least thirty languages.[x] Online fan fiction has therefore proven to be particularly attractive to English Language Learners, whose multilingual participation is valued and encouraged.[xi] As Rebecca Black has shown, female ELLs have utilized manga and anime fan fiction as a means to improve their English. In the case of the writer she identifies as "Grace," the production of her fan text "Heart Song," envisioned as part of the Card Captor Sakura manga series, required the use of multilingual and interpersonal registers. She not only humbly advertises herself as "the Fastest/quickest and yet Poor english Writer from the Philippines," but also invites her audience to "R+R" [read and review], assuming that her readers would be capable of negotiating between English and Japanese.[xii] Here is a brief excerpt of the beginning of her story:
“PLEASE WELCOME . . OUR VERY OWN!! . . . CHERRY BLOSSOM!!!!”
The crowds went wild, when the lights went out and then they heard a voice . . .
“Aitai na Aenai na . . . Setsunai na . . . Kono kimochi . . .”
Then one by one light went on . . and they saw a shower of Cherry Blossoms . . the audience gasp . . then they heard a voice again . . .
“Ienai no.... Iitai no... Chansu.. nogashite bakari”
Then a little spotlight focus on a girl . . going down, up from the sky. . wearing a pink dress with wings on the back, and a cherry blossoms at her hair . . she is sitting on a gold swing “Datte . . . . Datte . . . tsubasa hiroge futari de . . . Sora wo marason Yume wo yunizon shitai”
“WE LOVE YOU CHERRY!!!!”
The singer smiles at them, then when her feet touches the stage, she smiles and continues to sing the slow melody . . without music . . . “Hora Catch You Catch You Catch Me Catch Me Matte . . . ”[xiii]
Her linguistic transparency and editorial flexibility led to the creation of a fifteen-chapter text and a fifteen-chapter sequel demanded by a fan community who offered a staggering 1569 reviews of her work over a five-year period.[xiv] The sheer volume of response, which proved to be incredibly productive for Grace, demonstrates that crowds offer collective resources and motivations that individual experts or writing teachers cannot hope to match.
Over the past year in my teaching, I have adapted the spirit of open critique for what I would like to call "crowd pedagogy." While a blog can provide a suitable format for collaborative productions of knowledge, I have recently preferred the easily adaptable wiki page as the optimal space for encouraging discursive classroom commentary. My first attempt at such crowd sourcing resulted in a wiki page I playfully called "Romancing the Tome," which provided an online space for my "Understanding Literature" students to respond to course texts and each other on the same web page. To ensure transparency for the purposes of assessment and collegiality, students were required to identify themselves through their full names or initials. Because the responses could be tracked back to particular students and operated as a threaded discussion, students were often resistant to confront each other with refutations, challenges, or even minor critiques of previous comments. On the other hand, the wiki did allow students to elaborate on topics that we did not have time to fully address in class. In this sense, the wiki functioned as extension of classroom dialogue. Without this online space, such extensive elaborations or clarifications would not have been possible.
I realized, however, that the cost of transparency was often the loss of candor in the discussions. To recapture this feature often characteristic of anonymous blogging, I implemented a response structure that Peter Taylor used in his 2010 Inter-College Faculty Seminar in Humanities and Sciences, "Engaging Colleagues in Caring Collaborations" here at UMass-Boston. Peter asked the faculty in the seminar to respond to a piece of writing he had posted to a wiki through the following dialectical method developed by Peter Elbow and Patricia Belanoff called "Believing and Doubting":
Believing: Simply ask readers to believe everything you have written – and then tell you what that makes them notice. Even if they disagree strongly with what you have written, their job is to pretend to agree. In this way they will act as your ally: they can give you more reasons or evidence for what you have written; they can think of different and better ways of saying or thinking about what you have written.
Doubting: Now ask readers to pretend that everything is false, to find as many reasons as they can why you are wrong in what you say (or why your story doesn't make sense.)[xv]
To code each response, Peter asked us to color our "believing" comments with blue and our "doubting" comments with red. These markers clearly distinguished the responses from the text and visually displayed the balance or imbalance between the peer praise and critique of the work. Moreover, Peter asked us to include our initials at the end of each comment to make the process transparent.
This method so effectively and fairly managed the responses of our scholarly crowd that I stole the strategy and quickly applied it to a wiki assignment for my graduate seminar, "Teaching Literature." Since this course is designed to help preservice or inservice secondary or college teachers develop their literature pedagogy, I asked my students to observe the classes of their colleagues or professors and write observation reports. In the past, the reports were submitted exclusively to me for feedback, but their findings were so important that I felt that the benefits of the assignment could be maximized if they were shared and discussed on the course wiki.[xvi] Students were asked to respond using the "believing and doubting" heuristic, but I did not require that they include their initials because the comments could be tracked by clicking the "History" tab, which logged all edits to the page.[xvii] Even though contributors were capable of being identified, I now realize that not requiring clear identifications on the main page was a mistake because it forced users to expend an unnecessary amount of labor to distinguish the comments from each other.
Nevertheless, I was pleasantly surprised that some students embraced the task and readily employed the dialectical method to respond to each other's reports. For example, one student produced a provocative account about a listless class that seemed incapable of engaging with the course material.[xviii] The report placed some of the blame on the students, but many of the critiques were directed to the instructor, who refused to consider alternative approaches to encourage engagement. In particular, the report emphasized the instructor's initial failure to incorporate in-class activities that gave students an opportunity to get to know each other. This point prompted one responder to initiate the following exchange:
"I'm not sure that all the blame should be placed on the professor because the students didn't introduce themselves the first day of classes."
"I don't agree. I think that what happens on the first day of a class does rest in the teacher's hands. What can't be controlled is how the students will respond. I think, and this is just my opinion, that if a teacher makes it a point to do some sort of activity that will result in students sharing a little bit about themselves, it sets the tone for what is to come and it forces students to realize that they will be learning from each other, not just the teacher."[xix]
While the writer of the report defended many of her initial arguments, she also clarified or revised other claims based on the responses she received. For example, at one point she suggests that the lack of investment could be attributed to the lack of English majors in the course, which more than one responder challenged. She then revised her position by saying, "I agree. We all have taken courses that are not in our specific subject area and have done well because that is what you should do. It seemed in this course though that many of them were so disinterested about all of the reading, assignments, everything! I really have never seen anything like it."[xx] Even in this small-scale crowd review, the wiki provided a dynamic platform in which student work continued to produce benefits for the both the writers and their readers long after initial submission.
I would like to conclude by turning to the words of Henry Thoreau. In his address before the Concord Lyceum in April of 1838, he complained that "[t]he mass never comes up to the standard of its best member, but on the contrary degrades itself to a level with the lowest."[xxi] Some mobs, particularly the kinds that incite violent riots, indeed degrade their participants, warranting Thoreau's pessimism. In fact, while I find the spirit of Thoreau's remarks to be overly pessimistic, he may be technically correct. More often than not I find that a well-intentioned crowd does not "come[] up to the standard of its best member" – it exceeds it. What makes many of the networks in cyberspace so powerful and pervasive is that they do not insist on hierarchies, individual genius, or even equal contribution. As Clay Shirky suggests in his recent book Cognitive Surplus, crowd sourcing on sites like Wikipedia succeeds because
Wikipedia offers potential participants the ability to do as much writing or editing as they like, but also as little. If you fix a typo and never do anything on Wikipedia again, that still has more value than if you hadn't fixed it. Wikipedia makes it as easy as possible to effect these small changes, not even making users set up an account before they start editing. This low threshold of participation invites the accumulation of the smallest units of value – no one would create an account just to fix a single typo. By making the size of the smallest contribution very small, and by making the threshold for making that change small as well, Wikipedia maximizes contributions across an enormous range of participation.[xxii]
By making contribution to the crowd as labor intensive as the contributor desires, collaborative work becomes less a matter of community obligation and more a matter of personal investment. This means that the "imbalance" of effort inherent to group work – that some students will do more work than others – is not a problem for the work itself. This inequity only becomes an obstacle when we assess individual contribution, which is difficult, if not impossible, when contributions are anonymous. Transparency, on the other hand, adds an element of personal responsibility to collaborative work, which I believe to be especially productive within high-stakes modes such as peer review. If contributions can be tracked to their contributors, reviewers must carefully consider the delicate nature of sharing and responding to scholarly work. More importantly, when review becomes the responsibility of a crowd of identifiable participants, the value of the collective response can greatly exceed the sum of its parts.
[i] Jen Boyle and Martin Foys, "Editor's Vision Statement," Postmedieval–Crowd Review, http://postmedievalcrowdreview.wordpress.com/editors-vision-statement/ (accessed August 29, 2011).
[ii] James Surowiecki, The Wisdom of Crowds (New York: Doubleday, 2004).
[iii] The Roundtable, http://roundtableknights.blogspot.com/ (accessed August 29, 2011). An online screen capture of a conference session entitled "Arthurian Fan Fiction," which I created and presented with three of my students, can be found here: http://www.screencast.com/t/0PDLlx1F.
[iv] See Alex Mueller, "Wikipedia as Imago Mundi," Studies in Medieval and Renaissance Teaching 17.2 (Fall 2010): 11-25.
[v] See, for example, the following article on Sarah Palin's Wikipedia page: Noam Cohen, "Don't Like Palin's Wikipedia Story? Change It," New York Times, August 31, 2008, http://www.nytimes.com/2008/09/01/technology/01link.html (accessed August 31, 2008).
[vi] Henry Jenkins, "Confronting the Challenges of Participatory Culture: Media Education for the 21st Century," Building the Field of Digital Media and Learning (MacArthur Foundation: Chicago, 2006), 1-68, at 14, http://digitallearning.macfound.org/atf/cf/%7B7E45C7E0-A3E0-4B89-AC9C-E807E1B0AE4E%7D/JENKINS_WHITE_PAPER.PDF (accessed August 29, 2011).
[vii] Sherry Turkle, Life on the Screen: Identity in the Age of the Internet (New York: Simon & Schuster, 1995), 70.
[viii] Eddie Christie, "Writing in Wax, Writing in Water," Postmedieval–Crowd Review, http://postmedievalcrowdreview.wordpress.com/papers/christie/ (accessed August 29, 2011).
[ix] Camille Bacon-Smith, Enterprising Women: Television Fandom and the Creation of Popular Myth (Philadelphia: University of Pennsylvania Press, 1992).
[x] FanFiction.Net, http://www.fanfiction.net (accessed November 29, 2010).
[xi] Rebecca W. Black, Adolescents and Online Fan Fiction (New York: Peter Lang, 2008).
[xii] Rebecca Black, "Online Fan Fiction, Global Identities, and Imagination," Research in the Teaching of English 43.4 (May 2009): 397-425, at 409.
[xiii] Ibid., 409-10.
[xiv] Ibid., 410.
[xv] Peter Elbow and Patricia Belanoff, Sharing and Responding, (New York: Random House, 1989), 32; For the paper we commented upon in the seminar, see Peter Taylor's wiki page, 4Rs, http://ptaylor.wikispaces.umb.edu/4Rs (accessed August 30, 2011).
[xvi] Alex Mueller, Teaching Literature, http://engl611-mueller.wikispaces.umb.edu/Teaching+Literature (accessed September 1, 2011).
[xvii] Mueller, "History," Teaching Literature, http://engl611-mueller.wikispaces.umb.edu/page/history/Teaching+Literature (accessed September 1, 2011).
[xviii] Trina Johnson, "Mini Paper 4," Teaching Literature, http://engl611-mueller.wikispaces.umb.edu/Trina (accessed September 1, 2011).
[xix] Ibid.
[xx] Ibid.
[xxi] Henry David Thoreau, Journal, ed. John C. Broderick, et al, volume 1 (Princeton: Princeton University Press, 1981), 36. For a discussion of this speech, see Robert Milder's Reimagining Thoreau (Cambridge: Cambridge University Press, 1995), 9-11. See also Surowiecki's discussion of this quotation and others that lament the stupidity of "mobs" in The Wisdom of Crowds, xv-xvii.
[xxii] Clay Shirky, Cognitive Surplus (New York: Penguin Press, 2010), 200.
Alex:
ReplyDeleteThought-provoking stuff, as usual! Reading with doubting red ink in mind, I recall a fairly recent article in the New York Times that analyzed the "Wisdom of Crowds" model and argued that strong personalities and other factors can act as attractors within the system and undermine the quality of crowd responses. Not that I'm fully against it, but college and university discourse might be such an attractor: what you style "collegiality" may appeal to students' better angels regarding bullying and belligerence, but it also may pre-determine what kinds of comments and questions students deem "smart."
I couldn't find that piece, though I did come across something else, that switched my head from red to blue. (I wonder about the politics of the color-scheme?)
This 07.18.2009 Times piece, "The Crowd Is Wise (When It’s Focused)" features a link to an interestingly structuralist analysis of crowd-sourced activity called "Harnessing Crowds: Mapping the Genome of Collective Intelligence."
What struck me most about the latter paper is the question of WHY students would be driven to comment productively on others' work. So I pose a question to you--what incentives have students to participate productively within a wiki discourse like you describe? And how might one balance concerns about collegiality with the discursive prescriptions of academic communities?
I fear that standard grading metrics would inspire students to make the same kinds of moves that they often do inexpertly in papers. Is student output conditioned by the input they receive at the end of the process? Is there a way to incentivize student performance that remaps the terrain of the wiki in non-academic terms that will inspire creative responses from their pre- or para-classroom mindset and perhaps get them to approach thinking and writing about literature and teaching in ways that challenges or overturns their received notions of discursive appropriateness, but without inspiring rancor?
I realize there are a lot of presumption driving these questions, particularly regarding where and how students get ideas about what we SHOULD be doing in literature and education courses. But they seem like good questions to my bleary, pre-caffeinated morning eyes.
Love the way you use the believing/doubting heuristic to respond! I have wondered if the use of the "red" for doubting is too reminiscent of the red ink pen that students fear, but I hadn't thought about blue/red states. Excellent point - yesterday, I happened to drive by two benches that were painted blue and red and labeled accordingly, Democrats and Republicans.
ReplyDeleteIf you find that NYT article that critiques the "crowd" model, I'd love to see it. On the surface, I'm not sure why strong personalities would undermine a crowd review process. Do you mean that some voices in the crowd would be louder than others and ultimately obscure the variety of responses? I'm sure this happens, especially in a transparent review that identifies the commenters. If a star in your field weighs in, wouldn't that voice hold more weight?
I think your key question about WHY students would be compelled to respond is a very good one. I think crowdsourcing on the web works primarily because each contributor's motivations are often hidden and often altruistic (at least in the case of Wikipedia - there is an interesting article on this but I have to track it down). However, when the crowd is a group of students, it's difficult to get beyond "requirements" and "grades" as incentives. You are absolutely right that this kind of forced participation will often lead to "the same kinds of moves" that we see in standard peer review group workshops, which are often uninvested responses. Ideally, students will find like minds and then genuinely seek to help each other out. I haven't figured out a way to structure that, though. Any ideas?