Monday, August 13, 2012

Coursera Thoughts Part Two

This is Part Two.  You can read Part One here.

As before, I've summarized things at the end, for those without the time or inclination to read the essay from beginning to end.

One of the suggestions I made in my previous thoughts included offering sample essays, ones that would clearly show what an essay with a 1 in form and a 1 in content should be and comparing it with an essay that is 3/3. This would, of course, benefit both the essay writer and the peer reviewer.

Here are some of the key phrases culled from the “Work Expectations” in the “Peer Evaluation” section:

  • [The] essays will be anonymous to you.
  • [P]lease write a brief response (perhaps a sentence or two) that accomplishes two things: pointing out at least one aspect of the essay that is good and pointing out at least one aspect of the essay that would profit by improvement.
  • The Comment box is a place to offer your fellow student direct, helpful thoughts that may not fit neatly under the categories of Form and Content.
As before, I want to take these point-by-point.

[The] essays will be anonymous to you. 
This is an excellent decision on the part of coursera because, obviously, it would be too easy for people to adjust their grade based on personality. If they favorably recognize the writer’s name from the forums, the reviewer might give a 3 where one is not deserved. Or, if the reviewer has taken offense to something the writer has said, they could give a lower grade, even a 1. By keeping the writer anonymous, they allow the reviewer to retain some objectivity. However, one could argue that the reviewer should be objective, regardless of who the writer is.

[P]lease write a brief response (perhaps a sentence or two) that accomplishes two things: pointing out at least one aspect of the essay that is good and pointing out at least one aspect of the essay that would profit by improvement.
The wording varies slightly between the “form” and “content” sections but they pretty much say the same thing: write a response that points out one thing you like about the essay and one area for improvement. Sounds reasonable, right? Here are some examples of feedback that students have received that I found on the forum:
  • Nice
  • Nice work
  • Content was very good and detailed.  However, it did not answer the assignment question if this is for children or adults. 
  • Too wordy 
  • Your vocabulary is OK I’m sorry to be harsh but your essay is worthless
The proof is in the pudding and what I’ve been saying about how many of the students in the course do not understand what is expected of them because it hasn’t been clearly explained is evident in the above examples. For one thing, there was no “assignment question” but the reviewer gave one essay writer a lower grade because they didn’t answer it. And when did saying “nice” and “nice work” qualify as giving useful feedback? Given that the essays are limited to a mere 320 words, how can anyone's essay be "too wordy"?  Do I even need to comment on that last comment?

Then there's this:  the writer receives contradictory statements:
student1 → The development of the argument is fuzzy.
student3 → Clear and convincing argument.
What is the essay writer who receives this supposed to do to improve their writing for the next essay? The anonymity is mutual—both the writer of the essay and the peer reviewers do not know who is doing the writing. As you can see, however, this can be a double edged sword. While this allows students to be honest, it also allows them to tell someone that their writing is worthless.

The Comment box is a place to offer your fellow student direct, helpful thoughts that may not fit neatly under the categories of Form and Content.  This is optional and often the reviewer leaves this blank. Typically, this space adds nothing to what has previously been said even when it is not left blank, because the reviewer merely repeats what they have previously said in the comments.

In addition to the above, I’ve seen people state on the forum that they deducted a point because someone used citations or because they disagreed with the person’s thesis even though they say n the same post that the person wrote a good argument. In effect, they are admitting that they grade based on personal preferences.

I felt it would be important to pull some of the reviews off the forum for this course because sharing my own might make me seem embittered, assuming I didn’t get a perfect score. So these are the guidelines as established by coursera and here are the problems as I see them:


Summary of Problems:
  • Anonymous peer-reviews allow people to be assholes.
  • Not all reviewers understand how what is expected so they review based on what they “think” is right.
  • With no criteria, reviewers often grade subjectively rather than objectively.


And now, my suggestions for how to improve these issues.

Possible Solutions:

First, peer reviews should be kept anonymous; however, the essay writers should be allowed to review the reviews. 
The obvious problem here is that a person who receives a bad score may reciprocate in retaliation. However, there should be a means by which a student could report a peer-review which, like the example of saying the essay is “worthless.” When a peer-review is reported, someone would be able to assess whether it is a useful review or simply a “worthless” one. Here is how I would see this working:
1) Student A receives a review they feel is not meant to be constructive.
2) Student B, the anonymous peer-reviewer, would receive a notice that their review has been reported.
3) Coursera would assess the report. If they find that Student A is correct and Student B is just being an asshole, the following would happen:
a. Student B would lose all credit for their reviews for that unit. This will save coursera the trouble of reviewing everything the student reviewed to see if this is a pattern. Also, knowing this going into things, all students would be more considerate or at least try to be less abusive in their comments to their fellow students when writing a review.
b. Student B would also receive an email saying that if there is another report made against them, they will be expelled from the class. If this student has been expelled from another course for the same reason they are expelled from coursera altogether. 
4) Student A would know nothing about the outcome because Student B’s peer-review does not affect Student A’s final grade. 
Without any criteria, the students have no means of adequately assessing some of the essays. 
Speaking for myself, I have received compliments on my citations on an essay and for the same essay been told I need to learn how to cite properly. Why? I am assuming that one student uses the same citation style I do and the other does not so in the former student’s eyes what I did was perfect while in the second’s it was flawed. Both are correct. In my previous look at this course from coursera, I suggested that a prerequisite course, a Composition 101, should be mandatory. In this course, students would not only learn how they should cite an academic literature essay but they would also learn to be objective in how they assess one another’s writing. Nobody should receive feedback that says “nice” any more than they should receive feedback that says their essay is “worthless.”

By taking this prerequisite, coursera establishes an opportunity to learn how to objectively read an essay, setting aside your personal beliefs, your preference for one way of citation over another, etc. would allow all students to benefit from giving and receiving feedback.

However, barring their offering a prerequisite, I also suggested that a criterion needed to be established.  Even if all they do is show samples of essays (as I recommended in my previous post) and samples of peer reviews, that would help. But having neither, too many of the students participating in this course are not giving nor receiving the intended benefits of the course. And that is unfortunate.

Worse, and I’ve seen several people post this to the forum, students are dropping this class because of the abusive things being said to them in the peer-reviews. I realize that some people might argue that the writers need to develop thicker skin but I am not talking about someone giving constructive criticism. I’m not talking about that; I’m addressing myself to the vitriolic and unhelpful comments that some of the students have been receiving.

Abusive language would not be allowed in the classroom. It should not be allowed in coursera.

Because this piece focused on the receiving of peer-review, my next thoughts will be on giving the peer-review and some of the problems I’ve seen mentioned on the forum.


Summary of Problems:
  • Anonymous peer-reviews allow people to be assholes.
  • Not all reviewers understand how what is expected so they review based on what they “think” is right.
  • With no criteria, reviewers often grade subjectively rather than objectively.


Possible Solutions:

  • Essay writers should have a way of reporting abusive peer-reviews.
  • Criterion established of sample reviews that are acceptable and considered "constructive criticism."
  • Prerequisite wherein students would learn how to read an essay objectively.

I am hopeful that, at the course's conclusion, we will have an opportunity give some feedback on the peer-review process, allowing the students who received such comments as "your essay is worthless" a means of sharing what they think.




2 comments:

  1. I am still trying to grasp what the point of a peer based grading system actually is. These students should give feedback and discussion, but they should not have the ability to issue grades - that is what an instructor is for. I see this entire course as problematic.

    ReplyDelete
    Replies
    1. There are a lot of problems with this course but I've heard that other courses are organized differently. I'm not overly fond of the peer grading because I've seen the feedback some have received and it is outrageous what some people think they can say to one another. Of course, they are getting away with it so they can say what they want and there's no recourse.

      Delete