For blog post for Tuesday, March 21:
Evaluate this article as a piece of research in terms of strengths and weaknesses. In addition, what are its pedagogical implications? In other words, what does it say to you as both a writing teacher and a peer workshopper?
The study is similar to what Cumming wrote for his dissertation that examined the same groups of participants from two aspects. Two ANOVAs were run for beginning and intermediate students, and new and returning students. However, the weakness of the study is that because the authors brought up the the degree of familiarities with peer feedback and writing gains, it is necessary to point out how many students are new and how many are returning students. Otherwise, it is hard to fully convince readers of their findings regarding the proficiency levels. It could be that more intermediate level students are returning students, which explains why they had fewer gains compared with beginning level students.
ReplyDeleteAnother weakness of the study is its design. Because both receivers and givers were given the essay written by previous students, it might have less influence on givers than on receivers, especially when the latter group needed to revise the essays. For instance, each writer has different global and local writing preferences and skills, it can be hard for someone other than the writer to polish ideas and language even when they are in the same language proficiency level. Different from givers who only need to provide feedback on the essays, receivers need to put themselves in the previous writers’ shoes and revise the essays. Sometimes, the feedback may not be what the receivers really need for their own writing. Thus, when they were administered the post-task which was to assess their own writing skills, it could explain why they failed to demonstrate significant gains.
The study has kept me thinking about those local aspects of writing. When givers, especially from a beginning level, provided feedback, what types of feedback did they usually offer? There could be the possibility that providing global level feedback was easier than providing local level feedback. Or givers skipped the local level errors because they did not know how to correct them. After reading the study, I am very curious about these nuances behind the findings and I hope there could be a corpus study to investigate the preferences of feedback offered by different levels of writers and a qualitative study into why givers provided those types of feedback.
Additionally, I am wondering why new givers benefitted more than returning givers. I am sure there is more room for new givers to improve, given the nature of joint scaffolding and ZDP. However, I believe that returning students should keep doing it. Although their developing processes may be slow, offering feedback to others will eventually bring long-term positive influence on their own writing, such as being able to monitor in all phases of writing and translate problems.
As far as strengths, the one that jumps to mind for me is the authors’ attention to alternative interpretations of the data than the ones they are drawing. All research should do this, and most research probably does to some degree, but this article goes beyond what most do (in my experience, so maybe I'm just not reading the right stuff), and I really appreciate that. Another strength could be the lit review--it's pretty strong--and they do a nice job of creating a rationale for their study. I also appreciated the efforts they made in the name of design rigor. Did they fill a gap? I suppose they did, though I would characterize it more as lending quant-based support to previous studies and opening a door to future research.
ReplyDeleteAs far as weaknesses, Fang brought up a couple of questions that I also have. The fact that the receivers are revising drafts that are not their own is just odd. Unless you find yourself in the role of editor, it's unlikely you will do much of this in real life. I know they did this to control for confounding factors, but in this case I’m not sure this solves more issues than it creates. Fang also brings up the need to know more about the feedback given. I agree. What did it look like and how was it incorporated? How useful was it to the receivers? I'm assuming it was teacher-produced feedback, but maybe it was peer feedback? That matters, too.
Along the same lines, as I read, I kept waiting to learn more about the trainings, but the information never came. What if they were more successful at teaching students how to provide feedback and less successful at teaching students how to use it for revision? Or maybe the training was more geared towards beginners than intermediate students? I'd also want to know how well the trainings coordinated with the rubrics that would be used to evaluate the pre- and post-test essays. For example, the authors mentioned that lack of growth in mechanics could be in part because the lesson plans focused on only global concerns (do they mean the trainings? Or the class? It's a bit ambiguous, but either way, I think the point stands).
The question that looms largest in my mind at this point is whether it's prudent to assume that giving and receiving feedback are equivalent activities (two sides of the same coin), which is an assumption on which I think this study (design, RQs, methods) must rest. The authors' point about how these activities differ w regard to the ZPD (givers can tailor their feedback to their ZPD; receivers can't) lends some support to that idea. Also, in real classroom peer review activities, these two tasks will be carried out in conjunction, and the authors hypothesize that in a more naturalistic setting, where these activities are not artificially separated, gains could be even bigger. So how important is it to determine which is more effective?
I really liked your ideas about focusing more on the trainings themselves. I also wanted to see how they were training learners and where they were directing their attention.
DeleteI really appreciated the focus Lundstrom and Baker (2009) had with this paper because I feel like I often read how learners use feedback to improve writing and not enough about how learners can learning from giving it. Also as Amanda mentioned, they did a good job of addressing possible weakness of their study (e.g. experience with peer reviewing, alternative explanations for results, acknowledgement of different teaching styles) and then working those into the study if feasible.
ReplyDeleteThe main weakness for me was part of the design. As Fang and Amanda point out, it was odd that receivers were editing someone else’s paper. They both brought up the possibility that the receiver may have a completely different writing style and that simply editing another person’s paper may not be useful. This also made me question the motivation of the receiver since it really did not have anything to do with their own papers or expanding their understanding of their writing. It is possible that they did not really make gains in scores because they were not learning from the experience. How would this be different if it were their papers?
Similar to Fang, I also found the experience with peer reviewing as problematic. In the results of the intermediate group, the researchers note that the lack of difference between the givers and receivers could be due to experience learning and using peer reviewing. This not only made me think about the differences between the results of the beginner and intermediate learners, but also about within group differences. While they did acknowledge and test for differences due to one’s familiarity with peer reviewing, I wondered what the results would have looked like if they had initially controlled for this variable. Additionally, Fang pointed out that we don’t know the number of students with experience, which is important when making the claims that they do.
The main pedagogical take away for me is that training students to review documents may not only help them develop their individual writing skills through the transfer of critiquing ability, but that it might be more helpful then simply receiving feedback from the teachers and peers. I think this has some important implications because it means that the focus of teaching might not be on using feedback until something is learned, but instead on teaching the student to critique and edit independently in the initial stages. In this manner, the focus might shift from product revision to process development, which in turn gives more control to the student. Therefore, as a teacher my thorough feedback might not be as helpful as I thought, and as a peer reviewer, it might not be about what I can help my peer with but instead what I can learn from their papers.
On my opinion the strength of this article is, peer review for language two leaner’s give the students the opportunity to use language effectively. When a student is reviewing a paper he or she has to read it and to give an opinion about it which helps him to get more ideas on the topic. Students can also learn better writing skills that they can use to write other papers for other classes. I once reviewed my classmate paper and I found myself improving my language skills than before.
ReplyDeleteI also agree with Fang and Amanda on the weakness of this article, that students who received feedback revised on others student’s papers. On my opinion it is not easy to change or correct a paper you haven’t written, because you do not have the idea of what that person was thinking. I don’t understand how those students where correcting those papers. May be the teacher gave them a set of instructions that they used in revising the drafts. I think it was a challenge for the students, since most of them where native speakers of other languages not a specific language.
An implication of the article is on the participants who received peer feedback but did not give. It sounds like these students relied more on others. They didn’t get the opportunity to read what other students wrote. I think it was hard for them to improve their writing skills compared to the ones who gave peer feedback. I think students who read more documents increase their knowledge of understanding compared to the ones who don’t.
I think a teacher is a person who helps students to plan their ideas of the study. As a teacher I may emphasis my students on evaluating other students work. This will help them improve their own writing. This activity may be a challenge since, some students do not want their work to be evaluated by their fellow students but, a few who are willing may improve their writing. I usually observe my students when I return their papers in class. They are not ready to share them with their friends because they don’t want to be embarrassed.
I agree with Rocky that the focus the researchers had as one of the aspects I enjoyed the most while reading. I also consider that its main strength. I also agree with Amanda in the fact that Lundstrom and Baker did a good job at making a case for the need to complete this research, they present a solid lit review. Moreover, I considered a strength the fact that they provided students with training on peer review and how to use feedback.
ReplyDeleteHowever, there were two weaknesses I think are worth mentioning: First, the fact that they only analyzed one set of data per student. Second, the fact that they just assumed that what the students learned (their writing improvement) comes exclusively from the peer reviewing exercise, and in the discussion session the authors completely leave aside the fact that the students were enrolled in writing classes.
The results and discussion are based on the data from one pre- and one post assessment per student. I understand that in research sometimes time and resources are limited, but I believe more data should have been incorporated. One essay at the end of the semester may not be the best way to measure a student’s writing improvement. I wonder how hard could it have been to collect data throughout the semester and make a kind of continuous evaluation of each student. Or is that too much to ask? I have been told so many times that when it comes to writing we have to see the process, not the product, and this study concentrates so much on product.
The second weakness I mentioned is the fact that, as far as I understood, the authors just assumed the data result exclusively from the peer review experience, because all groups had similar lessons. I think this is not something to take lightly. What if the students were making progress and learning from the activities in class? (It was a writing class, after all). I know I’m going to sound like a broken record, but the students’ opinions were not taken into account either. I feel like a simple survey would have sufficed, asking them whether they thought providing feedback to peers had helped them improve their own writing.
As a person who has specialized in giving linguistic feedback for the last 15 years, I’m certain that I benefit more from the feedback I give than the feedback I receive. I’m also aware that it is very difficult to successfully organize students to respond meaningfully to one another.
ReplyDeleteI’d like to explore the benefits of giving peer feedback in my own research, but I think it’s critical to recognize that response to language is a literacy practice that cannot be limited to classrooms. For me, a primary weakness of this study is its limited scope (e.g., a 16-week semester, a classroom context). I speculate that providing feedback offers long-term benefits that should be explored in more meaningful and nuanced ways.
While I cite Bruffee all the time, I hadn’t realized that this train of research had really dropped off in the 1970s. As Amanda explained, Lundstrom and Baker (2009) is an important study extending research on the literacy practice of providing linguistic feedback. I don’t understand the importance of dichotomizing local and global levels in this study. What is the benefit? Why is it important to the study design?
The strengths of this study are, as Amanda mentioned, the lit review that they provided. They explained the gap between previous studies and convinced the necessity of using experimental research methods for L2 peer feedback studies in writing. I also like the design of this study that they pointed out the similarity of previous design and which points that the authors developed more than the previous study. Especially regarding the design quality, the study showed consistency across all aspect of the study (e.g. theory, hypothesis, findings, and interpretations of findings). For example, the way of conducting pre-and post-test and data analysis were so clear, which addressed the rigor of procedures as well as interpretive rigor in this study.
ReplyDeleteAnother strength is that the inference that they made from the results. Although this was experimental study, the authors provided supporting statements (interpretation) for each significant and insignificant result in detail. After finding the insignificant result from pre- and post- test in intermediate groups, they hypothesized if the difference might be derived from the beginning and returning groups in the intermediate group. They found significant results from the beginning intermediate group particularly in global aspects, whereas they found the insignificant result from the intermediate returning group.
I believe this study showed strong implication regarding providing writing feedback. However, I wondered why the intermediate level didn’t show significant results between pre- and post-test. There are other possibilities that result in insignificant results, such as teachers’ quality, instruction procedure, or students’ motivation. I also agreed one of the reasons that they mentioned was the growth of writing skills might take longer for the intermediate group. This study only observed one or two semesters so if the study period get longer, the result can be different. Another possibility of students might have during the study is a saturation. They might get tired keep doing same activities so their affective filter might become high during the study.
The findings of the study provided a suggestion for writing teachers how to develop their lesson plan and curriculums in the beginning stage of the course. Since both basic and intermediate level of students showed significant results in writing, providing feedback to peers may be beneficial for those who gain their writing skills in global aspects. However, I’m curious how basic level of the students can gain if the students make the same mistake or provide wrong feedback to their peer. Teacher engagement is extremely important in this case. For the intermediate level, some of the students might have strength in local aspects of writing, instead of global aspects. In this case, students can use their strength and can give benefit to others who need more support.