ARC: Connections: ELMAR: Posting

---

Are stats requirements in our educational journals stopping us from sharing great learning exercises?

Dear friends,

I would like to discuss whether statistical data analysis adds value to learning exercises or not. This is a very important question because a growing number of editors of educational journals started requiring data analysis to be included in manuscripts describing learning exercises. In my opinion such trend reflects only the editors’ desire to present their journals as highly scientific publications. I do not argue that all educational articles should skip data analysis part; however, I do not see any added value in situations when authors are forced to collect and analyze data for simple learning exercises. I think that any educator is capable of assessing the value of an exercise based on its description. Statistical analysis brings more unnecessary work and is very limited in its ability to prove anything.

What can potentially be analyzed? First, whether students enjoyed this exercise or not (some authors compare actual means with scales’ middle points to show significance of “enjoyment”). Second, it’s possible to assess different dimensions such as complexity of the exercise, intent to apply learned skills, evaluation of own learning success etc. Even if students do not enjoy exercise and there are no significant mean differences, does this mean that exercise is intrinsically bad? Students may react to the instructor (both positively and negatively) more than to the exercise and may equally evaluate different dimensions. Finally, one can collect data for several semesters or even years and compare different student groups. Such data analysis may potentially show how the author’s perfection of exercise resulted in students’ better acceptance of this exercise.

Ideally, it would be good to compare performance of two student groups (control and treatment) taught by the same instructor on some specially designed test. And this is a valid requirement for any research comparing “global” teaching innovations like traditional vs. online teaching etc. However, for simple learning exercises this is an absolutely unnecessary hurdle. First of all, one learning activity may not result in significantly better grades for all topics covered throughout the semester. Second, development of a special test that would measure the effectiveness of this particular exercise is beyond the scope of any author who just wants to share her way of enriching student learning. Any educator based on her experience can assess face value of simple learning exercises: potential outcomes and implementation problems.

I did not think seriously about this issue and always had a formal stat section in my papers to satisfy journals’ requirements until recently – when I received two consecutive rejections with almost the same reasoning. One editor wrote that “small sample and lack of significant results … does not meet the journal’s requirement”, another wrote that small sample “… may be appropriate for another less rigorous journal, but … does not meet our standards…” No single word about the exercise.

After getting two rejections I realized that this problem is much more serious than I previously thought. It looks like a primitive learning exercise supported by tons of non-adding value numbers has a better chance of being published than an outstanding one but without statistical data (I do not state here that my work is outstanding; this is a general thought).

I would like to suggest dropping statistical analysis requirement for all learning exercises. If I am missing something significant, may be you can tell me what it is?

Thank you,

[Name withheld by ELMAR moderator.

Note that as a general rule I don’t like to post anonymously. In this case the young scholar agreed to use his/her name, and at the last second I changed my mind and decided to let it go through without the name.

You can send either an ELMAR response posting or a personal note to the poster by replying to this email. Let me know which one you intend however.

–ch]

---