A USABILITY STUDY OF MICROSOFT OFFICE 2007 AND MICROSOFT OFFICE 2003 (F117) Nestor J. Rodriguez, Puerto Rico

Note: This was a straightforward study i found worth sharing with focus on the conclusion without going to much into the details and figures presented. To me this is also relevant eLearning-paper even though this was presented on the Interfaces and Human Computer Interaction conference.
Introduction: Upgrade included from Off 2003 to 2007 is a major change in user interface. Upgrade CAN have negative effects (Khoo, 1996).
Objective of study: Assess effects.
Subjects: 50 (college students, professors and staff), each with at least one year of experience as user. 25 in each version
Methods: Initially questionnaire for demography.
Tasks: ca 20-30 per application performed on Word, Excel and Powerpoint.
Post test new questionnaire, with alternatives 1-5, on difficulty and satisfaction
Statistical analysis: Dependent Sample T-test of time to completion
Result: ONLY Excel had significant difference in average completion time, and this was only related to one question of ca 20 («create a graph…»)
Only Word 2007 had significantly higher error rate compared to Word 2003.

Conclusion: NO added significant average user value of upgrade identified! Excel 2007 upgrade had a significant negative effect.

—
Question: I asked the presenter if his conclusion would change with the one outliner resulting from the test question about creating graphs set aside.
His reply: «Then there is no significant difference in average would result also for Excel.»

My remark here; I do not think they should lay so much weight on this one outliner data result, as a new set of questions could give a new outliner in PPT or Word.
This however does not take away my interest for the case study that points towards no real added value by upgrade. (But I do suppose MS would like to address the undisclosed graph tool challenge if they have not already done so in Office 2010. New comparisons 2010 vs 2007/2003 would be of some interest, I believe.)