Controversy continues to swirl around ZDNet‘s George Ou’s blog report that compares OpenOffice.org 2.0 with Microsoft Office. Here’s George’s latest entry, Performance analysis of OpenOffice and MS Office:
Microsoft Office came out very lean and fast while OpenOffice.org Office Suite was just the opposite. Some couldn’t accept the numbers and complained that the Task Manager numbers may be inaccurate and hiding memory usage. They demanded more proof, so here it is.
In my previous blog entry I warned readers not to base your choice on such a simple comparison as George Ou’s benchmark of Excel vs. OpenOffice.org Calc. George responded with, among other comments, “All I do is present the data. It’s up to you to interpret it. Just don’t make anything up to refute it.”
OK, George. I have a spare half-hour, so I tried a few unscientific tests that might put all this into perspective. You see, interpretation is exactly the problem. I don’t refute the data in George’s benchmarks, but I do question his conclusion that OpenOffice.org is so much slower and more wasteful of memory of resources than MS Office that Office is worth the $240 price tag.
First, let me point out that I’ve paid for MS Office (or Office upgrades) way more than once. The $240 price is for this year’s model. OpenOffice.org is free — forever. That means upgrades will also be free. Over time, as you learn to be productive with OpenOffice.org, you might save a great deal more than $240.
In fact, I haven’t bought the latest version of Office, so I decided to try my tests with the version I have. I used Word 2002 and Excel 2002.
Second, a real world test might include PCs that are not optimized or tweaked in any way. You might, for example, just choose a random PC and see what happens — a PC that might be infected, or a PC that’s locked down tight. The point is to try this test in the real world, not in a lab. So I chose my son’s Dell Inspiron 4100 laptop, circa 2002.
Power users might disagree with this approach, but most people have an average machine, probably a year old or more, and they have what came with it, which is likely to be an older version of MS Office than what’s available today. So they face a choice: buy the new version of MS Office, use the version of MS Office they already have, or get OpenOffice.org 2.0.
The laptop has a Pentium III (866 MHz) and 256K [correction: 256 MB] of RAM — nothing special. But consider this: the version of MS Office designed in that era for that environment should run well, while OpenOffice.org, designed with more modern PCs in mind (which have more RAM and faster drives), should run slower. This presumably gives MS Office the edge in this test. If OpenOffice.org is truly a memory and resource hog, it would not run as well on an older PC as the version of MS Office designed for that older PC.
So imagine, if you will, that you just want to test how well these programs run without getting into the specifics and details of what these programs do “under the hood”. I tested the Word and Excel components of MS Office vs. the Writer and Calc components of OpenOffice.org.
At first glance, the results seem useless and insignificant. But they point out the folly of relying on benchmarks that measure performance outside the norm. If the difference between one program’s save and another program’s save is the blink of an eye, the difference makes no difference.
1. Application load time. I downloaded and installed OpenOffice.org 2.0, and after installation I shut down and restarted the PC. I then launched Microsoft Word first, then OpenOffice.org 2.0, from the Start menu, and counted the seconds until I could start typing into a new document:
Word: 39 seconds
OpenOffice.org Writer: 27 seconds
2. Open a native text document. For Word, I used a 200 KB Word document — the first chapter of my book, Just Say No to Microsoft (I guess that shows my bias, right?). For OpenOffice.org, I used the exact same chapter saved in OpenOffice.org 1.14 native format, which is only 92 KB:
Word: 7 seconds
OpenOffice.org Writer: 10 seconds
3. Save the document in the native format. I used Word to save the document used in test #2 in the native Word format as a new file, and I used OpenOffice.org 2.0 to save the document in its native format, ODT (OpenDoc format):
Word: 1 second
OpenOffice.org Writer: 2 seconds
3a. At this point I wanted to test how long it took to save a document in a cross-platform format. OpenOffice.org 2.0 lets me save to the Word doc format as well as the XML-based Open Doc format and other formats, so I chose the Word doc format. Word 2002 doesn’t offer an XML-based format in the Save As dialog, so I chose a Mac Word format, which is a bit different than the PC Word format:
Word: 2 seconds
OpenOffice.org Writer: 2 seconds
By the way, it took 3 seconds for OpenOffice.org to open the Word format version of the same document.
4. For my Excel vs. OpenOffice.org Calc test, I used 10YEAR, a mortgage investment analysis spreadsheet, 68K in size, provided as a sample from Mortgage-Investments.com, Inc. I first launched Excel and OpenOffice.org Calc (I didn’t test application load time), then used them first to open the spreadsheet:
Excel: 9 seconds
OpenOffice.org Calc: 8 seconds
5. I then saved the spreadsheet as a new file in the respective programs’ native formats:
Excel: 1 second
OpenOffice.org Calc: 2 seconds
6. Finally, I saved the spreadsheet in a cross-platform XML format — in Word, I used Microsoft XML, and in OpenOffice.org 2.0, I used ODT:
Excel: 2 seconds
OpenOffice.org Calc: 3 seconds
Conclusion: What’s the big deal? OpenOffice.org 2.0 about as fast as MS Office, with very little if any speed differences in real-world environments and with typical files. Clearly there is no justification for declaring OpenOffice.org 2.0 to be more bloated and porky than MS Office. It hardly matters to the average user what the relative memory footprints and CPU usage measurements are — my PC with 256K [correction: 256 MB, must have K on the brain] RAM and a 866MHz Pentium III demonstrates that MS Office and OpenOffice.org are very similar in performance.
Note: I can make these files available to anyone who wants to try this same test. But remember, it’s just as easy to test the programs yourself with typical document and spreadsheet files.
So maybe you need to look at other factors before choosing which one to use. Such as the feature set, the document format, and the price of sticking with MS Office vs. OpenOffice.org, which is free.