My Real World Test: MS Office vs. OpenOffice

Controversy continues to swirl around ZDNet‘s George Ou’s blog report that compares OpenOffice.org 2.0 with Microsoft Office. Here’s George’s latest entry, Performance analysis of OpenOffice and MS Office:

Microsoft Office came out very lean and fast while OpenOffice.org Office Suite was just the opposite. Some couldn’t accept the numbers and complained that the Task Manager numbers may be inaccurate and hiding memory usage. They demanded more proof, so here it is.

In my previous blog entry I warned readers not to base your choice on such a simple comparison as George Ou’s benchmark of Excel vs. OpenOffice.org Calc. George responded with, among other comments, “All I do is present the data. It’s up to you to interpret it. Just don’t make anything up to refute it.”

OK, George. I have a spare half-hour, so I tried a few unscientific tests that might put all this into perspective. You see, interpretation is exactly the problem. I don’t refute the data in George’s benchmarks, but I do question his conclusion that OpenOffice.org is so much slower and more wasteful of memory of resources than MS Office that Office is worth the $240 price tag.

First, let me point out that I’ve paid for MS Office (or Office upgrades) way more than once. The $240 price is for this year’s model. OpenOffice.org is free — forever. That means upgrades will also be free. Over time, as you learn to be productive with OpenOffice.org, you might save a great deal more than $240.

In fact, I haven’t bought the latest version of Office, so I decided to try my tests with the version I have. I used Word 2002 and Excel 2002.

Second, a real world test might include PCs that are not optimized or tweaked in any way. You might, for example, just choose a random PC and see what happens — a PC that might be infected, or a PC that’s locked down tight. The point is to try this test in the real world, not in a lab. So I chose my son’s Dell Inspiron 4100 laptop, circa 2002.

Power users might disagree with this approach, but most people have an average machine, probably a year old or more, and they have what came with it, which is likely to be an older version of MS Office than what’s available today. So they face a choice: buy the new version of MS Office, use the version of MS Office they already have, or get OpenOffice.org 2.0.

The laptop has a Pentium III (866 MHz) and 256K [correction: 256 MB] of RAM — nothing special. But consider this: the version of MS Office designed in that era for that environment should run well, while OpenOffice.org, designed with more modern PCs in mind (which have more RAM and faster drives), should run slower. This presumably gives MS Office the edge in this test. If OpenOffice.org is truly a memory and resource hog, it would not run as well on an older PC as the version of MS Office designed for that older PC.

So imagine, if you will, that you just want to test how well these programs run without getting into the specifics and details of what these programs do “under the hood”. I tested the Word and Excel components of MS Office vs. the Writer and Calc components of OpenOffice.org.

At first glance, the results seem useless and insignificant. But they point out the folly of relying on benchmarks that measure performance outside the norm. If the difference between one program’s save and another program’s save is the blink of an eye, the difference makes no difference.

1. Application load time. I downloaded and installed OpenOffice.org 2.0, and after installation I shut down and restarted the PC. I then launched Microsoft Word first, then OpenOffice.org 2.0, from the Start menu, and counted the seconds until I could start typing into a new document:

Word: 39 seconds
OpenOffice.org Writer: 27 seconds

2. Open a native text document. For Word, I used a 200 KB Word document — the first chapter of my book, Just Say No to Microsoft (I guess that shows my bias, right?). For OpenOffice.org, I used the exact same chapter saved in OpenOffice.org 1.14 native format, which is only 92 KB:

Word: 7 seconds
OpenOffice.org Writer: 10 seconds

3. Save the document in the native format. I used Word to save the document used in test #2 in the native Word format as a new file, and I used OpenOffice.org 2.0 to save the document in its native format, ODT (OpenDoc format):

Word: 1 second
OpenOffice.org Writer: 2 seconds

3a. At this point I wanted to test how long it took to save a document in a cross-platform format. OpenOffice.org 2.0 lets me save to the Word doc format as well as the XML-based Open Doc format and other formats, so I chose the Word doc format. Word 2002 doesn’t offer an XML-based format in the Save As dialog, so I chose a Mac Word format, which is a bit different than the PC Word format:

Word: 2 seconds
OpenOffice.org Writer: 2 seconds

By the way, it took 3 seconds for OpenOffice.org to open the Word format version of the same document.

4. For my Excel vs. OpenOffice.org Calc test, I used 10YEAR, a mortgage investment analysis spreadsheet, 68K in size, provided as a sample from Mortgage-Investments.com, Inc. I first launched Excel and OpenOffice.org Calc (I didn’t test application load time), then used them first to open the spreadsheet:

Excel: 9 seconds
OpenOffice.org Calc: 8 seconds

5. I then saved the spreadsheet as a new file in the respective programs’ native formats:

Excel: 1 second
OpenOffice.org Calc: 2 seconds

6. Finally, I saved the spreadsheet in a cross-platform XML format — in Word, I used Microsoft XML, and in OpenOffice.org 2.0, I used ODT:

Excel: 2 seconds
OpenOffice.org Calc: 3 seconds

Conclusion: What’s the big deal? OpenOffice.org 2.0 about as fast as MS Office, with very little if any speed differences in real-world environments and with typical files. Clearly there is no justification for declaring OpenOffice.org 2.0 to be more bloated and porky than MS Office. It hardly matters to the average user what the relative memory footprints and CPU usage measurements are — my PC with 256K [correction: 256 MB, must have K on the brain] RAM and a 866MHz Pentium III demonstrates that MS Office and OpenOffice.org are very similar in performance.

Note: I can make these files available to anyone who wants to try this same test. But remember, it’s just as easy to test the programs yourself with typical document and spreadsheet files.

So maybe you need to look at other factors before choosing which one to use. Such as the feature set, the document format, and the price of sticking with MS Office vs. OpenOffice.org, which is free.

Share

Comments

My Real World Test: MS Office vs. OpenOffice — 15 Comments

  1. You answered your own question. “so I tried a few unscientific tests that might put all this into perspective”

    Unscientific is the key issue here, and it explains your erratic results. Like I said, doing a non-stressful benchmark test (besides being inaccurate) is like having a college entrance exam based only on simple single integer arithmetic.

  2. What question was that? And what was erratic about my results? They reflect real world activities using a commonplace PC and ordinary files. I think what you mean is that since it was “unscientific” it is also meaningless. Sigh. The point is that your conclusions don’t reflect real-world usage. For example, nothing in my test took 10 seconds (or more) except loading the apps themselves. Your contention that smaller files yield proportionately similar results just doesn’t seem to be true.

    Thanks for writing.

  3. Your 256k of memory on the machine seems implossible with a P3… For a 386 it even seems low.

    So I have to assume you mean 256mb (about 1 million times more memory).

    If I have to make this assumption, the rest of your results seem suspect as well.

    Unscientific is right… why not just use tarot cards next time?

  4. Thanks for pointing out my typos — I typed 256 K twice, meaning 256 MB.

    As for tarot cards, why not? The benchmarks I read in George Ou’s blog were about as useful. So have you tried opening regular documents in Word vs. OpenOffice.org Writer, or regular spreadsheets in Excel vs. OO.o Calc? I bet your results would not be that different from mine — using a Pentium III or more powerful machine.

  5. “The point is that your conclusions don’t reflect real-world usage.” Can you tell me why starting an application, loading and saving files doesn’t equate to real-world usage? I seem to end up doing those things while I’m at work…

    I would have liked to see some more comparisons, I may forgive OpenOffice’s long load times if it calculated the formulas I use in less time than excel.

  6. Tony, bloggers of all camps – greetings !

    Interestingly enough, I also did my real world tests : and here are my results (and some may then debate where the “real” bloat lies) :

    On my brand-spanking-new powerhouse : a shiny IBM 300GL (PII-333Mhz, 256Mb of ram, and 2 x 5400rpm disks (hda=4gb, hdb=2gb), I installed OpenOffice 1.1.4.
    I loaded a file, unusual by daily business operations (read I needed the biggest file sent my way in the last months, granted I am a UNIX Sys Admin, and don’t receive that much fluff mail), thus I got this server_room_temp.doc file created by a supplier with MS-Office/Word XP.
    The native file was a 128 page document (basically a table) and weighed in at 2976 Kb (almost 3 Mb).

    Here we go. On this vastly underpowered (by today’s power glutton standards)
    1) OpenOffice_Writer launched in 31 seconds
    2) it took 1m50s to load the initial file.
    3) another 1m03s to convert it to *.sxw
    -> which made it deflate to 35 kb. (85 times smaller !!!)
    4) reloading this new file (still all 128 pages of it) took yet another 1m30s.

    Handling more “common” documents, and since I live in a non-friendly M$-eclusive environment, I tend to open, edit, save all documents in *.doc format – which OpenOffice conveniently lets me set as a default. (I know, shame on me, but the client insists). Even so, and dealing my with app’s non-native format, I have yet to :
    1) hit a compatibility glitch
    2) have had to wait forever to load a file
    3) have a receiver, reply that my file was un-readable

    I have however, “fixed” files on more that 1 occaison, where my colleague opened a *.doc – that trashed his system (a PIV/1Gb ram). I simply opened it, saved-as (added “2” at the end) and returned it. Voilà, fixed !

    So bloat for bloat, and cost for cost, I will definately stick with my old iron, and slow/bloated OpenOffice.

    For those who says costs doesn’t matter, if it didn’t and performances was all that mattered, we would all drive Mercedes and Porsches … ahem !

    LB.
    _________________________________________________
    PS: Space on disk also counts : Office 2003 (approx 261MB) OpenOffice1.1.4 (approx 217MB).

  7. George Ou:
    Can you explain why your numbers are so high? 15 times more CPU time, that it 1500% difference!
    I’m not claiming your numbers are invalid, I’m just damn curious where is the difference coming from.
    BTW, I did my own test, on 10 mb spreadsheet and difference wasn’t that huge. That doesn’t change fact that OO suck ^_^

  8. Wow! I can’t believe the responses from George and company here.

    Your tests quite validly show that for normal usage, it really doesn’t matter. Only folks hanlding very large or very complex documents should pay attention to the differences between the two suites.

    In on of George’s blogs, he says why worry about the other 10% of users who don’t use .doc format? Well, I could easily turn that question around here. Why should the rest of us care about the 1 or 2% of the users who need to handle very large office files?

    The fact is, paying attention to such minutiae (sp?) means you’re paying attention to the computer equivalent of how many angels can dance on the head of a pin. So many other points are more important than these ones. Plus, what good is fast document handling if you lose all that saved time with your system being down and fixed by the company computer tech.

    Where I work we have IBM Thinkpad 42s. The ones running linux just work. period. They go in for work when the hardware goes bad, like a battery or hard drive failure. The ones running XP get to visit the help desk about once every three months for OS / Software failure. And about half the time the fix is to wipe and reinstall.

    So, I could argue that those people should all be running linux. But the XP users use software that’s not available on linux, or the linux running counterparts are not mature and don’t provide the function they need, so they run XP.

    If we were choosing simply on office suites, they’d probably be happy with Linux and OOo. The differences sited in George’s article would be considered “noise” in that environment.

    And George, questioning the science, when you trusted Microsoft to tell you the truth? Come on, they could easily have a have in their own OS that hides the memory usage or loads libs into “system” memory thus making the application look smaller.

    The only “scientific” way to know is to use a debugger (not the one that comes with VCC, it should be considered tainted) and SEE WHAT HAPPENS when you load and run the program. That would be sicentific. Trusting Microsoft is about as far from scientific as you can get…

  9. Here’s my real world *scientific* test.

    Since I don’t have MS here at home I’ll substitute Corel WordPerfect 10 for Office 2000 [used at the office].

    I’ll also benchmark a POT editor.
    POT is a telephone term for *plain ol’ telephone*, so here it is Plain Old Text.
    I used PSPad [www.pspad.com] which is only one of several I use depending on what I am doing as it opens files faster than others, but not if I open 100+ emails files at the same time which a different editor will do faster [that sounds a lot like this discussion-not just here but all over].

    My timer is setting my watch on the desk and pushing the tiny-weenie button to start and stop the timer using the left thumb nail while the other fingers secure the watch to the desk top.

    The time started while simulaneously clicking [After R-clicking the filename and using *open with* application] the mouse on the file while selecting it with Exploder [hate My’puter] and the watch button.
    The time stopped when the program had the file open and was ready to use.

    Each test was run about 3 times with a marginal, say +/- 10%, deviation.

    Test File: 23,715,840 bytes
    167,129 records [lines] of 512 character fixed length [name, address, phone…..] fields.
    This is the 911 phone records we work with everyday.

    System:
    Built: Feb 2005
    MB: ChainTech VN4F
    CPU: AMD64 3500+
    Mem: 1gig Rosewill Dual-Channel PC3200
    Vid: Radeon X400 PCIe
    HD[2]: Maxtor 200gig SATA150 [16meg cache]
    OS: ‘doze 2k SP4 [full patch]

    PSPad—————————
    Open: 1.6 seconds [smooth scrolling all the way to the bottom]
    just a plain text editor [www.pspad.com] version 4.3.3
    memory used 38,568K CPU 0% with 1sec time and never got above 5% while scrolling the file

    WP 10———————–
    Open: 6.5 seconds [scrolling sucks, jumps, hangs, etc for several seconds trying to get to bottom page 14857 pages]
    memory used 263,700K CPU idled at 0% after 11sec time

    OO2————————–
    Open: 18.1 seconds [smooth scrolling 3095 pages]
    memory used 124,124 after 1:00 cpu time cpu went to 99% and stalled meaning a “hard kill” from the taskbar button resulting in a “recover document” on restarting OO2

    The page counts are from the default margin and fonts from the initial install, I did not try to make them the same.

    After thoroughly reviewing all the test results my affirmed conclusion:

    scrap all the bloated, crash prone, crap-in-its-diaper, cuties-wootsy eye candy and just stay with a POT application.

    Put that in your pipe and smoke it…………..

    PS: I ablsolutely despise HTML email, I’m getting ocular diabeties from all the eye candy.

  10. Benchmarks don’t always reflect the real world. For example, the Win32 application MS Outlook 2003 vs the web based Exchange 2003 Web Access. They both provide access to may email data, they’re both fast enough, and the Exchange 2003 web access GUI looks and works just like the Outlook 2003 application, even down to right click context menus.

    There’s some difference, of course, but for reading my current inbox email, they both get the exact same job done.

    win32 outlook 2003
    private bytes 40mb
    peak private 40mb
    working set 64mb
    peaking working set 64mb
    peak cpu usage 85%
    load time 8seconds

    iexplore exchange 2003 web access
    private bytes 16mb
    peak private 16mb
    working set 27mb
    peak working 27mb
    peak cpu usage 37%
    web page load time (auto login), 4seconds.

    So based on this contrived example, we should immediately stop buying PC installable application and switch to web based GUI applications. That’s not the point however.

    Both applications get the job (reading email) done, both are responsive enough that they don’t feel sluggish (the resource hog is noticibly faster loading new emails), and even though the benchmarks identify one to be clearly faster load time and less of a resource hog, IT DOESN’T MATTER TO THE END USER.

    End user perception of performance counts far more than emperical data. If it looks fast, seems fast, then it IS fast. No, this isn’t scientific to base a benchmark on perception, but it’s how people in the real world will judge the software.

  11. “So I have to assume you mean 256mb (about 1 million times more memory).”
    You youngsters don’t know your numbers anymore. One K is 1024 bytes, one k 1000 bytes. One M is 1024 * 1024 bytes, one m (finally) one million. So if the admin says, he has 256 KB (not Kb, which is 1024 bits) and he has actually 256 MB, he has 1024 times more, not one million..

    Hans

  12. Good grief. Who cares if OpenOffice.Org is “slower” than MS Office? We’ve grow such short attention spans that we can’t stand to wait a few seconds with nothing to do.

    Who cares if OO uses more memory? RAM is cheap. I have tons of it, and so do most people these days.

    OO is fantastic, secure software that does everything I need and more—and it’s free. Download OO, or put $200 more in Bill Gates’s obscenely-swollen pockets? Next question?

  13. why does everyone keep bashing microsoft? people keep referring to microsoft bloat, and bill gates getting even more riches etc, but how many of you have stopped to consider that bill gates is one of the most philanthropic business men you’ll have have the pleasure of knowing? he has put literally billions of dollars into researching diseases that mainly affect poor people around the world, and mostly africans (not that any of you would care about that). this money has been used in fighting diseases such as malaria, small pox, aids prevention and lots more, as well as education charities – these are making real life differences to really poor and unfortunate people. some might argue that he has the money to do it – yes that’s true, but how many sun or oracle or even open source companies for that matter have made even a cent of a contribution to the worlds poor? you guys haven’t got a clue what you’re talking about – you hear people talking badly about ms, and you just jump on the bandwagon and follow like puppets. i buy microsoft products happily, ‘cos i know some of it is going to go to people in need – yes, a rather radical thought, but if you do some real research instaed of just “bashing the obvious”, perhaps you’ll spend some of your time and emoptional energy a little more positively – wake up and smell the coffee!

  14. DJ,
    Following your procession of thought I have deduced that it would be wiser to return my copy of M$Office 2003, use OOo as a free alternative, and give $240 directly to charities myself. It is in this way that 100% of my good intention is donated and 100% is actually received instead of a mere fraction of one percent.

    Also, I would like to point out that OOo is making real life differences to really poor and unfortunate people as well, specifically those who can’t afford $240 worth of [unreliable] software to manage their information and perform simple business math.

    Also, I think it will suffice to argue that your your time could be better spent being active in a community of producers rather than buying into the FUD, jumping on the proverbial band wagon, and handing your money over to someone who has adopted deception as a business strategy and delivers an unreliable product.

    -Princess Mike

    P.S. Be a consumer whore or I’ll trow a chair at you! I’ll fxcking bury you! *snickers*

  15. Relating to the previous statement: It’s easy to say that I would rather send my money directly to charity, but are you actually going to do it? I asked myself that question and i’m sorry to say the answer is no. I live in South Africa, more than once have I heard about MS donating money to some poor nation north of the border. So it’s not all bad.

    I’m an administrator of a company. I found the best balance in my network is to have a linux backbone and Windows Workstations. I don’t agree with the server licencing policies of Microsoft. Having the stability of Linux servers and the ease of use of Windows workstations have reduced my time spent on dealing with user issues.

    One thing that I have noticed. Users attitudes towards office software is the main problem. I forced the users to switch to OpenOfffice i’ve never had so many problems. Most problems from users who can’t or don’t want to remember what formats to send documents on. That resulted in our clients not being able to open mail. After repeated attempts to teach the users how to send and save in other formats, it only lasts for about a week.

    From a administration point of view, it was worth the money for the company to switch to microsoft due to the complexity and confusion of the formats the keeps changing on OpenOffice. The sad fact still stands that Microsoft formats are the standard in the world. I heard Microsoft agreeing that the world needs a standard format. I hope that becomes a reality so that people can enjoy true flexiblity in the choice of office suits.

    One thing I must mention is that the computers in my network running office tends to prove very reliable if you just keep the software updated. While users have complained about Open Office taking too long esp if you work with big spreadsheets and someone is waiting on the phone for you to open a document that is quite big. I can say that I have seen it in a real working world. You can do all the lab test you want. Speed is something they have to improve.

    Open Office is quite good with docoment recovery. Files that are corrupt in Office can be easily repaired and saved so I keep a copy of Open Office installed on my pc to help users when they receive corrupt files.

    Both suits has got benefits, it’s just up the environment and users if it’s going to work.

Leave a Reply

Your email address will not be published. Required fields are marked *