I am sincerely sorry that I missed this discussion when it occurred, and have only seen it resurface with a bump. Perhaps I should just leave this pass, but I can’t resist comment here. It’s so rare for a topic to rise where I genuinely feel like the authority.
It should have been (and was to all but one person) obvious that Excel was/is plenty sufficient for Yogi’s need. Aero51 probably was quite sincere in his comments, just completely misreading the situation in this thread. This is a familiar type in most workplaces, I am guessing - instead of just worrying about practical matters and getting shit done for the task at hand, they would rather bitch about how much better (thing they like) is than (thing they have to use). It’s the computer-programming version of being a hipster.
I work with another statistician like that - spends half the time bitching about how Microsoft products suck, says that everything should run on Linux and just generally talks in a similar tone to the tone Aero showed in this entire thread. You know what? He’s right - but, the cardiologists and surgeons that we work with don’t give a shit about how much better the hospital records would be if he could build a new Linux-based network to host them. They want to know, if they give me a data sheet in Excel (because that’s what they know how to use), can I produce the results that we need for our paper? And the answer is, yes, easily. If a higher level of analysis is needed (which it usually is), it will take me about 2 seconds to read the data into my fancy-statistical-software to do my fancy-statistical-stuff. I’ll usually do that anyway, because I can get results faster, but they don’t need to know that, they only need to know that I can get the results.
Basically echoing twojarslave and usmc, among others, Excel has limitations but there’s a pretty minimal learning curve to get up and running for basic functions - which is perfect. Excel is ubiquitous software. It’s on EVERYONE’s computer. It’s not built to house Big Data or to do PhD-level statistical analysis, but it is FINE for the purposes described here. If the desired task requires something that’s not easily workable in Excel, they should probably be hiring a professional statistician (or someone with the proper analytic and/or computer programming knowledge) to help them do it anyway.
Do I occasionally chuckle at some of the clumsy attempts to perform data-analysis in Excel? Ha, yes, of course, because I know better ways to get it done. But that doesn’t entirely invalidate the utility of Excel!
Aero, I’ll cut off your “how much REAL programming experience do you have?” question with “Plenty, and a PubMed search can attest to the actual results of my programming experience.” Do I know that other programs are far superior to Excel for certain functions? Of course! Do I perform any of my statistical analysis in Excel? NO! But am I willing to help doctors set things up in Excel, then take over with the more complex stuff? YES! That’s significantly easier and more productive than trying to make them learn something else they have no expertise in.
I worked on one project at my first job where the physicians had set up a complex data-entry interface through a third-party where the data entry was done on a web-based portal. They were paying some company $15,000 a month for this - just to build and host the site! They still had to pay their own research stuff to type the data into the thing. Honest to God, they could just as easily have typed the data into Excel sheets and called it a day, no need for the $15K monthly expense. The scale of the project and data complexity is a factor here - for major multi-site clinical trials, that setup makes a lot of sense. But for a simple study like they had, it was a waste of hundreds of thousands of dollars. That money should have been funneled into paying one additional research nurse to help with the data-entry burden, and a small chunk of my salary to run a couple of simple data-quality checks a month.
And, in conclusion, that brings us full circle: the scale of the problem matters, and for the type of problems being discussed here, Excel works JUST FUCKING FINE.
(Also, sidebar: I chuckled at usmc’s description of the cut/copy/paste function because I have seen MEDICAL DOCTORS do exactly the same thing - the “manual” copy and paste to try and move data around in their spreadsheet)