Google and IBM say we need to train more supercrunchers
There was an article in the New York Times today about the effort that companies like Google and IBM are making to allow university students access to very powerful computing environments to allow engineers and scientists to plow through massive data sets. Their argument is that students are being trained right now to think on a gigabyte scale (if they’re lucky enough to be trained how to analyze real data at all), when all the breakthroughs are happening with datasets in the tera and peta-byte scales.
I couldn’t agree more with this analysis. If people are serious about analyzing those “very rare events”, “long tails” or whatever that can make the difference between a profit and loss, success or failure, or even life or death, then we can’t continue running around assuming things because the model fits 80% of the time and anyways, it’s too hard to do that level of analysis. We all saw what happened with that idea.
When I was working at Lincoln, we created a highly accurate model of U.S. near mid-air collisions. We did this by analyzing about 5 terabytes worth of radar data from across the country (about 8 months worth). Nobody had ever done this before on anything close to that scale.
As a result, we had orders of magnitude more data on near mid-air collisions (a very rare event) than the last model in the early 90′s. Without this data, and the high-powered systems available at Lincoln that we used to analyze it, our model would have suffered from the same assumptions and modeling error as previous attempts, and that is just not good enough for developing something as important as the next generation of collision avoidance systems for manned and unmanned aircraft, which people are now doing at Lincoln, largely as a result of that effort.
The ability to analyze massive data sets has been proven again and again as a competitive advantage in bio-tech, finance (those who do it correctly), internet, and even marketing, making those companies who developed those competencies hundreds of billions of dollars.
Is it then a stretch to say that the next lucrative opportunity in operations management will be to develop the capabilities to harness the massive amounts of data companies already generate every day? I’m talking about everything from inventories to machine control outputs and even to intra-company emails. There are signals in that data, just as there are signals in everything from our DNA to the stock markets, if you look hard enough.
To be honest, I don’t know (I’m new to this stuff!) but that’s why I and several of my classmates are trying to start a new track for LGOs in the EECS department this year called Information and Decision Systems. The focus in this track is to develop the theoretical, practical and communication skills for students who want to take on this operations challenge in the real world, for real companies. That means not just studying and learning the algorithms, but also getting a design background in the networking, database and parallel computing systems that are critical enablers of this type of work. It also means developing specialized communication skills to explain the opportunities and the results, because like the NYT article said, most people have not been trained to think on this scale before.
I could talk for pages more about this topic, but lets just leave it at that for now. I just had to write something because I’m obsessed with this idea, and this article got me all excited. I’m definitely going to look into Hadoop…
MIT 100K Competition
I attended the finals for the MIT 100K business plan competition this past Wednesday. The competitors entries were all very impressive, but I was especially impressed (as was the rest of the audience – they won the audience prize) with the company Global Cycle Solutions who won the development track prize. They basically have come up with a design to easily convert energy from a person pedaling a bike to do other “work”, like charging a cell phone or shelling corn. The idea is that the bike is often the first capital investment that people in poorer countries can make, and this way, they can turn their bike into a mini-business. A person riding a bike in place with this special attachment can shell corn much easier and faster than the usual way (by hand).
Picture of someone using Global Solutions' corn sheller attachment
Although it seems as though not all the applications they purport to be able to do are ready, and the manufacturing of these attachments will be an issue, I think it is a cool idea. I also like the fact that they demonstrated the use of how a little clever engineering could potentially improve the lives of many people in developing countries in a way that is environmentally friendly and personally empowering to boot. Well done!
Overall, the experience was very interesting. The keynote speaker was Rodney Brooks, one of the founders of iRobot, and a professor at CSAIL at MIT – a lab where I hope to be spending a lot of time in the coming two years. He described his experiences as an entrepreneur, and outlined in a very entertaining speech (about halfway through that video) his tongue-in-cheek “PROP” method for starting a business, standing for Passion, Rejection, Opportunity, and Persistence. Having worked for a startup out of college, I can certainly vouch for those qualities being a pre-requisite for a successful venture. In deference to him, however, I would add an additional “M” and “T” in there, forming what I call the “PROMPT” method (my thoughts are purely from my experience – I’d be interested how this jives with the experts at Sloan).
Rodney Brooks
The “M” stands for Management. I think it is romantic to believe that an early-stage company can succeed with poor management, as long as the idea is good. I think this is an especially seductive thought for engineers especially, who are maybe used more to academia and certain work environments where good work is generally recognized and rewarded based on the merits of the work. And I am sure there are certainly plenty of cases where this is true. In my experience, however, poor management (mis-allocation of valuable resources, poor communication with employees, lack of professionalism) can easily sink an early stage company, even if the idea is sound. In addition, the management should be intimately familiar with the product, and ideally be working on the project alongside the employees in its development, at least at first. One question on my mind: can you learn to be a good manager? (Rhetorical answer: I certainly hope so, that’s why I’m going to Sloan!)
The “T” stands for Timing. This is a difficult principle, as timing involves an element of luck. However, when I look back on the development of our product, a telehealth system to deliver preventive care best practices for patients with chronic disease, I wonder what would have happened if we just started the company one or two years later (say 2005 instead of 2003). Back in 2003, the exploding health care costs in this country, although well known, had not seeped into the general consciousness of the country as a national crisis (as it should have been). Moreover, telehealth and data driven health solutions were sort of niche ideas, not a part of the mainstream solution of health care reform as they are now. As such, we had a difficult time selling our system in this country, because nobody was really motivated to pay for it – a well documented failing of the episodic based U.S. health care system as opposed to the “wellness” systems that exists in many other countries where the government is the single payor. In any event, we ramped up (despite the fact that we hadn’t secured any contracts) and by the time Obama got elected and the country finally came around on the idea, the company was bust, rather than waiting patiently with a solution in hand. Now other major players have entered the scene. There is certainly something to be said for the “stealth mode” that seems to be popular now – it certainly increases the flexibility of the company to better time entry into the market place. Operating within one’s means from the beginning, even if it means few employees making steady contributions, increases the liklihood the company will be around long enough to seize opportunities when they arise.
Anyways, I am interested to see how the companies involved in the 100K progress. They certainly seem to have bright futures, and the fact that they have gotten this far should indicate some grasp of the basic principles (aside from luck of course!) and that most valuable of commodities – good press!
