Doing “Big Science” In Academia

Recently, there has been a lot of handwringing in the systems community about the work that we can do in the age of mega-scale data centers and cloud computing.  The worry is that the really interesting systems today consist of tens of thousands of machines interconnected both within data centers and across the wide area.  Further, appropriate system architectures are heavily dependent on the workloads imposed by millions of users on particular software architectures.  The worry is that  we in academia cannot perform good research because we do not have access to either systems of the appropriate scale or application workloads to inform appropriate system architectures.

The concern further goes that systems research is increasingly being co-opted by industry, with many (sometimes most) of the papers in top systems and networking conferences being written by our colleagues in industry.

One of my colleagues hypothesized that perhaps the void in the systems community was partially caused by the void in “big funding” that was historically available to the academic systems community from DARPA. Starting in about 2000, DARPA moved to more focused funding to efforts likely to have direct impact in the near term.  Though, it looks that this policy is changing under new DARPA leadership, the effects in the academic community have yet to be felt.

My feeling is that all this worry is entirely misplaced.  I will outline some of the opportunities that go along with the challenges that we currently face in academic research.

First, for me, this may in fact be another golden age in systems research, borne out of tremendous opportunity to address a whole new scale of problems collaboratively between industry and academia. Personally, I find interactions with my colleagues in industry to be a terrific source of concrete problems to work on.  For example, our recent work on data center networking could never have happened without detailed understanding of the real problems faced in large-scale network deployments.  While we had to carry out a significant systems building effort as part of the work, we did not need to build a 10,000-node network to carry out interesting work in this space.  Even the terrific work coming out of Microsoft Research on related efforts such as VL2, DCell, and BCube typically employ relatively modest-sized system implementations as proofs of concepts of their designs.

A related approach is to draw inspiration from a famous baseball quote by Willie Keeler, “I keep my eyes clear and I hit ’em where they ain’t.” The analog in systems research is to focus on topics that may not currently be addressed by industry.  For example, while there has been tremendous interest and effort in building systems that scale seemingly arbitrarily, there has been relatively little focus on per-node efficiency.  So a recent focus of my group has been on building scalable systems that do not necessarily sacrifice efficiency.  More on this in a subsequent post.

The last, and perhaps best, strategy is to actively seek out collaborations with industry to increase overall impact on both sides. One of the best papers I read in the set of submissions to SIGCOMM 2010 was on DCTCP, a variant of TCP targeting the data center.  This work was a collaboration between Microsoft Research and Stanford with the protocol deployed live on a cluster consisting of thousands of machines.  The best paper award from IMC 2009 was on a system called WhyHigh, a system for diagnosing performance problems in Google’s Content Distribution Network.  This was a multi-way collaboration between Google, UC San Diego, University of Washington, and Stony Brook.  Such examples of fruitful collaborations abound.  Companies like Akamai and AT&T are famous for multiple very successful academic collaborations with actual impact on business operations.  I have personally benefitted from insights and collaborations with HP Labs on topics such as virtualization and system performance debugging.

I think the big thing to note is that industry and academia have long lived in a symbiotic relationship. When I was a PhD student at Berkeley, many of the must read systems papers came out of industry: the Alto, Grapevine, RPC, NFS, Firefly, Logic of Authentication, Pilot, etc., just as systems such as GFS, MapReduce, Dynamo, PNUTS, and Dryad are heavily influencing academic research today.  At the same time, GFS likely could not have happened without the lineage of academic file systems research, from AFS, Coda, LFS, and Zebra to xFS.  Similarly, Dynamo would not have been as straightforward if it had not been informed by Chord, Pastry, Tapestry, CAN, and all the peer to peer systems that came afterward.  The novel consistency model in PNUTS that enables its scalability was informed by decades of research in strong and weak data consistency models.

Sometimes things go entirely full circle multiple times between industry and academia.  IBM’s seminal work on virtual machines in the 1960’s lay dormant for a few decades before inspiring some of the top academic work of the 1990’s, SimOS and DISCO.  This work in turn led to the founding of VMWare, perhaps one of the most influential companies to directly come out of the systems community.  And of course, VMWare has helped define part of the research agenda for the system’s community in the past decade, through academic efforts like Xen.  Interestingly, academic work on Xen led to a second high-profile company, XenSource.

This is all to say that I believe that the symbiotic relationship between industry and academia in systems and networking will continue.  We in academia do not need a 100,000-node data center to do good research, especially by focusing on direct collaboration with industry where it makes sense and otherwise on topics that may not be being directly addressed by industry.  And the fact that there are so many great systems and networking papers from industry in top conferences should only serve as inspiration, both to define important areas for further research and to set the bar higher for the quality of our own work in academia.

Finally, and only partially in jest, all the fundamental work in industrial research is perhaps further affirmation of the important role that academia plays, since many of the people carrying out the work were MS and PhD students in academia not so long ago.

9 Responses to “Doing “Big Science” In Academia”

  1. 1 Matt Welsh June 15, 2010 at 7:53 pm

    Amin, since I’m the one who wrote the post which you’re referring to, I feel compelled to reply 🙂 I’m glad you are so positive about the symbiosis between industry and academia. I’ve gotten a lot out of my relationship with industry as well, although honestly the sensor nets area has had a lot less industry activity than, say, datacenters. Clearly I’ve been working on the wrong stuff. But the main question I have is whether academics should be working on these problems — whether they are not better left to the Googles and Microsofts and Yahoos of the world — whether academia is doing itself a disservice by “competing” with industry (I mean that somewhat in jest), rather than working on problems that are not yet on their radar screen.

    I’m all for a healthy relationship between industry and academia but I wonder if (a) industry isn’t better equipped to tackle those problems and (b) whether academia is selling itself short by working on the near-term stuff.

    Curious to know what you think.

    • 2 aminvahdat June 15, 2010 at 9:00 pm

      Matt, I think that the problems that the big industrial players can solve on their own, especially those that have more of a short-term flavor, are best left to industry. However, there are often opportunities for applying some “basic science” to industrial problems for mutual benefit. Let me give two examples, outside the context of my own work to make me seem impartial. First, as I referenced in my post, Harsha Madhyastha and his colleagues had been independently developing iPlane for many years, addressing basic questions in characterizing the Internet in a scalable, efficient manner. Through collaboration with Google, they found a way to apply similar techniques to a pressing problem that Google was facing with its CDN. Both parties won in this case. Harsha and colleagues got tremendous impact from their work and Google has improved performance for its infrastructure.

      As a second example, my colleagues at UC San Diego had been performing basic security research in using machine learning techniques to identify malicious URLs (see: A large Internet company became aware of the work and through an active collaboration, every piece of email received by this company is now run through the algorithms developed as part of this earlier work as part of Spam filtering. My colleagues are even getting feedback on what works and what does not work in refining their research moving forward.

      The moral of the story here is that “short-term” impact is not to be avoided if it is applying basic research to solve real problems. One should worry in academia in the case where the collaboration involves doing things that most engineers in the company could do anyway.

  2. 3 Rajesh Gupta June 23, 2010 at 7:53 am

    This is somewhat orthogonal, but sometimes I feel that we in computer science get too caught up in grandstanding implied by “Big” or “Transformational” or “Fundamental” when promoting projects.

    Real science is often very incremental, built upon current state of knowledge, with impacts that are cumulatively great. Nevertheless, I understand the spirit of the question, actually two questions: a. should we solve near-term industry problems in academia? b. should problems that require huge industrial resources be a barrier to selecting problems by the academia. The second is almost a rhetoric question since it is really a quality issue that reverts back to first question. On that count, ironically, sometimes I have seen choosing to solve precisely such a problem is cited as evidence of doing something relevant or useful. At DARPA, there used to be a measure sometimes used in describing a launched program: how many years does it accelerate the technology advancement compared to left as a problem to industry alone. That seems to me a good way to define a project worth pursuing.

    • 4 aminvahdat June 23, 2010 at 7:48 pm

      Hi Rajesh, you make some great points. I fully agree with you that great science is often incremental. Even when the P versus NP debate is finally settled, it will be on the back of thousands of individual important contributions that finally provide the right tools to solve the problem.

      I did not equate “big science” with going after some incredible breakthrough. Rather, I was addressing the question that a number of people have been grappling with in the systems/networking community: whether it makes sense to “compete” with industry on large-scale systems architecture. I definitely agree with you that one great metric for academic work is how far it can accelerate development in industry. Only trouble is that it is very difficult metric to quantify.

  3. 5 A.J. June 24, 2010 at 11:24 am

    I have one point that I hope is valid. Systems research (at least some of it) lags behind compared to the industry. There is a magnificent industry with millions of talented software engineers and architects whose primary job is to build massive, scalable, distributed architectures that are, at the same time, robust and (multi-platform) production quality. They do not publish/patent most of their work (even if they do, academics don’t read/cite patents anyway).

    Of course, we aren’t even talking about the emulations and prototypes that the systems research community does to support claims in papers; and the question of reproducibility of results. How many times have you had the chance to play with someone else’s research prototype? .. and how many times have you run a research prototype that worked perfectly well the 10 times you ran it (leave alone working in varied conditions)?

    Given that the research that goes on behind industry output is not published, it can be argued that my statement above of systems research lagging industry is not substantiated or cannot be verified. I have personally worked on projects that were in production, years before there were SOSP papers which “defined” the problem. What is the point of “repeating” something (to a mere prototype) that *may* have already been done to perfection?

    • 6 aminvahdat June 24, 2010 at 2:34 pm

      Certainly very good points. Unfortunately however, science does not progress unless the ideas are widely disseminated and published. How does the community know the important set of problems to focus on? The goal of basic research is to build the “baseline” upon which everyone can make progress. Otherwise, the rate of progress in industry would be much slower.

      Certainly your points regarding the maturity of research software relative to industrial software is very valid. And certainly there are cases where academia lags behind industry but as pointed out in my post above, there are also cases where academia leads industry (consider Google coming out of Stanford for instance, Inktomi out of Berkeley, or Akamai out of MIT). And there are of course cases where fundamental breakthroughs in academia, even if demonstrated through emulation or small-scale experiments, winds up heavily influencing industry (Distributed Hash Tables come to mind as a recent example from systems and networking).

      The point is that the relationship is synergistic.

      At the same time, there are cases where

      • 7 A.J. June 24, 2010 at 4:52 pm

        Amin, your reply seems to have been truncated (word limit?).

        I completely agree with you that there are many fundamental contributions that came from academia, which significantly changed (and at times created) the industry. My only concern is influence in the other direction: academia -> industry is a very open environment (the industry can actually patent small variations of significant academic papers); while industry -> academia is somewhat closed due to patents, intellectual property, non-disclosure, etc. This observation seems to hold more in systems than in other areas of CS. I guess there’s not much we can do to change that.

  4. 8 Donn Lee July 19, 2010 at 9:50 am

    Nice note, Amin. I enjoyed it.


  5. 9 anon February 16, 2011 at 12:05 am

    In data center networking, does academia lag behind industry a lot?

Comments are currently closed.

Amin Vahdat is a Professor in Computer Science and Engineering at UC San Diego.

June 2010

%d bloggers like this: