Abstract

Cluster computing is becoming increasingly important because the size of workloads continues to grow faster than the size of individual machines. In this talk I will argue that:

• The resource demands of emerging workloads (e.g., distributyed Graph analytics) look different from software traditionally deployed on clusters (HPC and distributed/replicated servers).

• With jobs spanning multiple machines, no individual system is in control of traditional OS functions such as scheduling and resource management. This leads to poor interactions (e.g., where cluster-wide scheduling of jobs to machines is unaware of the exact load on individual machines) and wasted resources (e.g., if machines or VMs are statically assigned, but resources go unused).

I will describe some of the trends I am seeing, and research directions I am exploring in the design of distributed runtime systems. This is an informal work-in-progress talk – feedback very welcome.