Virtuoso: Resource Management and Prediction for Distributed Computing Using Virtual Machines

Virtual machines provide a powerful new layer of abstraction in distributed computing environments. Entire "machines" running arbitrary operating systems and software can now be stored to disk files, migrated over the network, and instantiated on arbitrary physical machines. This freedom creates new opportunities and challenges for scheduling and resource management. Intriguingly, this is true both from the perspective of resources "looking up" at applications and applications "looking down" at resources. The Virtuoso project explores both of thse perspectives. From the resource perspective, we are developing mechanisms that will make it easy for resource owners to carefully control when and how the resources are used by virtual machines. From the application perspective, we are extending our existing work in the measurement and prediction of static and dynamic resource availability to include virtual machines.

This project is in collaboration with a companion project run by Renato Figueiredo and Jose Fortes at the University of Florida.

People

  • Peter Dinda
  • Ashish Gupta
  • Ananth Sundararaj
  • Bin Lin
  • Jack Lange
  • Alex Shoykhet
  • Brian Cornell
  • Sam Rossoff
  • Blair Heuer
  • Jay Bruins
  • Benjamin Prosnitz

    • Movies

  • Virtuoso Web Interface (AVI, 79 MB)
  • VNET Virtual Network (AVI, 51 MB)
  • VTTIF Inference Tool detecting all-to-all (AVI, 9 MB) and tree (AVI, 8 MB) topologies
  • VSched Real-time Scheduler (Real, 11 MB)

    • Talks

  • Transparent Network Services via a Virtual Traffic Layer for Virtual Machines, (HPDC 2007) (powerpoint)
  • Time-sharing Parallel Applications With Performance Isolation And Control, (ICAC 2007), (pdf)
  • The User In Experimental Computer Systems Research, (ExpCS 2007) (powerpoint)
  • Towards Scheduling Virtual Machines Based On Direct User Input, (VTDC 2006) (pdf slides)
  • Free Network Measurement for Adaptive Virtualized Distributed Computing , (IPDPS 2006) (powerpoint)
  • Hardness of Approximation and Greedy Algorithms for the Adaptation Problem in Virtual Environments, (ICAC 2006), (powerpoint)
  • Towards an Understanding of Decision Complexity in IT Configuration , (ICAC 2006), (powerpoint)
  • VSched: Mixing Batch and Interactive Virtual Machines Using Periodic Real-time Scheduling, (SC 2005) (powerpoint)
  • Free Network Measurement for Adaptive Virtualized Distributed Computing , (SC 2006) (powerpoint poster)
  • Increasing Application Performance in Virtual Environments Through Run-time Inference and Adaptation, (HPDC 2005) (powerpoint)
  • Automatic Dynamic Run-time Optical Network Reservations, (HPDC 2005) (powerpoint)
  • An Optimization Problem in Adaptive Virtual Environments, (MAMA 2005) (powerpoint)
  • User-driven Scheduling of Interactive Virtual Machines, (Grid 2004) (powerpoint) and related preliminary poster on VSched (powerpoint)
  • Dynamic Topology Adaptation of Virtual Networks of Virtual Machines, (LCR 2004) (powerpoint)
  • Addressing the Trust Asymmetry Problem in Grid Computing with Encrypted Computation, (LCR 2004) (powerpoint)
  • Adaptive Virtual Networking For Virtual Machine-based Distributed Computing, (University of Florida 2004) (powerpoint)
  • Wayback: A User-level Versioning File System For Linux, (USENIX 2004 (Best Freenix Track Paper) ) (pdf)
  • Inferring the Topology and Traffic Load of Parallel Programs Running in a Virtual Machine Environment (JSSPP 2004) (powerpoint)
  • Measuring and Understanding User Comfort With Resource Borrowing (HPDC 2004) (powerpoint)
  • Towards Virtual Networks For Virtual Machine Grid Computings (USENIX VM 2004) (powerpoint)
  • Virtuoso: Distributed Computing Using Virtual Machines (TTI / U. Chicago 2004) (powerpoint)
  • Virtuoso: Distributed Computing Using Virtual Machines (Purdue 2003) (powerpoint)
  • A Case for Grid Computing on Virtual Machines (ICDCS 2003, given by R. Figueiredo) (powerpoint)
  • Middleware for Grid Computing On Virtual Machines (ANIR PI Meeting 2003) (powerpoint poster)

    • Papers

  • A. Gupta, Black Box Methods for Inferring Parallel Applications' Properties in Virtual Environments, Doctoral Dissertation, Technical Report NWU-EECS-08-04, Department of Electrical Engineering and Computer Science, Northwestern University, May, 2008. pdf

  • B. Lin, Human-driven Optimization, Doctoral Dissertation, Technical Report NWU-EECS-07-04, Department of Electrical Engineering and Computer Science, Northwestern University, July, 2007. pdf

  • J. Lange, P. Dinda, F. Bustamante, Vortex: Enabling Cooperative Selective Wormholing for Network Security Systems, Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (RAID 2007), Accepted, to appear.

  • J. Lange, P. Dinda, Transparent Network Services via a Virtual Traffic Layer for Virtual Machines, Proceedings of the 16th IEEE International Symposium on High Performance Distributed Computing (HPDC 2007), June, 2007. pdf

  • B. Lin, A. Sundararaj, P. Dinda, Time-sharing Parallel Applications With Performance Isolation And Control, Proceedings of the 4th IEEE International Conference on Autonomic Computing (ICAC 2007), June, 2007. pdf

  • P. Dinda, G. Memik, R. Dick, B. Lin, A. Mallik, A. Gupta, S. Rossoff, The User In Experimental Computer Systems Research, Proceedings of the Workshop on Experimental Computer Science (ExpCS 2007), June, 2007. pdf

  • B. Prosnitz, Black Box No More: Reconstruction of Internal Virtual Machine State, Technical Report NWU-EECS-07-01, Department of Electrical Engineering and Computer Science, Northwestern University, March, 2007, pdf.

  • A. Sundararaj, Automatic, Run-time and Dynamic Adaptation of Distributed Applications Executing in Virtual Environments, Ph.D. Dissertation, Technical Report NWU-EECS-06-18, Department of Electrical Engineering and Computer Science, Northwestern University, December, 2006, pdf.

  • B. Lin, P. Dinda,Towards Scheduling Virtual Machines Based On Direct User Input, Proceedings of the 1st International Workshop on Virtualization Technology in Distributed Computing (VTDC 2006), November, 2006, pdf

  • S. Rossoff, and P. Dinda, Prospects for Speculative Remote Display, Technical Report NWU-EECS-06-08, Department of Electrical Engineering and Computer Science, Northwestern University, August, 2006, pdf.

  • B. Lin, and P. Dinda, Putting the User in Direct Control of CPU Scheduling, Technical Report NWU-EECS-06-07, Department of Electrical Engineering and Computer Science, Northwestern University, July, 2006. pdf.

  • A. Sundararaj, M. Sanghi, J. Lange, P. Dinda, Hardness of Approximation and Greedy Algorithms for the Adaptation Problem In Virtual Environments, Technical Report NWU-EECS-06-06, Department of Electrical Engineering and Computer Science, Northwestern University, July, 2006, pdf.

  • A. Sundararaj, M. Sanghi, J. Lange, P. Dinda, Hardness of Approximation and Greedy Algorithms for the Adaptation Problem in Virtual Environments (poster), Proceedings of the 3rd IEEE International Conference on Autonomic Computing (ICAC 2006), June, 2006. pdf. (See also NWU-EECS-06-06, above)

  • B. Lin, A. Brown, J. Hellerstein, Towards an Understanding of Decision Complexity in IT Configuration (poster), Proceedings of the 3rd IEEE International Conference on Autonomic Computing (ICAC 2006), June, 2006. pdf.

  • A. Gupta, M. Zangrilli, A. Sundararaj, A. Huang, P. Dinda, and B. Lowekamp, Free Network Measurment for Adaptive Virtualized Distributed Computing, Proceedings of the 20th International Parallel and Distributed Processing Symposium (IPDPS 2006), April, 2006. pdf A poster appeared at ACM/IEEE SC 2005, and a previous version is available as Technical Report NWU-CS-05-13

  • A. Gupta, M. Zangrilli, A. Sundararaj, P. Dinda, and B. Lowekamp, Free Network Measurment for Adaptive Virtualized Distributed Computing, Technical Report NWU-CS-05-13, June, 2005. pdf

  • B. Lin, and P. Dinda, VSched: Mixing Batch and Interactive Virtual Machines Using Periodic Real-time Scheduling, Proceedings of ACM/IEEE SC 2005 (Supercomputing), November, 2005. pdf.

  • A. Sundararaj, M. Sanghi, J. Lange, P. Dinda, An Optimization Problem in Adaptive Virtual Environments, Proceedings of the Seventh Workshop on Mathematical Performance Modeling and Analysis (MAMA 2005). pdf

  • R. Figueiredo, P. Dinda, J. Fortes, Resource Virtualization Renaissance, (Guest Editors' Introduction to the IEEE Computer Special Issue On Resource Virtualization), May, 2005. link

  • B. Lin, and P. Dinda, Vsched: Mixing Batch and Interactive Virtual Machines Using Periodic Real-time Scheduling, Technical Report NWU-CS-05-06, April, 2005, pdf.

  • A. Sundararaj, A. Gupta, and P. Dinda, Increasing Application Performance In Virtual Environments Through Run-time Inference and Adaptation, Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing (HPDC 2005). pdf

  • J. Lange, A. Sundararaj, and P. Dinda, Automatic Dynamic Run-time Optical Network Reservations, Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing (HPDC 2005). pdf

  • A. Sundararaj, A. Gupta, P. Dinda, Dynamic Topology Adaptation In Virtual Networks of Virtual Machines, Proceedings of the Seventh Workshop on Langauges, Compilers and Run-time Support for Scalable Systems (LCR 2004). pdf

  • P. Dinda, Addressing the Trust Asymmetry Problem in Grid Computing Using Encrypted Computation, Proceedings of the Seventh Workshop on Langauges, Compilers and Run-time Support for Scalable Systems (LCR 2004), pdf

  • B. Lin, P. Dinda, D. Lu, User-driven Scheduling of Interactive Virtual Machines, Proceedings of the Fifth International Workshop on Grid Computing (Grid 2004). pdf

  • A. Shoykhet, J. Lange, P. Dinda, Virtuoso: A System For Virtual Machine Marketplaces, Technical Report NWU-CS-04-39, July, 2004, pdf

  • A. Gupta, P. Dinda, Inferring the Topology and Traffic Load of Parallel Programs Running In a Virtual Machine Environment, Proceedings of the 10th Workshop on Job Scheduling Policies for Parallel Processing, June, 2004, pdf

  • A. Gupta, B. Lin, P. Dinda, Measuring And Understanding User Comfort With Resource Borrowing, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC 2004), pdf

  • B. Cornell, P. Dinda, F. Bustamante, Wayback: A User-level Versioning File System for Linux, Proceedings of the USENIX Annual Technical Conference 2004, pdf. Earlier version available as Technical Report NWU-CS-04-30, Department of Computer Science, Northwestern University, January, 2004, (pdf).

  • A. Gupta, B. Lin, and P. Dinda, A Framework and Toolkit for Understanding User Comfort with Resource Borrowing, Technical Report NWU-CS-04-28, Department of Computer Science, Northwestern University, February, 2004. pdf.

  • A. Sundararaj, P. Dinda, Exploring Inference-based Monitoring of Virtual Machine Resources, in submission.

  • A. Sundararaj, P. Dinda, Towards Virtual Networks for Virtual Machine Grid Computing, Proceedings of the third USENIX Virtual Machine Research and Technology Symposium (VM 04), May, 2004, (pdf) Earlier version available as Technical Report NWU-CS-03-27, Department of Computer Science, Northwestern University, October, 2003, (pdf).

  • R. Figueiredo, P. Dinda, J. Fortes, A Case for Grid Computing on Virtual Machines, Proceedings of the 23rd International Conference on Distributed Computing (ICDCS 2003), May, 2003, (pdf). Earlier version available as ACIS Technical Report TR_ACIS-02-001, University of Florida, August, 2002. (pdf)

    • Codes

  • VNET : A layer 2 virtual network system for virtual machines (vnet-0.9.tgz)
  • VSched: A periodic real-time scheduling system for virtual machines (vsched-0.9.tgz)
  • Wayback: A user-level versioning filesystem for Linux (external link)

    • Acknowledgement

    This material is based upon work supported by the National Science Foundation under NSF Middleware Initiative (NSF NMI) Grant No. ANI-0301108, NSF equipment grant EIA-0224449, a gift from VMware Corporation, and support from IBM Corporation and Comtech Group. The opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF), VMware, IBM, or Comtech Group.
    Peter Dinda
    Last modified: Sun Jun 1 15:16:27 CDT 2008