]> The University of Tokyo

7-3-1 Hongo Bunkyo-ku Tokyo 113-8656 JP +81 3 5841 6748 panda@hongo.wide.ad.jp

VMware Inc.

mrm@vmware.com

Jacobs University

Campus Ring 1 Bremen 28759 Germany j.schoenwaelder@jacobs-university.de

The University of Tokyo

2-11-16 Yayoi Bunkyo-ku Tokyo 113-8658 JP sekiya@wide.ad.jp

IIJ Innovation Institute Inc.

3-13 Kanda-Nishikicho Chiyoda-ku Tokyo 101-0054 JP keiichi@iijlab.net

Huawei Technologies (USA)

2330 Central Expressway Santa Clara CA 95050 USA tina.tsou.zouting@huawei.com

Huawei Technologies

Bantian, Longgang District Shenzhen 518129 P.R. China cathyzhou@huawei.com

The University of Tokyo

7-3-1 Hongo Bunkyo-ku Tokyo 113-8656 JP +81 3 5841 6748 hiroshi@wide.ad.jp

Operations and Management OPSAWG MIB Hypervisor Virtual machine This document defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, this specifies objects for managing virtual machines controlled by a hypervisor (a.k.a. virtual machine monitor).

This document defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, this specifies objects for managing virtual machines controlled by a hypervisor (a.k.a. virtual machine monitor). A hypervisor controls multiple virtual machines on a single physical machine by allocating resources to each virtual machine using virtualization technologies. Therefore, this MIB module contains information on virtual machines and their resources controlled by a hypervisor as well as hypervisor's hardware and software information. The design of this MIB module has been derived from enterprise specific MIB modules, namely a MIB module for managing guests of the Xen hypervisor, a MIB module for managing virtual machines controlled by the VMware hypervisor, and a MIB module using the libvirt programming interface to access different hypervisors. However, this MIB module attempts to generalize the managed objects to support other hypervisors.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

For a detailed overview of the documents that describe the current Internet-Standard Management Framework, please refer to section 7 of RFC 3410. Managed objects are accessed via a virtual information store, termed the Management Information Base or MIB. MIB objects are generally accessed through the Simple Network Management Protocol (SNMP). Objects in the MIB are defined using the mechanisms defined in the Structure of Management Information (SMI). This memo specifies a MIB module that is compliant to the SMIv2, which is described in STD 58, RFC 2578, STD 58, RFC 2579 and STD 58, RFC 2580.

On the common implementations of hypervisor softwares, a hypervisor allocates virtual resources such as virtual CPUs, virtual memory, virtual storage devices, and virtual network interfaces to virtual machines from physical resources. This document defines objects related to system and software information of a hypervisor, the list of virtual machines controlled by the hypervisor, and virtual resources allocated by the hypervisor to virtual machines. This document specifies four specific types of virtual resources that are common to general hypervisors; CPUs (processors), memory, network interfaces, and storage devices.

A hypervisor allocates virtual resources such as virtual CPUs, virtual memory, virtual storage devices, and virtual network interfaces to virtual machines from physical resources.

As shown in , the objects defined in this document are managed at a hypervisor and an SNMP agent is launched at the hypervisor to provide access to the objects. The objects are managed from the viewpoint of the operators of hypervisors, but not the operators of virtual machines; i.e., the objects do not take into account the actual resource utilization on each virtual machine but the resource allocation from the physical resources. For example, vmNetworIfIndex indicates the virtual interface associated with an interface of a virtual machine at the hypervisor, and consequently, the `in' and `out' directions denote `from a virtual machine to the hypervisor' and `from the hypervisor to a virtual machine', respectively. Moreover, vmStorageAllocatedSize denotes the size allocated by the hypervisor, but not the size actually used by the operating system on the virtual machine. This means that vmStorageDefinedSize and vmStorageAllocatedSize do not take different values when the vmStorageSourceType is `block' or `raw'. The other objects related to virtual machines such as management IP addresses of a virtual machine are not included in this MIB module because this MIB module defines the objects common to general hypervisors but they are specific to some hypervisors. They may be included in the entLogicalTable of ENTITY-MIB. The objects related to virtual switches are not also included in this MIB module though virtual switches shall be placed on a hypervisor. This is because the virtual network interfaces are the lowest abstraction of network resources allocated to a virtual machine. Instead of including the objects related to virtual switches, for example, BRIDGE-MIB and Q-BRIDGE-MIB could be used.

The MIB module is organized into a group of scalars and tables. The scalars below `hypervisor' provide basic information about the hypervisor. The `vmTable' lists the virtual machines (guests) that are known to the hypervisor. The `vmCpuTable' provides the mapping table of virtual CPUs to virtual machines, including CPU time used by each virtual CPU. The 'vmCpuAffinityTable' provides the affinity of each virtual CPU to a physical CPU. The `vmStorageTable' provides the list of virtual storage devices and their mapping to virtual machines. In case that an entry in the `vmStorageTable' has a corresponding parent physical storage device managed in `hrStorageTable' of HOST-RESOURCES-MIB, the entry contains a pointer `vmStorageParent' to the physical storage device. The `vmNetworkTable' provides the list of virtual network interfaces and their mapping to virtual machines. Each entry in the `vmNetworkTable' also provides a pointer `vmNetworIfIndex' to the corresponding entry in the `ifTable' of IF-MIB. In case that an entry in the `vmNetworkTable' has a corresponding parent physical network interface managed in `ifTable' of IF-MIB, the entry contains a pointer `vmNetworkParent' to the physical network interface.

| running |<-------------->| migrating | | !vmResuming | | !vmRunning | | !vmMigrating | + - - - - - - + +-------------+ + - - - - - - -+ | ^ *running ^ | | | | +-------------------+ | | | | v *shutdown *destroy v v + - - - - - - - - + +-------------+ | shuttingdown |--------->| shutdown | | !vmShuttingdown | | !vmShutdown | + - - - - - - - - + +-------------+ ^ | | v !vmDeleted + - - - - - -+ +------------+ + - - - - - - + (Deleted from | blocked | | crashed | | preparing | vmTable) | !vmBlocked | | !vmCrashed | | | + - - - - - -+ +------------+ + - - - - - - + ]]> The state transition of a virtual machine

The `vmAdminState' and `vmOperState' textual conventions define an administrative state and an operational state model for virtual machines. Events causing transitions between major operational states will cause the generation of notifications. Per virtual machine (per-VM) notifications (vmRunning, vmShutdown, vmPaused, vmSuspended, vmCrashed, vmDeleted) are generated if vmPerVMNotificationsEnabled is true(1). Bulk notifications (vmBulkRunning, vmBulkShutdown, vmBulkPaused, vmBulkSuspended, vmBulkCrashed, vmBulkDeleted) are generated if vmBulkNotificationsEnabled is true(1). The transition of `vmOperState' by the write access to `vmAdminState' and the notifications generated by the operational state changes are summarized in . Note that the notifications shown in this figure are per-VM notifications. In the case of Bulk notifications, the prefix `vm' is replaced with 'vmBulk'. The bulk notification mechanism is designed to reduce the number of notifications that are trapped by an SNMP manager. This is because the number of virtual machines managed by a bunch of hypervisors in a datacenter possibly becomes several thousands or more, and consequently, many notifications could be trapped if these virtual machines frequently change their administrative state. The per-VM notifications carry more detailed information, but the scalability shall be a problem. An implementation shall support both, either of, or none of per-VM notifications and bulk notifications. The notification filtering mechanism described in section 6 of RFC 3413 is used by the management applications to control the notifications. The MIB module provides a few writable objects that can be used to make non-persistent changes, e.g., changing the memory allocation or the CPU allocation. It is not the goal of this MIB module to provide a configuration interface for virtual machines since other protocols and data modeling languages are more suitable for this task. The OID tree structure of the MIB module is shown below.

The MIB module in this document uses the following IANA-assigned OBJECT IDENTIFIER values recorded in the SMI Numbers registry:

There are a number of management objects defined in this MIB that have a MAX-ACCESS clause of read-write and/or read-create. Such objects may be considered sensitive or vulnerable in some network environments. The support for SET operations in a non-secure environment without proper protection can have a negative effect on hypervisor and virtual machine operations. There are a number of managed objects in this MIB that may contain sensitive information. The objects in the vmHvSoftware and vmHvVersion list information about the hypervisor's software and version. Some may wish not to disclose to others which software they are running. Further, an inventory of the running software and versions may be helpful to an attacker who hopes to exploit software bugs in certain applications. Moreover, the objects in the vmTable, vmCpuTable, vmCpuAffinityTable, vmStorageTable and vmNetworkTable list information about the virtual machines and their virtual resource allocation. Some may wish not to disclose to others how many and what virtual machines they are operating. It is thus important to control even GET access to these objects and possibly to even encrypt the values of these object when sending them over the network via SNMP. Not all versions of SNMP provide features for such a secure environment. It is recommended that attention be specifically given to implementing the MAX-ACCESS clause in a number of objects, including vmAdminState, vmMinCpuNumber, vmMaxCpuNumber, vmMinMem, vmMaxMem, and vmCpuAffinity in scenarios that DO NOT use SNMPv3 strong security (i.e. authentication and encryption). Extreme caution must be used to minimize the risk of cascading security vulnerabilities when SNMPv3 strong security is not used. When SNMPv3 strong security is not used, these objects should have access of read-only, not read-create. SNMPv1 by itself is not a secure environment. Even if the network itself is secure (for example by using IPsec), even then, there is no control as to who on the secure network is allowed to access and GET/SET (read/change/create/delete) the objects in this MIB. It is recommended that the implementers consider the security features as provided by the SNMPv3 framework. Specifically, the use of the User-based Security Model and the View-based Access Control Model is recommended. It is then a customer/user responsibility to ensure that the SNMP entity giving access to an instance of this MIB, is properly configured to give access to the objects only to those principals (users) that have legitimate rights to indeed GET or SET (change/create/delete) them.

The authors like to thank Joe Marcus Clarke, Randy Presuhn, and David Black for providing helpful comments during the development of this specification. Juergen Schoenwaelder was partly funded by Flamingo, a Network of Excellence project (ICT-318488) supported by the European Commission under its Seventh Framework Programme.

&RFC2119; &RFC2578; &RFC2579; &RFC2580; &RFC2790; &RFC2863; &RFC3413; &RFC3414; &RFC3415; &RFC3418; &RFC4122; &RFC4133; &RFC4188; &RFC4363; &RFC3410;

State Action or (Event) Next state Notification suspended running resuming vmResuming | vmBulkResuming suspending (suspend operation completed) suspended vmSuspended | vmBulkSuspended running suspended suspending vmSuspending | vmBulkSuspending shutdown shuttingdown vmShuttingdown | vmBulkShuttingdown destroy shutdown vmShutdown | vmBulkShutdown (migration to other hypervisor initiated) migrating vmMigrating | vmBulkMingrating resuming (resume opeartion completed) running vmRunning | vmBulkRunning paused running running vmRunning | vmBulkRunning shuttingdown (shutdown operation completed) shutdown vmShutdown | vmBulkShutdown shutdown running running vmRunning | vmBulkRunning (if this state entry is created by a migration operation (*) migrating vmMigrating | vmBulkMigrating (deletion operation completed) (no state) vmDeleted | vmBulkDeleted migrating (migration from other hypervisor completed) running vmRunning | vmBulkRunning (migration to other hypervisor completed) shutdown vmShutdown | vmBulkShutdown preparing (preparation completed) shutdown vmShutdown | vmBulkShutdown blocked (blocking operation completed) (previous state) - crashed - - - (any) (blocking operation initiated) blocked vmBlocked | vmBulkBlocked (crashed) crashed vmCrashed | vmBulkCrashed (no state) (preparation initiated) preparing - (migrate from other hypervisor initiated) shutdown (*) vmShutdown | vmBulkShutdown