21.4. FreeBSD as a Host with bhyve

Starting with FreeBSD 10.0-RELEASE the BSD licensed hypervisor bhyve is part of the base system. bhyve supports a number of guests including FreeBSD, OpenBSD, and many flavors of Linux®. bhyve currently only supports a serial console and does not emulate a graphical console. bhyve is a legacy-free hypervisor, meaning that instead of translating instructions, and manually managing memory mappings, it relies on the virtualization offload features of newer CPUs. bhyve also avoids emulating compatible hardware for the guest, and instead relies on the paravirtualization drivers, called VirtIO.

Due to the design of bhyve, it is only possible to use bhyve on computers with newer processors that support Intel® EPT (Extended Page Tables) or AMD® RVI (Rapid Virtualization Indexing, also know as NPT or Nested Page Tables). Most newer processors, specifically the Intel® Core™ i3/i5/i7 and Intel® Xeon™ E3/E5/E7 support this feature, for a complete list of Intel® processors that support EPT see the Intel® ARK. RVI is found on the 3rd generation and later of the AMD Opteron™ (Barcelona) processors. The easiest way to check for support of EPT or RVI on a system is to look for the POPCNT processor feature flag on the Features2 line in dmesg or /var/run/dmesg.boot.

21.4.1. Preparing the Host

The first step to creating a virtual machine in bhyve is configuring the host system. Load the bhyve kernel module called vmm. Create a tap interface for the network device in the virtual machine to attach to. Optionally create a bridge interface and add the tap interface as well as the physical interface as members to allow the virtual machine to have access to the network.

# kldload vmm# kldload nmdm# ifconfig tap0 create# sysctl net.link.tap.up_on_open=1net.link.tap.up_on_open: 0 -> 1# ifconfig bridge0 create# ifconfig bridge0 addm igb0 addm tap0# ifconfig bridge0 up

21.4.2. Creating a FreeBSD Guest

Create a file to use as the virtual disk for the guest machine.

# truncate -s 16G guest.img

Download an installation image of FreeBSD to install:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.isoFreeBSD-10.0-RELEASE-amd64-bootonly.iso 100% of 209 MB 570 kBps 06m17s

FreeBSD comes with an example script for running a virtual machine in bhyve. The script will start the virtual machine and run it in a loop, so it will automatically restart if it crashes. The script takes a number of options to control the configuration of the machine. -c controls the number of virtual CPUs. -m limits the amount of memory available to the guest. -t defines which tap device to use. -d indicates which disk image to use. -i tells bhyve to boot from the CD image instead of the disk, and -I defines which CD image to use. Finally the last parameter is the name of the virtual machine, used to track the running machines. Start the virtual machine in installation mode:

# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso guestname

The system will boot and start the installer. After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose Yes. A small change needs to be made to make the system start with a serial console. Edit /etc/ttys and replace the existing console line with:

console "/usr/libexec/getty std.9600" xterm on secure

Reboot the virtual machine. Rebooting the virtual machine causes bhyve to exit. However the vmrun.sh script runs bhyve in a loop and will automatically restart it. When this happens, choose the reboot option from the boot loader menu, and this will escape the loop. Now the guest can be started from the virtual disk:

# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname

21.4.3. Creating a Linux® Guest

Note:

bhyve requires sysutils/grub2-bhyve in order to boot operating systems other than FreeBSD.

Create a file to use as the virtual disk for the guest machine.

# truncate -s 16G linux.img

Starting a virtual machine with bhyve is a two step process. First a kernel must be loaded, then the guest can be started. sysutils/grub2-bhyve is used to load the Linux® kernel. Create a device.map that grub will use to map the virtual devices to the files on the host system:

(hd0) ./linux.img(cd0) ./somelinux.iso

Use sysutils/grub2-bhyve to load the Linux® kernel from the ISO image:

# grub-bhyve -m device.map -r cd0 -M 1024M linuxguest

This will start grub. If the installation CD contains a grub.cfg then a menu will be displayed. If not, the vmlinuz and initrd files must be located and loaded manually:

grub> ls(hd0) (cd0) (cd0,msdos1) (host)grub> ls (cd0)/isolinuxboot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memtestsplash.jpg TRANS.TBL vesamenu.c32 vmlinuzgrub> linux (cd0)/isolinux/vmlinuzgrub> initrd (cd0)/isolinux/initrd.imggrub> boot

Now that the Linux® kernel is loaded, the guest can be started:

# bhyve -AI -H -P \-s 0:0,hostbridge \-s 1:0,lpc \-s 2:0,virtio-net,tap1 \-s 3:0,virtio-blk,./linux.img \-s 4:0,ahci-cd,./somelinux.iso \-l com1,stdio \-c 4 -m 1024M linuxguest

The system will boot and start the installer. After installing a system in the virtual machine, reboot the virtual machine. This will cause bhyve to exit. The instance of the virtual machine needs to be destroyed before it can be started again:

# bhyvectl --destroy --vm=linuxguest

Now the guest can be started directly from the virtual disk. Load the kernel:

# grub-bhyve -m device.map -r hd0,msdos1 -M 1024M linuxguestgrub> ls(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos1) (host)(lvm/VolGroup-lv_swap) (lvm/VolGroup-lv_root)grub> ls (hd0,msdos1)/lost+found/ grub/ efi/ System.map-2.6.32-431.el6.x86_64 config-2.6.32-431.el6.x86_64 symvers-2.6.32-431.el6.x86_64.gz vmlinuz-2.6.32-431.el6.x86_64initramfs-2.6.32-431.el6.x86_64.imggrub> linux (hd0,msdos1)/vmlinuz-2.6.32-431.el6.x86_64 root=/dev/mapper/VolGroup-lv_rootgrub> initrd (hd0,msdos1)/initramfs-2.6.32-431.el6.x86_64.imggrub> boot

Boot the virtual machine:

# bhyve -AI -H -P \-s 0:0,hostbridge \-s 1:0,lpc \-s 2:0,virtio-net,tap1 \-s 3:0,virtio-blk,./linux.img \-l com1,stdio \-c 4 -m 1024M linuxguest

Linux® will now boot in the virtual machine and eventually present you with the login prompt. Login and use the virtual machine. When you are finished, reboot the virtual machine to exit bhyve. Destroy the virtual machine instance:

# bhyvectl --destroy --vm=linuxguest

21.4.4. Creating an OpenBSD Guest

Note:

bhyve requires sysutils/grub2-bhyve in order to boot operating systems other than FreeBSD. Additionally, OpenBSD requires bhyve to emulate an AMD® hostbridge rather than a generic one and use only single-vector for MSI (-W).

Due to limitations in bhyve it is not currently possible to run the OpenBSD installer in bhyve. However, a disk image of an existing OpenBSD 5.5 or later system will run.

Starting a virtual machine with bhyve is a two step process. First a kernel must be loaded, then the guest can be started. sysutils/grub2-bhyve is used to load the kernel. Create a device.map that grub will use to map the virtual devices to the files on the host system:

(hd0) ./openbsd.img

Load the kernel:

# grub-bhyve -m device.map -r hd0,openbsd1 -M 1024M obsd55grub> ls(hd0) (hd0,openbsd1) (host)grub> kopenbsd -h com0 (hd0,openbsd1)/bsd.mpgrub> boot

Boot the virtual machine:

# bhyve -AI -H -P -W \-s 0:0,amd_hostbridge \-s 1:0,lpc \-s 2:0,virtio-net,tap1 \-s 3:0,virtio-blk,./openbsd.img \-l com1,stdio \-c 4 -m 1024M obsd55

OpenBSD will now boot in the virtual machine and eventually present you with the login prompt. Login and use the virtual machine. When you are finished, reboot the virtual machine to exit bhyve. Destroy the virtual machine instance:

# bhyvectl --destroy --vm=obsd55

21.4.5. Virtual Machines Consoles

It is advantageous to wrap the bhyve console in a session management tool such as sysutils/tmux or sysutils/screen in order to detach and reattach to the console. It is also possible to have the console of bhyve be a null modem device that can be accessed with cu. Load the nmdm kernel module, and replace -l com1,stdio with -l com1,/dev/nmdm0A. The /dev/nmdm devices are created automatically as needed, each is a pair, /dev/nmdm1A and /dev/nmdm1B corresponding to the two ends of the null modem cable. See nmdm(4) for more information.

# bhyve -AI -H -P \-s 0:0,hostbridge \-s 1:0,lpc \-s 2:0,virtio-net,tap1 \-s 3:0,virtio-blk,./linux.img \-l com1,/dev/nmdm0A \-c 4 -m 1024M linuxguest# cu -l /dev/nmdm0B -s 9600ConnectedUbuntu 13.10 handbook ttyS0handbook login:

21.4.6. Managing Virtual Machines

A device node is created in /dev/vmm for each virtual machine. This allows the administrator to easily see a list of the running virtual machines:

# ls -al /dev/vmmtotal 1dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestnamecrw------- 1 root wheel 0x19f Mar 17 12:19 linuxguestcrw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest

Virtual machines can be destroyed using bhyvectl:

# bhyvectl --destroy --vm=guestname

21.4.7. Persistent Configuration

In order to make the system able to start bhyve guests at boot time, the following configurations must be made in the specified files:

  1. /etc/sysctl.conf

    net.link.tap.up_on_open=1
  2. /boot/loader.conf

    vmm_load="YES"nmdm_load="YES"if_bridge_load="YES"if_tap_load="YES"
  3. /etc/rc.conf

    cloned_interfaces="bridge0 tap0"ifconfig_bridge0="addm igb0 addm tap0"

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.