Xen setup NetBSD Dom0 and DomU

This is a page I will always try to keep up to date with latest NetBSD and Xen versions. On this version I'm using NetBSD 8.0 and Xen 4.8.

NetBSD

Xen

This page is a summary of my experiences with NetBSD and Xen since NetBSD 3.1 and Xen 2.0, although it is a very simple experience, as the user of basic features only. Back to that days of NetBSD 3.1 we needed to install GRUB bootloader to be able to boot Xen. We don't need it anymore, since NetBSD comes with its own good bootloader since version 5.0.

GRUB

A very useful, more complete and probably up to date reference where I got much of the information is the NetBSD Xen HowTo:

NetBSD Xen HowTo

Installing Xen

After installing NetBSD in your computer (with a common kernel like GENERIC), you need to download and compile the xenkernel from pkgsrc::

    # cd /tmp
    # ftp http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
    # tar zxf pkgsrc.tar.gz -C /usr
    # cd /usr/pkgsrc/sysutils/xenkernel48
    # make install

Copy the kernel to the root directory::

    # cp /usr/pkg/xen48-kernel/xen.gz /

We now have the xen kernel built in /usr/pkg/xen48-kernel/xen.gz that we will use later. Now, build xentools::

    # cd /usr/pkgsrc/sysutils/xentools48
    # make install

Standard xen devices are created by default in /dev. You just need to call ./MAKEDEV script in /dev if you need more vnd devices than vnd[0123] or more xbd devices beyond xbd[0123]. See the NetBSD Xen HowTo for more information.

Copy daemon scripts in the right place and configure rc.conf to lauch xencommons at the system startup:

    # cp /usr/pkg/share/examples/rc.d/xen* /etc/rc.d/
    # echo 'xencommons=YES' >> /etc/rc.conf

Using the right kernel

The GENERIC kernel (the one that comes by default) cannot be used with Xen. You need to get another kernel. Just download it::

    # cd /
    # ftp ftp.netbsd.org/pub/NetBSD/NetBSD-8.0/amd64/binary/kernel/netbsd-XEN3_DOM0.gz

Note: It is important to note that if you are going to load kernel modules with modload, you'll have to load the ones specially built for Xen, because of ABI incompatibilities.

modload man page

Configure bootloader

Since NetBSD 5.0, we can use NetBSD's boot loader (you can optionally use GRUB if you prefer). What we need is to edit the /boot.cfg file, which is a very simple and plain text file. The man page of this file has some examples on how to setup it to load the xen kernel and the NetBSD Dom0 kernel to work with Xen.

boot.cfg man page

What we need to do is just to add the following menu entry::

    menu=Xen:load /netbsd-XEN3_DOM0.gz;multiboot /xen.gz dom0_mem=512M console=pc

And change the default parameter to where we just put this menu entry.

Reboot, select the right option at the boot menu and see the magic happening!

You can use xl command to get dom0 information like xl dmesg or xl info.

Network configuration

After having Xen up and running, you'll have to create a bridge to connect the interfaces of the virtual machine. I like to think a bridge as a switch internal to the host.

To create a bridge, type::

    # ifconfig bridge0 create
    # ifconfig bridge0 up

Then, add physical interfaces to that switch, so your virtual machines can talk to the external world. For instance, if your physical network interface is re0, do:

    # brconfig bridge0 add re0

This configuration will vanish at the next reboot. So make them persistent. Add the following configuration to /etc/ifconfig.bridge0::

    create
    up
    !brconfig $int add re0

Troubleshooting

Error when bridge is not created

Errors like::

    libxl: error: libxl_dm.c:2189:device_model_spawn_outcome: domain 1 device model: spawn failed (rc=-3)
    libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model did not start: -3
    libxl: error: libxl_dm.c:2303:kill_device_model: Device Model already exited
    libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1
    libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1
    libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed

Are not very helpful, but can mean that you did not create your bridge device. Check "Network configuration" section for more information.

`... libxl.c:363:libxl_get_physinfo: getting physinfo: Permission denied`

When issuing a xen command, like xl list, you get an error like that::

    libxl: error: libxl.c:363:libxl_get_physinfo: getting physinfo: Permission denied
    libxl: error: libxl_domain.c:315:libxl_list_domain: getting domain info list: Permission denied
    libxl_list_domain failed.

It probably means a version mismatch between xenkernel and xentools packages. See link below for more information.

Xen: libxl_list_domain: Permission denied