I would like to share my fresh experience with XEN domU migration between two linux machines. I found this important piece of knowledge:
Before you start to migrate xen domU image, please doublecheck your /etc/fstab at least !!! Sorry for potential boring the experienced geeks. After the reached finding I reinvented the feelings of a greenhorn. Attempts to migrate one domU vm machine running continously on one server were alternately succesful as well as failing in a longer period, causing big frustration as reliable backup/restore procedure was all the time beyond the horizon.
The term „migration" in my case means just only simple copy od disk image files by use of rsync via ssh tunnel. I redefined a new vm machine on the target host by use of xen vm manager GUI. After two days long intensive experiments, search and googling I started to doubt whether XEN and OpenSuse are reliable platforms. VNC console showed just only frozen OpenSuse chameleon. Xend.log showed only several messages about waiting for several devices with no result. I tried to use xm console, what finally helped me to find the reason, but several cutoffs had to be discovered before a light blinked at the end:
1st Cutoff – using xm console command I got as the first reponse several lines of text including this line:
Error: Device 51712 (vbd) could not be connected. losetup /dev/loop0 failed Googling for this text found many requests for help with failed domU migrations, mostly with very general recommendations, like to add no-xsave option to xen command line in grub definition as well as the compatibility between source and target dom0. Both are helpful in general but they did not help to fix my situation.
2nd Cutoff: I repeated my experiment with xm console further, until I received another oraculum:
[ 0.144387] PCI: Fatal: No config space access function found
[ 0.152178] Unable to read sysrq code in control/sysrq
[ 0.342964] i8042: No controller found
[ 0.444155] /home/abuild/rpmbuild/BUILD/kernel-xen-3.1.10/linux-3.1/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Very instructive again, another time losses with googling.
Aha moment: In all cases on entering xm console command systém responded with long delay. After display of error messages it seemed to be busy or frozen. Once I tried to apply Ctrl-C and I got a line:
Give root password for login: When the password for migrated vm was entered, login was successful but displyed command line started by
another system name than expected. And this was the first sign forcing me to seek for the explanation, why this can happen.
My Current View of the Primary Reason of XEN VM Migration Issues: When we define a new XEN domU (vm machine) by the use of GUI in vm manager, parts of the definition are reflected into the content of the files in the image of domU. Particularly there are the files /etc/fstab and /etc/HOSTNAME. If you change a definition of the domU (even when again GUI of vm manager used) later on, these files are not updated more, the original content remains. In addition you can modify these files later on either manually or via yast2, as me trying to add nfs client to domU.
When now after some time you want to „migrate" a domU to another host without a proper revision and correction of /etc/fstab (and others?), you risk a serious inconsistence which prevent a proper start of the vm in a target dom0. For example I found in fstab forgotten inactive nfs client definition as well as /dev/xvdc1 device entered at the first definition and later on improperly umounted.
Of course even we do not expect that XEN will solve these cases for us, when we start to think bit deeper. VM Manager GUI is a nice tool but with very basic functionality limited to the first definitions. That is good and sufficient I think. We have a right to update /etc/fstab even for domU anytime and it is good for us. This right is more important than any complex gui for vm manager, but we have to be responsible for it.
Conclusions:
- I am using Opensuse as xen dom0 platform for several cooperating servers for more than 2 years, but my experience is generally valid for other linux distributions I hope. In general OpenSuse provides good support for XEN dom0 as well as domU environment. In last years it is version 11.4 and 12.1 which is stable enough. I tested already 12.2 as domU and plan to upgrade dom0s to 12.2 soon. First months of any new release may bring some instabilities naturally, thus I do not like to hurry with upgrades of life critical servers.
- XEN si a very good virtualization tool, but any concept of virtualization brings another level of complexity. Migration of vm machines between the hosts is one of the most difficult tasks and it's success is dependent on several prerequisites fulfilment. A good checklist for some migration scenarios between two OpenSuse dom0 host wolud be appreciated.
Many thanks in advance for any comments, corrections and additions, Hynek;)
No comments:
Post a Comment