Unable to build qcow2 on [fedora 23] for ppc64le

[bssrikanth]

I am trying to build my own Atomic image for ppc64le arch.

I was able to build atomic image qcow2 for ppc64le on fedora 21. But I am not able to build qcow2 image for atomic host on fedora23 ppc64le. Below mentioned is the error which is being hit:

export LIBGUESTFS_BACKEND=direct

rpm-ostree-toolbox create-vm-disk /srv/rpm-ostree/repo fedora-atomic-host fedora-atomic/f23/ppc64le/docker-host f23-atomic.qcow2

output log of previous command:

/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py:29: PyGIWarning: OSTree was imported without specifying a version first. Use gi.require_version('OSTree', '1.0') before import to ensure that the right version gets loaded.
from gi.repository import Gio, OSTree, GLib # pylint: disable=no-name-in-module
/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/taskrunner.py:27: PyGIWarning: GSystem was imported without specifying a version first. Use gi.require_version('GSystem', '1.0') before import to ensure that the right version gets loaded.
from gi.repository import GLib,Gio,GSystem # pylint: disable=no-name-in-module
Formatting '/root/f23-atomic.qcow2.tmp', fmt=qcow2 size=17179869184 compat='0.10' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
bytesize: 17179869184 sectorsize: 512
boot: 409600 root: 33142784
Mounting /tmp/rpmostreetoolbox.BDZ9CY/mnt : guestmount -o allow_root --pid-file /tmp/rpmostreetoolbox.BDZ9CY/mnt.guestmount-pid -a /root/f23-atomic.qcow2.tmp --rw -m /dev/atomicos/root -m /dev/sda2:/boot /tmp/rpmostreetoolbox.BDZ9CY/mnt
Started child process 'ostree' 'admin' '--sysroot=/tmp/rpmostreetoolbox.BDZ9CY/mnt' 'init-fs' '/tmp/rpmostreetoolbox.BDZ9CY/mnt': pid=2497
Started child process 'ostree' 'admin' '--sysroot=/tmp/rpmostreetoolbox.BDZ9CY/mnt' 'os-init' 'fedora-atomic-host': pid=2499
/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host initialized as OSTree root
Started child process 'ostree' '--repo=/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/repo' 'pull-local' '--disable-fsync' '--remote=fedora-atomic-host' '/srv/rpm-ostree/repo' 'f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b': pid=2500

Workaround https://bugzilla.redhat.com/show_bug.cgi?id=1144766 : Reset mtime of 15767 files
Started child process 'ostree' 'admin' '--sysroot=/tmp/rpmostreetoolbox.BDZ9CY/mnt' 'deploy' '--karg=root=UUID=d230f7f0-99d3-4244-8bd9-665428054831' '--karg=rd.lvm.lv=atomicos/root' '--os=fedora-atomic-host' '--origin-file=/root/origin.tmp' 'f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b': pid=4054
Relabeling /var (no stamp file '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/var/.ostree-selabeled' found)
Transaction complete; bootconfig swap: yes deployment count change: 1
Labeling deployment root
Started child process 'ostree' 'admin' '--sysroot=/tmp/rpmostreetoolbox.BDZ9CY/mnt' 'instutil' 'selinux-ensure-labeled' '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0' '': pid=4157
Relabeling using policy 'targeted'

Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0/etc/fstab' (as '//etc/fstab') to 'system_u:object_r:etc_t:s0'
Doing initial labeling
Started child process 'ostree' 'admin' '--sysroot=/tmp/rpmostreetoolbox.BDZ9CY/mnt' 'instutil' 'selinux-ensure-labeled' '/tmp/rpmostreetoolbox.BDZ9CY/mnt' '': pid=4178
Relabeling using policy 'targeted'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt' (as '/') to 'system_u:object_r:root_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/boot' (as '//boot') to 'system_u:object_r:boot_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/boot/loader.1' (as '//boot/loader.1') to 'system_u:object_r:boot_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/boot/loader.1/entries' (as '//boot/loader.1/entries') to 'system_u:object_r:boot_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/boot/loader.1/entries/ostree-fedora-atomic-host-0.conf' (as '//boot/loader.1/entries/ostree-fedora-atomic-host-0.conf') to 'system_u:object_r:boot_t:s0'
..
..
..
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/repo/state' (as '//ostree/repo/state') to 'system_u:object_r:system_conf_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy' (as '//ostree/deploy') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host' (as '//ostree/deploy/fedora-atomic-host') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy' (as '//ostree/deploy/fedora-atomic-host/deploy') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0/proc' (as '//ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0/proc') to 'system_u:object_r:usr_t:s0'
..
..
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0.origin' (as '//ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0.origin') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/boot.1.1' (as '//ostree/boot.1.1') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/boot.1.1/fedora-atomic-host' (as '//ostree/boot.1.1/fedora-atomic-host') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/boot.1.1/fedora-atomic-host/ca84c575190fd83a355c18ec2b7b22e69c3192370c9f2ee5b41f7dfb316dae1e' (as '//ostree/boot.1.1/fedora-atomic-host/ca84c575190fd83a355c18ec2b7b22e69c3192370c9f2ee5b41f7dfb316dae1e') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/boot.1.1/fedora-atomic-host/ca84c575190fd83a355c18ec2b7b22e69c3192370c9f2ee5b41f7dfb316dae1e/0' (as '//ostree/boot.1.1/fedora-atomic-host/ca84c575190fd83a355c18ec2b7b22e69c3192370c9f2ee5b41f7dfb316dae1e/0') to 'system_u:object_r:usr_t:s0'
Set label of '/tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/boot.1' (as '//ostree/boot.1') to 'system_u:object_r:usr_t:s0'
Updating Docker storage: /tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0/etc/sysconfig/docker-storage
Started child process 'guestunmount' '-v' '/tmp/rpmostreetoolbox.BDZ9CY/mnt': pid=4240
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.BDZ9CY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
guestunmount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: fusermount: failed to unmount /tmp/rpmostreetoolbox.BDZ9CY/mnt: Device or resource busy
USER PID ACCESS COMMAND
/tmp/rpmostreetoolbox.BDZ9CY/mnt:
root kernel mount /tmp/rpmostreetoolbox.BDZ9CY/mnt
root 2391 f.... gjs

(gjs:2391): Gjs-WARNING **: JS ERROR: Error: Child process ["guestunmount","-v","/tmp/rpmostreetoolbox.BDZ9CY/mnt"]: Child process exited with code 2
_wait_sync_check_internal@/usr/share/rpm-ostree-toolbox/js/procutil.js:58
runSync@/usr/share/rpm-ostree-toolbox/js/procutil.js:73
GuestMount<.umount@/usr/share/rpm-ostree-toolbox/js/guestfish.js:127
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
CreateVmDisk<.execute@/usr/share/rpm-ostree-toolbox/js/builtins/create_vm_disk.js:96
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
Builtin<.main@/usr/share/rpm-ostree-toolbox/js/builtin.js:41
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
@/usr/share/rpm-ostree-toolbox/js/main.js:92
@/usr/share/rpm-ostree-toolbox/js/main.js:99

[bssrikanth]

I have tried to create vm disk multiple times, starting from scratch.. as well rebooting guest on which I am performing this task couple of times.

I am able to build fedora 21 atomic host for ppc64le, but not fedora23. Please advise.

[bssrikanth]

qcow2 disk image create logs for fedora 21 and fedora 23:

fedora 21:

No Docker storage config detected in /tmp/rpmostreetoolbox.TOQ3CY/mnt/ostree/deploy/fedora-atomic-host/deploy/47a47568933c7be09ebca95fab57b9eaa0e22eba18c2020b91a4218a3e6b0996.0/etc/sysconfig/docker-storage
Started child process 'guestunmount' '-v' '/tmp/rpmostreetoolbox.TOQ3CY/mnt': pid=2111
--> Unmounting goes fine at this point..

fedora 23:
qcow2 disk image create has below portion:

Updating Docker storage: /tmp/rpmostreetoolbox.BDZ9CY/mnt/ostree/deploy/fedora-atomic-host/deploy/f1d3c96a764056678762583aa07636b077faa7115d0a884868e6b053daca3f7b.0/etc/sysconfig/docker-storage
Started child process 'guestunmount' '-v' '/tmp/rpmostreetoolbox.BDZ9CY/mnt': pid=4240
-->Unmounting fails at this point complaining that partition is still busy [default 5 attempts]..

Could issue be related to docker storage related operation?

I did not include anything specific to docker storage in fedora 21 host json file compared to fedora 23 host json file...

[bssrikanth]

I removed Docker storage config for f23..

even then the qcow2 image creation fails with same error:

No Docker storage config detected in /tmp/rpmostreetoolbox.767ZCY/mnt/ostree/deploy/fedora-atomic-host/deploy/93b900860b860687c5fb7b2e45a8830b24779a03333cd9a85ec684d3c1e7bd97.0/etc/sysconfig/docker-storage
Started child process 'guestunmount' '-v' '/tmp/rpmostreetoolbox.767ZCY/mnt': pid=29669
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: running: fusermount -u /tmp/rpmostreetoolbox.767ZCY/mnt
fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
guestunmount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: fusermount: failed to unmount /tmp/rpmostreetoolbox.767ZCY/mnt: Device or resource busy
USER PID ACCESS COMMAND
/tmp/rpmostreetoolbox.767ZCY/mnt:
root kernel mount /tmp/rpmostreetoolbox.767ZCY/mnt
root 27769 f.... gjs

(gjs:27769): Gjs-WARNING **: JS ERROR: Error: Child process ["guestunmount","-v","/tmp/rpmostreetoolbox.767ZCY/mnt"]: Child process exited with code 2
_wait_sync_check_internal@/usr/share/rpm-ostree-toolbox/js/procutil.js:58
runSync@/usr/share/rpm-ostree-toolbox/js/procutil.js:73
GuestMount<.umount@/usr/share/rpm-ostree-toolbox/js/guestfish.js:127
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
CreateVmDisk<.execute@/usr/share/rpm-ostree-toolbox/js/builtins/create_vm_disk.js:96
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
Builtin<.main@/usr/share/rpm-ostree-toolbox/js/builtin.js:41
wrapper@resource:///org/gnome/gjs/modules/lang.js:169
@/usr/share/rpm-ostree-toolbox/js/main.js:92
@/usr/share/rpm-ostree-toolbox/js/main.js:99

[bssrikanth]

........ :-(

[dustymabe]

@bssrikanth did you ever figure out the problem? I might be able to help.

[jasonbrooks]

Sorry for ignoring this repo -- not sure if there's anything hw-specific about this, but I don't have access to a ppc64le machine.

[dustymabe]

@bssrikanth can you do your exact steps on x86_64 and get something working on that platform? then try to apply the same steps to ppc and we'll do a stare and compare?

[bssrikanth]

@dustymabe first of all sorry for delay in my response, I was out of office for couple of weeks.
Appreciate your willingness to help :-)
I did try with x86 server, it also fails with same issue where fusermount failed to unmount /tmp/rpmostreetoolbox.MWSWFY/mnt: Device or resource busy.

I have observed that this issue happens only with fedora 23 [either x86/ppc64le]. I tried building atomic image for fedora 24.. but that also hit roadblock with projectatomic/rpm-ostree-toolbox#86.

x86_fedora23_atomic_build_log.txt

[bssrikanth]

@jasonbrooks as mentioned above, same issue is seen on x86 as well.

[dustymabe]

@jasonbrooks do you have time to run through and see if you can reproduce?

@bssrikanth otherwise I will try to do it this weekend if that isn't too long for you to wait. There are now some instructions in the upstream pagure repo: https://pagure.io/fedora-atomic as well so that might be a place to look for any changes in the process.

[bssrikanth]

@dustymabe I can wait for you or @jasonbrooks to reproduce the issue. Many thanks for your time.

[jasonbrooks]

I spent some time on this today, trying to produce the images the same way we do with centos atomic images, but I'm hitting errors. The way fedora makes its images has changed at least once since I wrote up these image building steps.

[bssrikanth]

@jasonbrooks thanks for your time. Are you hitting same error which I have posted against fedora 23?

[dustymabe]

@bssrikanth @jasonbrooks so I am getting a different error right now. I'm having trouble just building the ostree using rpm-ostree compose --repo=repo tree fedora-atomic/fedora-atomic-docker-host.json after I did a git checkout on the f23 branch. The error I am getting looks like:

  zlib-1.2.8-9.fc23.x86_64

Downloading packages [==============================================================================================================================================================================================================   ]  99%

Installing packages [===============================================================================================================================================================================================                   ]  91%
error: Error -1 running transaction
[root@f23 ~]#

@jasonbrooks did you have success building the ostree?

@cgwalters do you know about this issue? strace output from running this command is here: https://dustymabe.fedorapeople.org/straceout.zip

[jasonbrooks]

@dustymabe I was able to build the tree, I had to install all updates to f23 first, though. If I recall correctly, this was an selinux issue... I think I once figured out how to build the images how fedora releng does it, but I need to go re-discover that.

[dustymabe]

@jasonbrooks you are right.. updating all packages to latest f23 seems to work now.

[dustymabe]

@bssrikanth @jasonbrooks I was able to actually get an ostree compose and an image built today. I'll try to document the steps and share them at some point.. i'm exhausted today.

[bssrikanth]

@dustymabe that is a great news!!! many thanks for your time. Looking forward for your documentation.

[jasonbrooks]

I've got it working, too. I'll update the directions in here. In the meantime, here's everything wrapped up into a Vagrantfile: https://gist.github.com/jasonbrooks/f3b405a43a5500b09085238528ab87ac

[bssrikanth]

@jasonbrooks many thanks for your time :)