I am using openstack-victoria (packstack) in CentOS 8 for a school assignment. I am trying to create an instance out of a qcow2 image I made, the image size is somewhere around 1.9GB (CentOS 7 - template). I made this image out of an volume that was 30GB.
After creating the instance it takes around 4-5min, then I get this error message in the openstack dashboard:
Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Build of instance 2b5bf737-dbd3-4173-afd0-3cb95c7d236b aborted: Volume e87171f3-99bc-4ef1-bdcc-eed55461cd18 did not finish being created even after we waited 188 seconds or 61 attempts. And its status is downloading.].
The /var/log/cinder/volume.log file contains:
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server cinder.exception.ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing at byte 20401094656: No space left on device
This is strange because I have plenty of storage left
Controller/computenode packstack.openstack.local:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 12G 0 12G 0% /dev
tmpfs 12G 4.0K 12G 1% /dev/shm
tmpfs 12G 9.4M 12G 1% /run
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/mapper/cl-root 162G 27G 135G 17% /
/dev/mapper/cl-home 30G 246M 30G 1% /home
/dev/loop0 1.9G 6.1M 1.7G 1% /srv/node/swiftloopback
/dev/sda1 1014M 254M 761M 25% /boot
//192.168.178.103/RAIDZ1_0 11T 8.7T 1.6T 85% /mnt
tmpfs 7.1G 0 7.1G 0% /run/user/0
NOTE: I am also running 1 compute node, I dont know if it matters but this is the df of that VM:
compute.openstack.local: Compute Node
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.1M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/cl_compute-root 17G 2.9G 15G 17% /
/dev/sda1 1014M 254M 761M 25% /boot
tmpfs 372M 0 372M 0% /run/user/0
I am trying to create the instance on the controller/compute node. Could it be that the qcow2 image I made is not right? I can create a instance from the cirros image without any problems, but I think it has to do with the size of the volume I am trying to make.
I tried changing "image_conversion_dir" in the cinder.config file, but this did not help. I also looked what happened in iotop, you can see that there is a file writen to loop1 with a size of around 19GB. And I also see some cinder threads in iotop:
qemu-img convert -O raw -t none -f qcow2 /var/lib/cinder/conversion/tmpsak_p8tcpackstac~.local@lvm /dev/mapper/cinder--volumes-volume--1beee724--bedc--4ea8--8c45--ef0406339597
This is the full log (/var/log/cinder/volume.log):
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Failed to copy image 90772271-6a81-475a-82dd-69e04796079f to volume: e87171f3-99bc-4ef1-bdcc-eed55461cd18: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /mnt/openstackvm/tmpbpz37frrpackstack.openstack.local@lvm /dev/mapper/cinder--volumes-volume--e87171f3--99bc--4ef1--bdcc--eed55461cd18
Exit code: 1
Stdout: ''
Stderr: 'qemu-img: error while writing at byte 20401094656: No space left on device\n'
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils Traceback (most recent call last):
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1144, in copy_image_to_volume
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils context, volume, image_service, image_id)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/lvm.py", line 523, in copy_image_to_volume
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils size=volume['size'])
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 623, in fetch_to_raw
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils size=size, run_as_root=run_as_root)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 729, in fetch_to_volume_format
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils run_as_root=run_as_root)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 407, in convert_image
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils src_passphrase_file=src_passphrase_file)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 349, in _convert_image
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils utils.execute(*cmd, run_as_root=run_as_root)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils return processutils.execute(*cmd, **kwargs)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils cmd=sanitized_cmd)
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /mnt/openstackvm/tmpbpz37frrpackstack.openstack.local@lvm /dev/mapper/cinder--volumes-volume--e87171f3--99bc--4ef1--bdcc--eed55461cd18
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils Exit code: 1
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils Stdout: ''
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils Stderr: 'qemu-img: error while writing at byte 20401094656: No space left on device\n'
2021-03-26 15:11:35.881 3029 ERROR cinder.volume.volume_utils
2021-03-26 15:11:35.899 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ca0d7bf4-66a3-45aa-a5f4-b52335b741a3) transitioned into state 'FAILURE' from state 'RUNNING'
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Traceback (most recent call last):
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1144, in copy_image_to_volume
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager context, volume, image_service, image_id)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/lvm.py", line 523, in copy_image_to_volume
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager size=volume['size'])
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 623, in fetch_to_raw
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager size=size, run_as_root=run_as_root)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 729, in fetch_to_volume_format
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager run_as_root=run_as_root)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 407, in convert_image
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager src_passphrase_file=src_passphrase_file)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 349, in _convert_image
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager utils.execute(*cmd, run_as_root=run_as_root)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager cmd=sanitized_cmd)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /mnt/openstackvm/tmpbpz37frrpackstack.openstack.local@lvm /dev/mapper/cinder--volumes-volume--e87171f3--99bc--4ef1--bdcc--eed55461cd18
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Exit code: 1
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Stdout: ''
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Stderr: 'qemu-img: error while writing at byte 20401094656: No space left on device\n'
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager Traceback (most recent call last):
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager result = task.execute(**arguments)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1134, in execute
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager **volume_spec)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 683, in _wrapper
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager return r.call(f, *args, **kwargs)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager do = self.iter(retry_state=retry_state)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 356, in iter
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager return fut.result()
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager return self.__get_result()
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager raise self._exception
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager result = fn(*args, **kwargs)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1034, in _create_from_image
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager image_service)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 925, in _create_from_image_cache_or_download
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager image_service
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 738, in _create_from_image_download
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager image_service)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1149, in copy_image_to_volume
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager raise exception.ImageCopyFailure(reason=ex.stderr)
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager cinder.exception.ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing at byte 20401094656: No space left on device
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager
2021-03-26 15:11:35.899 3029 ERROR cinder.volume.manager
2021-03-26 15:11:35.906 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ca0d7bf4-66a3-45aa-a5f4-b52335b741a3) transitioned into state 'REVERTED' from state 'REVERTING'
2021-03-26 15:11:35.935 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d9c32ff4-91dd-4f98-bc02-4731fb5912bc) transitioned into state 'REVERTED' from state 'REVERTING'
2021-03-26 15:11:35.937 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (2ef60d35-ad24-472e-b465-a53e040c2f4d) transitioned into state 'REVERTED' from state 'REVERTING'
2021-03-26 15:11:35.948 3029 ERROR cinder.volume.flows.manager.create_volume [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Volume e87171f3-99bc-4ef1-bdcc-eed55461cd18: create failed
2021-03-26 15:11:35.950 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (2cb69002-2c95-42f7-8a7d-79418a320b59) transitioned into state 'REVERTED' from state 'REVERTING'
2021-03-26 15:11:35.954 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (29e0fce7-6629-4a2e-9f58-d7d4de96d4c4) transitioned into state 'REVERTED' from state 'REVERTING'
2021-03-26 15:11:35.959 3029 WARNING cinder.volume.manager [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Flow 'volume_create_manager' (23afb573-ff8f-43df-b65f-1ce3355be8d0) transitioned into state 'REVERTED' from state 'RUNNING'
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server [req-87414510-3b58-4b2f-beed-5da4cd50cf1c afcf8e04d2cc4a33ba5193c8ee593d7b c107b21eb64941168b780871fc5315e8 - - -] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing at byte 20401094656: No space left on device
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1144, in copy_image_to_volume
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server context, volume, image_service, image_id)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/lvm.py", line 523, in copy_image_to_volume
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server size=volume['size'])
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 623, in fetch_to_raw
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server size=size, run_as_root=run_as_root)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 729, in fetch_to_volume_format
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server run_as_root=run_as_root)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 407, in convert_image
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server src_passphrase_file=src_passphrase_file)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 349, in _convert_image
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server utils.execute(*cmd, run_as_root=run_as_root)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -t none -f qcow2 /mnt/openstackvm/tmpbpz37frrpackstack.openstack.local@lvm /dev/mapper/cinder--volumes-volume--e87171f3--99bc--4ef1--bdcc--eed55461cd18
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Exit code: 1
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Stdout: ''
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Stderr: 'qemu-img: error while writing at byte 20401094656: No space left on device\n'
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-751>", line 2, in create_volume
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 212, in wrapper
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 748, in create_volume
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server _run_flow()
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 740, in _run_flow
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server flow_engine.run()
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server failures[0].reraise()
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server raise value
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1134, in execute
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server **volume_spec)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 683, in _wrapper
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 356, in iter
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server return fut.result()
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server return self.__get_result()
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server raise self._exception
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1034, in _create_from_image
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server image_service)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 925, in _create_from_image_cache_or_download
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server image_service
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 738, in _create_from_image_download
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server image_service)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1149, in copy_image_to_volume
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server raise exception.ImageCopyFailure(reason=ex.stderr)
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server cinder.exception.ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing at byte 20401094656: No space left on device
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server
2021-03-26 15:11:35.972 3029 ERROR oslo_messaging.rpc.server
