Hi, We are performing FPGA image update following the procedure defined below https://docs.starlingx.io/node_management/starlingx-kubernetes/hardware_acce... After running the below command, we see that image status stays in pending state for the last 2 images (although we see the first image update is completed). Although the doc says that it may take about 40 minutes, this has been there for 2 hours the image is in pending state. Any suggestions are appreciated. More information below. ~(keystone)admin)$ system host-device-image-update controller-0 [sysadmin@controller-0 ~(keystone_admin)]$ date;system device-image-state-list Wed Jan 27 22:41:30 UTC 2021 +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ | hostname | PCI device address | Device image uuid | status | Update start time | updated_at | +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ | controller-0 | 0000:1c:00.0 | 58bcd3f5-213b-4c16-ac5e-6d6313cc2712 | completed | 2021-01-27T20:18:43.269482+00:00 | 2021-01-27T20:18:50.399593+00:00 | | controller-0 | 0000:1c:00.0 | d31086d2-f7d6-4129-8421-a0d298fa21f1 | pending | None | None | | controller-0 | 0000:1c:00.0 | 3a04429f-916e-46b0-bff9-2f26c0a597c5 | pending | None | None | +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-0 |grep device_image_update | device_image_update | in-progress The image update was triggered at the below timestamp. bash info 2021-01-27T20:22:48.000 controller-0 -sh: info HISTORY: PID=1380366 UID=42425 system host-device-image-update controller-0 sysinv API has been triggered at the below timestamp. 2021-01-27 20:22:49.440 87255 INFO sysinv.api.controllers.v1.host [-] device_image_update host_uuid=26a50590-bbaa-41e8-a902-533250cca708 Checked after 2 hours nearly and the status is still the same. Wed Jan 27 22:41:30 UTC 2021 +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ | hostname | PCI device address | Device image uuid | status | Update start time | updated_at | +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ | controller-0 | 0000:1c:00.0 | 58bcd3f5-213b-4c16-ac5e-6d6313cc2712 | completed | 2021-01-27T20:18:43.269482+00:00 | 2021-01-27T20:18:50.399593+00:00 | | controller-0 | 0000:1c:00.0 | d31086d2-f7d6-4129-8421-a0d298fa21f1 | pending | None | None | | controller-0 | 0000:1c:00.0 | 3a04429f-916e-46b0-bff9-2f26c0a597c5 | pending | None | None | +--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+ Our understanding is that "host-device-image-update" is the command triggered once for all the pending images for controller-0 and that should automatically update all the pending images in one go. Is there a place in /var/log where we can see the logs on the progress and what might be going wrong?. Is it a good idea to do a force lock & unlock to recover from this state? Below are the images that are being updated. controller-0:/opt/platform/device_images# ls -lrt total 44044 -rw-r--r-- 1 root root 1152 Jan 27 19:35 root-key-8086-0b30-58bcd3f5-213b-4c16-ac5e-6d6313cc2712.bit -rw-r--r-- 1 root root 45089792 Jan 27 19:40 functional-8086-0b30-d31086d2-f7d6-4129-8421-a0d298fa21f1.bit -rw-r--r-- 1 root root 1152 Jan 27 19:41 key-revocation-8086-0b30-3a04429f-916e-46b0-bff9-2f26c0a597c5.bit Venkata Veldanda