[Starlingx-discuss] N3000 FPGA image update doesn't complete

[Venkata Ramana Veldanda]

Hi,

We are performing FPGA image update following the procedure defined below
https://docs.starlingx.io/node_management/starlingx-kubernetes/hardware_acceleration_devices/updating-an-intel-n3000-fpga-image.html

After running the below command, we see that image status stays in pending state for the last 2 images (although we see the first image update is completed). Although the doc says that it may take about 40 minutes, this has been there for 2 hours the image is in pending state. Any suggestions are appreciated. More information below.
~(keystone)admin)$ system host-device-image-update controller-0

[sysadmin at controller-0 ~(keystone_admin)]$ date;system device-image-state-list
Wed Jan 27 22:41:30 UTC 2021
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+
| hostname     | PCI device address | Device image uuid                    | status    | Update start time                | updated_at                       |
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+
| controller-0 | 0000:1c:00.0       | 58bcd3f5-213b-4c16-ac5e-6d6313cc2712 | completed | 2021-01-27T20:18:43.269482+00:00 | 2021-01-27T20:18:50.399593+00:00 |
| controller-0 | 0000:1c:00.0       | d31086d2-f7d6-4129-8421-a0d298fa21f1 | pending   | None                             | None                             |
| controller-0 | 0000:1c:00.0       | 3a04429f-916e-46b0-bff9-2f26c0a597c5 | pending   | None                             | None                             |
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+

[sysadmin at controller-0 ~(keystone_admin)]$ system host-show controller-0 |grep device_image_update
| device_image_update   | in-progress


The image update was triggered at the below timestamp.
bash info
2021-01-27T20:22:48.000 controller-0 -sh: info HISTORY: PID=1380366 UID=42425 system host-device-image-update controller-0

sysinv API has been triggered at the below timestamp.
2021-01-27 20:22:49.440 87255 INFO sysinv.api.controllers.v1.host [-] device_image_update host_uuid=26a50590-bbaa-41e8-a902-533250cca708

Checked after 2 hours nearly and the status is still the same.
Wed Jan 27 22:41:30 UTC 2021
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+
| hostname     | PCI device address | Device image uuid                    | status    | Update start time                | updated_at                       |
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+
| controller-0 | 0000:1c:00.0       | 58bcd3f5-213b-4c16-ac5e-6d6313cc2712 | completed | 2021-01-27T20:18:43.269482+00:00 | 2021-01-27T20:18:50.399593+00:00 |
| controller-0 | 0000:1c:00.0       | d31086d2-f7d6-4129-8421-a0d298fa21f1 | pending   | None                             | None                             |
| controller-0 | 0000:1c:00.0       | 3a04429f-916e-46b0-bff9-2f26c0a597c5 | pending   | None                             | None                             |
+--------------+--------------------+--------------------------------------+-----------+----------------------------------+----------------------------------+

Our understanding is that "host-device-image-update" is the command triggered once for all the pending images for controller-0 and that should automatically update all the pending images in one go.

Is there a place in /var/log where we can see the logs on the progress  and what might be going wrong?. Is it a good idea to do a force lock & unlock to recover from this state?

Below are the images that are being updated.
controller-0:/opt/platform/device_images# ls -lrt
total 44044
-rw-r--r-- 1 root root 1152 Jan 27 19:35 root-key-8086-0b30-58bcd3f5-213b-4c16-ac5e-6d6313cc2712.bit
-rw-r--r-- 1 root root 45089792 Jan 27 19:40 functional-8086-0b30-d31086d2-f7d6-4129-8421-a0d298fa21f1.bit
-rw-r--r-- 1 root root 1152 Jan 27 19:41 key-revocation-8086-0b30-3a04429f-916e-46b0-bff9-2f26c0a597c5.bit


Venkata Veldanda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210127/f379c330/attachment-0001.html>