September 2022 - Starlingx-discuss

[Starlingx-discuss] TSC & Community call in the new alternating time slot
by Ildiko Vancsa 07 Sep '22

07 Sep '22

Hi StarlingX Community, It is a friendly reminder that this is the first occasion when we will have the TSC & Community call in the new APAC-friendly time slot. __The call will happen at 7pm PDT / 10pm EDT / 0200 UTC / (Thursday) 10am CST.__ For details please check the project’s Meetings wiki page: https://wiki.openstack.org/wiki/Starlingx/Meetings#7pm_Pacific_on_Every_Fir… Thanks and Best Regards, Ildikó ——— Ildikó Váncsa Senior Manager, Community & Ecosystem Open Infrastructure Foundation

1 0

[Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 337 - Still Failing!
by starlingx.build＠gmail.com 06 Sep '22

06 Sep '22

Project: STX_build_layer_containers_master_master Build #: 337 Status: Still Failing Timestamp: 20220905T230914Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false

3 2

[Starlingx-discuss] OpenInfra Board of Directors - StarlingX Sync
by Ildiko Vancsa 06 Sep '22

06 Sep '22

Hi StarlingX Community, I’m reaching out to you to organize a call between the StarlingX project and the OpenInfra Foundation Board of Directors (BoD). The Open Infrastructure Foundation BoD is looking into engaging in regular check-ins with official OpenInfra projects in an online format. The goal of these meetings is to establish a closer connection and collaboration between the OpenInfra projects, including StarlingX, and the OpenInfra BoD group. This will help the Board to have a clearer view about each project’s state, achievements and challenges and it will be in a better position to support all the communities. At the same time, the projects will get a better understanding of the Board’s structure and responsibilities and have a better alignment with the group. The aim is to schedule a 1-hour long call some time in the upcoming weeks that will involve members of the Board as well as the StarlingX TSC along with further interested community members from the project. There is no expectation for a formal presentation from the projects and with that no materials have to be prepared in advance. If the community would like to provide an update in the form of a presentation that should work too. I created an etherpad to collect potential topics for the conversation and build an agenda for the call: https://etherpad.opendev.org/p/2022-summer-bod-starlingx-sync I also created a poll to try to find time slot options that we can share with the OpenInfra Board of Directors. The poll is covering options for August, if we fail to find good options for that month, I will extend it to September. Please find the poll here: https://doodle.com/meeting/participate/id/b48Pv7Vb Both our BoD and community are distributed all around the globe and there is no time slot that works well well everywhere. In that sense, please mark ALL the time slots when you are available, so we can find one that is accessible for most of our Board and community members. __Please submit your preferences by the end of day on July 20, 2022.__ Please let me know if you have any questions. Best Regards, Ildikó ——— Ildikó Váncsa Senior Manager, Community & Ecosystem Open Infrastructure Foundation

2 12

[Starlingx-discuss] FW: Incorrect public key for signed starlingX 5.0 iso .
by Waines, Greg 06 Sep '22

06 Sep '22

Forwarding …. I think scott meant this for the starlingx discuss list. Greg. From: Little, Scott <Scott.Little(a)windriver.com> Sent: Wednesday, July 6, 2022 2:37 PM To: Waines, Greg <Greg.Waines(a)windriver.com> Subject: Re: [Starlingx-discuss] Incorrect public key for signed starlingX 5.0 iso . I'll attempt to answer this at a high level. Secure boot signing uses a client/server approach. build-pkgs acts as the client. The server holds the signing scripts and the keys used for signing. The client side code was released to open source as part of StarlingX. The server side code was not released. The interface between client and server is currently undocumented. I'll try to give an overview as best I can. Basically it is a series of ssh/scp calls made into signing server. 'build-pkgs' requires that the following environment variables be defined before it will attempt to request a secure boot signing. export SIGNING_SERVER=<signing-host> export SIGNING_USER=<signing-user> export SIGNING_SERVER_SCRIPT=<path-to-signing-script> 'build-pkgs' further requires that "$USER" == "jenkins", and export FORMAL_BUILD=1 If that is satisfied, it calls into 'sign-secure-boot' 'sign-secure-boot' contains a fairly good set of comments describing what happens next. The client side call sequence looks like this ... # Set up the server side directory for files transfers. UPLOAD_PATH=`ssh $SIGNING_USER@$SIGNING_SERVER sudo $SIGNING_SCRIPT -r` # upload the original package scp -q $FILE $SIGNING_USER@$SIGNING_SERVER:$UPLOAD_PATH # Request that the package be signed ssh $SIGNING_USER@$SIGNING_SERVER sudo $SIGNING_SCRIPT -v -i $UPLOAD_PATH/$(basename $FILE) $UNSIGNED_OPTION -t $TYPE > $TMPFILE # Download the file from the signing server DOWNLOAD_FILENAME=$(basename $OUTPUT_FILE) scp -q $SIGNING_USER@$SIGNING_SERVER:$OUTPUT_FILE $(dirname $FILE) Within the signing server there are two keys used for signing, known as the 'boot' key and the 'shim' key. The public half of the 'boot' key must manually added to the secure boot keychain in firmware. The 'boot' key signs first executable loaded, contained in the 'shim' package. The first executable must then install the public half of the shim key (automatically) before passing control to grub, and ultimately the kernel, both of which are signed by the 'shim' key. Three packages need to be passed to the signing server. The rpms need to be unpacked, the relevant binaries signed with correct keys, and the rpms reassembled. package key files to sign ========= ==== =========================== shim boot BOOTX64, shim, shimx64 shim MokManager, fallback, mmx64, fbx64 grub shim grubx64.efi, gcdx64.efi kernel shim vmlinuz NOTE: shim files to sign might include a '.efi' or '.EFI' suffix. Sign those as well. NOTE: some files may be absent in newer packages. It is probably ok if some of the above are missing. Signing looks like this ... sbsign --key $KEYPATH/$KEYNAME.key --cert $KEYPATH/$KEYNAME.crt --output $SIGNEDFILE $UNSIGNEDFILE On keys and certs ... boot.crt - Certificate to boot (to be programmed in firmware) boot.key - Private key with which to sign shim shim.crt - Certificated embedded within shim used to validate kernel, grub shim.key - Private key with which to sign kernel/grub key generation ... openssl req -new -x509 -newkey rsa:2048 -keyout $KEY.key -out $KEY.pem -days 3650 openssl x509 -in $KEY.pem -out $KEY.crt -outform DER Note: boot.crt should be copied to cgcs-root/build-tools/certificates/TiBoot.crt for inclusion during the 'build-iso' step Hope this helps Scott On 2022-07-05 03:19, Kumar, Chandan wrote: [Please note: This e-mail is from an EXTERNAL e-mail address] Hi Greg, Thanks for acknowledgement. I have raised starlingx launchpad for document update. Meanwhile it would be really great if you can share the steps which has to be done to integrate signing of images as part of StarlingX build infrastructure. Regards, Chandan Kumar. From: Waines, Greg <Greg.Waines(a)windriver.com><mailto:Greg.Waines@windriver.com> Sent: Monday, July 4, 2022 6:00 PM To: starlingx-discuss(a)lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Little, Scott <Scott.Little(a)windriver.com><mailto:Scott.Little@windriver.com> Subject: Re: [Starlingx-discuss] Incorrect public key for signed starlingX 5.0 iso . Hey Chandan, Thanks for bringing this up. I chatted with Scott about this. Currently the StarlingX public builds on CENGN are not building a secure boot capable load … i.e. signing the appropriate items to enable secure boot. Although the StarlingX build infrastructure allows it to be added. Neither of these points are properly documented in docs.starlingx.io ☹ I can’t remember the details of why we are not doing this. I am guessing that the thinking was that a user of StarlingX that wanted to use UEFI Secure Boot, would want to sign with his own private key. Can you raise a starlingx launchpad ( https://launchpad.net/starlingx<https://urldefense.com/v3/__https:/secure-web.cisco.com/1mVLF-b4c-tBpQaEbNZ…> ) to address the docs.starlingx.io<https://urldefense.com/v3/__http:/docs.starlingx.io__;!!AjveYdw8EvQ!ftQxVQD…> documentation issue, i.e. to indicate that the StarlingX CENGN builds are not signed to support uefi secure boot, and describe how a starlingx user could add signing to their StarlingX build environment in order to sign for uefi secure boot with their own private key ? Greg. From: Scott Little <scott.little(a)windriver.com<mailto:scott.little@windriver.com>> Sent: Wednesday, June 29, 2022 10:42 AM To: starlingx-discuss(a)lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Incorrect public key for signed starlingX 5.0 iso . On 2022-06-02 13:12, Kumar, Chandan wrote: [Please note: This e-mail is from an EXTERNAL e-mail address] Hi, I am trying to validate signed starlingX 5.0 iso on a secureboot enabled setup against public key(TiBoot.crt) embedded inside iso. After rebooting , operating system is not coming up with error screen saying “Verification failed: (0X1A) Security Violation”. However, when I signed images with self-generated private key and validated against corresponding public key, system is able to boot up successfully. Signing is done after flashing the iso on a server. Please find attached procedure for signing images inside iso. I believe, the public key provided with iso is not correct. Can you please confirm ? Regards, Chandan Kumar. _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss(a)lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__https:/secure-web.cisco.com/1-KIhdgnDGw2i_w1bSf…> Thanks for the report. I'll look into it Scott Little

5 5

[Starlingx-discuss] [stable] [build-report] STX_build_docker_flock_images - Build # 580 - Still Failing!
by starlingx.build＠gmail.com 06 Sep '22

06 Sep '22

Project: STX_build_docker_flock_images Build #: 580 Status: Still Failing Timestamp: 20220906T001905Z Branch: Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… -------------------------------------------------------------------------------- Parameters WEB_HOST: mirror.starlingx.cengn.ca MY_WORKSPACE: /localdisk/loadbuild/jenkins/master-containers/20220905T230914Z OS: centos MY_REPO: /localdisk/designer/jenkins/master-containers/cgcs-root BASE_VERSION: master-stable-20220905T230914Z PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… REGISTRY_USERID: slittlewrs LATEST_PREFIX: master PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/containers/20220905T230914Z/logs PUBLISH_TIMESTAMP: 20220905T230914Z FLOCK_VERSION: master-centos-stable-20220905T230914Z WEB_HOST_PORT: 80 PREFIX: master TIMESTAMP: 20220905T230914Z BUILD_STREAM: stable REGISTRY_ORG: starlingx PUBLISH_OUTPUTS_BASE: /export/mirror/starlingx/master/centos/containers/20220905T230914Z/outputs REGISTRY: docker.io

2 1

[Starlingx-discuss] FW: Green: AIO-SX STX_7.0_build_layer_flock (202208311118)
by Cooke, Rob 06 Sep '22

06 Sep '22

Hi, Below are the results of the Sanity execution on the latest STX7.0 Release Candidate build, http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/7.0/centos/flock/20220…. We had a successful sanity. We ran into a few issues on the STX+Openstack sanity run and are in the process of re-executing that run. More to come for openstack. Thanks, Rob From: Winnicki, Chris <Chris.Winnicki(a)windriver.com> Sent: Wednesday, August 31, 2022 1:26 PM Subject: Green: AIO-SX STX_7.0_build_layer_flock (202208311118) Overall Status: GREEN Lab: WCP_111 Node Config: AIO-SX Load: r/stx.7.0 Job: STX_7.0_build_layer_flock Test Logs: abstract log: test_steps.log, pytestlog.log full log: TIS_AUTOMATION.log http://128.224.186.235/auto_logs/wcp_111/202208311623 All Cases: PASS testcases/functional/common/test_system_health.py::test_system_health_pre_session[pods] PASS testcases/functional/common/test_system_health.py::test_system_health_pre_session[alarms] PASS testcases/functional/common/test_system_health.py::test_system_health_pre_session[system_apps] PASS testcases/functional/mtc/test_lock_unlock_host.py::test_lock_unlock_host[controller] PASS testcases/functional/networking/test_pod_to_pod.py::TestPodtoPod::test_pod_to_pod_connection PASS testcases/functional/networking/test_pod_to_pod.py::TestPodtoPod::test_pod_to_service_connection PASS testcases/functional/networking/test_pod_to_pod.py::TestPodtoPod::test_host_to_service_connection PASS testcases/wrcp/regression/horizon/test_hosts.py::test_horizon_host_inventory_display http://128.224.150.21/jenkins/job/WRCP_SANITY/147/artifact/report.html Chris Winnicki chris.winnicki(a)windriver.com<mailto:chris.winnicki@windriver.com> 613-963-1329

1 2

[Starlingx-discuss] [stable] [build-report] master STX_build_docker_images_layered - Build # 336 - Still Failing!
by starlingx.build＠gmail.com 05 Sep '22

05 Sep '22

Project: STX_build_docker_images_layered Build #: 336 Status: Still Failing Timestamp: 20220905T234656Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… -------------------------------------------------------------------------------- Parameters BRANCH: master MY_WORKSPACE: /localdisk/loadbuild/jenkins/master-containers/20220905T230914Z OS: centos MUNGED_BRANCH: master MY_REPO: /localdisk/designer/jenkins/master-containers/cgcs-root PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… MASTER_BUILD_NUMBER: 337 PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/containers/20220905T230914Z/logs MASTER_JOB_NAME: STX_build_layer_containers_master_master MY_REPO_ROOT: /localdisk/designer/jenkins/master-containers PUBLISH_DISTRO_BASE: /export/mirror/starlingx/master/centos/containers PUBLISH_TIMESTAMP: 20220905T230914Z DOCKER_BUILD_ID: jenkins-master-containers-20220905T230914Z-builder TIMESTAMP: 20220905T230914Z OS_VERSION: 7.5.1804 BUILD_STREAM: stable PUBLISH_INPUTS_BASE: /export/mirror/starlingx/master/centos/containers/20220905T230914Z/inputs LAYER: containers PUBLISH_OUTPUTS_BASE: /export/mirror/starlingx/master/centos/containers/20220905T230914Z/outputs

1 0

[Starlingx-discuss] Attempt to PCI Passthrough in Stx ended up with "no valid host available" error
by open infra 05 Sep '22

05 Sep '22

Hi, I recently noticed that newly added workers not able to host any openstack instance but only few workers (which created very early in our setup) able to host instanes created in openstack side. system application-apply stx-openstack is failed at downloading docker images though controller can ping outside. No specific information found on system inventory log and no armada log at all for stx-openstack-apply. When I looking at stx active alarms, there was a failure of stx-openstack apply, which was on the day apply changes to enable PCI Pasthrough. [1]. I can see all the workers when I check nodes from kubernetes side. 01. How to get the stx-openstack application apply without fail at docker image download? I tried to remove the line added into helm override yaml to add pci passthrough device and apply the application but it failed. 02. I need to enable Nvidia T10 cards in selected nodes as PCI Passtrough. Should we add all selected gpu pci addresses to controller? Is there a complete guide for enabling PCI Passthorugh with GPU cards in starlingx side as well as openstack setup as well? [1] https://docs.starlingx.io/node_management/openstack/configure-pci-passthrou… Regards, Danishka

1 0

[Starlingx-discuss] [build-report] master STX_build_master_master - Build # 1399 - Failure!
by starlingx.build＠gmail.com 03 Sep '22

03 Sep '22

Project: STX_build_master_master Build #: 1399 Status: Failure Timestamp: 20220903T043002Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: false FORCE_BUILD: true

1 0

[Starlingx-discuss] [build-report] STX_build_pre_installer - Build # 1542 - Failure!
by starlingx.build＠gmail.com 03 Sep '22

03 Sep '22

Project: STX_build_pre_installer Build #: 1542 Status: Failure Timestamp: 20220903T044455Z Branch: Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/master/20220903T043002Z DOCKER_BUILD_ID: jenkins-master-20220903T043002Z-builder OS: centos MY_REPO: /localdisk/designer/jenkins/master/cgcs-root PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/… PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/monolithic/20220903T043002Z/logs MASTER_JOB_NAME: STX_build_master_master MY_REPO_ROOT: /localdisk/designer/jenkins/master

1 0