November 2023 - Starlingx-discuss

Bi-Weekly StarlingX OS Distro & Multi-OS meeting Oct 4, 2023
by Wang, Linda 16 Nov '23

16 Nov '23

Hi, Below is the meeting minutes for OS Distro & Multi-OS meeting of Oct 4th: Oct 4, 2023 Attendees: Charles Short, Davlet, Elaine, Juanita, Mark A. * 1.2 Log into the git before download the source. * How to pass the credential through the tooling layer when log into git repo. * workaround: currently use repo sync outside of the containers. * AI: Scott to write up a feature request for the tool - file a request in Story Board. * Created: https://storyboard.openstack.org/#!/story/2010777, (create a jira for it) * 1.3 Pre-patch ISO: * The Prepatched ISO will contain an ostree-repo with a ‘single’ commit containing the final state of the rootfs after all the patches have been applied * i.e. note that for size reasons, we should not have a commit for each patch and metadata for all the patches that have been applied. * In summary, the GA ISO and Prepatched ISOs will generally be the same in that they both have an ostree-repo with a single ostree-commit, the prepatched iso * will have the GA content + all patches and will use apt-ostree (Charles Short). * Open Questions: How to patch grub and other files that are not managed by ostree * A: this is covered in the upstream generic OSTree documentation. OSTree can cover the /boot directory when needed. 2. ARM Support * Skip due to CDC national holidays * 3. 3. StarlingX 8.0 * SPEC from Chuck on OSTree generation on the controller, pending review. * Work is in progress. Next Meeting: Oct 18, 2023

1 0

[Important] Please include download and build tests
by Scott Little 16 Nov '23

16 Nov '23

Hi all The November 14 build failed on a newly added package. The code, as submitted, failed to build on the 'download' step. Please verify that your code can pass a download and build before submitting it for gerrit review. Scott

1 0

Bi-Weekly StarlingX OS Distro & Multi-OS meeting Sept 20, 2023
by Wang, Linda 16 Nov '23

16 Nov '23

Hi, Below is the Meeting Minutes for OS Distro & Multi-OS meeting minutes on September 20: (Please ignore if this is a duplicate) Sept 20, 2023 Attendees: Davelt, Charles Short, Elaine, Elisamara, Mark, Tianyu, Scott 1. Gernerl Topics 1.1 Split package list betweeen repositories (ISO and container images) * Separeation of the list is done per Davlet. * AI: Devlet will check the documentation on package list. [Will need to bring this up in other forums, and will not track in this meeting] * 1.2 Log into the git before download the source. * How to pass the credential through the tooling layer when log into git repo. * workaround: currently use repo sync outside of the containers. * AI: Scott to write up a feature request for the tool - file a request in Story Board. * Created: https://storyboard.openstack.org/#!/story/2010777, (create a jira for it) * 1.3 Pre-patch ISO: * The Prepatched ISO will contain an ostree-repo with a ‘single’ commit containing the final state of the rootfs after all the patches have been applied * i.e. note that for size reasons, we should not have a commit for each patch and metadata for all the patches that have been applied. * In summary, the GA ISO and Prepatched ISOs will generally be the same in that they both have an ostree-repo with a single ostree-commit, the prepatched iso * will have the GA content + all patches and will use apt-ostree (Charles Short). * Open Questions: How to patch grub and other files that are not managed by ostree * A: this is covered in the OSTree documentation. OSTree can cover the /boot directory when needed. 2. ARM Support * a. Equipment * Steve gave the addresses for the location of the hosts, 4 in Alameda, and 1 in Ottawa lab, 1 remains to confirm. * ARM is sending those machines to these locations. Steve update the address of the Ottawa office. * pre-installed? * b. CI/CD updates * CI/CD try to set up the workflow in ARM's local lab (ARM's Shanghai Lab), applied Jackie's patches, but may still need more patches to do builds. * Tianyu is trying to reproduce the environment in local lab, but still have some failures. Limited bandwidth. * c. Porting still in progreess (Jakie) * rewrote the coding to support build system for both x86 and arm * build system will detect the arch, and if it is arm, it will build arm arch. * Wrote up the SPEC on the implementtaion., and now waiting on SPEC to be review * SPEC review first then will send code for code review. * Some kernel modules don't work * QEMU is not working yet. Waiting on kernel up-rev, then and see if QEMU can be fixed then. * Created 2 wikipages for setting up the build environment for ARM arch: * https://wiki.openstack.org/wiki/StarlingX/StarlingX_Debian_on_ARM_Informati… * https://wiki.openstack.org/wiki/StarlingX/DebianArmBuildEnvironment * SPEC file review continoues, Kernel SPEC file needs reviewer. * Need more community reviewers. * AI: check with Jackie to see if the porting is finished. * AI: Mark A. will like to review the SPECC before it is commited. * d. Packages for x86_64 are in CNGN, need to push ARM packages there. * Set up the own (CNGN like) website, push the packages to the website as part of CI/CD website (/export/mirror) * Secureboot is disabled for now. Will do it in the future. * ELF SDK is used to build image. Need to push the SDK to CNGN, no permission to push to CNGN yet, it is on local disk. * AI: Need to provide permission to Jackie so that he can push ELF SDK to CNGN. * Need to change meta-lat layer for ELF SDK, but want to link the task to different repo, but can't. * meta-lat layer is not in the StarlingX repo, but its in github repo. fork it.. do a pull request to github lab for any changes. * meta-lat used is with the WR's repo, not community one. Will migrate later. * e. OSTree * removing OSTree from the installer ISO. (not from the StarlingX) * Work in progress, not sure when it will be ready. Possible early next year, StarlingX 9. 3. StarlingX 8.0 * SPEC from Chuck on OSTree generation on the controller, pending review. * Work is in progress. Next Meeting: Oct 4, 2023

1 0

[Starlingx-discuss] Mail delivery issues and DMARC settings - VOTE NEEDED
by Ildiko Vancsa 15 Nov '23

15 Nov '23

Hi StarlingX Community, I’m reaching out to you about the starlingx-discuss mailing list. It’s been an issue for some subscribers in the community to receive emails from the mailing list, which can potentially be improved by changing some of the configuration options. Some mail servers on the client side either mark emails as spam or reject incoming emails depending on dmarc configuration settings. There are multiple options to try and be compatible to most mail server configurations. Options, based on other OpenInfra communities’ approaches: 1. Turn off various list features which alter the message headers and body (disabling the list features that add a prefix to the subject, that append a footer with the unsubscribe instructions, and setting it not to add a reply-to header) - Communities using this option: OpenDev, Zuul, OpenStack 2. Replace From: with list address as the mitigation action, and not doing it unconditionally - Communities using this option: Kata Containers Both options change how the mailing list alters and delivers the messages. What is your preference moving forward? __Please vote with either ‘option 1’ or ‘option 2’, or ’none’ if you would prefer a different approach.__ Please let me know if you have any questions. Best Regards, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation

1 2

[Starlingx-discuss] Recap of StarlingX presentation and discussion at OCP Edge monthly call
by Ildiko Vancsa 15 Nov '23

15 Nov '23

Hi StarlingX Community, I’m reaching out you about a recent OCP Edge WG meeting that I attended, and gave a presentation about StarlingX to raise awareness about the project in that community and seek for cross-community collaboration opportunities. Brief recap about the presentation and the meeting: - The main topic during the call was an introductory presentation about StarlingX. The outline of topics that I covered: * Introduction of the project as an open source, distributed cloud platform: ~ Debian integrated as base operating system ~ Support for both hardware and software infrastructure management * It utilizes container technologies on both the platform and application layers ~ The infrastructure services are running in containers ~ It supports running only containerized workloads at edge sites for small footprint * Some details about HW support and requirements in the platform * I highlighted that StarlingX is currently running in production at large telecom operators such as Verizon, Vodafone, T-Systems and more - Questions and collaboration areas that came up during the call included: * Is StarlingX supporting or looking into Terraform (now OpenTofu) and Ansible? * Redfish profiles and sensor support in StarlingX ~ What is supported? ~ Working on Redfish profiles together can be a collaboration opportunity * What support and mechanisms are there for firmware updates in StarlingX? * StarlingX is working on ARM support, which was in interest for the OCP Edge group As a next step, we discussed having someone from OCP Edge to give an overview presentation at an upcoming StarlingX call or at the PTG. I’m currently waiting for a response from the group in terms of what they would prefer. Please share any information and comments you might have about the questions that came up and further notes, as well as any questions you might have about the above summary. Thanks and Best Regards, Ildikó ——— Ildikó Váncsa Director of Community Open Infrastructure Foundation

1 2

[Starlingx-discuss] Sanity Master Test LAYERED build ISO 20231112T070059Z
by Peng, Peng 15 Nov '23

15 Nov '23

Sanity Test from November 14, 2023 (https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic…) Status: Yellow SX sanity Passed: 17 (100.0%) Failed: 0 (0.0%) Total Executed: 17 List of Test Cases: ------------------------------------------------------ PASS test_system_health_pre_session[pods] PASS test_system_health_pre_session[alarms] PASS test_system_health_pre_session[system_apps] PASS test_horizon_host_inventory_display PASS test_lock_unlock_host PASS test_pod_to_pod_connection PASS test_pod_to_service_connection PASS test_host_to_service_connection PASS test_push_docker_image_to_local_registry_active PASS test_upload_charts_via_helm_upload PASS test_host_operations_with_custom_kubectl_app PASS test_isolated_2p_2_big_pod_best_effort_HT_AIO PASS test_sriovdp_netdev_single_pod[1-1-lock/unlock] PASS test_sriovdp_netdev_connectivity_ipv4[1-1-calico-ipam] PASS test_sriovdp_mixed_add_vf_interface[1] PASS test_system_coredumps_and_crashes[core_dumps] PASS test_system_coredumps_and_crashes[crash_reports] A new Launchpad opened for the DX issue, https://bugs.launchpad.net/starlingx/+bug/2043506 - DX after host-swact, standby controller-services in disabled-failed status Regards, PV team

1 0

[Starlingx-discuss] Minutes: Community Call (Nov 8, 2023)
by Khalil, Ghada 14 Nov '23

14 Nov '23

Etherpad: https://etherpad.opendev.org/p/stx-status Minutes from the community call Nov 8, 2023 Standing topics - Build - Main Branch Debian Builds - Green - Build Output: https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/ - Antelope Feature Branch Debian Builds - No longer needed. Antelope openstack content merged to the main branch. - Build Output: https://mirror.starlingx.cengn.ca/mirror/starlingx/feature/antelope/debian/ - stx.6.0 Weekly RC Builds - Red. Issue with the repo tool. Davlet implemented a workaround and will monitor the next weekly build. - Build Output: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/6.0/ - Note: No container image builds are done for stx.6.0 - stx.7.0 Weekly RC Builds - Red. Issue with the repo tool. Davlet implemented a workaround and will monitor the next weekly build. - Build Output: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/7.0/ - Note: No container image builds are done for stx.7.0 - stx.8.0 Weekly RC Builds - Green - Build Output: https://mirror.starlingx.cengn.ca/mirror/starlingx/rc/8.0/ - Sanity - Debian Main Branch Platform Sanity - Last sanity email sent on Nov 7: https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Green for SX. Red for DX. - LP: https://bugs.launchpad.net/starlingx/+bug/2042953 - Introduced by recent merge on Nov 2; assigned to the networking team - Debian stx-openstack (Antelope, Main Branch) Sanity - Last sanity email sent on Nov 8: https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Red - LP: https://bugs.launchpad.net/starlingx/+bug/2042957 - appears to be introduced recently since this issue was not seen in the Oct 31 sanity. - There is an nginx upversion recently, so this could be the reason. Thales/Ghada to follow-up with the developer. - Gerrit Reviews in Need of Attention - Doc Reviews: - https://review.opendev.org/q/project:starlingx/docs+status:open - ARM port Reviews: - https://review.opendev.org/q/topic:arm64%252F20230725-stx-master-native - Reference Links: - Active Branch (open): https://review.opendev.org/q/projects:starlingx+is:open+branch:+master - Active Branch (merged): https://review.opendev.org/q/projects:starlingx+is:merged+branch:master Topics for this week - Concern about Starlingx Reviews with too many reviewers - e.g. https://review.opendev.org/c/starlingx/tools/+/897100 - unclear if we have the Gerrit Review process documented for StarlingX ... which would include such recommendations as what reviewers to include on Gerrit reviews - Greg to check with Ghada on whether this is currently documented or not. - No there are no explicit guidelines regarding the max number of reviewers. - opendev Code Review guidelines: - https://docs.opendev.org/opendev/infra-manual/latest/developers.html#code-r… - https://docs.openstack.org/contributors/code-and-documentation/using-gerrit… - StarlingX Contributor Guide / Code Submission Guideline - https://docs.starlingx.io/contributor/ - https://docs.starlingx.io/developer_resources/code-submission-guide.html - Action: SteveG volunteered to work with the doc team to update the current StarlingX code submission guidelines for this topic - ScottK's team is investigating support for rook-ceph - Scott is looking for someone more familiar with system applications / previous rook-ceph integration. - Greg suggested to reach out to Bob Church. Action: Greg to setup an email intro (cc'ing the mailing list) ARs from Previous Meetings - StarlingX Mailing List Issues - WR email accounts not receiving all emails from the mailing list; Jeremy investigating list settings - Ildiko sent a summary with options to the mailing list and matrix. No feedback received. Deadline for feedback is Nov 10. - Reference: https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - StarlingX Presentation in OCP Edge - Presented by Ildiko. Readout will be sent to the mailing list. - Interest was expressed in StarlingX; specifically the upcoming kata containers integration - Ildiko is also talking to OCP Edge to do a presentation to StarlingX and discuss collaboration ideas - No further updates Open Requests for Help - StarlingX Installation Help - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Open - Some further discussion on the thread from Bruno and the reporter. Unclear if anything is still outstanding. - Bruno: I tried to get more detailed info with more direct questions but answers were confusing. In my last email (October 20th) I asked them to try with version 8, no response so far (and I just noticed that I didn't include the list in this message). - ARM port - Issue w/ Debian mirror snapshots - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Complete. Davlet addressed the encountered bug and the fix was merged. - Request for assistance with reviews - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Moved to the "Gerrit reviews in need of attention" - Direction on the CI/CD setup, if someone can help answer his question: - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… - Status: Open?? Some responses provided on the thread. - Requires Build Team (Davlet) to provide input ... busy, but will try to provide input.

1 0

[Starlingx-discuss] Sanity and Regression - StarlingX MASTER build [20231029T060059Z] + STX-Openstack [20231111T] results - Nov-13
by Calixto de Paula, Gabriel 13 Nov '23

13 Nov '23

Hi all, Just a quick note, this sanity had to be executed using the StarlingX master Oct-29 build, as the builds after this date won't apply stx-openstack, more info at: #2042957 - STX-O Master failing to apply : "validate.nginx.ingress.kubernetes.io" denied the request <https://bugs.launchpad.net/starlingx/+bug/2042957> <https://bugs.launchpad.net/starlingx/+bug/2042957> StarlingX master + STX-Openstack MASTER sanity results: Overall Status: Yellow Build Info: Build date: Oct-29 - 20231029T060059Z ISO:https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20231029T060059Z/outputs/iso/starlingx-intel-x86-64-cd.iso STX-O master custom build date:Nov-11 AIO-DX Baremetal with VSWITCH_TYPE=OVS Sanity Results: Overall Status: Yellow Automated Test Results Summary: ------------------------------------------------------ Passed: 13 (86.67%) Failed: 2 (13.33%) Total Executed: 15 List of Test Cases: ------------------------------------------------------ PASS 20231113 12:13:38 test_ssh_to_hosts PASS 20231113 12:14:57 test_lock_unlock_host PASS 20231113 12:30:49 test_openstack_services_healthy PASS 20231113 12:31:47 test_reapply_stx_openstack_no_change[controller-0] PASS 20231113 12:33:48 test_reapply_stx_openstack_no_change[controller-1] PASS 20231113 12:41:42 test_horizon_create_delete_instance PASS 20231113 12:44:16 test_swact_controllers PASS 20231113 12:51:27 test_ping_between_two_vms[tis-centos-guest-virtio-virtio] FAIL 20231113 12:56:43 test_migrate_vm[tis-centos-guest-live-None] PASS 20231113 13:04:51 test_nova_actions[tis-centos-guest-dedicated-pause-unpause] PASS 20231113 13:08:24 test_nova_actions[tis-centos-guest-dedicated-suspend-resume] FAIL 20231113 13:11:34 test_evacuate_vms PASS 20231113 13:44:53 test_system_coredumps_and_crashes[core_dumps] PASS 20231113 13:45:08 test_system_coredumps_and_crashes[crash_reports] PASS 20231113 13:45:14 test_system_alarms ----------------------------------------------------------------------- #2043389 - STX-O | VM Failed to allocate the network <https://bugs.launchpad.net/starlingx/+bug/2043389> caused the 2 failures found in this run. Thanks STX-Openstack Distro Team

1 0

[Starlingx-discuss] [build-report] r/stx.7.0 STX_7.0_build_layer_distro - Build # 58 - Failure!
by starlingx.build＠gmail.com 12 Nov '23

12 Nov '23

Project: STX_7.0_build_layer_distro Build #: 58 Status: Failure Timestamp: 20231112T005033Z Branch: r/stx.7.0 Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/7.0/centos/distro/2023… -------------------------------------------------------------------------------- Parameters FULL_BUILD: false FORCE_BUILD: true

1 0

[Starlingx-discuss] [build-report] r/stx.7.0 STX_repo_sync_layered - Build # 5313 - Failure!
by starlingx.build＠gmail.com 12 Nov '23

12 Nov '23

Project: STX_repo_sync_layered Build #: 5313 Status: Failure Timestamp: 20231112T005049Z Branch: r/stx.7.0 Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/7.0/centos/distro/2023… -------------------------------------------------------------------------------- Parameters BRANCH: r/stx.7.0 MANIFEST: distro.xml PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/7.0/centos/distro/2023… PUBLISH_LOGS_BASE: /export/mirror/starlingx/rc/7.0/centos/distro/20231112T005033Z/logs MY_REPO_ROOT: /localdisk/designer/jenkins/rc-7.0-distro

1 0