Hi, Paolo. You are right, stx-openstack from stx.11.0 will fail to upload/apply in a Starlingx from stx.10.0. I forgot that in my first email, but there is some code depending on stx.11.0 for the new stx-openstack application. Since your StarlingX stx.10.0 without OpenStack is already struggling with your host-based Ceph, I would suggest to explore the Ceph config and we could eventually ping someone from the community that maintains the storage functions of StarlingX. Another option would be exploring an upgrade from stx.10.0 to stx.11.0, for which I would suggest waiting the official lstx.11.0 release. For both, please join our stx-openstack project room so we can try to help you directly: https://matrix.to/#/#starlingx-openstack:opendev.org Cheers, Thales. ________________________________ From: PAOLO NAPOLI <Paolo.Napoli@nttdata.com> Sent: Monday, August 4, 2025 10:11 AM To: Cervi, Thales Elero <ThalesElero.Cervi@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Cc: STEFANO VELTRI <stefano.veltri@nttdata.com>; GIUSEPPE DEL GAUDIO <Giuseppe.DelGaudio@nttdata.com> Subject: Re: Issue Uploading OpenStack Package on StarlingX Release 10 with Rook-Ceph CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi, Dear Thales Elero, let me explain in more details our experience with bare metal ceph on STX10. As said also by Giuseppe, we initially tried to use bare metal ceph (v14) on STX 10 but, also before installing stx-openstack (that is to say, no workloads on Kubernetes except the platform ones), we had a problem of CEPH intermittent connectivity: about every 3 or 5 minute the command ceph -s showed HEALTH_WARN since 1 OSD was up and 1 down. After some minutes, the alarm was cleared and ceph automatically updated the status to HEALTH_OK. After some minutes we get randomly again the same failure. Within this failure timeframe, something like this was logged in the sysinv.log: 2025-07-21 16:20:41.617972 mgr.controller-0 (mgr.4416) 136728 : cluster [DBG] pgmap v136800: 192 pgs: 192 active+clean; 12 KiB data, 31 GiB used, 4.3 TiB / 4.4 TiB avail 2025-07-21 16:20:43.618575 mgr.controller-0 (mgr.4416) 136729 : Cluster [DBG] pgmap v136801: 192 pgs: 192 active+clean; 12 KiB data, 31 GiB used, 4.3 TiB / 4.4 TiB avail 2025-07-21 16:20:52.223339 mon.controller-0 (mon.0) 105968 : cluster [DBG] osd.1 reported immediately failed by osd.0 2025-07-21 16:20:52.223410 mon.controller-0 (mon.0) 105969 : cluster [INF] osd.1 failed (root-storage-tier, chassis=group-0, host=controller-1) (connection refused reported by osd.0) 2025-07-21 16:20:52.223536 mon.controller-0 (mon.0) 105970 : cluster [DBG] osd.1 reported immediately failed by osd.0 2025-07-21 16:20:52.422884 mon.controller-0 (mon.0) 105971 : cluster [DBG] osd.1 reported immediately failed by osd.0 2025-07-21 16:20:52.423676 mon.controller-0 (mon.0) 105972 : cluster [D8G] osd.1 reported immediately failed by osd.0 2025-07-21 16:20:52.677873 mon.controller-0 (mon.0) 105973 : Cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2025-07-21 16:20:52.677131 mon.controller-0 (mon.0) 105974 : cluster [WRN] Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN) 2025-07-21 16:20:52.687625 mon.controller-0 (mon.0) 105975 : cluster [DBG] osdmap e109: 2 total, 1 up, 2 in 2025-07-21 16:20:45.619637 mgr.controller-0 (mgr.4416) 136730 : cluster [DBG] pgmap v136802: 192 pgs: 192 active+clean; 12 KiB data, 31 GiB used, 4.3 TiB / 4.4 TiB avail That causes some Events and consequently some Alarms: SET (MAJOR) 800.011 Loss of replication in replication group group-0: ODs are down SET (WARNING) 800.001 Storage Alarm Condition: HEALTH_WARN [PGs are degraded/stuck be undersized), Please check 'ceph -s' for more details CLEAR (WARNING) 800.001 Storage Alarm Condition: HEALTH_WARN [PGs are degraded/stuck or undersized), Please check 'ceph -s' for more details We also made some test disabling the Management IPSEC vpn (thinking we had a low throughput on 1gbps management network) and just use standard flat network as in previous releases. This was not useful because the intermittent behaviour was still happening on the flat network also with low throughput (few MBps). For this reason and due to deprecation changelog information, we decided to give a try to rook-ceph v18 (without Management IPSEC vpns) and the Kube cluster was stable (no events) but we cannot install stx-openstack on rook-ceph due to the incompatibilities shown by Giuseppe. Summing up, in older versions of STX7 or STX9 we were using bare metal ceph and Openstack without any issue, but with STX10 we cannot get a stable cluster (with or without Openstack installed). We have also tried one of the latest stx-openstack TGZ (probably designed and implemented on beta STX-11 reference platform) but it raised error on upload, so it was impossible to use that newer version of stx-openstack with STX 10. So, the question is: is there a workaround or a way to tune some ceph params/config to use bare metal ceph without this intermittent problem in STX 10 with stx-openstack? We haven’t found similar problems or solutions in STX website (changelog, docs, …) and the bug tracking system. We just tried to tune some timeout (eg: osd_heartbeat_grace=60) but with no luck. We have another question: reinstalling the whole lab each time, we tested and installed almost every stable release since STX3, but we never tested or installed a pre-relase. Currenlty we have STX 10 installed but without openstack. Is there a way to cleanly and successfully upgrade to the pre-release (or main branch) mentioned by you or we must reinstall the whole distributed cloud using that pre-release? Our next step, assuming we have a functional Openstack Cluster, is to configure SRIOV passthrough inside OpenStack for some Nvidia GPU to support some Client AI use cases. Any help will be appreciated. Thank you in advance. Best Regards, Paolo Napoli – Cloud Architect | Cloud & Digital Architecture [signature_1835094065] From: Cervi, Thales Elero <ThalesElero.Cervi@windriver.com> Date: Saturday, 2 August 2025 at 02:18 To: GIUSEPPE DEL GAUDIO <Giuseppe.DelGaudio@emeal.nttdata.com>, starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Cc: PAOLO NAPOLI <Paolo.Napoli@emeal.nttdata.com>, STEFANO VELTRI <stefano.veltri@emeal.nttdata.com> Subject: Re: Issue Uploading OpenStack Package on StarlingX Release 10 with Rook-Ceph NTT DATA Security Awareness - This is an incoming mail from an EXTERNAL SENDER (prvs=0309342ff8=thaleselero.cervi@windriver.com). Please verify sender before you open attachments or access links. ________________________________ Hi Giuseppe, Glad to hear you are using stx-openstack! Unfortunately, the stx-openstack from stx.10.0 release does not support rook-ceph. This support is being added as part of the stx.11.0 release. It is already merged into the main branch if you want to give it a try. Please keep us in the loop! Thales Elero Cervi StarlingX TSC Member | StarlingX OpenStack Project Lead ________________________________ From: GIUSEPPE DEL GAUDIO <Giuseppe.DelGaudio@nttdata.com> Sent: Friday, August 1, 2025 3:38 PM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Cc: PAOLO NAPOLI <Paolo.Napoli@nttdata.com>; STEFANO VELTRI <stefano.veltri@nttdata.com> Subject: Issue Uploading OpenStack Package on StarlingX Release 10 with Rook-Ceph CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Dear StarlingX Community, We are currently testing the StarlingX + OpenStack Release 10 (v24.09) solution on our labs. During the installation process (on bare metal), we encountered some issues—initially related to OSD synchronization using host-based Ceph. For this reason, we switched to Rook-Ceph, as host-based Ceph will be deprecated. However, after switching, we were unable to upload the OpenStack package (via system upload) and proceed with the installation. I’ve attached both an installation report and the full sysinv log showing the error that occurred during the upload of the OpenStack installation package. Has anyone experienced a similar issue? Thank you very much for your support! Best regards, Giuseppe Del Gaudio - Cloud Engineer | Cloud & Digital Architecture [cid:2165bbb9-c072-4d13-8124-38abc1d85e42] [cid:033b377a-7ed2-497f-85c7-0bddbe4a422a] Via Bastioni 14/ S.Michele 10, Salerno, Italia Email: giuseppe.delgaudio@nttdata.com Tel: +39 3346368189 Learn more at www.nttdata.com/it<https://urldefense.com/v3/__http:/www.nttdata.com/it__;!!AjveYdw8EvQ!aArk9I22gzMmwxVJx15ur7bESSSkTUVxVe7BEc6oUDfS5Ge8SolCYXv2PhuRVcMtcoekBLoWGKns02-N9VReIeHrH3vc5lUK0s7t$> ________________________________ NTT DATA Italia S.p.A. - Società per azioni, soggetta ad attività di direzione e coordinamento di NTT DATA EMEA Ltd. - Sede legale a Milano, in Via Ernesto Calindri, 4 – codice fiscale e iscrizione al Registro delle imprese di Milano Monza Brianza Lodi n. 00513990010, iscritta al Repertorio economico Amministrativo n. 974124 - Capitale sociale di euro 40.970.700,00, interamente versato. ________________________________ NTT DATA Italia S.p.A. - Società per azioni, soggetta ad attività di direzione e coordinamento di NTT DATA EMEA Ltd. - Sede legale a Milano, in Via Ernesto Calindri, 4 – codice fiscale e iscrizione al Registro delle imprese di Milano Monza Brianza Lodi n. 00513990010, iscritta al Repertorio economico Amministrativo n. 974124 - Capitale sociale di euro 40.970.700,00, interamente versato.