So we went to try to deploy a new starlingx 6.0 AIO bare metal in the lab two times, however both times, ceph fails to properly complete configuration, we looked at a previous 6.0 server deployed 120+ days ago and found they differ greatly, so i have to ask whats changed that would break this after the release? and is there a way to resolv it manually as from what ican see a fresh 6.0 install fails to really complete prod server where ceph works [sysadmin@controller-0 ~(keystone_admin)]$ ceph status cluster: id: 67f59a3e-bd10-433e-b5da-c16bb88053cb health: HEALTH_OK kube-system ceph-pools-audit-27797180-fl2wh 0/1 Completed 0 12m kube-system ceph-pools-audit-27797185-vdggr 0/1 Completed 0 7m49s kube-system ceph-pools-audit-27797190-79bq4 0/1 Completed 0 2m48s kube-system cephfs-provisioner-bd46f868-vn7x8 1/1 Running 0 103d kube-system rbd-provisioner-759dfb8b6b-mgdj6 1/1 Running 0 103d [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get storageclass --all-namespaces NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE cephfs ceph.com/cephfs Delete Immediate false 110d general (default) ceph.com/rbd Delete Immediate false 110d [sysadmin@controller-0 ~(keystone_admin)]$ system application-list +--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+ | application | version | manifest name | manifest file | status | progress | +--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+ | cert-manager | 1.0-26 | cert-manager-manifest | certmanager-manifest. | applied | completed | | | | | yaml | | | | | | | | | | | nginx-ingress-controller | 1.1-18 | nginx-ingress-controller-manifest | nginx_ingress_control | applied | completed | | | | | ler_manifest.yaml | | | | | | | | | | | oidc-auth-apps | 1.0-61 | oidc-auth-manifest | manifest.yaml | uploaded | completed | | platform-integ-apps | 1.0-44 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted | | | | | | | , check logs for | | | | | | | detail | | | | | | | | | rook-ceph-apps | 1.0-14 | rook-ceph-manifest | manifest.yaml | uploaded | completed | | stx-openstack | 1.0-140-centos-stable- | openstack-manifest | stx-openstack.yaml | applied | completed | | | versioned | | | | | | | | | | | | +--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ dev server where ceph is broken [sysadmin@controller-0 ~(keystone_admin)]$ ceph status cluster: id: 06ba8bae-bb17-43ef-96c2-c2ec136a6041 health: HEALTH_OK 0 ceph/rbd servies in k8s kubectl get storageclass --all-namespaces No resources found [sysadmin@controller-0 ~(keystone_admin)]$ system application-list +--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+ | application | version | manifest name | manifest file | status | progress | +--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+ | cert-manager | 1.0-26 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed | | nginx-ingress-controller | 1.1-18 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest.yaml | applied | completed | +--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+ [sysadmin@controller-0 ~(keystone_admin)]$