[Starlingx-discuss] StarlingX 6.0 - Ceph deployment fails now

[Outback Dingo]

So we went to try to deploy a new starlingx 6.0 AIO bare metal in the
lab two times, however both times, ceph fails to properly complete
configuration, we looked at a previous 6.0 server deployed 120+ days
ago and found they differ greatly, so i have to ask whats changed that
would break this after the release? and is there a way to resolv it
manually as from what ican see a fresh 6.0 install fails to really
complete

prod server where ceph works
[sysadmin at controller-0 ~(keystone_admin)]$ ceph status
  cluster:
    id:     67f59a3e-bd10-433e-b5da-c16bb88053cb
    health: HEALTH_OK


kube-system    ceph-pools-audit-27797180-fl2wh
0/1     Completed           0          12m
kube-system    ceph-pools-audit-27797185-vdggr
0/1     Completed           0          7m49s
kube-system    ceph-pools-audit-27797190-79bq4
0/1     Completed           0          2m48s
kube-system    cephfs-provisioner-bd46f868-vn7x8
1/1     Running             0          103d
kube-system    rbd-provisioner-759dfb8b6b-mgdj6
1/1     Running             0          103d

[sysadmin at controller-0 ~(keystone_admin)]$ kubectl get storageclass
--all-namespaces
NAME                PROVISIONER       RECLAIMPOLICY
VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
cephfs              ceph.com/cephfs   Delete          Immediate
   false                  110d
general (default)   ceph.com/rbd      Delete          Immediate
   false                  110d


[sysadmin at controller-0 ~(keystone_admin)]$ system application-list
+--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+
| application              | version                       | manifest
name                     | manifest file         | status       |
progress          |
+--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+
| cert-manager             | 1.0-26                        |
cert-manager-manifest             | certmanager-manifest. | applied
  | completed         |
|                          |                               |
                        | yaml                  |              |
            |
|                          |                               |
                        |                       |              |
            |
| nginx-ingress-controller | 1.1-18                        |
nginx-ingress-controller-manifest | nginx_ingress_control | applied
  | completed         |
|                          |                               |
                        | ler_manifest.yaml     |              |
            |
|                          |                               |
                        |                       |              |
            |
| oidc-auth-apps           | 1.0-61                        |
oidc-auth-manifest                | manifest.yaml         | uploaded
  | completed         |
| platform-integ-apps      | 1.0-44                        |
platform-integration-manifest     | manifest.yaml         |
apply-failed | operation aborted |
|                          |                               |
                        |                       |              | ,
check logs for  |
|                          |                               |
                        |                       |              |
detail            |
|                          |                               |
                        |                       |              |
            |
| rook-ceph-apps           | 1.0-14                        |
rook-ceph-manifest                | manifest.yaml         | uploaded
  | completed         |
| stx-openstack            | 1.0-140-centos-stable-        |
openstack-manifest                | stx-openstack.yaml    | applied
  | completed         |
|                          | versioned                     |
                        |                       |              |
            |
|                          |                               |
                        |                       |              |
            |
+--------------------------+-------------------------------+-----------------------------------+-----------------------+--------------+-------------------+
[sysadmin at controller-0 ~(keystone_admin)]$

dev server where ceph is broken
[sysadmin at controller-0 ~(keystone_admin)]$ ceph status
  cluster:
    id:     06ba8bae-bb17-43ef-96c2-c2ec136a6041
    health: HEALTH_OK

0 ceph/rbd servies in k8s

kubectl get storageclass --all-namespaces
No resources found

[sysadmin at controller-0 ~(keystone_admin)]$ system application-list
+--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+
| application              | version | manifest name
  | manifest file                          | status  | progress  |
+--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+
| cert-manager             | 1.0-26  | cert-manager-manifest
  | certmanager-manifest.yaml              | applied | completed |
| nginx-ingress-controller | 1.1-18  |
nginx-ingress-controller-manifest |
nginx_ingress_controller_manifest.yaml | applied | completed |
+--------------------------+---------+-----------------------------------+----------------------------------------+---------+-----------+
[sysadmin at controller-0 ~(keystone_admin)]$