[Starlingx-discuss] RBD provisioner changes

[Church, Robert]

I have up for review a series of commits [1] that will "decouple the RBD
provisioner from the stx-openstack application" [2] and provide a separate
application that will be installed as part platform provisioning.

The goal is to provide support for k8s persistent volume claims independent of
the openstack application which can be utilized by additional applications or
individual helm charts.

The rbd-provisioner and the ceph-pools-audit charts that were previously part of
the stx-openstack application are now provided as part of a 'platform-integ-apps'
application. The application tarball will install charts in a new helm repo,
'stx-platform', which will hold all the charts that we provide for platform
integration and are not considered user application specific [3].

A new sysinv audit task has been created to automatically upload and install this
application after the active controller is unlocked/enabled. As this is a k8s
application, it will follow existing application patterns and install a
manifest, download docker images, generate system overrides, and apply the
platform integration application.

Support has been added to add ceph secrets to application namespaces to enable 
access to the provisioner and use of the 'general' storage class as part of 
application-apply.

Prior to running any additional user applications (including the stx-openstack
application), you will want to make sure that the platform application has been
applied [4] to ensure that persistent volume claims will be serviced. Other than
this check, no other additional changes are required from an automation
perspective to launch the stx-openstack application.

The stx-openstack application has also been updated to support using the default
system provisioner.

Once the platform application has been applied, persistent volume claims can be
filled from the following namespaces without further configuration: kube-system,
default, and kube-public. Any helm charts launched in these namespaces should use 
the 'general' storage class and volumes will be created in the kube-rbd ceph 
pool. [5]

We can also launch additional dedicated provisioners to support a specific
applications using a dedicated pool, storage class, and namespace. [6]

Testing has been performed with config_controller and ansible for initial system
provisioning.

Known issues (to be addressed with follow-up tasks/commits):
- The system helm-override API currently only supports the stx-openstack
  application. Because of this we cannot add a user override to enable
  additional namespaces for the provisioner and re-apply the application. 
- As currently written, the rbd-provisioner helm chart is not upgradable.
  Additional modifications are required to add upgrade hooks for the
  initialization job.

[1] Reviews: https://review.opendev.org/#/q/topic:decouple_rbd_provisioner+(status:open+OR+status:merged
[2] Storyboard: https://storyboard.openstack.org/#!/story/2005424
[3] Example: Available helm repos: http://paste.openstack.org//show/750846/
[4] Example: platform application: http://paste.openstack.org//show/750847/
[5] Example: Helm chart using storageclass: http://paste.openstack.org//show/750848/
[6] Example: Launching additional provisioners: http://paste.openstack.org//show/750850/

Thanks,
Bob