[Starlingx-discuss] question about ceph or storage

Chen, Haochuan Z haochuan.z.chen at intel.com
Mon Dec 23 07:18:35 UTC 2019 

Sorry to disturb!

Merry Christmas to all StarlingX member!

Martin, Chen
SSP, Software Engineer
021-61164330

From: Chen, Haochuan Z
Sent: Monday, December 23, 2019 3:17 PM
To: Poncea, Ovidiu <Ovidiu.Poncea at windriver.com>; Church, Robert <Robert.Church at windriver.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: RE: question about ceph or storage

Hi Ovidiu

For "system storage-backend-add -c <ceph_conf>", user could deploy another ceph cluster, and add as storage backend, correctly.
And this is this command's intention, correct?

Martin, Chen
SSP, Software Engineer
021-61164330

From: Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>
Sent: Wednesday, December 18, 2019 7:47 PM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>; Church, Robert <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Subject: RE: question about ceph or storage

Hi Chen, see inline.
________________________________
From: Chen, Haochuan Z [haochuan.z.chen at intel.com]
Sent: Tuesday, December 17, 2019 10:46 AM
To: Church, Robert; Poncea, Ovidiu
Cc: 'starlingx-discuss at lists.starlingx.io'
Subject: question about ceph or storage
Hi Bob & Ovidiu

Some question about ceph or storage.

1, What's storage tier and storage profile? What's the
[Ovi] Storage tiering is equivalent with this: https://ceph.io/planet/deploying-ceph-with-storage-tiering/ .
[Ovi] Profiles are managed by system storprofile-* and system host-apply-profile and are used to copy configuration from one node to another, identical node on initial provisioning. These profiles are only in system inventory, there is no Ceph equivalent.

2, why for duplex it request such puppet class dependency in ceph.pp? Is this request make all drbd config before class ceph? [Ovi] ceph-mon in AIO-DX is DRBD managed and it has a single, floating, monitor. On DX, when you swact, the monitor is stopped on the active controller and started on the standby controller.
    Drbd::Resource <| |> -> Class['::ceph']
    And flag file ".node_ceph_configured", to inform drbd make init setup before ceph config?

3, To launch ceph-mon, create a logical volume "ceph-mon-lv" and mount to /var/lib/ceph/mon, not directly mkdir "/var/lib/ceph/mon" for ceph-mon [Ovi] Ceph monitors have their own logical volume. They are managed through "system ceph-mon*" commands.

Thanks!

Martin, Chen
SSP, Software Engineer
021-61164330

From: Chen, Haochuan Z
Sent: Wednesday, December 11, 2019 4:46 PM
To: Church, Robert <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>; Qi, Mingyuan <mingyuan.qi at intel.com<mailto:mingyuan.qi at intel.com>>
Subject: RE: ceph ops enabling in sysinv-conductor

Hi Bob

Some question, what's storage tier and storage profile? As you said, we no longer manage pool and pg num, is this also unnecessary and we should remove it?

BR!

Martin, Chen
SSP, Software Engineer
021-61164330

From: Church, Robert <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Sent: Wednesday, December 4, 2019 12:09 AM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>
Subject: Re: ceph ops enabling in sysinv-conductor

See inline...

From: "Chen, Haochuan Z" <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>
Date: Tuesday, December 3, 2019 at 2:18 AM
To: Robert Church <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, Ovidiu Poncea <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>
Subject: RE: ceph ops enabling in sysinv-conductor

Hi Bob

1, we could update default pg num in sysinv/helm/rbd_provisioner.py. But when platform-integ-apps applied, maybe user has not add osded, so the default pg num is still for user reference.
And for user override, I think it should add in ceph-pool-audit, stx-platform-helm/helm-charts/ceph-pools-audit/templates/job-ceph-pools-audit.yaml. but pg num update will make rebalance which cause management network jam.
[RTC] For a more robust solution, consider the following:

*        We can update _met_app_apply_prerequisites () in sysinv/conductor.py for platform-integ-apps to require OSDs to be provisioned prior to applying the application. This will ensure an accurate OSD view when the application is initially applied.

*        The provisioner system overrides (in sysinv/helm/rbd_provisioner.py) should have the ability to calculate and set an optimal PG number to avoid generating a warning.

*        The rbd provisioner chart should also support setting a new chunk size (if greater than the existing size) to update the PG num. This will support user PG updates from the helm-overrides API.

*        I think it's potentially a good idea for ceph-pool-audit to support adjusting the PG numbers as well but this is tricky as I don't think we can reduce PG_num in Mimic without creating a new pool and copying the contents (pg_autoscaling is added in Nautilis). This audit code would have to be very specific in adjusting the PG num as installing applications that add additional pools will change the PG_num distribution.

2, the above case is only for system application. Any more alarm for PG number, request user should manage.

BR!

Martin, Chen
SSP, Software Engineer
021-61164330

From: Church, Robert <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Sent: Tuesday, December 3, 2019 10:34 AM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>
Subject: Re: ceph ops enabling in sysinv-conductor

Hi Martin,

In STX 1.0,  we ran bare-metal openstack services for which we created and managed specific ceph storage pools for glance, cinder, nova,  and swift. In this environment, we managed their creation and parameters (PGs and quotas) as the cluster scaled up.

With the move to containerizing the openstack services in STX 2.0, pool creation is now driven mostly by helm charts packaged into an application (radosgw pools are the exception). Since we don't know what additional application(s) will be deployed and what pools may be created, we are currently longer managing pools as in STX 1.0.

These related functions in sysinv/conductor/ceph.py are remaining from STX 1.0 and need to be removed and/or repurposed to meet any new requirements.

I took a look at the LP logs and it looks like we only have a single application applied: platform-integ-apps. This currently results in a single pool for 6 OSDs

  cluster:
    id:     6231df84-33be-4aa4-82ea-7408e0f2421c
    health: HEALTH_WARN
            too few PGs per OSD (21 < min 30)

  services:
    mon: 3 daemons, quorum controller-0,controller-1,storage-0
    mgr: controller-0(active), standbys: controller-1
    osd: 6 osds: 6 up, 6 in

  data:
    pools:   1 pools, 64 pgs
    objects: 0  objects, 0 B
    usage:   645 MiB used, 5.4 TiB / 5.4 TiB avail
    pgs:     64 active+clean


Since every installation will install platform-integ-apps, I think we should do the following:

1.      Update the chunk_size calculation in sysinv/helm/rbd_provisioner.py to be dynamically calculated based on the number OSDs provisioned in the cluster. As this may be the only pool created, it should meet the minimum size characteristics to avoid a ceph warning.

2.      Update the rbd-provisioner helm chart to support explicitly setting the pg_num based on the chunk size. Do this to allow setting a user override for the chunk size so that we can re-apply platform-integ-apps and explicitly set new values.

Regards,
Bob

From: "Chen, Haochuan Z" <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>
Date: Monday, December 2, 2019 at 1:15 AM
To: Robert Church <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, Ovidiu Poncea <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>
Subject: ceph ops enabling in sysinv-conductor

Hi Bob

I find some many function in sysinv-conductor/ceph.py, which could manage ceph cluster, such as create/delete/configure pool, audit pg etc.
But why these function is not enabled? Or plan to request user to manage ceph cluster, such as create pool and configure pg num?

Now I checked this issue, pg too few, as user maybe deploy few osd, which make alarm. So for such issue, request user to decide correct  pg num or user could ignore?
https://bugs.launchpad.net/starlingx/+bug/1844164

BR!

Martin, Chen
SSP, Software Engineer
021-61164330



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191223/ba503736/attachment-0001.html>

More information about the Starlingx-discuss mailing list