[Starlingx-discuss] rook deployment for duplex

Church, Robert Robert.Church at windriver.com
Tue Nov 10 18:48:20 UTC 2020

Hi Martin,

I’m open all Friday morning, if that works for you, starting at 8am CST/2pm UTC.


From: "Chen, Haochuan Z" <haochuan.z.chen at intel.com>
Date: Tuesday, November 10, 2020 at 8:00 AM
To: "Sun, Austin" <austin.sun at intel.com>, "Qi, Mingyuan" <mingyuan.qi at intel.com>, Robert Church <Robert.Church at windriver.com>, Ovidiu Poncea <Ovidiu.Poncea at windriver.com>, "Waines, Greg" <Greg.Waines at windriver.com>, "Voiculeasa, Dan" <Dan.Voiculeasa at windriver.com>
Cc: "Hu, Yong" <yong.hu at intel.com>, "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io>
Subject: RE: rook deployment for duplex

[Please note this e-mail is from an EXTERNAL e-mail address]
Hi Ovidiu & Bob

This Thursday morning(Bob’s time), Thursday afternoon(Ovidiu), when you are ok to book a meeting.
We could sync rook status. I will update duplex deployment update info.


Martin, Chen
IOTG, Software Engineer

From: Chen, Haochuan Z
Sent: Friday, November 6, 2020 10:24 PM
To: Sun, Austin <austin.sun at intel.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'Church, Robert' <Robert.Church at windriver.com>; 'Poncea, Ovidiu' <Ovidiu.Poncea at windriver.com>; 'Waines, Greg' <Greg.Waines at windriver.com>; 'Voiculeasa, Dan' <Dan.Voiculeasa at windriver.com>
Cc: Hu, Yong <yong.hu at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: RE: rook deployment for duplex

Hi Ovidiu, Bob and Greg

I introduce my design for duplex to Dan. If you think necessary, I could book a meeting next Tuesday to explain again.


Martin, Chen
IOTG, Software Engineer

From: Chen, Haochuan Z
Sent: Thursday, November 5, 2020 4:22 PM
To: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>; Qi, Mingyuan <mingyuan.qi at intel.com<mailto:mingyuan.qi at intel.com>>; Church, Robert <Robert.Church at windriver.com<mailto:Robert.Church at windriver.com>>; Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>; Waines, Greg <Greg.Waines at windriver.com<mailto:Greg.Waines at windriver.com>>; Voiculeasa, Dan <Dan.Voiculeasa at windriver.com<mailto:Dan.Voiculeasa at windriver.com>>
Cc: Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: rook deployment for duplex

Hi Ovidiu

I confirm the issue you raise for duplex. So I change rook deployment for duplex as this.

1, duplex will use host network, ceph mon ip is controller floating ip
2, there is only one monitor pod on active controller
3, same as native ceph cluster, there is a drbd filesystem, named drbd-cephmon, to sync mon data on both controllers, mount to folder /var/lib/ceph/mon-a/data
4, there is a cronjob, periodic check these is host offline or lock. If host offline and mon pod deployed on that host, delete mon deployment and rook-ceph operator will make mon deployment again on the other controller
# because when rook-operator launch mon deployment, it nodeselector for this deployment will bind to this host. So must delete this mon deployment to make operator reschedule the mon deployment to the other controller
# this is similar what sm do for native ceph

We could discuss tomorrow.

Task items for above design:
1, update sysinv to enable  drbd ceph-mon for rook
    1), add system storage-backend-add rook --confirmed
    2), in cgtsclient add storage_rook.py
    3), in sysinv-api, add storage_rook.py, will call sysinv-conductor update_rook_config
    4), in sysinv-conductor update_rook_config,
           if cutils.is_aio_duplex_system(self.dbapi):
                # On 2 node systems we have a floating Ceph monitor.

2, update rook-app override
    for duplex, use hostnetwork, mon: 1.

3, add a pre install job in rook-ceph helm chart

    # controller-1 is active controller
    # is floating ip
    # rook-operator will read this configmap and the only monitor pod, mon-a will deployed on active controller,
    export ROOK_EXTERNAL_MAPPING='{"node":{"a":{"Name":"controller-1","Hostname":"controller-1","Address":""}}}'

    kubectl -n kube-system create configmap rook-ceph-mon-endpoints   \
            --from-literal=data="$ROOK_EXTERNAL_CEPH_MON_DATA"        \
            --from-literal=mapping=”$ROOK_EXTERNAL_MAPPING”           \

4, add a cron job in rook-ceph-provision helm chart
    If any host down, and mon-a pod deployed on that host(at this time, mon-a pod is terminating and pending)
      a), delete mon-a pod and mon-a deployment.
      b), update confimap rook-ceph-mon-endpoints, field mapping.
          '{"node":{"a":{"Name":"controller-0","Hostname":"controller-0","Address":""}}}'        # hostname change to active controller-0
      c), delete operator pod, wait operator pods will launch and recreate mon-a deployment


Martin, Chen
IOTG, Software Engineer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201110/15bb53f2/attachment-0001.html>

More information about the Starlingx-discuss mailing list