Thanks Bart, all clear. Mingyuan From: Wensley, Barton <Barton.Wensley@windriver.com> Sent: Friday, February 28, 2020 22:25 To: Qi, Mingyuan <mingyuan.qi@intel.com>; Wang, Jing (Angie) <Angie.Wang@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Dale, Kristal <kristal.dale@intel.com> Subject: RE: [docs]Provide doc input for 2006781 Good questions Mingyuan. Since I did the original design for k8s upgrades I'll reply to your questions (see below). Bart From: Qi, Mingyuan [mailto:mingyuan.qi@intel.com] Sent: February 28, 2020 5:01 AM To: Wang, Jing (Angie); starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Dale, Kristal Subject: Re: [Starlingx-discuss] [docs]Provide doc input for 2006781 Hi Angie, The feature looks great that we can now upgrade the k8s cluster at runtime. Regarding the details, I and other folks have several questions that we didn't find clues in the doc. Could you please give us the info from designer's view? * Why the upgrade complete cmd line needed? Any essential checks? [Bart] This command does a final check to verify that all the k8s components are now running the new release and then updates the state to upgrade-complete. * If the first master node upgrade completed on the active controller, that means the active control plane is upgraded to the new version. Then how to ensure the compatibility between upgraded controller plane and un-upgraded kubelet before step 12? [Bart] The k8s version skew policy (https://kubernetes.io/docs/setup/release/version-skew-policy) specifies that the kubelet may be up to two minor versions older than the kube-apiserver. Any k8s upgrades done to StarlingX deployments must respect those rules. * What will happen if some nodes restart unexpectedly between step 11 and 12? [Bart] The kubelet on the node that restarts would come up running the new version, but would be reading the old format of the kubelet config file. This should be supported (new values in the config file will be defaulted). The user can still run the kube-host-upgrade command after this to upgrade the kubelet config file. * How to deal with the certificates and keys of Kubernetes during upgrading? [Bart] As mentioned in the spec (https://docs.starlingx.io/specs/specs/stx-4.0/approved/containerization-2006...), we are using the kubeadm command to perform the upgrade. It will take care of anything certificate and key related. * Is upgrade rollback supported regardless the upgrade completion? [Bart] There is no support for rollback. I'm not sure if we will add this in the future - the general consensus in the k8s community is that k8s downgrades are not supported. * Is kubelet/kubectl patch marked as in-service patch or reboot-required patch? [Bart] It is an in-service patch. * Why lock/unlock is needed to upgrade kubelet? Is there a way to avoid that? Since this will impact the running pods on all nodes. [Bart] It is not safe to upgrade the kubelet while there are pods running on the node. The lock will move the pods off the node. In the future, when we add orchestration of k8s upgrades (with the VIM), we will likely add support to move the pods off the node without locking it, which would avoid the reboot of the node when the kubelet was upgraded. Thanks, Mingyuan From: Wang, Jing (Angie) <Angie.Wang@windriver.com<mailto:Angie.Wang@windriver.com>> Sent: Thursday, February 27, 2020 23:17 To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Dale, Kristal <kristal.dale@intel.com<mailto:kristal.dale@intel.com>> Subject: [Starlingx-discuss] [docs]Provide doc input for 2006781 Hi All, The attached file is the documentation changes required for the story "Kubernetes upgrade support in StarlingX" (https://storyboard.openstack.org/#!/story/2006781). Thanks, Angie