[Starlingx-discuss] Minutes: StarlingX Containerization Meeting
Thanks to all who participated in today's meeting. Good progress across the stx.3.0 features. Minutes for Oct 22: 1. Updates on stx.3.0 container features for MS3 milestone: 2004008: [Feature] Create HELM chart for Fault project [Ran An] - one horizon related commit needs a new approach which cannot be done in stx.3.0 timeframe Decision for stx.3.0 is to keep FM container code but disable it so that FM runs natively for stx.3.0 - then address horizon issue in stx.4.0 and re-enable FM containers. Ran has proposed doing this by bringing back the "fault_management_pod_disabled" flag that was introduced in stx.2.0 and removed in stx.3.0 Ran's forecast for completing this and posting gerrit review is end of week, ie Oct 25 2005937: Intel GPU K8s device plugin support in StarlingX [An Ran] - Ran confirmed this feature is now merged Below 2 patches are for k8s device plugin enabling mechanism in ansible and sysinv, add intel gpu device plugin as well --- merged - https://review.opendev.org/#/c/666511/ - https://review.opendev.org/#/c/666510/ This patch is for align all device plugins in one method. Will be revisit when default device plugins require to be align. --- abandoned - https://review.opendev.org/#/c/685592/ 2005514: QAT K8S device plugin support in StarlingX Mingyuan: status of 3 patches which depended on Ran's 666510/666511 - https://review.opendev.org/684890 --> merged - https://review.opendev.org/688344 --> Cores have +2'd -- just need WFL +1 - https://review.opendev.org/688363 --> Cores have +2'd -- just need WFL +1 2005860: Upversion container components [Al Bailey] --> merged 2004761: Integrate Backup & Restore with Containers (for platform) [Ovidiu Poncea] --> merged 2. Complete review of KATA containers feature with container Cores. - Shuicheng's update: Here is the latest test status for kata integration in StarlingX. (1) Pass Sanity test for AIO-SX/AIO-DX/Multi/Multi+Storage. (2) Tested private registry (insecure registry) enabled/disabled. (3) Tested docker proxy enabled/disabled. (4) Tested swact/lock & unlock/remove & apply application. (5) Tested local registry (registry.local:9001) with image pull/push, and access both from kubernetes and crictl (CLI). For regression test, I checked container related test cases in below doc, and executed most of them. No issue is found yet. https://docs.google.com/spreadsheets/d/1dwcBwY4Yq1Lo9Der4RylzQ6KYp0BsMHohhEm... Currently, I meet a deploy issue with IPv6 environment, and the issue could be reproduced with master daily build also. I am checking whether it is caused by my environment or a real issue of master now. Here is the patch list waiting review: https://review.opendev.org/#/q/topic:kata+(status:open) Some questions/comments from TL/Brent: - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) - What is the plan to remove dockerd? - Plan is to remove in a separate commit after the main code merges. Shuicheng plan is to wait till stx.4.0 to remove - is this acceptable? - Shuicheng plan is to have the dockerd removal commit ready by end of November. Plan then to merge this into stx.4.0 once it is open. - We agreed this would be acceptable. - Looks like there is one upstream patch we are carrying (in containerd and runc?). What is the plan to upstream these patches? - Action: Ensure there is a task to upversion docker distribution registry to remove the need to carry this patch. - This too will be done early in stx.4.0. Shuicheng to update us in 1 week on when this task can be completed. Decision to allow into stx.3.0? Three requirements: - Shuicheng to complete testing for IPv6 and external registry with username/pwd authentication - Cores to complete their reviews - Frank to confirm with Ada that testing of this feature can be done in time for stx.3.0 - Backup plan: If instability is seen in stx.3.0 or significant testing by Ada results in failures then the feature would need to be backed up and fixed/delivered in stx.4.0 Frank Miller, Manager, Engineering, Wind River direct 613.963.1389 zoom frank.miller@windriver.com 350 Terry Fox Drive, Suite 200, Kanata, ON K2K 2W5
Hi Frank & Brent, " - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) " For external registry with authentication, the status is below: 1. Pull image by crictl is supported. Cmd like: crictl pull --creds USERNAME:PASSWORD image_url 2. Pull image by Kubernetes is supported. It is following standard Kubernetes operation. A Secret need be created to store the username/password, and pod yaml file will use it. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-... Since localhost.yml file for ansible playbook doesn't support authentication info, it means StarlingX cannot be deployed by registry with authentication. Registry with authentication should be used for 3rd application only. Is it right? Please share me your test cases for registry with authentication if you still have concern on it. Thanks. Best Regards Shuicheng From: Miller, Frank <Frank.Miller@windriver.com> Sent: Wednesday, October 23, 2019 8:11 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Minutes: StarlingX Containerization Meeting Thanks to all who participated in today's meeting. Good progress across the stx.3.0 features. Minutes for Oct 22: 1. Updates on stx.3.0 container features for MS3 milestone: 2004008: [Feature] Create HELM chart for Fault project [Ran An] - one horizon related commit needs a new approach which cannot be done in stx.3.0 timeframe Decision for stx.3.0 is to keep FM container code but disable it so that FM runs natively for stx.3.0 - then address horizon issue in stx.4.0 and re-enable FM containers. Ran has proposed doing this by bringing back the "fault_management_pod_disabled" flag that was introduced in stx.2.0 and removed in stx.3.0 Ran's forecast for completing this and posting gerrit review is end of week, ie Oct 25 2005937: Intel GPU K8s device plugin support in StarlingX [An Ran] - Ran confirmed this feature is now merged Below 2 patches are for k8s device plugin enabling mechanism in ansible and sysinv, add intel gpu device plugin as well --- merged - https://review.opendev.org/#/c/666511/ - https://review.opendev.org/#/c/666510/ This patch is for align all device plugins in one method. Will be revisit when default device plugins require to be align. --- abandoned - https://review.opendev.org/#/c/685592/ 2005514: QAT K8S device plugin support in StarlingX Mingyuan: status of 3 patches which depended on Ran's 666510/666511 - https://review.opendev.org/684890 --> merged - https://review.opendev.org/688344 --> Cores have +2'd -- just need WFL +1 - https://review.opendev.org/688363 --> Cores have +2'd -- just need WFL +1 2005860: Upversion container components [Al Bailey] --> merged 2004761: Integrate Backup & Restore with Containers (for platform) [Ovidiu Poncea] --> merged 2. Complete review of KATA containers feature with container Cores. - Shuicheng's update: Here is the latest test status for kata integration in StarlingX. (1) Pass Sanity test for AIO-SX/AIO-DX/Multi/Multi+Storage. (2) Tested private registry (insecure registry) enabled/disabled. (3) Tested docker proxy enabled/disabled. (4) Tested swact/lock & unlock/remove & apply application. (5) Tested local registry (registry.local:9001) with image pull/push, and access both from kubernetes and crictl (CLI). For regression test, I checked container related test cases in below doc, and executed most of them. No issue is found yet. https://docs.google.com/spreadsheets/d/1dwcBwY4Yq1Lo9Der4RylzQ6KYp0BsMHohhEm... Currently, I meet a deploy issue with IPv6 environment, and the issue could be reproduced with master daily build also. I am checking whether it is caused by my environment or a real issue of master now. Here is the patch list waiting review: https://review.opendev.org/#/q/topic:kata+(status:open) Some questions/comments from TL/Brent: - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) - What is the plan to remove dockerd? - Plan is to remove in a separate commit after the main code merges. Shuicheng plan is to wait till stx.4.0 to remove - is this acceptable? - Shuicheng plan is to have the dockerd removal commit ready by end of November. Plan then to merge this into stx.4.0 once it is open. - We agreed this would be acceptable. - Looks like there is one upstream patch we are carrying (in containerd and runc?). What is the plan to upstream these patches? - Action: Ensure there is a task to upversion docker distribution registry to remove the need to carry this patch. - This too will be done early in stx.4.0. Shuicheng to update us in 1 week on when this task can be completed. Decision to allow into stx.3.0? Three requirements: - Shuicheng to complete testing for IPv6 and external registry with username/pwd authentication - Cores to complete their reviews - Frank to confirm with Ada that testing of this feature can be done in time for stx.3.0 - Backup plan: If instability is seen in stx.3.0 or significant testing by Ada results in failures then the feature would need to be backed up and fixed/delivered in stx.4.0 Frank Miller, Manager, Engineering, Wind River direct 613.963.1389 zoom frank.miller@windriver.com<mailto:frank.miller@windriver.com> 350 Terry Fox Drive, Suite 200, Kanata, ON K2K 2W5
Shuicheng, The localhost.yml file does support authentication info - please take a look at: ansible-playbooks/playbookconfig/src/playbooks/host_vars/bootstrap/default.yml You need to test installation of the system for the case where the system images are coming from an external authenticated registry. Bart From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: October 26, 2019 4:02 AM To: Miller, Frank; starlingx-discuss@lists.starlingx.io; Rowsell, Brent Subject: Re: [Starlingx-discuss] Minutes: StarlingX Containerization Meeting Hi Frank & Brent, " - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) " For external registry with authentication, the status is below: 1. Pull image by crictl is supported. Cmd like: crictl pull --creds USERNAME:PASSWORD image_url 2. Pull image by Kubernetes is supported. It is following standard Kubernetes operation. A Secret need be created to store the username/password, and pod yaml file will use it. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-... Since localhost.yml file for ansible playbook doesn't support authentication info, it means StarlingX cannot be deployed by registry with authentication. Registry with authentication should be used for 3rd application only. Is it right? Please share me your test cases for registry with authentication if you still have concern on it. Thanks. Best Regards Shuicheng From: Miller, Frank <Frank.Miller@windriver.com> Sent: Wednesday, October 23, 2019 8:11 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Minutes: StarlingX Containerization Meeting Thanks to all who participated in today's meeting. Good progress across the stx.3.0 features. Minutes for Oct 22: 1. Updates on stx.3.0 container features for MS3 milestone: 2004008: [Feature] Create HELM chart for Fault project [Ran An] - one horizon related commit needs a new approach which cannot be done in stx.3.0 timeframe Decision for stx.3.0 is to keep FM container code but disable it so that FM runs natively for stx.3.0 - then address horizon issue in stx.4.0 and re-enable FM containers. Ran has proposed doing this by bringing back the "fault_management_pod_disabled" flag that was introduced in stx.2.0 and removed in stx.3.0 Ran's forecast for completing this and posting gerrit review is end of week, ie Oct 25 2005937: Intel GPU K8s device plugin support in StarlingX [An Ran] - Ran confirmed this feature is now merged Below 2 patches are for k8s device plugin enabling mechanism in ansible and sysinv, add intel gpu device plugin as well --- merged - https://review.opendev.org/#/c/666511/ - https://review.opendev.org/#/c/666510/ This patch is for align all device plugins in one method. Will be revisit when default device plugins require to be align. --- abandoned - https://review.opendev.org/#/c/685592/ 2005514: QAT K8S device plugin support in StarlingX Mingyuan: status of 3 patches which depended on Ran's 666510/666511 - https://review.opendev.org/684890 --> merged - https://review.opendev.org/688344 --> Cores have +2'd -- just need WFL +1 - https://review.opendev.org/688363 --> Cores have +2'd -- just need WFL +1 2005860: Upversion container components [Al Bailey] --> merged 2004761: Integrate Backup & Restore with Containers (for platform) [Ovidiu Poncea] --> merged 2. Complete review of KATA containers feature with container Cores. - Shuicheng's update: Here is the latest test status for kata integration in StarlingX. (1) Pass Sanity test for AIO-SX/AIO-DX/Multi/Multi+Storage. (2) Tested private registry (insecure registry) enabled/disabled. (3) Tested docker proxy enabled/disabled. (4) Tested swact/lock & unlock/remove & apply application. (5) Tested local registry (registry.local:9001) with image pull/push, and access both from kubernetes and crictl (CLI). For regression test, I checked container related test cases in below doc, and executed most of them. No issue is found yet. https://docs.google.com/spreadsheets/d/1dwcBwY4Yq1Lo9Der4RylzQ6KYp0BsMHohhEm... Currently, I meet a deploy issue with IPv6 environment, and the issue could be reproduced with master daily build also. I am checking whether it is caused by my environment or a real issue of master now. Here is the patch list waiting review: https://review.opendev.org/#/q/topic:kata+(status:open) Some questions/comments from TL/Brent: - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) - What is the plan to remove dockerd? - Plan is to remove in a separate commit after the main code merges. Shuicheng plan is to wait till stx.4.0 to remove - is this acceptable? - Shuicheng plan is to have the dockerd removal commit ready by end of November. Plan then to merge this into stx.4.0 once it is open. - We agreed this would be acceptable. - Looks like there is one upstream patch we are carrying (in containerd and runc?). What is the plan to upstream these patches? - Action: Ensure there is a task to upversion docker distribution registry to remove the need to carry this patch. - This too will be done early in stx.4.0. Shuicheng to update us in 1 week on when this task can be completed. Decision to allow into stx.3.0? Three requirements: - Shuicheng to complete testing for IPv6 and external registry with username/pwd authentication - Cores to complete their reviews - Frank to confirm with Ada that testing of this feature can be done in time for stx.3.0 - Backup plan: If instability is seen in stx.3.0 or significant testing by Ada results in failures then the feature would need to be backed up and fixed/delivered in stx.4.0 Frank Miller, Manager, Engineering, Wind River direct 613.963.1389 zoom frank.miller@windriver.com<mailto:frank.miller@windriver.com> 350 Terry Fox Drive, Suite 200, Kanata, ON K2K 2W5
Hi Shuicheng, Ansible bootstrap supports to deploy with authenticated registry. To deploy with authenticated registry, you will need to specify username and password in localhost.yml, Ie... docker_registries: defaults: url: registry.com username: registryUser password: registryPass You can also take a look playbooks/host_vars/bootstrap/default.yml for details. To test authenticated registry: - Bootstrap system with authenticated registry, verify system images are pulled from private authenticated registry and pushed to local registry. - After bootstrap is done, verify "auth-secret" is stored for each public registry in service-paramater. - Verify platform-integ-apps applied and images are pulled from private authenticated registry and pushed to local registry. - Deploy stx-monitor/stx-openstack app, verify images are pulled from private authenticated registry and pushed to local registry. Thanks, Angie From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: October-26-19 4:02 AM To: Miller, Frank; starlingx-discuss@lists.starlingx.io; Rowsell, Brent Subject: Re: [Starlingx-discuss] Minutes: StarlingX Containerization Meeting Hi Frank & Brent, " - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) " For external registry with authentication, the status is below: 1. Pull image by crictl is supported. Cmd like: crictl pull --creds USERNAME:PASSWORD image_url 2. Pull image by Kubernetes is supported. It is following standard Kubernetes operation. A Secret need be created to store the username/password, and pod yaml file will use it. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-... Since localhost.yml file for ansible playbook doesn't support authentication info, it means StarlingX cannot be deployed by registry with authentication. Registry with authentication should be used for 3rd application only. Is it right? Please share me your test cases for registry with authentication if you still have concern on it. Thanks. Best Regards Shuicheng From: Miller, Frank <Frank.Miller@windriver.com> Sent: Wednesday, October 23, 2019 8:11 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Minutes: StarlingX Containerization Meeting Thanks to all who participated in today's meeting. Good progress across the stx.3.0 features. Minutes for Oct 22: 1. Updates on stx.3.0 container features for MS3 milestone: 2004008: [Feature] Create HELM chart for Fault project [Ran An] - one horizon related commit needs a new approach which cannot be done in stx.3.0 timeframe Decision for stx.3.0 is to keep FM container code but disable it so that FM runs natively for stx.3.0 - then address horizon issue in stx.4.0 and re-enable FM containers. Ran has proposed doing this by bringing back the "fault_management_pod_disabled" flag that was introduced in stx.2.0 and removed in stx.3.0 Ran's forecast for completing this and posting gerrit review is end of week, ie Oct 25 2005937: Intel GPU K8s device plugin support in StarlingX [An Ran] - Ran confirmed this feature is now merged Below 2 patches are for k8s device plugin enabling mechanism in ansible and sysinv, add intel gpu device plugin as well --- merged - https://review.opendev.org/#/c/666511/ - https://review.opendev.org/#/c/666510/ This patch is for align all device plugins in one method. Will be revisit when default device plugins require to be align. --- abandoned - https://review.opendev.org/#/c/685592/ 2005514: QAT K8S device plugin support in StarlingX Mingyuan: status of 3 patches which depended on Ran's 666510/666511 - https://review.opendev.org/684890 --> merged - https://review.opendev.org/688344 --> Cores have +2'd -- just need WFL +1 - https://review.opendev.org/688363 --> Cores have +2'd -- just need WFL +1 2005860: Upversion container components [Al Bailey] --> merged 2004761: Integrate Backup & Restore with Containers (for platform) [Ovidiu Poncea] --> merged 2. Complete review of KATA containers feature with container Cores. - Shuicheng's update: Here is the latest test status for kata integration in StarlingX. (1) Pass Sanity test for AIO-SX/AIO-DX/Multi/Multi+Storage. (2) Tested private registry (insecure registry) enabled/disabled. (3) Tested docker proxy enabled/disabled. (4) Tested swact/lock & unlock/remove & apply application. (5) Tested local registry (registry.local:9001) with image pull/push, and access both from kubernetes and crictl (CLI). For regression test, I checked container related test cases in below doc, and executed most of them. No issue is found yet. https://docs.google.com/spreadsheets/d/1dwcBwY4Yq1Lo9Der4RylzQ6KYp0BsMHohhEm... Currently, I meet a deploy issue with IPv6 environment, and the issue could be reproduced with master daily build also. I am checking whether it is caused by my environment or a real issue of master now. Here is the patch list waiting review: https://review.opendev.org/#/q/topic:kata+(status:open) Some questions/comments from TL/Brent: - Have we tested with external registry and where username/pwd is required? Answer: Not yet. Action: Request is for Shuicheng to test (maybe work with Yong) - What is the plan to remove dockerd? - Plan is to remove in a separate commit after the main code merges. Shuicheng plan is to wait till stx.4.0 to remove - is this acceptable? - Shuicheng plan is to have the dockerd removal commit ready by end of November. Plan then to merge this into stx.4.0 once it is open. - We agreed this would be acceptable. - Looks like there is one upstream patch we are carrying (in containerd and runc?). What is the plan to upstream these patches? - Action: Ensure there is a task to upversion docker distribution registry to remove the need to carry this patch. - This too will be done early in stx.4.0. Shuicheng to update us in 1 week on when this task can be completed. Decision to allow into stx.3.0? Three requirements: - Shuicheng to complete testing for IPv6 and external registry with username/pwd authentication - Cores to complete their reviews - Frank to confirm with Ada that testing of this feature can be done in time for stx.3.0 - Backup plan: If instability is seen in stx.3.0 or significant testing by Ada results in failures then the feature would need to be backed up and fixed/delivered in stx.4.0 Frank Miller, Manager, Engineering, Wind River direct 613.963.1389 zoom frank.miller@windriver.com<mailto:frank.miller@windriver.com> 350 Terry Fox Drive, Suite 200, Kanata, ON K2K 2W5
participants (4)
-
Lin, Shuicheng
-
Miller, Frank
-
Wang, Jing (Angie)
-
Wensley, Barton