[Starlingx-discuss] k8s pods cant use application-isolated cpus

Chris Friesen chris.friesen at windriver.com
Wed Feb 24 17:47:58 UTC 2021


Upstream Kubernetes doesn't really know how to handle "isolated" CPUs 
separately from "normal" CPUs. Because of this sysinv will reserve the 
"application-isolated" CPUs from use by Kubernetes when using the 
"static" CPU Manager policy.

If you set the kube-cpu-mgr-policyhost label to "none" (via Horizon or 
the "sysinv" CLI) Kubernetes will use the default policy and should 
include the "application-isolated" CPUs as part of the "Allocatable" cpu 
count.  You'll have to handle CPU affinity of containerized processes 
explicitly though to ensure that they don't interfere with each other.

Commercial products based on StarlingX have support for isolated CPUs 
with the "static" CPU Manager policy by modifying upstream Kubernetes.

Hope that helps,

Chris

On 2/18/2021 12:00 PM, Dharwadkar, Sriram wrote:
>
> Hi,
>
> We are using Distributed-StarlingX-4.0.
>
> Our use case requires pods to use isolated cpu’s, so executed the
>
> “system host-cpu-modify -f application-isolated -p0 20 controller-0” 
> command to isolate the cpus.
>
> system host-cpu-list controller-1
>
> +--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+
>
> | uuid                                 | log_core | processor | 
> phy_core | thread | processor_model                           | 
> assigned_function    |
>
> +--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+
>
> | e64e0004-3096-40c7-804c-d9c79d2ceb17 | 0        | 0         | 
> 0        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Platform             |
>
> | 28697ba9-c5e5-4d9b-9d97-346ec61c1fa7 | 1        | 0         | 
> 1        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Platform             |
>
> | 1b111e2e-84e4-4414-870f-73c759ecbac0 | 2        | 0         | 
> 2        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 66933896-5c9f-4e0a-9527-72025172e1f6 | 3        | 0         | 
> 3        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 027258c8-392e-4497-9e46-6acb007e759f | 4        | 0         | 
> 4        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 5d88a23a-68de-4062-9313-173fd91a4bc5 | 5        | 0         | 
> 5        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | dd804bbb-191c-4499-bda1-272f9bb923af | 6        | 0         | 
> 6        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 7ae2a90b-59ca-4af0-b20e-20635af0aceb | 7        | 0         | 
> 8        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | ada68f62-4d06-43eb-8ed9-cd6ec6d33487 | 8        | 0         | 
> 9        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 8c2612fd-8a7f-4732-82ff-fb4717ab9275 | 9        | 0         | 
> 10       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 01412e4d-c9ff-41e8-8fd3-c2a0d27ac7a3 | 10       | 0         | 
> 11       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | c8e8b15b-538b-422e-9d72-2ba96982054a | 11       | 0         | 
> 12       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | b2efd166-3125-46b9-95ac-4fb00a0de3c7 | 12       | 0         | 
> 13       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | c33f3dd7-0b06-40c3-93b3-82f065c6db05 | 13       | 0         | 
> 16       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | e33cda3e-7162-45d4-af79-4dff7f0bb07f | 14       | 0         | 
> 17       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 551b7272-371e-4c05-b5c8-7708f95d3a97 | 15       | 0         | 
> 18       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | d0522e8f-ebd4-49fd-8a0e-26de857f2caf | 16       | 0         | 
> 19       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 84dd71f2-ad7a-46bb-8ae2-e62b2a817cc5 | 17       | 0         | 
> 20       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | f4761605-3208-4758-b31c-d679cb0b631e | 18       | 0         | 
> 21       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 66423b6f-134b-4166-bdff-9643d4ec8d4d | 19       | 0         | 
> 25       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 45d38571-ca97-4b89-b442-fe5f5282bc5f | 20       | 0         | 
> 26       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | fc87bbd5-2954-4730-9f67-b8fe8820faa7 | 21       | 0         | 
> 27       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application-isolated |
>
> | 6b3a8a78-d191-47fa-af82-15166a7924fc | 22       | 0         | 
> 28       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application          |
>
> | 57739379-ab8f-4d3d-847a-f5aa9663bcfb | 23       | 0         | 
> 29       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | 
> Application          |
>
> +--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+
>
> But the k8s is not able to allocate isolated cpus to pods. Allocatable 
> cpus are only 2.
>
> controller-0:/home/sysadmin# kubectl get node controller-0 -o json | 
> jq '.status.allocatable'
>
> {
>
> *  "cpu": "2",*
>
>   "ephemeral-storage": "9391196145",
>
>   "hugepages-1Gi": "20Gi",
>
>   "hugepages-2Mi": "0",
>
>   "intel.com/pci_sriov_net_physnet0": "32",
>
>   "memory": "70207428Ki",
>
>   "pods": "110"
>
> }
>
> I saw this bug report, which seems to be talking about the same issue.
>
> https://bugs.launchpad.net/starlingx/+bug/1894173 
> <https://bugs.launchpad.net/starlingx/+bug/1894173>
>
> Kindly let me know when the fix is planned.
>
> If there is any work around to solve this issue, do let me know.
>
> Regards,
>
> Sirram
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210224/45ce13ae/attachment-0001.html>


More information about the Starlingx-discuss mailing list