Upstream Kubernetes doesn't really know how to handle "isolated" CPUs separately from "normal" CPUs.  Because of this sysinv will reserve the "application-isolated" CPUs from use by Kubernetes when using the "static" CPU Manager policy.

If you set the kube-cpu-mgr-policy host label to "none" (via Horizon or the "sysinv" CLI) Kubernetes will use the default policy and should include the "application-isolated" CPUs as part of the "Allocatable" cpu count.  You'll have to handle CPU affinity of containerized processes explicitly though to ensure that they don't interfere with each other.

Commercial products based on StarlingX have support for isolated CPUs with the "static" CPU Manager policy by modifying upstream Kubernetes.


Hope that helps,

Chris

On 2/18/2021 12:00 PM, Dharwadkar, Sriram wrote:

Hi,

 

We are using Distributed-StarlingX-4.0.

Our use case requires pods to use isolated cpu’s, so executed the

“system host-cpu-modify -f application-isolated -p0 20 controller-0” command to isolate the cpus.

 

system host-cpu-list controller-1

+--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+

| uuid                                 | log_core | processor | phy_core | thread | processor_model                           | assigned_function    |

+--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+

| e64e0004-3096-40c7-804c-d9c79d2ceb17 | 0        | 0         | 0        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Platform             |

| 28697ba9-c5e5-4d9b-9d97-346ec61c1fa7 | 1        | 0         | 1        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Platform             |

| 1b111e2e-84e4-4414-870f-73c759ecbac0 | 2        | 0         | 2        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 66933896-5c9f-4e0a-9527-72025172e1f6 | 3        | 0         | 3        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 027258c8-392e-4497-9e46-6acb007e759f | 4        | 0         | 4        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 5d88a23a-68de-4062-9313-173fd91a4bc5 | 5        | 0         | 5        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| dd804bbb-191c-4499-bda1-272f9bb923af | 6        | 0         | 6        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 7ae2a90b-59ca-4af0-b20e-20635af0aceb | 7        | 0         | 8        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| ada68f62-4d06-43eb-8ed9-cd6ec6d33487 | 8        | 0         | 9        | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 8c2612fd-8a7f-4732-82ff-fb4717ab9275 | 9        | 0         | 10       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 01412e4d-c9ff-41e8-8fd3-c2a0d27ac7a3 | 10       | 0         | 11       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| c8e8b15b-538b-422e-9d72-2ba96982054a | 11       | 0         | 12       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| b2efd166-3125-46b9-95ac-4fb00a0de3c7 | 12       | 0         | 13       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| c33f3dd7-0b06-40c3-93b3-82f065c6db05 | 13       | 0         | 16       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| e33cda3e-7162-45d4-af79-4dff7f0bb07f | 14       | 0         | 17       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 551b7272-371e-4c05-b5c8-7708f95d3a97 | 15       | 0         | 18       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| d0522e8f-ebd4-49fd-8a0e-26de857f2caf | 16       | 0         | 19       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 84dd71f2-ad7a-46bb-8ae2-e62b2a817cc5 | 17       | 0         | 20       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| f4761605-3208-4758-b31c-d679cb0b631e | 18       | 0         | 21       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 66423b6f-134b-4166-bdff-9643d4ec8d4d | 19       | 0         | 25       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 45d38571-ca97-4b89-b442-fe5f5282bc5f | 20       | 0         | 26       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| fc87bbd5-2954-4730-9f67-b8fe8820faa7 | 21       | 0         | 27       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated |

| 6b3a8a78-d191-47fa-af82-15166a7924fc | 22       | 0         | 28       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application          |

| 57739379-ab8f-4d3d-847a-f5aa9663bcfb | 23       | 0         | 29       | 0      | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application          |

+--------------------------------------+----------+-----------+----------+--------+-------------------------------------------+----------------------+

But the k8s is not able to allocate isolated cpus to pods. Allocatable cpus are only 2.

controller-0:/home/sysadmin# kubectl get node controller-0 -o json | jq '.status.allocatable'

{

  "cpu": "2",

  "ephemeral-storage": "9391196145",

  "hugepages-1Gi": "20Gi",

  "hugepages-2Mi": "0",

  "intel.com/pci_sriov_net_physnet0": "32",

  "memory": "70207428Ki",

  "pods": "110"

}

 

 

I saw this bug report, which seems to be talking about the same issue.

https://bugs.launchpad.net/starlingx/+bug/1894173

Kindly let me know when the fix is planned.

 

If there is any work around to solve this issue, do let me know.

 

Regards,

Sirram

 


_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss