[Starlingx-discuss] CPU Isolation discrepancy
Hi I am running StarlingX 4.0 in Low Latency Mode in AIO-SX. We have isolated CPUs for application as below using the supported way. However, we can see that kubelet treads are scheduled on application isolated vcpus - Ex: 49, 50, 75. Could someone explain the behavior here?. Full log is attached. controller-0:/home/sysadmin# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 root=UUID=837a0268-a574-45ea-b560-a6e5afc483dd ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=tty0 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on user_namespace.enable=1 skew_tick=1 hugepagesz=1G hugepages=47 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0,24,48,72 rcu_nocbs=1-23,25-47,49-71,73-95 nohz_full=1-23,25-47,49-71,73-95 isolcpus=1-22,25-46,49-70,73-94 kthread_cpus=0,24,48,72 nopti nospectre_v2 nospectre_v1 [root@controller-0 sysadmin(keystone_admin)]# pidstat -t -p 134731 Linux 4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 (controller-0) 03/03/2021 _x86_64_(96 CPU) 01:26:28 AM 0 - 145642 0.00 0.00 0.00 0.00 50 |__kubelet 01:26:28 AM 0 - 145643 0.01 0.01 0.00 0.02 72 |__kubelet 01:26:28 AM 0 - 146196 0.05 0.03 0.00 0.08 24 |__kubelet 01:26:28 AM 0 - 147027 0.00 0.00 0.00 0.00 75 |__kubelet 01:26:28 AM 0 - 147028 0.00 0.00 0.00 0.00 49 |__kubelet system host-cpu-list controller-0 | 456f11f6-231f-4c6a-b9d9-fa35a6f6bff6 | 49 | 0 | 1 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | | | | | | | @ 3.00GHz | | | | | | | | | | | d491dd29-8603-4bf1-8620-9fbdde5d5d38 | 50 | 0 | 2 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | | 94bb8566-e726-418a-ae1f-3264807bfbcd | 75 | 1 | 3 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | Venkata Veldanda
The reason I bring out this concern is even dockerd seem to be running on isolated cores. For some reason the scheduler seem to be completely ignoring isolcpus. Is that correct? The impact here is that we have certain 100% poll threads that needs to be scheduled on couple of CPUs and we do not have a clue on where to allocate because of this behaviour. Any comments are appreciated. [cid:image001.jpg@01D7114F.2BCF1840] From: Venkata Ramana Veldanda Sent: Thursday, March 4, 2021 7:21 PM To: starlingx-discuss@lists.starlingx.io Cc: Srinivas Sadagopan <Srinivas.Sadagopan@radisys.com> Subject: CPU Isolation discrepancy Hi I am running StarlingX 4.0 in Low Latency Mode in AIO-SX. We have isolated CPUs for application as below using the supported way. However, we can see that kubelet treads are scheduled on application isolated vcpus - Ex: 49, 50, 75. Could someone explain the behavior here?. Full log is attached. controller-0:/home/sysadmin# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 root=UUID=837a0268-a574-45ea-b560-a6e5afc483dd ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=tty0 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on user_namespace.enable=1 skew_tick=1 hugepagesz=1G hugepages=47 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0,24,48,72 rcu_nocbs=1-23,25-47,49-71,73-95 nohz_full=1-23,25-47,49-71,73-95 isolcpus=1-22,25-46,49-70,73-94 kthread_cpus=0,24,48,72 nopti nospectre_v2 nospectre_v1 [root@controller-0 sysadmin(keystone_admin)]# pidstat -t -p 134731 Linux 4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 (controller-0) 03/03/2021 _x86_64_(96 CPU) 01:26:28 AM 0 - 145642 0.00 0.00 0.00 0.00 50 |__kubelet 01:26:28 AM 0 - 145643 0.01 0.01 0.00 0.02 72 |__kubelet 01:26:28 AM 0 - 146196 0.05 0.03 0.00 0.08 24 |__kubelet 01:26:28 AM 0 - 147027 0.00 0.00 0.00 0.00 75 |__kubelet 01:26:28 AM 0 - 147028 0.00 0.00 0.00 0.00 49 |__kubelet system host-cpu-list controller-0 | 456f11f6-231f-4c6a-b9d9-fa35a6f6bff6 | 49 | 0 | 1 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | | | | | | | @ 3.00GHz | | | | | | | | | | | d491dd29-8603-4bf1-8620-9fbdde5d5d38 | 50 | 0 | 2 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | | 94bb8566-e726-418a-ae1f-3264807bfbcd | 75 | 1 | 3 | 1 | Intel(R) Xeon(R) Gold 6248R CPU | Application-isolated | Venkata Veldanda
participants (1)
-
Venkata Ramana Veldanda