Hi Sriram, we discussed this on the community call today…

 

Nobody on the call had any specific suggestions for you, but I will ask the Containers team to weigh in on this.

 

Thanks, Bill...  

 

From: Dharwadkar, Sriram <Sriram.Dharwadkar@commscope.com>
Sent: Friday, May 21, 2021 3:21 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] [Distributed Edge cloud] Edge cloud: kubelet-taking 100% in edge cloud

 

[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi Team,

 

We have deployed Distributed StarlingX in one of our systems (with the load 20.06). Below is the version

OS="centos"

SW_VERSION="20.06"

BUILD_TARGET="Host Installer"

BUILD_TYPE="Formal"

BUILD_ID="r/stx.4.0"

 

JOB="STX_4.0_build_layer_flock"

BUILD_BY="starlingx.build@cengn.ca"

BUILD_NUMBER="22"

BUILD_HOST="starlingx_mirror"

BUILD_DATE="2020-08-05 12:25:52 +0000"

 

Edge cloud is running in AIO-Duplex low latency mode in 2 controllers with 24 cores each.

When we deploy CPU intensive pods attached to SRIOV-vfs, we are observing an issue, where kubelet is taking 100% of CPU.

Below are the details captured.

 

ontroller-1:/proc/2455541/fd# ps -ealf | grep -i kubelet

4 S root     2453203       1 39  80   0 - 265440 -     12:47 ?        00:31:18 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/var/run/containerd/containerd.sock --cni-bin-dir=/usr/libexec/cni --node-ip=10.222.35.3 --volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ --feature-gates TopologyManager=true --cpu-manager-policy=static --topology-manager-policy=single-numa-node --system-reserved=memory=7000Mi --reserved-cpus=0-2 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock

0 S root     3554919 2551522  0  80   0 - 28181 -      14:06 pts/2    00:00:00 grep --color=auto -i kubelet

 

controller-1:/proc/2455541/fd# ps -eLP | grep 2453203

2453203 2453203   0 ?        00:00:00 kubelet

2453203 2453206   2 ?        00:00:19 kubelet

2453203 2453207   1 ?        00:00:00 kubelet

2453203 2453208   1 ?        00:00:12 kubelet

2453203 2453209   0 ?        00:00:00 kubelet

2453203 2453210   1 ?        00:00:00 kubelet

2453203 2453211   1 ?        00:00:00 kubelet

2453203 2453215   2 ?        00:00:00 kubelet

2453203 2453218   2 ?        00:00:00 kubelet

2453203 2453254   1 ?        00:00:00 kubelet

2453203 2453255   1 ?        00:00:00 kubelet

2453203 2453287   0 ?        00:00:00 kubelet

2453203 2453290   1 ?        00:00:19 kubelet

2453203 2455519   1 ?        00:00:17 kubelet

2453203 2455540   2 ?        00:00:14 kubelet

2453203 2455541   0 ?        00:28:24 kubelet

…..

 

There are around 31 threads of kubelet running. One of the thread(2455541) is taking 100% cpu.  

Strace done on kubelet, shows the below output. It is trying to read something from the fd 32 continously

controller-1:/proc/2455541/fd# strace -p 2455541

strace: Process 2455541 attached

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

read(32, 0xc002125900, 4608)            = ? ERESTARTNOINTR (To be restarted)

 

controller-1:/proc/2455541# cd fd

controller-1:/proc/2455541/fd# ls

0  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  29  3  30  31  32  34  4  5  6  7  8  9

 

Fd 32 seems to be a calico virtual interface. Kubelet is spinning on this interface continuously which shoots up the CPU usage to 100%

controller-1:/proc/2455541/fd# ls -l 32

lr-x------ 1 root root 64 May 21 13:44 32 -> /sys/devices/virtual/net/cali0db93ba21d0/speed

controller-1:/proc/2455541/fd#

 

Within few mins, we cannot do ssh to controller-1. In console, top shows the kubelet taking 100%.

 

Any clues on this issue would be of great help, let me know if any information is required.

 

Regards,
Sriram