Hi Team,

We have deployed Distributed StarlingX in one of our systems (with the load 20.06). Below is the version

OS="centos"

SW_VERSION="20.06"

BUILD_TARGET="Host Installer"

BUILD_TYPE="Formal"

BUILD_ID="r/stx.4.0"

JOB="STX_4.0_build_layer_flock"

BUILD_BY="starlingx.build@cengn.ca"

BUILD_NUMBER="22"

BUILD_HOST="starlingx_mirror"

BUILD_DATE="2020-08-05 12:25:52 +0000"

Edge cloud is running in AIO-Duplex low latency mode in 2 controllers with 24 cores each.

When we deploy CPU intensive pods attached to SRIOV-vfs, we are observing an issue, where kubelet is taking 100% of CPU.

Below are the details captured.

ontroller-1:/proc/2455541/fd# ps -ealf | grep -i kubelet

4 S root 2453203 1 39 80 0 - 265440 - 12:47 ? 00:31:18 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/var/run/containerd/containerd.sock --cni-bin-dir=/usr/libexec/cni --node-ip=10.222.35.3 --volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ --feature-gates TopologyManager=true --cpu-manager-policy=static --topology-manager-policy=single-numa-node --system-reserved=memory=7000Mi --reserved-cpus=0-2 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock

0 S root 3554919 2551522 0 80 0 - 28181 - 14:06 pts/2 00:00:00 grep --color=auto -i kubelet

controller-1:/proc/2455541/fd# ps -eLP | grep 2453203

2453203 2453203 0 ? 00:00:00 kubelet

2453203 2453206 2 ? 00:00:19 kubelet

2453203 2453207 1 ? 00:00:00 kubelet

2453203 2453208 1 ? 00:00:12 kubelet

2453203 2453209 0 ? 00:00:00 kubelet

2453203 2453210 1 ? 00:00:00 kubelet

2453203 2453211 1 ? 00:00:00 kubelet

2453203 2453215 2 ? 00:00:00 kubelet

2453203 2453218 2 ? 00:00:00 kubelet

2453203 2453254 1 ? 00:00:00 kubelet

2453203 2453255 1 ? 00:00:00 kubelet

2453203 2453287 0 ? 00:00:00 kubelet

2453203 2453290 1 ? 00:00:19 kubelet

2453203 2455519 1 ? 00:00:17 kubelet

2453203 2455540 2 ? 00:00:14 kubelet

2453203 2455541 0 ? 00:28:24 kubelet

…..

There are around 31 threads of kubelet running. One of the thread(2455541) is taking 100% cpu.

Strace done on kubelet, shows the below output. It is trying to read something from the fd 32 continously

controller-1:/proc/2455541/fd# strace -p 2455541

strace: Process 2455541 attached

read(32, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)

controller-1:/proc/2455541# cd fd

controller-1:/proc/2455541/fd# ls

0 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 23 24 25 26 27 29 3 30 31 32 34 4 5 6 7 8 9

Fd 32 seems to be a calico virtual interface. Kubelet is spinning on this interface continuously which shoots up the CPU usage to 100%

controller-1:/proc/2455541/fd# ls -l 32

lr-x------ 1 root root 64 May 21 13:44 32 -> /sys/devices/virtual/net/cali0db93ba21d0/speed

controller-1:/proc/2455541/fd#

Within few mins, we cannot do ssh to controller-1. In console, top shows the kubelet taking 100%.

Any clues on this issue would be of great help, let me know if any information is required.

Regards,
Sriram