<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#2E74B5">Hi Sriram, we discussed this on the community call today…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#2E74B5"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#2E74B5">Nobody on the call had any specific suggestions for you, but I will ask the Containers team to weigh in on this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#2E74B5"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#2E74B5">Thanks, Bill... <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#2E74B5"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Dharwadkar, Sriram <Sriram.Dharwadkar@commscope.com>
<br>
<b>Sent:</b> Friday, May 21, 2021 3:21 PM<br>
<b>To:</b> starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> [Starlingx-discuss] [Distributed Edge cloud] Edge cloud: kubelet-taking 100% in edge cloud<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><span style="font-size:10.0pt;color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]<o:p></o:p></span></p>
</div>
<p class="MsoNormal">Hi Team,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have deployed Distributed StarlingX in one of our systems (with the load 20.06). Below is the version<o:p></o:p></p>
<p class="MsoNormal">OS="centos"<o:p></o:p></p>
<p class="MsoNormal">SW_VERSION="20.06"<o:p></o:p></p>
<p class="MsoNormal">BUILD_TARGET="Host Installer"<o:p></o:p></p>
<p class="MsoNormal">BUILD_TYPE="Formal"<o:p></o:p></p>
<p class="MsoNormal">BUILD_ID="r/stx.4.0"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">JOB="STX_4.0_build_layer_flock"<o:p></o:p></p>
<p class="MsoNormal">BUILD_BY="<a href="mailto:starlingx.build@cengn.ca">starlingx.build@cengn.ca</a>"<o:p></o:p></p>
<p class="MsoNormal">BUILD_NUMBER="22"<o:p></o:p></p>
<p class="MsoNormal">BUILD_HOST="starlingx_mirror"<o:p></o:p></p>
<p class="MsoNormal">BUILD_DATE="2020-08-05 12:25:52 +0000"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Edge cloud is running in AIO-Duplex low latency mode in 2 controllers with 24 cores each.
<o:p></o:p></p>
<p class="MsoNormal">When we deploy CPU intensive pods attached to SRIOV-vfs, we are observing an issue, where kubelet is taking 100% of CPU.<o:p></o:p></p>
<p class="MsoNormal">Below are the details captured.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">ontroller-1:/proc/2455541/fd# ps -ealf | grep -i kubelet<o:p></o:p></p>
<p class="MsoNormal">4 S root 2453203 1 39 80 0 - 265440 - 12:47 ? 00:31:18 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml
--container-runtime=remote --container-runtime-endpoint=/var/run/containerd/containerd.sock --cni-bin-dir=/usr/libexec/cni --node-ip=<a href="https://urldefense.com/v3/__http:/10.222.35.3__;!!AjveYdw8EvQ!Kyn7Pb6HI6QFExJsdXfs1By48FX3eETwJFAWF5u2PM132m3QF_eHiny5IN_bnKol6l7a$">10.222.35.3</a>
--volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ --feature-gates TopologyManager=true --cpu-manager-policy=static --topology-manager-policy=single-numa-node --system-reserved=memory=7000Mi --reserved-cpus=0-2 --container-runtime=remote
--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock<o:p></o:p></p>
<p class="MsoNormal">0 S root 3554919 2551522 0 80 0 - 28181 - 14:06 pts/2 00:00:00 grep --color=auto -i kubelet<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541/fd# ps -eLP | grep 2453203<o:p></o:p></p>
<p class="MsoNormal">2453203 2453203 0 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453206 2 ? 00:00:19 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453207 1 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453208 1 ? 00:00:12 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453209 0 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453210 1 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453211 1 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453215 2 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453218 2 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453254 1 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453255 1 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453287 0 ? 00:00:00 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2453290 1 ? 00:00:19 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2455519 1 ? 00:00:17 kubelet<o:p></o:p></p>
<p class="MsoNormal">2453203 2455540 2 ? 00:00:14 kubelet<o:p></o:p></p>
<p class="MsoNormal"><b>2453203 2455541 0 ? 00:28:24 kubelet<o:p></o:p></b></p>
<p class="MsoNormal">…..<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are around 31 threads of kubelet running. One of the thread(2455541) is taking 100% cpu. <o:p></o:p></p>
<p class="MsoNormal">Strace done on kubelet, shows the below output. It is trying to read something from the fd 32 continously<o:p></o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541/fd# strace -p 2455541<o:p></o:p></p>
<p class="MsoNormal">strace: Process 2455541 attached<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal">read(<b>32</b>, 0xc002125900, 4608) = ? ERESTARTNOINTR (To be restarted)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541# cd fd<o:p></o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541/fd# ls<o:p></o:p></p>
<p class="MsoNormal">0 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 23 24 25 26 27 29 3 30 31
<b>32</b> 34 4 5 6 7 8 9<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Fd 32 seems to be a calico virtual interface. Kubelet is spinning on this interface continuously which shoots up the CPU usage to 100%<o:p></o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541/fd# ls -l <b>32</b><o:p></o:p></p>
<p class="MsoNormal">lr-x------ 1 root root 64 May 21 13:44 32 -> /sys/devices/virtual/net/cali0db93ba21d0/speed<o:p></o:p></p>
<p class="MsoNormal">controller-1:/proc/2455541/fd#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Within few mins, we cannot do ssh to controller-1. In console, top shows the kubelet taking 100%.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any clues on this issue would be of great help, let me know if any information is required.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<br>
Sriram<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>