<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi All,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is a piece of information I hope can help trouble-shooting and avoid coding mistakes:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In stx system, there is a /tmp dir which is the system default temporary file location. System command mktemp and mkstemp() in python for example will create a temp file in there if temp dir is not explicitly set by either environment variable or in the function
call. We see some programs in the system are doing that.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
But the /tmp dir is considered as "temporary" files holder. CentOS has systemd-tmpfiles-clean.timer enabled, and it will wake systemd-tmpfiles-clean.service every day, to cleanup files in /tmp that are older than 10 days (configured in /etc/tmpfiles.d/tmp.conf).
The consequence is that some programs start failing or generate errors as they assume their temp files are still there. This is the root cause for [1].<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In [1], sysinv-conductor uses kubernetes python client to access k8s services. The kubernetes client caches credentials in /tmp files. If the code to access k8s services are not triggered for more than 10 days, the /tmp files are removed as part of the systemd
tmpfiles cleanup. Then the client used by sysinv-conductor stops working, complaining its temp files are not found. This is a known issue with kubernetes python client as in [2].<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So the conclusion are:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<ul>
<li>don't assume files in /tmp will be "permanent", be careful when use it for long lasting processes.</li><li>as kubernetes python client caches info in /tmp by default, either release the client after each use of it, or change its default temp dir by setting environment variable such as TMPDIR.<br>
</li></ul>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt">[1] <a href="https://bugs.launchpad.net/starlingx/+bug/1883599" id="LPNoLP580442">
https://bugs.launchpad.net/starlingx/+bug/1883599</a><br>
</span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<font size="2"><span style="font-size:11pt">[2] <a href="https://github.com/kubernetes-client/python/issues/765" target="_blank" rel="nofollow" id="LPlnk141222">
https://github.com/kubernetes-client/python/issues/765</a></span></font><br>
</div>
<br>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hope this help,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Andy<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><font size="2" face="Calibri"><span style="font-size:11pt"><font face="Calibri">/tmp is cleaned up every 10 days and point to the LP and explain the issue we had in this particular case. This impacts any StarlingX component that (mistakenly) assumes that
the /tmp dir is an OK place to anything but very short-lived files.</font></span></font></span><br>
</div>
</body>
</html>