<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-style-priority:99;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:1.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Times New Roman",serif;}
span.emailstyle20
{mso-style-name:emailstyle20;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.emailstyle21
{mso-style-name:emailstyle21;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.emailstyle22
{mso-style-name:emailstyle22;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.emailstyle24
{mso-style-name:emailstyle24;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.emailstyle25
{mso-style-name:emailstyle25;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Hi Brent and Ovidiu,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Could you kindly help to clear following concern?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">5. needs the shadow tenant created before use => puppet / helm chart chart update (for --kubernates)</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">#5: This is mandatory, otherwise cinder's caching won't work at all.</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] It has to set cinder_internal_tenant_project_id and cinder_internal_tenant_user_id before enabling cache images. As this user can manage these
cached image volumes. Why can’t it work with Kubernetes? </span></i></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Li, Xiaoyan [mailto:xiaoyan.li@intel.com]
<br>
<b>Sent:</b> Tuesday, November 27, 2018 10:30 AM<br>
<b>To:</b> Poncea, Ovidiu <Ovidiu.Poncea@windriver.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io><br>
<b>Subject:</b> Re: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Ovidiu,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As far as I’m concerned, Cinder image cache is an cache mechanism. So overall, users don’t need to clean it manually.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Currently when capacity for cache is full, it removes the cached image volumes with LRU policy. More detailed please see the following comments.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Poncea, Ovidiu [<a href="mailto:Ovidiu.Poncea@windriver.com">mailto:Ovidiu.Poncea@windriver.com</a>]
<br>
<b>Sent:</b> Monday, November 26, 2018 11:15 PM<br>
<b>To:</b> Li, Xiaoyan <<a href="mailto:xiaoyan.li@intel.com">xiaoyan.li@intel.com</a>>; Rowsell, Brent <<a href="mailto:Brent.Rowsell@windriver.com">Brent.Rowsell@windriver.com</a>>; 'starlingx-discuss@lists.starlingx.io' <<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Cc:</b> Miller, Frank <<a href="mailto:Frank.Miller@windriver.com">Frank.Miller@windriver.com</a>><br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Hi Lisa,<br>
<br>
Yeah, even if we refactor raw caching, it's most likely going to be rejected by upstream due to replicating existing functionality in cinder. Yet, imho, we should have an working replacement before retiring raw caching and we should have some agreed mitigations
in place for cinder's disadvantages (if we can't live with them, Brent please help here). See my questions bellow & inline. Also, please correct text bellow if I made wrong assumptions as you know cinder's caching better than me.<br>
<br>
Short comparison of the two:<br>
<br>
<b>Raw caching</b><br>
<br>
Uses --raw-cache cli option in Glance to trigger a background process that converts the image. Once cached, new volumes get created on Ceph instantly by levereging Ceph's copy-on-write. Cache is allocated from the "images" RBD pool.<br>
<br>
<i>Advantages:</i><br>
- user can select the images it wants to cache<br>
- user can monitor the progress and can check used space for each image (cli + dashboard).<br>
- on image delete the cache is also cleared if there is no volume using it. Else it is cleared with the last volume keeping the cache data in-use.<br>
- no wasted space<br>
- complete control by user<br>
<br>
<i>Disadvantages:</i><br>
- There is almost no way this is going to be accepted upstream. Maybe, yet with small hopes, if we refactor everything as a 3rd party glance feature, but we may need to push some hooks upstream to make it work.<br>
- Ceph only<br>
<br>
<b>Cinder's caching</b><br>
<br>
Uses a "shadow" tenant to store shadow volumes. Cache is created with the first volume from that images. Next volume will be created instantly by leveraging copy-on-write if backend provides support for it (e.g. on Ceph). Space for cache is allocated on one
of the cinder backends, has a configurable threshold.<br>
<br>
<i>Advantages:</i><br>
- already upstream<br>
- works with all backends<br>
- all cached images are displayed for the "admin" if he changes to the shadow tenant and lists volumes.<br>
- admin (not user, only admin) can free cache by deleting volumes of the shadow tenant (need confirmation)<br>
<i><br>
Disadvantages:</i><br>
1. it's either globally enabled or disabled => needs sysinv configuration option<br>
2. it caches every image. No way to select what image to cache nor with what backend (question bellow) => space waste<br>
3. cached images are not removed. It needs to hit a space provision to do that, and it will remove the oldest image, although that image cache may be important.<br>
4. less control: Images are cached on first use and are removed when provisioned space hits threshold. This means that user does not have control over what images are converted and what images are in cache. So, sometimes volume creation works fast, other times
it's slow. This can be a problem especially on parallel volume creation through helm charts as, if the image did not have a cache, then stack creation may timeout. Another problem may be if cache is small and images get rotated in the cache => we need alarms
when threshold is hit.<br>
5. needs the shadow tenant created before use => puppet / helm chart chart update (for --kubernates)<br>
<br>
<b><i>Mitigations of disadvantages above - possible solutions and alternatives:</i></b><br>
#1: Customers may not want to enable it, we should allow customers to choose when to enable it (it can be added as a custom capabilities parameter to "system storage-backend-add/system storage-backend-modify")</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] Currently image cache can be enabled/disabled per backend storage.
<a href="https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html">
https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html</a><o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I think it is enough.
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
#2: No workaround comes to my mind - we can probably live with it</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
#3: A simple solution would be to implement a cron job to clean cache periodically or a more elaborate solution would be remove the cache with the last volume that used that image (need a cinder upstream feature for it)</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] From doc, currently Cinder removes cached images from least used to recently used. Every time Cinder uses a cached image volume, it updates
its last_used field. This is the normal policy for data eviction. <o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As it is cache and should be transparent for users, why do we need users to evict data?<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
#4: Two options comes to mind:<br>
1. To get some control we should not limit the cache size, given that we do propper cleanup in #3.</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] Even we do cleanup, the limit can’t be removed.
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
2. If we limit the cache, we have to make the limit configurable and raise an alarm once cache gets near full so that admin takes preventive measures and either increases provisioned space or</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
#5: This is mandatory, otherwise cinder's caching won't work at all.</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] It has to set cinder_internal_tenant_project_id and cinder_internal_tenant_user_id before enabling cache images. As this user can manage these
cached image volumes. Why can’t it work with Kubernetes?</span></i></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
<br>
<b><i>Questions, (maybe if you get time to play with cinder's caching to get a better understanding):</i></b><br>
1. How does cinder's caching behaves when multiple volumes are created in parallel from a newly created image? Will it wait for the cache to be created before creating the volumes or just start all volume creations in parallel?</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] Inside a volume service it is sequential to run volume creation tasks. But as we have HA. For image cache, it creates an entry in cinder db
at first and then creates volumes. The primary key is not image_id+backend_storage.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It is possible that several entries or volumes will be created in same backend storage.
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><br>
2. What is the cinder backend that store the cache? If it is the one used by the volume, will this lead to multiple cached volumes of the same image? Can we chose the backend?<o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] We can set whether enabled cache per backend. If users create a volume in backend ceph from an image, an cached image volume will be created
in Ceph if it is enabled. Next time if users create a volume in IBM storage from the same image, it will create another image cached volume in IBM storage if it is enabled.
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">3. How is cache space provisioned? Do we need to restart cinder-volume for changes to take effect?<o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] These config needs to be done in config file. So it needs to restart cinder volume services once config are changed.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">4. Is admin able to clean up individual cached images in the shadow tenant? Maybe also user?</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1F497D"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Li, Xiaoyan] Admin and shadow tenants can both do cleanup.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Ovidiu<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div id="divRpF227160">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Li, Xiaoyan [xiaoyan.li@intel.com]<br>
<b>Sent:</b> Thursday, November 22, 2018 2:41 AM<br>
<b>To:</b> Poncea, Ovidiu; Rowsell, Brent; 'starlingx-discuss@lists.starlingx.io'<br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Brent and Ovidiu,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As this email has a long history, I re-summarize the raw cache in StarlingX and Cinder upstream image cache.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Please vote whether we can abandon raw cache in StarlingX.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">StarlingX</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Create an image cache in ceph when Glance creates an image.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">And delete the cached image in ceph when deleting the original image in Glance.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Cinder:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When creating a volume from an image in a backend storage at the first time, Cinder creates a volume from this image, and uses it as the image cache.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">So next time if users create another volume from this image in the same backend storage, Cinder at first finds out the cached image volume and clones a new volume
from it.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Cinder allows capacity configuration for cached images. If the space is used up, Cinder will evict the cached image volumes.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">From my viewpoint,
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Cinder image cache can achieve same functionality as Raw cache in StarlingX with more enhancement. It is for all Cinder supported backend storage, not just for
Ceph.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Li, Xiaoyan
<br>
<b>Sent:</b> Monday, November 19, 2018 9:44 AM<br>
<b>To:</b> Poncea, Ovidiu <<a href="mailto:Ovidiu.Poncea@windriver.com">Ovidiu.Poncea@windriver.com</a>>; Rowsell, Brent <<a href="mailto:Brent.Rowsell@windriver.com">Brent.Rowsell@windriver.com</a>>; 'starlingx-discuss@lists.starlingx.io' <<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Ovidiu,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A cached image ( new volume from this image) is created on a storage backend when Cinder firstly creates a volume in the same backend storage from the image.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">All the information are stored in Cinder, including volume id, image id etc.
<a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/manager.py#L1368" target="_blank">
https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/manager.py#L1368</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L82" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L82</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A cached image is deleted when the configure space for cache is used up. So currently Cinder doesn’t delete the cached image volumes even if the image is deleted.
But this can be an enhancement of current cinder image cache.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L117" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L117</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/manager.py#L1351" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/manager.py#L1351</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Poncea, Ovidiu [<a href="mailto:Ovidiu.Poncea@windriver.com" target="_blank">mailto:Ovidiu.Poncea@windriver.com</a>]
<br>
<b>Sent:</b> Friday, November 16, 2018 4:57 PM<br>
<b>To:</b> Li, Xiaoyan <<a href="mailto:xiaoyan.li@intel.com" target="_blank">xiaoyan.li@intel.com</a>>; Rowsell, Brent <<a href="mailto:Brent.Rowsell@windriver.com" target="_blank">Brent.Rowsell@windriver.com</a>>; 'starlingx-discuss@lists.starlingx.io' <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Hi Li,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Quick question: Is cache going to be freed when an image is deleted from glance? It would be a waste to cache images that are no longer needed.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Ovidiu</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div id="divRpF502789">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Li, Xiaoyan [xiaoyan.li@intel.com]<br>
<b>Sent:</b> Tuesday, November 13, 2018 9:19 AM<br>
<b>To:</b> Rowsell, Brent; 'starlingx-discuss@lists.starlingx.io'<br>
<b>Subject:</b> Re: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">About the raw cache function in StarlingX Cinder and Glance, I would like to remove it as Cinder has similar function. Please see following detail.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">And if I would like to remove the function in StarlingX, there are two methods:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Submit a patch to revert the changes in Glance and Cinder.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ignore these patches during upgrading StarlingX/Cinder to new Cinder release.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Which way do we prefer to?</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Li, Xiaoyan
<br>
<b>Sent:</b> Thursday, September 20, 2018 10:17 AM<br>
<b>To:</b> Rowsell, Brent <<a href="mailto:Brent.Rowsell@windriver.com" target="_blank">Brent.Rowsell@windriver.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Hi, Brent</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">The following are mechanism of Cinder volume cache.</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Creation of cached volume:
</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">It creates a cached volume in the backend storage when creating from an image.</span><span style="color:black"><o:p></o:p></span></p>
<p style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">1.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Create_from_image:
</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L890" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L890</a></span><span style="color:black"><o:p></o:p></span></p>
<p style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">2.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Return image cache entry: If not existed, it creates a new entry.</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L746" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L746</a></span><span style="color:black"><o:p></o:p></span></p>
<p style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">3.</span><span style="font-size:7.0pt;color:black">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Create a new image-volume and cache entry for it:</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L872" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L872</a></span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Use a cached volume when creating a volume:</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L723-L735" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/flows/manager/create_volume.py#L723-L735</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Delete the cache volume: When capacity and number of cache entries exceed specified limit, it deletes cache entries (cached volumes).
</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L164" target="_blank">https://github.com/openstack/cinder/blob/stable/pike/cinder/image/cache.py#L164</a>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best wishes</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lisa</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Rowsell, Brent [<a href="mailto:Brent.Rowsell@windriver.com" target="_blank">mailto:Brent.Rowsell@windriver.com</a>]
<br>
<b>Sent:</b> Thursday, September 6, 2018 10:02 AM<br>
<b>To:</b> Li, Xiaoyan <<a href="mailto:xiaoyan.li@intel.com" target="_blank">xiaoyan.li@intel.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> RE: [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We would need to review this feature to ensure it provides equivalent functionality first.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If it does, great, we can look at reverting and enabling this cinder functionality.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Brent
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Li, Xiaoyan [<a href="mailto:xiaoyan.li@intel.com" target="_blank">mailto:xiaoyan.li@intel.com</a>]
<br>
<b>Sent:</b> Wednesday, September 5, 2018 9:59 PM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> [Starlingx-discuss] [StarlingX] Use Cinder generic image cache to replace raw caching</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Hi all,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">This email is about Raw caching function in StarlingX. This feature is to cache an image in backend storage like Ceph when we first create a volume in this backend
storage.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">In fact, Cinder upstream has already had a similar function in Pike release.
<a href="https://specs.openstack.org/openstack/cinder-specs/specs/liberty/image-volume-cache.html" target="_blank">
<span style="color:#0563C1">https://specs.openstack.org/openstack/cinder-specs/specs/liberty/image-volume-cache.html</span></a>
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">So I want to revert Raw caching function in StarlingX, and use Cinder generic image cache instead.
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">The problem is that we need to update Cinder config in StarlingX. Any comments?
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Best wishes</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Lisa</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>