[Starlingx-discuss] [Containers] Worker nodes pulling from external registry instead from the internal

[Miller, Frank]

Erich - FYI - I re-gated this to stx.3.0 as this is an optimization and not urgent for StarlingX.

Frank

-----Original Message-----
From: Wang, Jing (Angie) [mailto:Angie.Wang at windriver.com] 
Sent: Monday, August 19, 2019 12:35 PM
To: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] [Containers] Worker nodes pulling from external registry instead from the internal

Hi Erich,

There is no logic problem of the download_an_image function. This function is only used at the application time (system application-apply). 

The issue here is after unlocking worker nodes, k8s pods(ie.. calico-node, kube-proxy ...) failed to start up due to image download failed. Currently, the k8s base images are pulled from external registry at bootstrap/puppet time, it depends on external network, so the issue happens if there has networking issue. This LP is to update to pull those base images from local registry instead to bring up k8s static and dynamic pods.

Thanks,
-Angie

-----Original Message-----
From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran at intel.com]
Sent: August-15-19 7:08 PM
To: starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] [Containers] Worker nodes pulling from external registry instead from the internal

Hi. 

I'm working on this bug: 
https://bugs.launchpad.net/starlingx/+bug/1817958 and I have a proposal that I would like to discuss.

In this bug, if an external docker registry is being used, all worker nodes will retrieve containers from the external registry instead of using the registry.local. This happens because the download_an_image[0] function has the following logic. 


If container has registry.local, then
    Try to download
    If not found in registry.local, then
        Try to download from public/private registry.
        Push image to registry.local
Else
    Try to download from public/private registry.


Here there are two bugs: 

1. If the container doesn't has the registry.local prefix, then the image won't be push into the local registry. This can be easily fix by adding the client.push call.
2. Worker nodes will still try to download from external registry even if the image was pushed to the local registry. This is because the images won't have the registry.local prefix.


So, to fix this I would like to propose the following: 

1. Ensure that all images should have the registry.local prefix, regardless if a private is defined or not. I'm not sure yet how to do this and I will appreciate some help point me to the right direction. 
2. Change the logic in download_an_image to do this: 

Try to download image:
If not found then:
    Remove registry.local prefix
    Try to download from public/private registry
    Retag and push to local registry.

By this way all nodes will try registry.local first and then public/private if not found.

What do you think about this fix?

Thank you in advance. 

-Erich

- [0] https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py#L2592



_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss