Hi,

We are running the distributed cloud system, and maintain all the starlingx images in the private repository. The repository information is provided like this in the bootstrap.yml file of an edge.

docker_registries:

k8s.gcr.io:

url: 10.222.22.1:30003/starlingx

quay.io:

url: 10.222.22.1:30003/starlingx

docker.io:

url: 10.222.22.1:30003/starlingx

gcr.io:

url: 10.222.22.1:30003/starlingx

defaults:

url: 10.222.22.1:30003/starlingx

username: admin

password: XXXXXXX

Image path in the repository: starlingx/external_storage/rbd-provisioner:v2.1.1-k8s1.11

For some edges the rbd-provisioner image failed to downloads or failed to pushed to the local image repository but for some edge it is downloaded and pushed to the local repository. The image repository and registry configuration is common for all the edges.

I am not able to understand and debug what is going wrong here. However if I modified the deployment of rbd-provisioner to use the image directly from external private repository. It is able to download the image.

Steps to update the deployment:

kubectl edit deployment rbd-provisioner -n kube-system
update image: “registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11” to “image: 10.222.22.1:30003/starlingx/external_storage/rbd-provisioner:v2.1.1-k8s1.11”

Image error:

kube-system rbd-provisioner-77bfb6dbb-z6dp8 0/1 ImagePullBackOff 0 20d

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Pulling 15m (x1603 over 5d16h) kubelet, controller-0 Pulling image "registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11"

Normal BackOff 5m14s (x36129 over 5d16h) kubelet, controller-0 Back-off pulling image "registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11"

Warning Failed 16s (x36150 over 5d16h) kubelet, controller-0 Error: ImagePullBackOff

controller-0:/$

Any idea what is going wrong here and how to debug this issue?

Thanks,

Ankush