[Starlingx-discuss] [Containers] Deployment status

[Lemus Contreras, Cristopher J]

Hello All,

** Baremetal Environment **

Simplex configuration on BareMetal servers without internet access.

SUMMARY:


• config_controller is over-writing /etc/hosts and /etc/docker/daemon.json, which, depending of the user environment, might need special settings.
• config_controller is not pulling images from local registry, using a local/mirror registry is a viable option? 
• Errors during system application-apply related to docker images not being pulled.



During the install, on servers that doesn’t have internet connection, config_controller will fail because it will try to pull docker images, which is an expected behavior. Docker offers a solution that is basically defining a local registry (mirror registry) from where the infrastructure can pull the images. In our environment we have http://registry.localdomain defined as a registry, /etc/docker/daemon.json can be defined like this:

{
    "registry-mirrors": ["https://registry.localdomain"]
}

With this method, we can pull all docker images without issues:

• One from http://quay.io

localhost:~# docker pull calico/node:v3.1.4
v3.1.4: Pulling from calico/node
c67f3896b22c: Pull complete
1386084f19d9: Pull complete
3a59f3ebe6d0: Pull complete
0fc581a09289: Pull complete
Digest: sha256:87dd10c421dff39c5546c888ef0905c1b8e9bf7d7c8c4668f5dc3842753bdab9
Status: Downloaded newer image for calico/node:v3.1.4
localhost:~#

• One from gcr.io/kubernetes-helm/tiller  

localhost:~# docker pull kubernetes-helm/tiller:v2.12.1
v2.12.1: Pulling from kubernetes-helm/tiller
c67f3896b22c: Already exists
de5dfb1eb274: Pull complete
d144c197b919: Pull complete
a8ecec0345d6: Pull complete
Digest: sha256:3060d5de6614e96adf540c10528929a056b1a7751e781beff4cc00d2e47e114c
Status: Downloaded newer image for kubernetes-helm/tiller:v2.12.1

localhost:~# docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
kubernetes-helm/tiller   v2.12.1             def589f37fc4        5 weeks ago         81.4MB
calico/node              v3.1.4              715f56d3ab67        2 months ago        241MB
localhost:~#

However, config_controller will try to pull them from internet (a couple of failures from /var/log/puppet/latest/puppet.log):

2019-01-30T20:35:02.282 ^[[mNotice: 2019-01-30 20:35:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:      [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.12.3: output: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/images/create?fromImage=k8s.gcr.io%2Fkube-apiserver&tag=v1.12.3: EOF^[[0m
2019-01-30T20:35:02.287 ^[[mNotice: 2019-01-30 20:35:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:      [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.12.3: output: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/images/create?fromImage=k8s.gcr.io%2Fkube-controller-manager&tag=v1.12.3: EOF^[[0m

I’ll continue debugging above error to confirm if there are issues with the local registry configuration, however,  if we add a couple of extra options, we are able to pass through this issue, on daemon.json, the idea is to put in place an entry on /etc/hosts to redirect the requests to the local registry:


{
    "insecure-registries" : [ "k8s.gcr.io","quay.io","gcr.io" ],
    "registry-mirrors": ["https://registry.localdomain"]
}




And, on /etc/hosts:


192.168.100.60    registry.localdomain k8s.gcr.io quay.io http://gcr.io


Above configuration works to get a successful  config_controller setup. However, we need to update both files during config_controller execution because both are replaced. 



After controller-0 is unlocked, trying to do a system application_apply fails with errors similar to this one:

Pull image "gcr.io/kubernetes-helm/tiller at sha256:022ce9d4a99603be1d30a4ca96a7fa57a45e6f2ef11172f4333c18aaae407f5b" failed: rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/kubernetes-helm/tiller at sha256:022ce9d4a99603be1d30a4ca96a7fa57a45e6f2ef11172f4333c18aaae407f5b not found

Image is on controller-0:

 gcr.io/kubernetes-helm/tiller                                      v2.12.1                                    def589f37fc4        6 weeks ago         81.4MB


Further investigation will be performed to check about registry, however, if you have any advice or thoughts about using a registry-mirror will be appreciated.

Regards,

Cristopher Lemus