Hi Cristopher, The story 2004711[0] is to resolve these issues if deployment with poor network condition or without internet access. [0] https://storyboard.openstack.org/#!/story/2004711 Mingyuan -----Original Message----- From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com] Sent: Thursday, January 31, 2019 12:41 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Deployment status Hello All, ** Baremetal Environment ** Simplex configuration on BareMetal servers without internet access. SUMMARY: • config_controller is over-writing /etc/hosts and /etc/docker/daemon.json, which, depending of the user environment, might need special settings. • config_controller is not pulling images from local registry, using a local/mirror registry is a viable option? • Errors during system application-apply related to docker images not being pulled. During the install, on servers that doesn’t have internet connection, config_controller will fail because it will try to pull docker images, which is an expected behavior. Docker offers a solution that is basically defining a local registry (mirror registry) from where the infrastructure can pull the images. In our environment we have http://registry.localdomain defined as a registry, /etc/docker/daemon.json can be defined like this: { "registry-mirrors": ["https://registry.localdomain"] } With this method, we can pull all docker images without issues: • One from http://quay.io localhost:~# docker pull calico/node:v3.1.4 v3.1.4: Pulling from calico/node c67f3896b22c: Pull complete 1386084f19d9: Pull complete 3a59f3ebe6d0: Pull complete 0fc581a09289: Pull complete Digest: sha256:87dd10c421dff39c5546c888ef0905c1b8e9bf7d7c8c4668f5dc3842753bdab9 Status: Downloaded newer image for calico/node:v3.1.4 localhost:~# • One from gcr.io/kubernetes-helm/tiller localhost:~# docker pull kubernetes-helm/tiller:v2.12.1 v2.12.1: Pulling from kubernetes-helm/tiller c67f3896b22c: Already exists de5dfb1eb274: Pull complete d144c197b919: Pull complete a8ecec0345d6: Pull complete Digest: sha256:3060d5de6614e96adf540c10528929a056b1a7751e781beff4cc00d2e47e114c Status: Downloaded newer image for kubernetes-helm/tiller:v2.12.1 localhost:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kubernetes-helm/tiller v2.12.1 def589f37fc4 5 weeks ago 81.4MB calico/node v3.1.4 715f56d3ab67 2 months ago 241MB localhost:~# However, config_controller will try to pull them from internet (a couple of failures from /var/log/puppet/latest/puppet.log): 2019-01-30T20:35:02.282 ^[[mNotice: 2019-01-30 20:35:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.12.3: output: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/images/create?fromImage=k8s.gcr.io%2Fkube-apiserver&tag=v1.12.3: EOF^[[0m 2019-01-30T20:35:02.287 ^[[mNotice: 2019-01-30 20:35:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.12.3: output: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/images/create?fromImage=k8s.gcr.io%2Fkube-controller-manager&tag=v1.12.3: EOF^[[0m I’ll continue debugging above error to confirm if there are issues with the local registry configuration, however, if we add a couple of extra options, we are able to pass through this issue, on daemon.json, the idea is to put in place an entry on /etc/hosts to redirect the requests to the local registry: { "insecure-registries" : [ "k8s.gcr.io","quay.io","gcr.io" ], "registry-mirrors": ["https://registry.localdomain"] } And, on /etc/hosts: 192.168.100.60 registry.localdomain k8s.gcr.io quay.io http://gcr.io Above configuration works to get a successful config_controller setup. However, we need to update both files during config_controller execution because both are replaced. After controller-0 is unlocked, trying to do a system application_apply fails with errors similar to this one: Pull image "gcr.io/kubernetes-helm/tiller@sha256:022ce9d4a99603be1d30a4ca96a7fa57a45e6f2ef11172f4333c18aaae407f5b" failed: rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/kubernetes-helm/tiller@sha256:022ce9d4a99603be1d30a4ca96a7fa57a45e6f2ef11172f4333c18aaae407f5b not found Image is on controller-0: gcr.io/kubernetes-helm/tiller v2.12.1 def589f37fc4 6 weeks ago 81.4MB Further investigation will be performed to check about registry, however, if you have any advice or thoughts about using a registry-mirror will be appreciated. Regards, Cristopher Lemus _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss