[Starlingx-discuss] [Containers] Unable to pull docker image from a worker node
Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi Bin, Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check 1. if your worker nodes can access external network through oam 2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes. Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 8:21 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi Yi, Thanks for the response. I did test if worker node could access to oam network by ping the docker proxy ip, and the result turned out that worker node cannot reach to that ip. I did the same test over controllers and the controllers could reach to that ip. So I guess this is not an issue of docker settings, it might be something wrong with controllers if controllers are deemed to NAT the traffic from worker nodes to OAM network. Any suggestion on how to check that? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 24, 2019 9:26 AM To: Yang, Bin; Xie, Cindy; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Bin, Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check 1. if your worker nodes can access external network through oam 2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes. Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 8:21 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
And could you kindly educate me how to setup a local docker register over oam network? The network latency results a lot of issue during my installation of stx controller nodes. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Yang, Bin Sent: Wednesday, July 24, 2019 9:46 AM To: 'Wang, Yi C'; Xie, Cindy; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Yi, Thanks for the response. I did test if worker node could access to oam network by ping the docker proxy ip, and the result turned out that worker node cannot reach to that ip. I did the same test over controllers and the controllers could reach to that ip. So I guess this is not an issue of docker settings, it might be something wrong with controllers if controllers are deemed to NAT the traffic from worker nodes to OAM network. Any suggestion on how to check that? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 24, 2019 9:26 AM To: Yang, Bin; Xie, Cindy; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Bin, Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check 1. if your worker nodes can access external network through oam 2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes. Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 8:21 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi Bin, I can't ping my local registry too. I didn't check iptables rules of controller nodes. But I guess icmp was blocked by some rules. I can connect to external hosts by ssh from my worker nodes. You can refer to below link on local registry deploy. https://docs.docker.com/registry/deploying/ Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 9:47 AM To: Wang, Yi C <yi.c.wang@intel.com>; Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node And could you kindly educate me how to setup a local docker register over oam network? The network latency results a lot of issue during my installation of stx controller nodes. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Yang, Bin Sent: Wednesday, July 24, 2019 9:46 AM To: 'Wang, Yi C'; Xie, Cindy; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Yi, Thanks for the response. I did test if worker node could access to oam network by ping the docker proxy ip, and the result turned out that worker node cannot reach to that ip. I did the same test over controllers and the controllers could reach to that ip. So I guess this is not an issue of docker settings, it might be something wrong with controllers if controllers are deemed to NAT the traffic from worker nodes to OAM network. Any suggestion on how to check that? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 24, 2019 9:26 AM To: Yang, Bin; Xie, Cindy; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Bin, Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check 1. if your worker nodes can access external network through oam 2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes. Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 8:21 AM To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
Hi Yi, Ok, then perhaps I need to test it in other ways. Thanks for your help Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 24, 2019 10:08 AM To: Yang, Bin; Xie, Cindy; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Bin, I can't ping my local registry too. I didn't check iptables rules of controller nodes. But I guess icmp was blocked by some rules. I can connect to external hosts by ssh from my worker nodes. You can refer to below link on local registry deploy. https://docs.docker.com/registry/deploying/ Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 9:47 AM To: Wang, Yi C <yi.c.wang@intel.com>; Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node And could you kindly educate me how to setup a local docker register over oam network? The network latency results a lot of issue during my installation of stx controller nodes. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Yang, Bin Sent: Wednesday, July 24, 2019 9:46 AM To: 'Wang, Yi C'; Xie, Cindy; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Yi, Thanks for the response. I did test if worker node could access to oam network by ping the docker proxy ip, and the result turned out that worker node cannot reach to that ip. I did the same test over controllers and the controllers could reach to that ip. So I guess this is not an issue of docker settings, it might be something wrong with controllers if controllers are deemed to NAT the traffic from worker nodes to OAM network. Any suggestion on how to check that? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 24, 2019 9:26 AM To: Yang, Bin; Xie, Cindy; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Bin, Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check 1. if your worker nodes can access external network through oam 2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes. Thanks. Yi From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Wednesday, July 24, 2019 8:21 AM To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi Cindy, Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes. The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, July 23, 2019 10:00 PM To: Yang, Bin; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Hi, Bin I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab. Thx. - cindy From: Yang, Bin [mailto:Bin.Yang@windriver.com] Sent: Tuesday, July 23, 2019 8:53 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node Dear experts, I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation. As I investigate the root cause, I found out that the worker node is not in ready status: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes NAME STATUS ROLES AGE VERSION compute-0 NotReady <none> 3h32m v1.13.5 controller-0 Ready master 14d v1.13.5 controller-1 Ready master 13d v1.13.5 The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy: [sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) compute-0:~# docker pull k8s.gcr.io/pause:3.1 Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) The traceroute to the docker proxy : compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://128.224.230.5:9090" Environment="HTTPS_PROXY=http://128.224.230.5:9090" Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27" compute-0:~# traceroute 128.224.230.5 traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets 1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms 2 * * * 3 * * * Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue? Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Tuesday, July 23, 2019 7:49 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs PASS ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs PASS ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] Standard - External Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs PASS ] ----------------------------------------------------------------------------------- Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241 Regards Maria G.
participants (3)
-
Wang, Yi C
-
Xie, Cindy
-
Yang, Bin