[Starlingx-discuss] [Containers] Unable to pull docker image from a worker node
Wang, Yi C
yi.c.wang at intel.com
Wed Jul 24 01:25:55 UTC 2019
Hi Bin,
Worker nodes can access external network through controller nodes. In our deployment, we use a local registry which is accessible via OAM. Worker nodes do can pull images from the local registry. So I suggest you check
1. if your worker nodes can access external network through oam
2. check the docker configuration "/etc/docker/daemon.json" on your worker nodes.
Thanks.
Yi
From: Yang, Bin [mailto:Bin.Yang at windriver.com]
Sent: Wednesday, July 24, 2019 8:21 AM
To: Xie, Cindy <cindy.xie at intel.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node
Hi Cindy,
Thanks for the response. I do locate in China, and I have setup a proxy to overcome the connectivity issue. That was verified to work by installing controllers, and by executing docker pull image on controller nodes.
The issue I am experiencing is that 'a worker node' have no access to oam network but trying to pull docker image while I am trying to deploy openstack helm using 'system application-apply stx-openstack' commands.
Thanks
Best Regards,
Bin Yang, Solution Engineering Team, Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189
Skype: yangbincs993
From: Xie, Cindy [mailto:cindy.xie at intel.com]
Sent: Tuesday, July 23, 2019 10:00 PM
To: Yang, Bin; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node
Hi, Bin
I guess you're located in China and have firewall blocking some docker image, right? You may have to setup your local registry in your lab.
Thx. - cindy
From: Yang, Bin [mailto:Bin.Yang at windriver.com]
Sent: Tuesday, July 23, 2019 8:53 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [Containers] Unable to pull docker image from a worker node
Dear experts,
I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki. Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation.
As I investigate the root cause, I found out that the worker node is not in ready status:
[sysadmin at controller-0 ~(keystone_admin)]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
compute-0 NotReady <none> 3h32m v1.13.5
controller-0 Ready master 14d v1.13.5
controller-1 Ready master 13d v1.13.5
The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy:
[sysadmin at controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods kube-proxy-pftk8
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 38s (x462 over 3h36m) kubelet, compute-0 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
compute-0:~# docker pull k8s.gcr.io/pause:3.1
Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
The traceroute to the docker proxy :
compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://128.224.230.5:9090"
Environment="HTTPS_PROXY=http://128.224.230.5:9090"
Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27"
compute-0:~# traceroute 128.224.230.5
traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets
1 controller-0 (192.168.204.3) 0.200 ms 0.256 ms 0.208 ms
2 * * *
3 * * *
Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue?
Thanks
Best Regards,
Bin Yang, Solution Engineering Team, Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189
Skype: yangbincs993
From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com]
Sent: Tuesday, July 23, 2019 7:49 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722
Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>)
Status: RED
===========================================
Sanity Test is executed in a Containers - Bare Metal Environment
AIO - Simplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
AIO - Duplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
Standard - Dedicated Storage (2+2+2)
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Sanity Test is executed in a Containers - Virtual Environment
AIO - Simplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
AIO - Duplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
Standard - Local Storage (2+2)
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
Standard - External Storage (2+2+2)
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] 24 TCs FAIL
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
-----------------------------------------------------------------------------------
Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241
Regards
Maria G.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190724/3b241c3a/attachment-0001.html>
More information about the Starlingx-discuss
mailing list