[Starlingx-discuss] [Containers] Unable to pull docker image from a worker node

[Yang, Bin]

Dear experts,

I have been trying to install stx milestone3 over virtualbox with standard modes, I managed to install 2 controller nodes and 1 worker node, and provisioned them according to the instructions on wiki.  Then I uploaded openstack helm charts then apply it, then it failed to accomplish that operation.

As I investigate the root cause, I found out that the worker node is not in ready status:


[sysadmin at controller-0 ~(keystone_admin)]$ kubectl get  nodes
NAME           STATUS     ROLES    AGE     VERSION
compute-0      NotReady   <none>   3h32m   v1.13.5
controller-0   Ready      master   14d     v1.13.5
controller-1   Ready      master   13d     v1.13.5

The root cause is that worker node is lacking of access to external network directly, hence unable to pull docker image via the proxy:


[sysadmin at controller-0 ~(keystone_admin)]$ kubectl -n kube-system describe pods  kube-proxy-pftk8



Events:

  Type     Reason                  Age                    From                Message

  ----     ------                  ----                   ----                -------

  Warning  FailedCreatePodSandBox  38s (x462 over 3h36m)  kubelet, compute-0  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)





compute-0:~# docker pull k8s.gcr.io/pause:3.1

Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The traceroute to the docker proxy :

compute-0:~# cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://128.224.230.5:9090"
Environment="HTTPS_PROXY=http://128.224.230.5:9090"
Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.2,192.168.204.3,10.0.2.25,10.0.2.26,192.168.204.4,10.0.2.27"


compute-0:~# traceroute 128.224.230.5
traceroute to 128.224.230.5 (128.224.230.5), 30 hops max, 60 byte packets
1  controller-0 (192.168.204.3)  0.200 ms  0.256 ms  0.208 ms
2  * * *
3  * * *

Can anybody help explain how a worker node without access to external network could pull docker images? How can I workaround this issue?

Thanks

Best Regards,
Bin Yang,    Solution Engineering Team,    Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126    Mobile +86,13811391682    Fax +86,10,64398189
Skype: yangbincs993

From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com]
Sent: Tuesday, July 23, 2019 7:49 AM
To: starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190722

Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jul-22 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190721T233000Z/outputs/iso/>)



Status: RED



===========================================



Sanity Test is executed in a Containers - Bare Metal Environment


AIO - Simplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 61 TCs PASS ]


AIO - Duplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 64 TCs PASS ]



Standard - Dedicated Storage (2+2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] 24 TCs FAIL

Sanity Platform   09 TCs [PASS]



TOTAL: [ 66 TCs PASS ]


===========================================



Sanity Test is executed in a Containers - Virtual Environment


AIO - Simplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 61 TCs PASS ]



AIO - Duplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] 24 TCs FAIL

Sanity Platform   07 TCs [PASS]



TOTAL: [ 64 TCs PASS ]



Standard - Local Storage (2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]



TOTAL: [ 65 TCs PASS ]



Standard - External Storage (2+2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] 24 TCs FAIL

Sanity Platform   08 TCs [PASS]



TOTAL: [ 65 TCs PASS ]



-----------------------------------------------------------------------------------



Create instance from Image or from Volume fails https://bugs.launchpad.net/starlingx/+bug/1837241

Regards
Maria G.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190723/979a29d8/attachment-0001.html>