[Starlingx-discuss] controller-1 failed after unlock

Yang, Bin Bin.Yang at windriver.com
Thu Jul 18 01:15:11 UTC 2019


Hi Bin,

Forget my previous question, I think I understand the root cause: the oam network was not provisioned appropriately. I will figure out how to fix that.

Thanks

Best Regards,
Bin Yang,    Solution Engineering Team,    Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126    Mobile +86,13811391682    Fax +86,10,64398189
Skype: yangbincs993


-----Original Message-----
From: Yang, Bin 
Sent: Thursday, July 18, 2019 9:13 AM
To: 'Yang, Bin'
Cc: starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] controller-1 failed after unlock

Hi Bin,

Thanks a lot for the help. Indeed there is error there, it seems a timeout issue to pull docker images (again, I have met such issue during ansible-playbook on controller-0, the workaround is to increase the timeout in ansible playbook),  any suggestion to workaround this issue? Thanks


2019-07-16T11:07:02.080 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.085 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.091 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.096 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.101 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.105 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.109 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.113 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.117 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.121 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.125 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns:     [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.130 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.134 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`ESC[0m
2019-07-16T11:07:02.138 ESC[1;31mError: 2019-07-16 11:07:02 +0000 kubeadm init --config=/etc/kubernetes/kubeadm.yaml returned 1 instead of one of [0]

BTW, It is very nice to talk to you, we are both 'Yang Bin' :) 

Best Regards,
Bin Yang,    Solution Engineering Team,    Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126    Mobile +86,13811391682    Fax +86,10,64398189
Skype: yangbincs993

-----Original Message-----
From: Yang, Bin [mailto:bin.yang at intel.com] 
Sent: Wednesday, July 17, 2019 9:28 PM
To: Yang, Bin
Cc: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] controller-1 failed after unlock

Hi Bin,
	This file is generated by kubeadm. And kubeadm is executed by puppet. 
	Normally, you should find below log in /var/log/puppet/:
		Executing: 'kubeadm init --config=/etc/kubernetes/kubeadm.yaml'
		...
		/Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
		...

	If kubeadm does not run properly, you should find some error log.

thanks,
Bin

On Wed, Jul 17, 2019 at 09:11:16AM +0000, Yang, Bin wrote:
>    Dear StarlingX experts,
> 
>     
> 
>    I am exercising installing starlingx (with latest milestone3  image) with
>    virtualbox following the wiki:
>     https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard
> 
>     
> 
>    I am able to install/provision controller-0, compute-0, but failed to
>    unlock controller-1, the issue seems to be kubelet.service cannot start
>    due to missing of config file at /var/lib/kubelet/config.yaml :
> 
>     
> 
>    controller-1:~# ls /var/lib/kubelet/ -al
> 
>    total 8
> 
>    drwxr-xr-x.  2 root root 4096 Jun 21 02:27 .
> 
>    drwxr-xr-x. 60 root root 4096 Jul 16 10:43 ..
> 
>     
> 
>     
> 
>    Could anyone who have insight of this issue shed some lights on this issue
>    ?  Thanks in advance
> 
>     
> 
>    Below is the daemon log:
> 
>     
> 
>    2019-07-17T02:06:13.623 controller-1 systemd[1]: info kubelet.service
>    holdoff time over, scheduling restart.
> 
>    2019-07-17T02:06:13.623 controller-1 systemd[1]: warning Cannot add
>    dependency job for unit dev-hugepages.mount, ignoring: Unit is masked.
> 
>    2019-07-17T02:06:13.641 controller-1 systemd[1]: info Stopped Kubernetes
>    Kubelet Server.
> 
>    2019-07-17T02:06:13.647 controller-1 systemd[1]: info Starting Kubernetes
>    Kubelet Server...
> 
>    2019-07-17T02:06:13.000 controller-1 root: info
>    /usr/bin/kubelet-cgroup-setup.sh(481470): Nothing to do, already
>    configured: /sys/fs/cgroup/cpuset/k8s-infra.
> 
>    2019-07-17T02:06:13.664 controller-1 systemd[1]: info Started Kubernetes
>    Kubelet Server.
> 
>    2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag
>    --feature-gates has been deprecated, This parameter should be set via the
>    config file specified by the Kubelet's --config flag. See
>    https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
>    for more information.
> 
>    2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag
>    --cpu-manager-policy has been deprecated, This parameter should be set via
>    the config file specified by the Kubelet's --config flag. See
>    https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
>    for more information.
> 
>    2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info F0717
>    02:06:13.831961  481475 server.go:189] failed to load Kubelet config file
>    /var/lib/kubelet/config.yaml, error failed to read kubelet config file
>    "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml:
>    no such file or directory
> 
>    2019-07-17T02:06:13.837 controller-1 systemd[1]: notice kubelet.service:
>    main process exited, code=exited, status=255/n/a
> 
>    2019-07-17T02:06:13.847 controller-1 systemd[1]: notice Unit
>    kubelet.service entered failed state.
> 
>    2019-07-17T02:06:13.847 controller-1 systemd[1]: warning kubelet.service
>    failed.
> 
>     
> 
>     
> 
>     
> 
>     
> 
>    Best Regards,
> 
>    Bin Yang
> 
>     

> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss




More information about the Starlingx-discuss mailing list