[Starlingx-discuss] [Containers] LP 1812519: config_controller --kubernetes fails at step 06
Erich: You indicated you saw this failure when using an ISO from Jan 17: https://bugs.launchpad.net/starlingx/+bug/1812519 Al cannot reproduce this. The one difference between Al's environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation 2. Mingyuan can you tell us if you are able to get config_controller -kubernetes to succeed with a load from Jan 17th or later? Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not. Frank
My latest try is built from last Wednesday's code. I'll try the latest code today along with the proxy patch. Mingyuan From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Tuesday, January 22, 2019 3:39 To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Containers] LP 1812519: config_controller --kubernetes fails at step 06 Erich: You indicated you saw this failure when using an ISO from Jan 17: https://bugs.launchpad.net/starlingx/+bug/1812519 Al cannot reproduce this. The one difference between Al's environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation 2. Mingyuan can you tell us if you are able to get config_controller -kubernetes to succeed with a load from Jan 17th or later? Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not. Frank
On Tue, 2019-01-22 at 09:20 +0800, Qi, Mingyuan wrote:
My latest try is built from last Wednesday’s code. I’ll try the latest code today along with the proxy patch.
Mingyuan
From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Tuesday, January 22, 2019 3:39 To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; 'starlingx-d iscuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
Erich:
You indicated you saw this failure when using an ISO from Jan 17: htt ps://bugs.launchpad.net/starlingx/+bug/1812519
Al cannot reproduce this. The one difference between Al’s environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation
I'm using a libvirt/qemu setup. I wasn't unable to set a NAT network but I define some iptables rules to get internet access from the VM. I set the proxy settings on /etc/environment and /etc/systemd/system/docker.service.d/ This was my initial no_proxy no_proxy=localhost,127.0.0.1,192.168.206.2,172.16.0.0/16,10.96.0.0/12 I started adding some networks there as I saw errors on the puppet.log. At this point I can curl any host on internet, so external networking is working. With this setup I run the config_controller --kubernetes command and I see the behavior described in the initial bug report. Then I tried a different no_proxy value, Jose told me that he as able to pass config_controller on virtualbox using only no_proxy=127.0.0.1. With no_proxy=127.0.0.1 the config_controller --kubernetes succeed but I can't do a 'source /etc/platform/openrc', I get the following errors: controller-0:~$ source /etc/platform/openrc Openstack Admin credentials can only be loaded from the active controller. I tried to force execution of the controller_config script but I get: ***************************************************** ***************************************************** Unable to get IP from host: controller-0 ***************************************************** ***************************************************** Pausing for 5 seconds... and checking /etc/hosts the file is incomplete: # HEADER: This file was autogenerated at 2019-01-21 18:59:58 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 localhost localhost.localdomain controller ::1 controller Anyway, something bad happened even with the config_controller running without errors. Also, I tried the VirtualBox in a non-proxy environment and everything worked. Today I'll try VirtualBox behind a proxy. As my libvirt/qemu setup has internet access I can't think on reason on why it works with virtualbox, but no with my setup. I'll confirm this.
2. Mingyuan can you tell us if you are able to get config_controller –kubernetes to succeed with a load from Jan 17th or later?
Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not.
Frank
Erich, Current docker version 18.03.1-ce in starlingx doesn't support wildcard or CIDR notation subnet in no_proxy. Go-lang net lib used by docker supports CIDR notation in this commit[0] which is after docker 18.03.1-ce. Have you tried with a particular ip address in no_proxy? [0] https://github.com/golang/net/blob/c21de06aaf072cea07f3a65d6970e5c7d8b6cd6d/... Mingyuan -----Original Message----- From: Cordoba Malibran, Erich Sent: Tuesday, January 22, 2019 23:53 To: Frank.Miller@windriver.com; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Al.Bailey@windriver.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Containers] LP 1812519: config_controller --kubernetes fails at step 06 On Tue, 2019-01-22 at 09:20 +0800, Qi, Mingyuan wrote:
My latest try is built from last Wednesday’s code. I’ll try the latest code today along with the proxy patch.
Mingyuan
From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Tuesday, January 22, 2019 3:39 To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; 'starlingx-d iscuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
Erich:
You indicated you saw this failure when using an ISO from Jan 17: htt ps://bugs.launchpad.net/starlingx/+bug/1812519
Al cannot reproduce this. The one difference between Al’s environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation
I'm using a libvirt/qemu setup. I wasn't unable to set a NAT network but I define some iptables rules to get internet access from the VM. I set the proxy settings on /etc/environment and /etc/systemd/system/docker.service.d/ This was my initial no_proxy no_proxy=localhost,127.0.0.1,192.168.206.2,172.16.0.0/16,10.96.0.0/12 I started adding some networks there as I saw errors on the puppet.log. At this point I can curl any host on internet, so external networking is working. With this setup I run the config_controller --kubernetes command and I see the behavior described in the initial bug report. Then I tried a different no_proxy value, Jose told me that he as able to pass config_controller on virtualbox using only no_proxy=127.0.0.1. With no_proxy=127.0.0.1 the config_controller --kubernetes succeed but I can't do a 'source /etc/platform/openrc', I get the following errors: controller-0:~$ source /etc/platform/openrc Openstack Admin credentials can only be loaded from the active controller. I tried to force execution of the controller_config script but I get: ***************************************************** ***************************************************** Unable to get IP from host: controller-0 ***************************************************** ***************************************************** Pausing for 5 seconds... and checking /etc/hosts the file is incomplete: # HEADER: This file was autogenerated at 2019-01-21 18:59:58 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 localhost localhost.localdomain controller ::1 controller Anyway, something bad happened even with the config_controller running without errors. Also, I tried the VirtualBox in a non-proxy environment and everything worked. Today I'll try VirtualBox behind a proxy. As my libvirt/qemu setup has internet access I can't think on reason on why it works with virtualbox, but no with my setup. I'll confirm this.
2. Mingyuan can you tell us if you are able to get config_controller –kubernetes to succeed with a load from Jan 17th or later?
Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not.
Frank
Hi Mingyuan, My issue is solved now, I didn't use a NAT network and that caused the problem. Regarding the no_proxy, did you were able to pass the 'system application-apply stx-openstack' step? I'm seeing a problem on tiller trying to get install osh-openstack- mariadb, which I think is related to not having a proper no_proxy value. Is there a way to get all the IP needed to configure no_proxy, so the gRPC (and other mechanisms) can connect each other in the k8s cluster? Thanks -Erich On Wed, 2019-01-23 at 09:30 +0800, Qi, Mingyuan wrote:
Erich,
Current docker version 18.03.1-ce in starlingx doesn't support wildcard or CIDR notation subnet in no_proxy. Go-lang net lib used by docker supports CIDR notation in this commit[0] which is after docker 18.03.1-ce. Have you tried with a particular ip address in no_proxy?
[0] https://github.com/golang/net/blob/c21de06aaf072cea07f3a65d6970e5 c7d8b6cd6d/http/httpproxy/proxy.go#L39-L52
Mingyuan
-----Original Message----- From: Cordoba Malibran, Erich Sent: Tuesday, January 22, 2019 23:53 To: Frank.Miller@windriver.com; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Al.Bailey@windriver.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
On Tue, 2019-01-22 at 09:20 +0800, Qi, Mingyuan wrote:
My latest try is built from last Wednesday’s code. I’ll try the latest code today along with the proxy patch.
Mingyuan
From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Tuesday, January 22, 2019 3:39 To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; 'starlingx-d iscuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
Erich:
You indicated you saw this failure when using an ISO from Jan 17: htt ps://bugs.launchpad.net/starlingx/+bug/1812519
Al cannot reproduce this. The one difference between Al’s environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation
I'm using a libvirt/qemu setup. I wasn't unable to set a NAT network but I define some iptables rules to get internet access from the VM.
I set the proxy settings on /etc/environment and /etc/systemd/system/docker.service.d/
This was my initial no_proxy no_proxy=localhost,127.0.0.1,192.168.206.2,172.16.0.0/16,10.96.0.0/12
I started adding some networks there as I saw errors on the puppet.log.
At this point I can curl any host on internet, so external networking is working.
With this setup I run the config_controller --kubernetes command and I see the behavior described in the initial bug report.
Then I tried a different no_proxy value, Jose told me that he as able to pass config_controller on virtualbox using only no_proxy=127.0.0.1.
With no_proxy=127.0.0.1 the config_controller --kubernetes succeed but I can't do a 'source /etc/platform/openrc', I get the following errors:
controller-0:~$ source /etc/platform/openrc Openstack Admin credentials can only be loaded from the active controller.
I tried to force execution of the controller_config script but I get:
***************************************************** ***************************************************** Unable to get IP from host: controller-0 ***************************************************** ***************************************************** Pausing for 5 seconds...
and checking /etc/hosts the file is incomplete:
# HEADER: This file was autogenerated at 2019-01-21 18:59:58 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 localhost localhost.localdomain controller ::1 controller
Anyway, something bad happened even with the config_controller running without errors.
Also, I tried the VirtualBox in a non-proxy environment and everything worked. Today I'll try VirtualBox behind a proxy.
As my libvirt/qemu setup has internet access I can't think on reason on why it works with virtualbox, but no with my setup. I'll confirm this.
2. Mingyuan can you tell us if you are able to get config_controller –kubernetes to succeed with a load from Jan 17th or later?
Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not.
Frank
Erich, They are separate issues. 1. For system application-apply, I've finished the apply and openstack are running good. No need to set proxy to the system env. All images are pulled by docker through docker proxy setting. So just watch out for the golang net lib limitation I've mentioned. 2. For the system proxy config, centos supports CIDR notation in no_proxy. But which step do you need to set system proxy config? I do encountered an issue during sysinv interaction because httplib2(which get system env proxy settings) used by cgts-client doesn't support CIDR notation :( , which means you still have to set a specific ip address in system no_proxy. https://github.com/httplib2/httplib2/blob/master/python2/httplib2/__init__.p... Fortunately, it doesn't block config_controller and application-apply. Mingyuan -----Original Message----- From: Cordoba Malibran, Erich Sent: Thursday, January 24, 2019 1:30 To: Frank.Miller@windriver.com; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Al.Bailey@windriver.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Containers] LP 1812519: config_controller --kubernetes fails at step 06 Hi Mingyuan, My issue is solved now, I didn't use a NAT network and that caused the problem. Regarding the no_proxy, did you were able to pass the 'system application-apply stx-openstack' step? I'm seeing a problem on tiller trying to get install osh-openstack- mariadb, which I think is related to not having a proper no_proxy value. Is there a way to get all the IP needed to configure no_proxy, so the gRPC (and other mechanisms) can connect each other in the k8s cluster? Thanks -Erich On Wed, 2019-01-23 at 09:30 +0800, Qi, Mingyuan wrote:
Erich,
Current docker version 18.03.1-ce in starlingx doesn't support wildcard or CIDR notation subnet in no_proxy. Go-lang net lib used by docker supports CIDR notation in this commit[0] which is after docker 18.03.1-ce. Have you tried with a particular ip address in no_proxy?
[0] https://github.com/golang/net/blob/c21de06aaf072cea07f3a65d6970e5 c7d8b6cd6d/http/httpproxy/proxy.go#L39-L52
Mingyuan
-----Original Message----- From: Cordoba Malibran, Erich Sent: Tuesday, January 22, 2019 23:53 To: Frank.Miller@windriver.com; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Al.Bailey@windriver.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
On Tue, 2019-01-22 at 09:20 +0800, Qi, Mingyuan wrote:
My latest try is built from last Wednesday’s code. I’ll try the latest code today along with the proxy patch.
Mingyuan
From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Tuesday, January 22, 2019 3:39 To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Qi, Mingyuan <mingyuan.qi@intel.com> Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; 'starlingx-d iscuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Containers] LP 1812519: config_controller --kubernetes fails at step 06
Erich:
You indicated you saw this failure when using an ISO from Jan 17: htt ps://bugs.launchpad.net/starlingx/+bug/1812519
Al cannot reproduce this. The one difference between Al’s environment and yours is Al does not use a proxy. Mingyuan used a proxy a few weeks ago and this worked for him and he added steps to use a proxy to the wiki. I have 2 questions: 1. Erich can you explain any changes you had to make in your environment that are not listed on the current wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/Installation
I'm using a libvirt/qemu setup. I wasn't unable to set a NAT network but I define some iptables rules to get internet access from the VM.
I set the proxy settings on /etc/environment and /etc/systemd/system/docker.service.d/
This was my initial no_proxy no_proxy=localhost,127.0.0.1,192.168.206.2,172.16.0.0/16,10.96.0.0/12
I started adding some networks there as I saw errors on the puppet.log.
At this point I can curl any host on internet, so external networking is working.
With this setup I run the config_controller --kubernetes command and I see the behavior described in the initial bug report.
Then I tried a different no_proxy value, Jose told me that he as able to pass config_controller on virtualbox using only no_proxy=127.0.0.1.
With no_proxy=127.0.0.1 the config_controller --kubernetes succeed but I can't do a 'source /etc/platform/openrc', I get the following errors:
controller-0:~$ source /etc/platform/openrc Openstack Admin credentials can only be loaded from the active controller.
I tried to force execution of the controller_config script but I get:
***************************************************** ***************************************************** Unable to get IP from host: controller-0 ***************************************************** ***************************************************** Pausing for 5 seconds...
and checking /etc/hosts the file is incomplete:
# HEADER: This file was autogenerated at 2019-01-21 18:59:58 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 localhost localhost.localdomain controller ::1 controller
Anyway, something bad happened even with the config_controller running without errors.
Also, I tried the VirtualBox in a non-proxy environment and everything worked. Today I'll try VirtualBox behind a proxy.
As my libvirt/qemu setup has internet access I can't think on reason on why it works with virtualbox, but no with my setup. I'll confirm this.
2. Mingyuan can you tell us if you are able to get config_controller –kubernetes to succeed with a load from Jan 17th or later?
Hopefully answers to the above 2 questions will point to why you are seeing failures while others are not.
Frank
participants (3)
-
Cordoba Malibran, Erich
-
Miller, Frank
-
Qi, Mingyuan