Hi Tee,

 

I created a new launchpad to track this issue using private registry separately from the proxy issue: https://bugs.launchpad.net/starlingx/+bug/1830826

 

Thanks & Regards,

 

Cristopher Lemus

 

From: "Ngo, Tee" <Tee.Ngo@windriver.com>
Date: Tuesday, May 28, 2019 at 4:31 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Church, Robert" <Robert.Church@windriver.com>, "Chen, Yan" <yan.chen@intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza@intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>
Subject: RE: ansible deployment for duplex/standard

 

Hi Cristopher,

 

You included a link to a LP that is related to docker_proxy. Is this related to docker_proxy or private registry? Can we separate the two issues?

 

Tee

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: May-28-19 5:01 PM
To: Ngo, Tee; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard

 

Hi Tee,

 

Besides the errors on the screenshot where it says that prerequisites are not met, there is no other error.

 

We tried to manually do the apply, but it fails (goes to apply-failed status), there are some errors, this one I think is the first one:

 

2019-05-28 12:43:28.431 106547 ERROR sysinv.conductor.kube_app [-] Received a false positive response from Docker/Armada. Failed to apply application manifest /manifests/platform-integ-apps/1.0-5/platform-integ-apps-manifest.yaml: 2019-05-28 12:37:19.800 42 DEBUG armada.handlers.document [-] Resolving reference /manifests/platform-integ-apps/1.0-5/platform-integ-apps-manifest.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49

 

Further details related to “apply-failed” are on this bug: https://bugs.launchpad.net/starlingx/+bug/1830290

 

Now, about LAG or AE , is this link aggregation? If that;s the case, no, we are not using it. We are directly using eno2(baseboard ethernet) for Management. Attached is the config.ini that we used for our daily sanity, which worked (Green status) until last week on the same physical servers.

 

Thanks & Regards,

 

Cristopher Lemus

 

 

 

 

From: "Ngo, Tee" <Tee.Ngo@windriver.com>
Date: Tuesday, May 28, 2019 at 3:49 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Church, Robert" <Robert.Church@windriver.com>, "Chen, Yan" <yan.chen@intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza@intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>
Subject: RE: ansible deployment for duplex/standard

 

By any chance are you using LAG or AE interface on mgmt? Can I also see the .ini file that you would have used for config_controller --force in this lab?

 

Tee

 

From: Ngo, Tee
Sent: May-28-19 4:11 PM
To: 'Lemus Contreras, Cristopher J'; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: RE: ansible deployment for duplex/standard

 

Any errors in sysinv.log that could explain the stalling of upload?

 

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: May-28-19 4:10 PM
To: Ngo, Tee; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard

 

Here’s the output:

 

[wrsroot@controller-0 ~(keystone_admin)]$ sudo cat /etc/sysinv/sysinv.conf | grep armada

Password:

armada_image_tag=192.168.100.60/airshipit/armada:af8a9ffd0873c2fbc915794e235dbd357f2adab1

 

The system was able to pull the image:

 

[wrsroot@controller-0 ~(keystone_admin)]$ sudo docker images |grep -i armada

192.168.100.60/airshipit/armada                            af8a9ffd0873c2fbc915794e235dbd357f2adab1   7e106423a07d        6 weeks ago         462MB

[wrsroot@controller-0 ~(keystone_admin)]$

 

 

 

 

From: "Ngo, Tee" <Tee.Ngo@windriver.com>
Date: Tuesday, May 28, 2019 at 3:07 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Church, Robert" <Robert.Church@windriver.com>, "Chen, Yan" <yan.chen@intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza@intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>
Subject: RE: ansible deployment for duplex/standard

 

+ Yan and Bob

 

Yan was able to bring up the standard system with the private registry workaround including stx-openstack applied.

 

Can you run the following command?

sudo cat /etc/sysinv/sysinv.conf | grep armada

 

to verify that the it has your private registry in its path?

 

What kind of errors do you see in sysinv.log? Bob, is there anything else we need to check for possible cause?

 

Tee

 

 

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: May-28-19 3:56 PM
To: Ngo, Tee
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard

 

Hi Tee,

 

By applying the workarounds, we were able to move forward with our setup, however, that `platform-integ-apps` is not automatically applied, it remains on “uploaded” status and it breaks the install of secondary nodes:

 

cid:image002.png@01D5156F.6A031E00

 

Basically, the nodes booted to management, but the install fails because `platform-integ-apps` requirement is not met (I assume it needs to be applied).

 

Following the wiki instructions (https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard), is as follows:

 

  1. Execute Setup steps of controller-0
  2. Unlock controller-0
  3. Automatically, ` platform-integ-apps` is uploaded AND applied. Currently, is just on “uploaded” status.
  4. Continue with the install of remaining hosts. Fails because `platform-integ-apps` is not applied.

 

The first time that we found about  `platform-integ-apps` was on May/13, using config_controller it is automatically uploaded and applied since then. Using ansible, it seems that it remains on uploaded status.

 

Thanks & Regards,

 

Cristopher Lemus

 

 

From: "Ngo, Tee" <Tee.Ngo@windriver.com>
Date: Tuesday, May 28, 2019 at 12:17 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza@intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>
Subject: RE: ansible deployment for duplex/standard

 

Let me know how your testing of private registry (without proxy) with the workaround goes J.

 

Tee

 

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: May-27-19 9:30 PM
To: Ngo, Tee
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard

 

Thanks Tee,

 

We’ll take a look at this workaround, and also, to the other one that you sent us in the morning.

 

Regards,

 

Cristopher Lemus

 

From: "Ngo, Tee" <Tee.Ngo@windriver.com>
Date: Monday, May 27, 2019 at 4:57 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza@intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>
Subject: FW: ansible deployment for duplex/standard

 

Cristopher,

 

In case you want to test a private/unified registry, there is an outstanding issue https://bugs.launchpad.net/starlingx/+bug/1830319, the fix is in review https://review.opendev.org/661657. It has been verified by Yan – the LP originator.

 

You can modify the following two files on the host right after the fresh install and before running the bootstrap playbook to move forward. See small changes in the review:

/usr/share/ansible/stx-ansible/playbooks/bootstrap/roles/persist-config/files/populate_initial_config.py

/usr/lib64/python2.7/site-packages/sysinv/common/service_parameter.py

 

Hope it will get merged tomorrow.

 

Tee

From: Chen, Yan [mailto:yan.chen@intel.com]
Sent: May-27-19 2:44 AM
To: Ngo, Tee
Subject: RE: ansible deployment for duplex/standard

 

Yes, this new script works.

 

I tried simplex with 0523 cengn image, with your new python file.

The armada registry in sysinv.conf was right and the applications were uploaded and applied successfully.

 

controller-0:~$ sudo cat /etc/sysinv/sysinv.conf | grep armada

Password:

armada_image_tag=edgehost01.sh.intel.com:5000/airshipit/armada:af8a9ffd0873c2fbc915794e235dbd357f2adab1

 

 

Yan

 

From: Chen, Yan
Sent: Monday, May 27, 2019 12:20
To: 'Ngo, Tee' <Tee.Ngo@windriver.com>
Subject: RE: ansible deployment for duplex/standard

 

Sure, I will have a try.

 

Thanks for your help and have a good sleep!

 

 

Yan

 

From: Ngo, Tee [mailto:Tee.Ngo@windriver.com]
Sent: Monday, May 27, 2019 12:04
To: Chen, Yan <yan.chen@intel.com>
Subject: RE: ansible deployment for duplex/standard
Importance: High

 

Hi Yan,

 

Even if the application apply fails with config_controller --force, can you still try my change for Ansible to see if you get the same result as with config_controller --force? It’s mid-night here so I’m going to bed. Let me know how it goes.

 

Thanks,

Tee

 

From: Ngo, Tee
Sent: May-26-19 11:44 PM
To: 'Chen, Yan'
Subject: RE: ansible deployment for duplex/standard

 

Hi Yan,

 

Right after a fresh install of controller-0, please replace the following file on the host at /usr/share/ansible/stx-ansible/playbooks/bootstrap/roles/persist-config/files/populate_initial_config.py with the version attached to this email and let me know if it works for you using Ansible bootstrap?.

 

If you are testing in vbox environment, take a snapshot right after the fresh install of controller-0 so you can switch back to it.

 

Thanks,

Tee

 

From: Chen, Yan [mailto:yan.chen@intel.com]
Sent: May-26-19 10:02 PM
To: Ngo, Tee
Subject: RE: ansible deployment for duplex/standard

 

Hi, Tee

 

Sorry for late reply.

 

Attached is the result from “config_controller --force".

 

But on this deployment I still met some issue after the application-upload:

| platform-integ-apps | 1.0-5                       | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |

I will try another deployment and if it can pass I will send you again.

 

Sysinv.log error like this:

 

2019-05-24 08:59:20.609 47 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.

2019-05-24 08:59:20.609 47 ERROR armada.cli Traceback (most recent call last):

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 37, in _load_yaml_file

2019-05-24 08:59:20.609 47 ERROR armada.cli     with open(doc) as f:

2019-05-24 08:59:20.609 47 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/helm-toolkit-helm-toolkit.yaml'

2019-05-24 08:59:20.609 47 ERROR armada.cli

2019-05-24 08:59:20.609 47 ERROR armada.cli During handling of the above exception, another exception occurred:

2019-05-24 08:59:20.609 47 ERROR armada.cli

2019-05-24 08:59:20.609 47 ERROR armada.cli Traceback (most recent call last):

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 39, in safe_invoke

2019-05-24 08:59:20.609 47 ERROR armada.cli     self.invoke()

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 209, in invoke

2019-05-24 08:59:20.609 47 ERROR armada.cli     resp = self.handle(documents, tiller)

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 63, in func_wrapper

2019-05-24 08:59:20.609 47 ERROR armada.cli     return future.result()

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result

2019-05-24 08:59:20.609 47 ERROR armada.cli     return self.__get_result()

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result

2019-05-24 08:59:20.609 47 ERROR armada.cli     raise self._exception

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run

2019-05-24 08:59:20.609 47 ERROR armada.cli     result = self.fn(*self.args, **self.kwargs)

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 251, in handle

2019-05-24 08:59:20.609 47 ERROR armada.cli     target_manifest=self.target_manifest)

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 85, in __init__

2019-05-24 08:59:20.609 47 ERROR armada.cli     values=values).update_manifests()

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 161, in update_manifests

2019-05-24 08:59:20.609 47 ERROR armada.cli     merging_values = self._load_yaml_file(value)

2019-05-24 08:59:20.609 47 ERROR armada.cli   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 40, in _load_yaml_file

2019-05-24 08:59:20.609 47 ERROR armada.cli     raise override_exceptions.InvalidOverrideFileException(doc)

2019-05-24 08:59:20.609 47 ERROR armada.cli armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.

2019-05-24 08:59:20.609 47 ERROR armada.cli

 

 

Yan

 

From: Ngo, Tee [mailto:Tee.Ngo@windriver.com]
Sent: Saturday, May 25, 2019 06:10
To: Chen, Yan <yan.chen@intel.com>
Subject: RE: ansible deployment for duplex/standard
Importance: High

 

Hi,

 

Can you provide the output of system service-parameter-list using config_controller --force?

 

Thanks,

Tee

 

From: Chen, Yan [mailto:yan.chen@intel.com]
Sent: May-24-19 1:40 AM
To: Chen, Yan; Poncea, Ovidiu; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] ansible deployment for duplex/standard

 

I tried to config oam/mgmt. for duplex/standard after ansible, and the controller1 and other nodes can be installed now.

 

But we still met issue when doing “system application-upload” because we have to use private registry in SH, but private registry is not successfully configed by ansible, please check this LP:

https://bugs.launchpad.net/starlingx/+bug/1830319

 

Thanks!

 

Yan

 

From: Chen, Yan [mailto:yan.chen@intel.com]
Sent: Friday, May 24, 2019 10:32
To: Poncea, Ovidiu <Ovidiu.Poncea@windriver.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] ansible deployment for duplex/standard

 

Thanks!

I will try.

 

 

Yan

 

From: Poncea, Ovidiu [mailto:Ovidiu.Poncea@windriver.com]
Sent: Thursday, May 23, 2019 18:21
To: Chen, Yan <yan.chen@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: ansible deployment for duplex/standard

 

On controller-0 you need to delete the loopback interface and then configure the other nets on ctrl-0 with something like:

 

system host-if-modify controller-0 lo -c none
system host-if-modify controller-0 enp0s3 -c platform --networks oam
system host-if-modify controller-0 enp0s8 -c platform --networks mgmt
system host-if-modify controller-0 enp0s8 -c platform --networks cluster-host

system host-addr-list controller-0

 

The rest of the process is identical on the other nodes.

 

Ovidiu

From: Chen, Yan [yan.chen@intel.com]
Sent: Wednesday, May 22, 2019 10:12 AM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] ansible deployment for duplex/standard

Hi

 

I wonder if anyone can help me on the ansible deployment for duplex/standard.

 

I installed ISO (about 0519) on vms and run ansible with my yml file as below, but the 2nd controller failed to be installed via pxe.

After checked controller0, I found that mgmt network is configured to lo.

But use “config_controller --force" everything’s ok.

 

Am I doing anything wrong with the yml file?

There’s no instruction on WIKI for duplex + ansible yet. (?)

 

---

system_mode: duplex

 

management_subnet: 192.178.204.0/28

 

external_oam_subnet: 10.10.10.0/24

external_oam_gateway_address: 10.10.10.1

external_oam_floating_address: 10.10.10.2

external_oam_node_0_address: 10.10.10.3

external_oam_node_1_address: 10.10.10.4

 

 

 

 

Yan