[Starlingx-discuss] ansible deployment for duplex/standard
Lemus Contreras, Cristopher J
cristopher.j.lemus.contreras at intel.com
Wed May 29 03:23:06 UTC 2019
Hi Tee,
I created a new launchpad to track this issue using private registry separately from the proxy issue: https://bugs.launchpad.net/starlingx/+bug/1830826
Thanks & Regards,
Cristopher Lemus
From: "Ngo, Tee" <Tee.Ngo at windriver.com>
Date: Tuesday, May 28, 2019 at 4:31 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, "Church, Robert" <Robert.Church at windriver.com>, "Chen, Yan" <yan.chen at intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza at intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>
Subject: RE: ansible deployment for duplex/standard
Hi Cristopher,
You included a link to a LP that is related to docker_proxy. Is this related to docker_proxy or private registry? Can we separate the two issues?
Tee
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: May-28-19 5:01 PM
To: Ngo, Tee; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard
Hi Tee,
Besides the errors on the screenshot where it says that prerequisites are not met, there is no other error.
We tried to manually do the apply, but it fails (goes to apply-failed status), there are some errors, this one I think is the first one:
2019-05-28 12:43:28.431 106547 ERROR sysinv.conductor.kube_app [-] Received a false positive response from Docker/Armada. Failed to apply application manifest /manifests/platform-integ-apps/1.0-5/platform-integ-apps-manifest.yaml: 2019-05-28 12:37:19.800 42 DEBUG armada.handlers.document [-] Resolving reference /manifests/platform-integ-apps/1.0-5/platform-integ-apps-manifest.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49
Further details related to “apply-failed” are on this bug: https://bugs.launchpad.net/starlingx/+bug/1830290
Now, about LAG or AE , is this link aggregation? If that;s the case, no, we are not using it. We are directly using eno2(baseboard ethernet) for Management. Attached is the config.ini that we used for our daily sanity, which worked (Green status) until last week on the same physical servers.
Thanks & Regards,
Cristopher Lemus
From: "Ngo, Tee" <Tee.Ngo at windriver.com>
Date: Tuesday, May 28, 2019 at 3:49 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, "Church, Robert" <Robert.Church at windriver.com>, "Chen, Yan" <yan.chen at intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza at intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>
Subject: RE: ansible deployment for duplex/standard
By any chance are you using LAG or AE interface on mgmt? Can I also see the .ini file that you would have used for config_controller --force in this lab?
Tee
From: Ngo, Tee
Sent: May-28-19 4:11 PM
To: 'Lemus Contreras, Cristopher J'; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: RE: ansible deployment for duplex/standard
Any errors in sysinv.log that could explain the stalling of upload?
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: May-28-19 4:10 PM
To: Ngo, Tee; Church, Robert; Chen, Yan
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard
Here’s the output:
[wrsroot at controller-0 ~(keystone_admin)]$ sudo cat /etc/sysinv/sysinv.conf | grep armada
Password:
armada_image_tag=192.168.100.60/airshipit/armada:af8a9ffd0873c2fbc915794e235dbd357f2adab1
The system was able to pull the image:
[wrsroot at controller-0 ~(keystone_admin)]$ sudo docker images |grep -i armada
192.168.100.60/airshipit/armada af8a9ffd0873c2fbc915794e235dbd357f2adab1 7e106423a07d 6 weeks ago 462MB
[wrsroot at controller-0 ~(keystone_admin)]$
From: "Ngo, Tee" <Tee.Ngo at windriver.com>
Date: Tuesday, May 28, 2019 at 3:07 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, "Church, Robert" <Robert.Church at windriver.com>, "Chen, Yan" <yan.chen at intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza at intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>
Subject: RE: ansible deployment for duplex/standard
+ Yan and Bob
Yan was able to bring up the standard system with the private registry workaround including stx-openstack applied.
Can you run the following command?
sudo cat /etc/sysinv/sysinv.conf | grep armada
to verify that the it has your private registry in its path?
What kind of errors do you see in sysinv.log? Bob, is there anything else we need to check for possible cause?
Tee
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: May-28-19 3:56 PM
To: Ngo, Tee
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard
Hi Tee,
By applying the workarounds, we were able to move forward with our setup, however, that `platform-integ-apps` is not automatically applied, it remains on “uploaded” status and it breaks the install of secondary nodes:
[cid:image002.png at 01D5156F.6A031E00]
Basically, the nodes booted to management, but the install fails because `platform-integ-apps` requirement is not met (I assume it needs to be applied).
Following the wiki instructions (https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard), is as follows:
1. Execute Setup steps of controller-0
2. Unlock controller-0
3. Automatically, ` platform-integ-apps` is uploaded AND applied. Currently, is just on “uploaded” status.
4. Continue with the install of remaining hosts. Fails because `platform-integ-apps` is not applied.
The first time that we found about `platform-integ-apps` was on May/13, using config_controller it is automatically uploaded and applied since then. Using ansible, it seems that it remains on uploaded status.
Thanks & Regards,
Cristopher Lemus
From: "Ngo, Tee" <Tee.Ngo at windriver.com>
Date: Tuesday, May 28, 2019 at 12:17 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza at intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>
Subject: RE: ansible deployment for duplex/standard
Let me know how your testing of private registry (without proxy) with the workaround goes ☺.
Tee
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: May-27-19 9:30 PM
To: Ngo, Tee
Cc: Perez Carranza, Jose; Alonso, Juan Carlos; Cabrales, Ada
Subject: Re: ansible deployment for duplex/standard
Thanks Tee,
We’ll take a look at this workaround, and also, to the other one that you sent us in the morning.
Regards,
Cristopher Lemus
From: "Ngo, Tee" <Tee.Ngo at windriver.com>
Date: Monday, May 27, 2019 at 4:57 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>
Cc: "Perez Carranza, Jose" <jose.perez.carranza at intel.com>, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>
Subject: FW: ansible deployment for duplex/standard
Cristopher,
In case you want to test a private/unified registry, there is an outstanding issue https://bugs.launchpad.net/starlingx/+bug/1830319, the fix is in review https://review.opendev.org/661657. It has been verified by Yan – the LP originator.
You can modify the following two files on the host right after the fresh install and before running the bootstrap playbook to move forward. See small changes in the review:
/usr/share/ansible/stx-ansible/playbooks/bootstrap/roles/persist-config/files/populate_initial_config.py
/usr/lib64/python2.7/site-packages/sysinv/common/service_parameter.py
Hope it will get merged tomorrow.
Tee
From: Chen, Yan [mailto:yan.chen at intel.com]
Sent: May-27-19 2:44 AM
To: Ngo, Tee
Subject: RE: ansible deployment for duplex/standard
Yes, this new script works.
I tried simplex with 0523 cengn image, with your new python file.
The armada registry in sysinv.conf was right and the applications were uploaded and applied successfully.
controller-0:~$ sudo cat /etc/sysinv/sysinv.conf | grep armada
Password:
armada_image_tag=edgehost01.sh.intel.com:5000/airshipit/armada:af8a9ffd0873c2fbc915794e235dbd357f2adab1
Yan
From: Chen, Yan
Sent: Monday, May 27, 2019 12:20
To: 'Ngo, Tee' <Tee.Ngo at windriver.com>
Subject: RE: ansible deployment for duplex/standard
Sure, I will have a try.
Thanks for your help and have a good sleep!
Yan
From: Ngo, Tee [mailto:Tee.Ngo at windriver.com]
Sent: Monday, May 27, 2019 12:04
To: Chen, Yan <yan.chen at intel.com<mailto:yan.chen at intel.com>>
Subject: RE: ansible deployment for duplex/standard
Importance: High
Hi Yan,
Even if the application apply fails with config_controller --force, can you still try my change for Ansible to see if you get the same result as with config_controller --force? It’s mid-night here so I’m going to bed. Let me know how it goes.
Thanks,
Tee
From: Ngo, Tee
Sent: May-26-19 11:44 PM
To: 'Chen, Yan'
Subject: RE: ansible deployment for duplex/standard
Hi Yan,
Right after a fresh install of controller-0, please replace the following file on the host at /usr/share/ansible/stx-ansible/playbooks/bootstrap/roles/persist-config/files/populate_initial_config.py with the version attached to this email and let me know if it works for you using Ansible bootstrap?.
If you are testing in vbox environment, take a snapshot right after the fresh install of controller-0 so you can switch back to it.
Thanks,
Tee
From: Chen, Yan [mailto:yan.chen at intel.com]
Sent: May-26-19 10:02 PM
To: Ngo, Tee
Subject: RE: ansible deployment for duplex/standard
Hi, Tee
Sorry for late reply.
Attached is the result from “config_controller --force".
But on this deployment I still met some issue after the application-upload:
| platform-integ-apps | 1.0-5 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |
I will try another deployment and if it can pass I will send you again.
Sysinv.log error like this:
2019-05-24 08:59:20.609 47 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.
2019-05-24 08:59:20.609 47 ERROR armada.cli Traceback (most recent call last):
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 37, in _load_yaml_file
2019-05-24 08:59:20.609 47 ERROR armada.cli with open(doc) as f:
2019-05-24 08:59:20.609 47 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/helm-toolkit-helm-toolkit.yaml'
2019-05-24 08:59:20.609 47 ERROR armada.cli
2019-05-24 08:59:20.609 47 ERROR armada.cli During handling of the above exception, another exception occurred:
2019-05-24 08:59:20.609 47 ERROR armada.cli
2019-05-24 08:59:20.609 47 ERROR armada.cli Traceback (most recent call last):
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 39, in safe_invoke
2019-05-24 08:59:20.609 47 ERROR armada.cli self.invoke()
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 209, in invoke
2019-05-24 08:59:20.609 47 ERROR armada.cli resp = self.handle(documents, tiller)
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 63, in func_wrapper
2019-05-24 08:59:20.609 47 ERROR armada.cli return future.result()
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2019-05-24 08:59:20.609 47 ERROR armada.cli return self.__get_result()
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2019-05-24 08:59:20.609 47 ERROR armada.cli raise self._exception
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2019-05-24 08:59:20.609 47 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 251, in handle
2019-05-24 08:59:20.609 47 ERROR armada.cli target_manifest=self.target_manifest)
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 85, in __init__
2019-05-24 08:59:20.609 47 ERROR armada.cli values=values).update_manifests()
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 161, in update_manifests
2019-05-24 08:59:20.609 47 ERROR armada.cli merging_values = self._load_yaml_file(value)
2019-05-24 08:59:20.609 47 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/override.py", line 40, in _load_yaml_file
2019-05-24 08:59:20.609 47 ERROR armada.cli raise override_exceptions.InvalidOverrideFileException(doc)
2019-05-24 08:59:20.609 47 ERROR armada.cli armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.
2019-05-24 08:59:20.609 47 ERROR armada.cli
Yan
From: Ngo, Tee [mailto:Tee.Ngo at windriver.com]
Sent: Saturday, May 25, 2019 06:10
To: Chen, Yan <yan.chen at intel.com<mailto:yan.chen at intel.com>>
Subject: RE: ansible deployment for duplex/standard
Importance: High
Hi,
Can you provide the output of system service-parameter-list using config_controller --force?
Thanks,
Tee
From: Chen, Yan [mailto:yan.chen at intel.com]
Sent: May-24-19 1:40 AM
To: Chen, Yan; Poncea, Ovidiu; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] ansible deployment for duplex/standard
I tried to config oam/mgmt. for duplex/standard after ansible, and the controller1 and other nodes can be installed now.
But we still met issue when doing “system application-upload” because we have to use private registry in SH, but private registry is not successfully configed by ansible, please check this LP:
https://bugs.launchpad.net/starlingx/+bug/1830319
Thanks!
Yan
From: Chen, Yan [mailto:yan.chen at intel.com]
Sent: Friday, May 24, 2019 10:32
To: Poncea, Ovidiu <Ovidiu.Poncea at windriver.com<mailto:Ovidiu.Poncea at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] ansible deployment for duplex/standard
Thanks!
I will try.
Yan
From: Poncea, Ovidiu [mailto:Ovidiu.Poncea at windriver.com]
Sent: Thursday, May 23, 2019 18:21
To: Chen, Yan <yan.chen at intel.com<mailto:yan.chen at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: ansible deployment for duplex/standard
On controller-0 you need to delete the loopback interface and then configure the other nets on ctrl-0 with something like:
system host-if-modify controller-0 lo -c none
system host-if-modify controller-0 enp0s3 -c platform --networks oam
system host-if-modify controller-0 enp0s8 -c platform --networks mgmt
system host-if-modify controller-0 enp0s8 -c platform --networks cluster-host
system host-addr-list controller-0
The rest of the process is identical on the other nodes.
Ovidiu
________________________________
From: Chen, Yan [yan.chen at intel.com]
Sent: Wednesday, May 22, 2019 10:12 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] ansible deployment for duplex/standard
Hi
I wonder if anyone can help me on the ansible deployment for duplex/standard.
I installed ISO (about 0519) on vms and run ansible with my yml file as below, but the 2nd controller failed to be installed via pxe.
After checked controller0, I found that mgmt network is configured to lo.
But use “config_controller --force" everything’s ok.
Am I doing anything wrong with the yml file?
There’s no instruction on WIKI for duplex + ansible yet. (?)
---
system_mode: duplex
management_subnet: 192.178.204.0/28
external_oam_subnet: 10.10.10.0/24
external_oam_gateway_address: 10.10.10.1
external_oam_floating_address: 10.10.10.2
external_oam_node_0_address: 10.10.10.3
external_oam_node_1_address: 10.10.10.4
Yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190529/7b36d4ae/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 120502 bytes
Desc: image001.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190529/7b36d4ae/attachment-0001.png>
More information about the Starlingx-discuss
mailing list