[Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
I think the puppet 5 upgrade ends up being one of the more complicated parts of this activity.
It looks like for 5.0.x you need to be cautious about is all the deprecated features in 4 that were removed in 5. It would not surprise me if we were still using some of those deprecated parts. https://github.com/puppetlabs/docs-archive/blob/master/puppet/5.0/release_no...
Puppet 5 also uses a newer ruby. We ship ruby 2.0 now. The notes for 5.0 indicate ruby 2.4, which is extra work and can take a while if existing ruby code needs to be changed.
Puppet 5 also claims to have hiera 5 built in. I don't know what that means, we ship hiera 1.3.4 separarately, so there may be packaging concerns there as well.
Last time when I upversioned puppet from puppet 3 to 4 the most unexpected thing was not the code changes required, but the runtime behavior was different. Example: https://opendev.org/starlingx/integ/src/branch/master/config/puppet-4.8.2/ce...
Experiment with installing train keystone/barbican/horizon with the existing puppet modules. If a config value is incompatible, then it may require a local change, or an upversion.
Al
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 9:43 AM To: Chen, Yan; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
We discussed this topic at the Release meeting today. The conclusion was that we do not want to upgrade puppet or ruby for this release. It's too risky.
As one of the last features gating our 3.0 release there is some urgency in figuring out the right approach to this upgrade.
Where we need help is figuring out if it is possible to divide the OpenStack world into two parts and treat each part separately.
Most of the OpenStack services running in StarlingX are in containers and should not depend on puppet at all. The 3 services that are not in containers (keystone, barbican and horizon) do depend on puppet.
Is it possible / practical to run the Train versions of the containerized services, while still running the Stein versions of keystone, barbican and horizon in the platform layer?
Or is there another creative solution to the problem?
brucej
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, October 17, 2019 7:47 AM To: Jones, Bruce E bruce.e.jones@intel.com; Chen, Yan yan.chen@intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
I think the puppet 5 upgrade ends up being one of the more complicated parts of this activity.
It looks like for 5.0.x you need to be cautious about is all the deprecated features in 4 that were removed in 5. It would not surprise me if we were still using some of those deprecated parts. https://github.com/puppetlabs/docs-archive/blob/master/puppet/5.0/release_no...
Puppet 5 also uses a newer ruby. We ship ruby 2.0 now. The notes for 5.0 indicate ruby 2.4, which is extra work and can take a while if existing ruby code needs to be changed.
Puppet 5 also claims to have hiera 5 built in. I don't know what that means, we ship hiera 1.3.4 separarately, so there may be packaging concerns there as well.
Last time when I upversioned puppet from puppet 3 to 4 the most unexpected thing was not the code changes required, but the runtime behavior was different. Example: https://opendev.org/starlingx/integ/src/branch/master/config/puppet-4.8.2/ce...
Experiment with installing train keystone/barbican/horizon with the existing puppet modules. If a config value is incompatible, then it may require a local change, or an upversion.
Al
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 9:43 AM To: Chen, Yan; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
On 10/17/19 1:51 PM, Jones, Bruce E wrote:
We discussed this topic at the Release meeting today. The conclusion was that we do not want to upgrade puppet or ruby for this release. It's too risky.
Agreed there.
As one of the last features gating our 3.0 release there is some urgency in figuring out the right approach to this upgrade.
Where we need help is figuring out if it is possible to divide the OpenStack world into two parts and treat each part separately.
Most of the OpenStack services running in StarlingX are in containers and should not depend on puppet at all. The 3 services that are not in containers (keystone, barbican and horizon) do depend on puppet.
I still want to be really clear here, the services themselves do not depend on Puppet, the StarlingX deployment of them does.
Is it possible / practical to run the Train versions of the containerized services, while still running the Stein versions of keystone, barbican and horizon in the platform layer?
I thought we had already decided to do that, but I may be behind in my news.
OpenStack clients are all written with serious backward-compatibility in mind at the REST level. OSC and the SDK both are known to still work on juno clouds for example, not all of the python-*client libs go that far back but most do. Forward compatibility is also reliable for most APIs, those that use microversioning especially so (Cinder, Ironic, Nova, etc).
In the list at play here, Horizon is itself a client so it needs to be considered the same as the CLIs not as a service. That leaves Barbican and Keystone. I do not know much about the Barbican API other than it is not under heavy development. I would be shocked if it did not work rolling forward a version or two. Keystone also has a mostly-stable API, the risks here are if we need to make use of anything new in Keystone Train specifically around the distcloud functionality.
I thought I saw a decision to wait on Train for the platform components until CentOS had packaging for Train available. Is that approach of just installing those packages a possibility?
dt
Bruce,
To build on your point below, the problem should be split into 2 parts.
1) Openstack Cloud support. Openstack is containerized, it does not use puppet
2) There are a small set of openstack components that run on bare metal that provide services for the "flock" components.
My 2 cents is #1 is the most important. There is no burning need to do #2 as there are no new capabilities in Train that are required.
Brent
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 2:52 PM To: Bailey, Henry Albert (Al) Al.Bailey@windriver.com; Chen, Yan yan.chen@intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
We discussed this topic at the Release meeting today. The conclusion was that we do not want to upgrade puppet or ruby for this release. It's too risky.
As one of the last features gating our 3.0 release there is some urgency in figuring out the right approach to this upgrade.
Where we need help is figuring out if it is possible to divide the OpenStack world into two parts and treat each part separately.
Most of the OpenStack services running in StarlingX are in containers and should not depend on puppet at all. The 3 services that are not in containers (keystone, barbican and horizon) do depend on puppet.
Is it possible / practical to run the Train versions of the containerized services, while still running the Stein versions of keystone, barbican and horizon in the platform layer?
Or is there another creative solution to the problem?
brucej
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, October 17, 2019 7:47 AM To: Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
I think the puppet 5 upgrade ends up being one of the more complicated parts of this activity.
It looks like for 5.0.x you need to be cautious about is all the deprecated features in 4 that were removed in 5. It would not surprise me if we were still using some of those deprecated parts. https://github.com/puppetlabs/docs-archive/blob/master/puppet/5.0/release_no...
Puppet 5 also uses a newer ruby. We ship ruby 2.0 now. The notes for 5.0 indicate ruby 2.4, which is extra work and can take a while if existing ruby code needs to be changed.
Puppet 5 also claims to have hiera 5 built in. I don't know what that means, we ship hiera 1.3.4 separarately, so there may be packaging concerns there as well.
Last time when I upversioned puppet from puppet 3 to 4 the most unexpected thing was not the code changes required, but the runtime behavior was different. Example: https://opendev.org/starlingx/integ/src/branch/master/config/puppet-4.8.2/ce...
Experiment with installing train keystone/barbican/horizon with the existing puppet modules. If a config value is incompatible, then it may require a local change, or an upversion.
Al
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 9:43 AM To: Chen, Yan; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
Brent, thank you, that makes sense.
Yan & the openstack-distro team - please focus on 1) below -the containerized openstack services.
brucej
From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Thursday, October 17, 2019 3:55 PM To: Jones, Bruce E bruce.e.jones@intel.com; Bailey, Henry Albert (Al) Al.Bailey@windriver.com; Chen, Yan yan.chen@intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
Bruce,
To build on your point below, the problem should be split into 2 parts.
1) Openstack Cloud support. Openstack is containerized, it does not use puppet
2) There are a small set of openstack components that run on bare metal that provide services for the "flock" components.
My 2 cents is #1 is the most important. There is no burning need to do #2 as there are no new capabilities in Train that are required.
Brent
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 2:52 PM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
We discussed this topic at the Release meeting today. The conclusion was that we do not want to upgrade puppet or ruby for this release. It's too risky.
As one of the last features gating our 3.0 release there is some urgency in figuring out the right approach to this upgrade.
Where we need help is figuring out if it is possible to divide the OpenStack world into two parts and treat each part separately.
Most of the OpenStack services running in StarlingX are in containers and should not depend on puppet at all. The 3 services that are not in containers (keystone, barbican and horizon) do depend on puppet.
Is it possible / practical to run the Train versions of the containerized services, while still running the Stein versions of keystone, barbican and horizon in the platform layer?
Or is there another creative solution to the problem?
brucej
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, October 17, 2019 7:47 AM To: Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
I think the puppet 5 upgrade ends up being one of the more complicated parts of this activity.
It looks like for 5.0.x you need to be cautious about is all the deprecated features in 4 that were removed in 5. It would not surprise me if we were still using some of those deprecated parts. https://github.com/puppetlabs/docs-archive/blob/master/puppet/5.0/release_no...
Puppet 5 also uses a newer ruby. We ship ruby 2.0 now. The notes for 5.0 indicate ruby 2.4, which is extra work and can take a while if existing ruby code needs to be changed.
Puppet 5 also claims to have hiera 5 built in. I don't know what that means, we ship hiera 1.3.4 separarately, so there may be packaging concerns there as well.
Last time when I upversioned puppet from puppet 3 to 4 the most unexpected thing was not the code changes required, but the runtime behavior was different. Example: https://opendev.org/starlingx/integ/src/branch/master/config/puppet-4.8.2/ce...
Experiment with installing train keystone/barbican/horizon with the existing puppet modules. If a config value is incompatible, then it may require a local change, or an upversion.
Al
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 9:43 AM To: Chen, Yan; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
Thank you all for making this clear.
Currently Zhipeng is working on containerized OpenStack upgrade. And 3 patches for Horizon upgrade for StarlingX dashboard are also under review.
Yan
From: Jones, Bruce E bruce.e.jones@intel.com Sent: Friday, October 18, 2019 07:45 To: Rowsell, Brent Brent.Rowsell@windriver.com; Bailey, Henry Albert (Al) Al.Bailey@windriver.com; Chen, Yan yan.chen@intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
Brent, thank you, that makes sense.
Yan & the openstack-distro team - please focus on 1) below -the containerized openstack services.
brucej
From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Thursday, October 17, 2019 3:55 PM To: Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
Bruce,
To build on your point below, the problem should be split into 2 parts.
1. Openstack Cloud support. Openstack is containerized, it does not use puppet 2. There are a small set of openstack components that run on bare metal that provide services for the "flock" components.
My 2 cents is #1 is the most important. There is no burning need to do #2 as there are no new capabilities in Train that are required.
Brent
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 2:52 PM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
We discussed this topic at the Release meeting today. The conclusion was that we do not want to upgrade puppet or ruby for this release. It's too risky.
As one of the last features gating our 3.0 release there is some urgency in figuring out the right approach to this upgrade.
Where we need help is figuring out if it is possible to divide the OpenStack world into two parts and treat each part separately.
Most of the OpenStack services running in StarlingX are in containers and should not depend on puppet at all. The 3 services that are not in containers (keystone, barbican and horizon) do depend on puppet.
Is it possible / practical to run the Train versions of the containerized services, while still running the Stein versions of keystone, barbican and horizon in the platform layer?
Or is there another creative solution to the problem?
brucej
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, October 17, 2019 7:47 AM To: Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Chen, Yan <yan.chen@intel.commailto:yan.chen@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Train upgrade for openstack clients
I think the puppet 5 upgrade ends up being one of the more complicated parts of this activity.
It looks like for 5.0.x you need to be cautious about is all the deprecated features in 4 that were removed in 5. It would not surprise me if we were still using some of those deprecated parts. https://github.com/puppetlabs/docs-archive/blob/master/puppet/5.0/release_no...
Puppet 5 also uses a newer ruby. We ship ruby 2.0 now. The notes for 5.0 indicate ruby 2.4, which is extra work and can take a while if existing ruby code needs to be changed.
Puppet 5 also claims to have hiera 5 built in. I don't know what that means, we ship hiera 1.3.4 separarately, so there may be packaging concerns there as well.
Last time when I upversioned puppet from puppet 3 to 4 the most unexpected thing was not the code changes required, but the runtime behavior was different. Example: https://opendev.org/starlingx/integ/src/branch/master/config/puppet-4.8.2/ce...
Experiment with installing train keystone/barbican/horizon with the existing puppet modules. If a config value is incompatible, then it may require a local change, or an upversion.
Al
From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, October 17, 2019 9:43 AM To: Chen, Yan; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Train upgrade for openstack clients
Yan, thank you for making this very clear.
As I read this, the puppet modules we use for the openstack clients are several releases old and the update to the Train version would be painful, er I mean lots of fun.
I think we need some input from the broader community.
What are the costs / risks of upgrading to puppet 5.0?
Should we try to upgrade the client puppet modules ourselves (and contribute the changes upstream)? Or is there a better long way we should launch these clients, e.g. should we just write Ansible installers for them?
My thanks to Dean for helping me understand what questions to ask :)
brucej
From: Chen, Yan [mailto:yan.chen@intel.com] Sent: Tuesday, October 15, 2019 10:43 PM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Train upgrade for openstack clients
Hi,
I'm now trying to upgrade the openstack clients installed on StarlingX to Train version. Currently I see the clients are Stein version, but the puppet packages (for those modules that we need to config on StarlingX host) are still old versions. [See the list below.] But for Train, I met a problem when I wanted to upgrade python-openstackclient/keystone to Train (4.0.0), the old puppet packages cannot work with Train clients and failed when running puppet-manifest-apply.sh. See from Stein keystone log, some of the deprecated cmds and options are planned to be removed in Train.
If we want to make Train clients work, we may need to do the following steps:
1. Upgrade puppet itself (required 5.x by Train) 2. Upgrade all the OpenStack related puppet packages accordingly, and update StarlingX patches. 3. If flock puppet pkgs are based on puppet 4.8.2, we also need to upgrade our code. 4. If other puppet pkgs are based on puppet 4.8.2, we also need to upgrade those pkgs.
So is it possible that we keep on using Stein clients with limited test now (need test with Train images, but suppose old functions should still work)?
Here's the list of the puppet packages and StarlingX patches (for Stein it should be 14.1.0 - 14.4.0, and for Train 15.0.0 - 15.4.0): Pkg
Current Version
Train Upgrade to
Meta Patch
Patch
puppet
4.8.2
5.x
2
5
puppet-keystone
11.3.0
15.x
6
5
puppet-oslo
11.3.0
15.x
3
2
puppet-openstacklib
11.5.0
15.x
2
1
puppet-horizon
11.5.0
15.x
2
1
puppet-barbican
11.3.0
15.x
0
0
Stein Keystone client logs: keystone:log 2019-10-10 10:04:44.050 111346 WARNING keystone.server.flask.request_processing.middleware.auth_context [-] The use of the '[DEFAULT] admin_token' configurationoption presents a significant security risk and should not be set. This option is deprecated in favor of using 'keystone-manage bootstrap' and will be removed in a future release.
Yan
participants (5)
-
Bailey, Henry Albert (Al)
-
Chen, Yan
-
Dean Troyer
-
Jones, Bruce E
-
Rowsell, Brent