[Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Legacy, Allain
Allain.Legacy at windriver.com
Mon Dec 17 14:32:49 UTC 2018
Sorry, I think I confused things by mentioning the scenario of launching VM instances directly on an external network.
The original scenario that was being discussed was two different VM instances, on two different tenant networks, communicating via their respective FIP address over the same external tenant network. In that scenario, it is the respective virtual routers of both VM instances that will be sending out ARP requests for the destination FIP addresses. It is those broadcast ARP requests that need to be eliminated by using updated L2POP FDB information. Therefore, the FDB updates need to be generated properly at the neutron-server and received by the neutron-layer2-agent on the node which is hosting the virtual routers; not necessarily the nodes hosting the VM instances. In the following diagram, that means that the FDB on network node 1 and 2 need to be updated with information for FIP1 and FIP2. The FDB on compute node 1 and 2 do not need to be updated because there are no ports directly on the external network on those nodes.
FIP1 FIP2
+ +
| |
+-------+ +------+ | | +------+ +-------+
| | | | | | | | | |
| VM1 +---network1--+ R1 +-+-ext-network-+-+ R2 +----network2-----+ VM2 |
| | | | | | | |
+---+---+ +---+--+ +---+--+ +---+---+
| | | |
| | | |
v v v v
compute network network compute
node 1 node 1 node 2 node 2
The other scenario that I mentioned, which was to launch VM instances directly on the external network, was only to illustrate that it is possible for a VM to communicate directly for another server’s FIP rather than having to go through a virtual router. In this type of scenario, the FDB of the node hosting the VM instance that resides directly on the external network needs to be updated whenever the FIP changes state. In the following diagram, that means that the FDB on compute2 needs to be updated with information for FIP1, and the FDB on network node 1 needs to be updated with information for VM2’s port on the external network. The FDB on compute node 1 does not need to be updated for VM2’s port information because there are no ports on compute node 1 that are directly attached to the external network.
FIP1
+
|
+-------+ +------+ | +-------+
| | | | | | |
| VM1 +---network1--+ R1 +-+-ext-network---+ VM2 |
| | | | | |
+---+---+ +---+--+ +---+---+
| | |
| | |
v v v
compute network compute
node 1 node 1 node 2
Allain
Allain Legacy, Software Developer, Wind River
direct 613.270.2279 fax 613.492.7870 skype allain.legacy
350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5
[WIND]<http://www.windriver.com/>
From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Monday, December 17, 2018 5:35 AM
To: Legacy, Allain
Cc: Peters, Matt; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Allain,
Thank you for your advice! By following your suggestion, the FDB for floating IP can be installed on compute node. However there is a problem within the use case for RFE “Add l2pop support for floating IP resources”. The use case can be summarized as two VM instances, residing on different networks, communicate via their respective FIP addresses. The flows for floating IP on compute node:
[cid:image002.jpg at 01D495EB.7889B190]
To use the flows for floating IP, the VM on compute node needs to have the same dl_vlan (local VLAN ID) which is 1. According to your previous email, the VM must be launched on the external network. Thus for the use case, the two VM resides on different external networks where the floating IPs for two VMs are. Thus another VM’s dl_vlan won’t be 1 and the flows for floating IP can’t be matched.
Based on the above analysis, the two VM instances, residing on different networks, can’t communicate via their respective FIP addresses. Could you please help review and comment?
Best Regards,
Xu, Chenjie
From: Legacy, Allain [mailto:Allain.Legacy at windriver.com]
Sent: Friday, December 14, 2018 8:55 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Cc: Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
An FDB entry for a Floating IP resource would only be required on nodes with ports attached to the related external network. Normally that means only the network nodes that are hosting virtual routers attached to those external networks rather than compute nodes. However, there are less frequently used scenarios that involve customers launching VM instances directly on external networks and in those scenarios the external network will be instantiated in some compute nodes as well as the network nodes.
In your test scenario, the external network will only be present on the network node and therefore only the agent running on the network node will be capable of processing any related FDB entries.
Regards,
Allain
Allain Legacy, Software Developer, Wind River
direct 613.270.2279 fax 613.492.7870 skype allain.legacy
350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5
[WIND]<http://www.windriver.com/>
From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Friday, December 14, 2018 3:24 AM
To: Legacy, Allain
Cc: Peters, Matt; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: FW: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Allain,
As you suggested, I’m testing the 2 changes together. However I find that the fdb for floatingip can’t be installed on the br-tun of the compute node. Because the network-id in fdb is not in the LocalVlanManager. The details are below and could you please help review and comment?
The environment: latest DevStack with 1 controller node and 1 compute node
The network topology is below:
[cid:image002.jpg at 01D49632.CEB88F40]
Steps:
1. Create an external network external-net
neutron net-create external-net --router:external True --provider:network_type vxlan
2. Create a subnet on external-net
neutron subnet-create external-net 192.168.25.0/24 --name external-subnet --allocation-pool start=192.168.25.200,end=192.168.25.250
3. Create an internal network, create a subnet on net4
neutron net-create net4
neutron subnet-create net4 192.168.2.0/24 --name subnet4
4. Create a router, set router gateway as external-net, add subnet4 to router
neutron router-create router
neutron router-gateway-set $router-id $external-net-id
neutron router-interface-add $router-id $subnet4-id
5. Create a VM vm-1 on net4 (vm-1 runs on compute node)
6. Allocate floating IP FIP-1 on external-net through horizon
7. Associate FIP-1 with vm-1 through horizon
The fdb is below:
[cid:image003.jpg at 01D49632.CEB88F40]
The OVS agent on the compute node receives the FDB:
[cid:image004.jpg at 01D49632.CEB88F40]
In OVS agent, LocalVlanManager is used to map tunnel ids or vlan ids to internal vlans:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/vlanmanager.py
After OVS agent receiving the FDB, it will try to get LocalVlanMapping from LocalVlanMannager:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L558
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/l2pop/rpc_manager/l2population_rpc.py#L237
However the network-id f2ebf82a-e788-4456-8516-c95b12f91d49 is not in the LocalVlanManager. Thus the fdb can’t be installed on br-tun. The mapping in the LocalVlanManager in the OVS agent which is on the compute node is below:
[cid:image005.jpg at 01D49632.CEB88F40]
Analysis:
The vm-1 is created on internal network net4 . When creating vm-1, a port in the net4 will be bound to vm-1. Thus the network-id can be added to the LocalVlanManager. The network id for floating IP is used in FDB but the corresponding network is an external network. And this external network’s network id is not in the LocalVlanManager in the OVS agent which is on the compute node.
Best Regard,
Xu, Chenjie
From: Legacy, Allain [mailto:Allain.Legacy at windriver.com]
Sent: Friday, December 7, 2018 10:20 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
The change that is being reviewed here was originally a part of a larger commit (9f926a5d253<http://stash.wrs.com/projects/CGCS/repos/github.com.openstack.neutron/commits/9f926a5d2539b0b0451c50053244a8fa6c43bf3c>). They should be implemented together or at least tested together. I seem to remember that there was information missing in case 1 that prevented a proper FDB notification from being generated.
Please retest your scenarios and capture the input parameters to add_fdb_entries(), remove_fdb_entries(), and update_fdb_entries() in neutron/plugins/ml2/drivers/l2pop/rpc.py:L2populationAgentNotifyAPI to be sure that expected notifications are published.
Regards,
Allain
Allain Legacy, Software Developer, Wind River
direct 613.270.2279 fax 613.492.7870 skype allain.legacy
350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5
[WIND]<http://www.windriver.com/>
From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Friday, December 07, 2018 3:42 AM
To: Peters, Matt
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Matt,
Ryan Tidwell comments on this patch and he thinks that AFTER_DELETE notification can be used to trigger l2pop.
https://review.openstack.org/#/c/611261/
https://review.openstack.org/#/c/611261/4/neutron/db/l3_db.py
From the comment in the following line:
https://github.com/starlingx-staging/stx-neutron/blob/master/neutron/services/l3_router/service_providers/l2pop.py#L276
It seems that the router_id and port_id in AFTER_DELETE notification are None. As a result of that, the last_known_router_id and last_fixed_port_id should be used to construct FDB entries which are used to remove FDBs on each host. However, I print the notification in the following 2 cases:
Case-1:
1) Allocate floating ip fip-1
2) Associate fip-1 with vm-1
3) Delete fip-1
Case-2:
1) Allocate floating ip fip-1
2) Associate fip-1 with vm-1
3) Disassociate fip-1 with vm-1
4) Delete fip-1
The notification for case1 and case 2 are attached. router_id and port_id are not None in case-1 and are None in case-2. Thus in case-1, AFTER_DELETE notification can be used. In case-2, FDB will be removed by step 3, thus no need to remove again.
Based on the above analysis, I think we can use AFTER_DELETE notification. Could you please comment and review?
Best Regards,
Xu, Chenjie
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Monday, November 12, 2018 11:19 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Legacy, Allain <Allain.Legacy at windriver.com<mailto:Allain.Legacy at windriver.com>>
Subject: Re: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Chenjie,
The latest RFE looks good to me.
Regards, Matt
From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Monday, November 12, 2018 at 1:23 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>
Cc: "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, Allain Legacy <Allain.Legacy at windriver.com<mailto:Allain.Legacy at windriver.com>>
Subject: RE: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Matt,
The RFE has been updated and is attached. Could you please help review and comment?
Best Regards,
Xu, Chenjie
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Friday, November 9, 2018 9:22 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Legacy, Allain <Allain.Legacy at windriver.com<mailto:Allain.Legacy at windriver.com>>
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Chenjie,
The RFE looks good. The use cases are clear and detailed.
I only have a few minor review comments (see attached).
Regards, Matt
From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Thursday, November 1, 2018 at 4:28 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, Allain Legacy <Allain.Legacy at windriver.com<mailto:Allain.Legacy at windriver.com>>
Cc: "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Subject: [Starlingx-discuss] Analysis of patch 9f926a5 for StartlingX upstreaming
Hi Matt/Allain,
We analyze the patch 9f926a5 related to l2pop. An RFE “Add l2pop support for floating ip resources” has been written and is attached. The test case is provided by Allain. Could you please help to review and comment? Thanks very much!
Best Regards,
Xu, Chenjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1807 bytes
Desc: image001.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.jpg
Type: image/jpeg
Size: 5143 bytes
Desc: image008.jpg
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0005.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.jpg
Type: image/jpeg
Size: 11643 bytes
Desc: image009.jpg
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0006.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.jpg
Type: image/jpeg
Size: 11184 bytes
Desc: image010.jpg
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0007.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image011.jpg
Type: image/jpeg
Size: 10148 bytes
Desc: image011.jpg
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0008.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 36643 bytes
Desc: image002.jpg
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181217/a9a6371a/attachment-0009.jpg>
More information about the Starlingx-discuss
mailing list