[Starlingx-discuss] a question about starlingx error handling behavior
Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
Hi Eric, Thank you, Eric! I will collect all the information and get back to you soon. I confirm that I physically pulled the cable. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Tuesday, July 2, 2019 7:54 PM To: Wang, Yi C <yi.c.wang@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] a question about starlingx error handling behavior Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
Hi Eric, I retested it. And here is all the information you requested. I uploaded them to my google drive. Below is the link. https://drive.google.com/open?id=1LiCgPz5iS3SApb0Em8ZyA56lryAU-3Vm I conducted two tests. For case 1, the system can recover. For case 2, it can't. Case 1: pull the active controller management cable for longer than 30s, and then reinsert it Case 2: pull the active controller management cable for less than 30s, and then reinsert it For case 2, since controller-1 was shown as "offline" on controller-0. "collect all" can't get the information of controller-1. So I copied the whole folder "/var/log" of controller-1. It is included in the shared zip package. If you need more information, let me know. Thanks for your help again! Thanks. Yi From: Wang, Yi C Sent: Tuesday, July 2, 2019 9:21 PM To: MacDonald, Eric <Eric.MacDonald@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Hi Eric, Thank you, Eric! I will collect all the information and get back to you soon. I confirm that I physically pulled the cable. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Tuesday, July 2, 2019 7:54 PM To: Wang, Yi C <yi.c.wang@intel.com<mailto:yi.c.wang@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] a question about starlingx error handling behavior Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
OK, give me some time to fit this analysis into my existing task list. I'll be in touch before the end of the week. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 03, 2019 11:34 AM To: MacDonald, Eric Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I retested it. And here is all the information you requested. I uploaded them to my google drive. Below is the link. https://drive.google.com/open?id=1LiCgPz5iS3SApb0Em8ZyA56lryAU-3Vm I conducted two tests. For case 1, the system can recover. For case 2, it can't. Case 1: pull the active controller management cable for longer than 30s, and then reinsert it Case 2: pull the active controller management cable for less than 30s, and then reinsert it For case 2, since controller-1 was shown as "offline" on controller-0. "collect all" can't get the information of controller-1. So I copied the whole folder "/var/log" of controller-1. It is included in the shared zip package. If you need more information, let me know. Thanks for your help again! Thanks. Yi From: Wang, Yi C Sent: Tuesday, July 2, 2019 9:21 PM To: MacDonald, Eric <Eric.MacDonald@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Hi Eric, Thank you, Eric! I will collect all the information and get back to you soon. I confirm that I physically pulled the cable. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Tuesday, July 2, 2019 7:54 PM To: Wang, Yi C <yi.c.wang@intel.com<mailto:yi.c.wang@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] a question about starlingx error handling behavior Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
Hi Yi, Seems like your issue is a duplicate of another LP that both myself and one of my colleagues are (currently) working on a fix for. https://bugs.launchpad.net/starlingx/+bug/1815969 The above LP requires the following LP that I'm working on a fix for before it can be delivered. https://bugs.launchpad.net/starlingx/+bug/1835268 Please retest once the above 2 LP's have updates delivered against them. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 03, 2019 11:34 AM To: MacDonald, Eric Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I retested it. And here is all the information you requested. I uploaded them to my google drive. Below is the link. https://drive.google.com/open?id=1LiCgPz5iS3SApb0Em8ZyA56lryAU-3Vm I conducted two tests. For case 1, the system can recover. For case 2, it can't. Case 1: pull the active controller management cable for longer than 30s, and then reinsert it Case 2: pull the active controller management cable for less than 30s, and then reinsert it For case 2, since controller-1 was shown as "offline" on controller-0. "collect all" can't get the information of controller-1. So I copied the whole folder "/var/log" of controller-1. It is included in the shared zip package. If you need more information, let me know. Thanks for your help again! Thanks. Yi From: Wang, Yi C Sent: Tuesday, July 2, 2019 9:21 PM To: MacDonald, Eric <Eric.MacDonald@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Hi Eric, Thank you, Eric! I will collect all the information and get back to you soon. I confirm that I physically pulled the cable. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Tuesday, July 2, 2019 7:54 PM To: Wang, Yi C <yi.c.wang@intel.com<mailto:yi.c.wang@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] a question about starlingx error handling behavior Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
Hi Eric, Thank you very much! Sure, I will monitor the two LPs. Once they are resolved, I will recheck my LP. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Thursday, July 4, 2019 7:49 PM To: Wang, Yi C <yi.c.wang@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Qian, Bin <Bin.Qian@windriver.com> Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Hi Yi, Seems like your issue is a duplicate of another LP that both myself and one of my colleagues are (currently) working on a fix for. https://bugs.launchpad.net/starlingx/+bug/1815969 The above LP requires the following LP that I'm working on a fix for before it can be delivered. https://bugs.launchpad.net/starlingx/+bug/1835268 Please retest once the above 2 LP's have updates delivered against them. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Wednesday, July 03, 2019 11:34 AM To: MacDonald, Eric Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I retested it. And here is all the information you requested. I uploaded them to my google drive. Below is the link. https://drive.google.com/open?id=1LiCgPz5iS3SApb0Em8ZyA56lryAU-3Vm I conducted two tests. For case 1, the system can recover. For case 2, it can't. Case 1: pull the active controller management cable for longer than 30s, and then reinsert it Case 2: pull the active controller management cable for less than 30s, and then reinsert it For case 2, since controller-1 was shown as "offline" on controller-0. "collect all" can't get the information of controller-1. So I copied the whole folder "/var/log" of controller-1. It is included in the shared zip package. If you need more information, let me know. Thanks for your help again! Thanks. Yi From: Wang, Yi C Sent: Tuesday, July 2, 2019 9:21 PM To: MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] a question about starlingx error handling behavior Hi Eric, Thank you, Eric! I will collect all the information and get back to you soon. I confirm that I physically pulled the cable. Thanks. Yi From: MacDonald, Eric [mailto:Eric.MacDonald@windriver.com] Sent: Tuesday, July 2, 2019 7:54 PM To: Wang, Yi C <yi.c.wang@intel.com<mailto:yi.c.wang@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] a question about starlingx error handling behavior Please perform both cases and for each - indicate what cables/interfaces were pulled - indicate what hosts the cables were pulled from - indicate approximate timestamp of when the cable was pulled - indicate approximate timestamp of when the cable was reinserted - Run 'collect all' and provide me access to the collect tarball Also, just to be sure ... please confirm that you are physically pulling the cable and not just ifdowning the interface. Eric. From: Wang, Yi C [mailto:yi.c.wang@intel.com] Sent: Sunday, June 30, 2019 9:45 PM To: MacDonald, Eric Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] a question about starlingx error handling behavior Importance: High Hi Eric, I am working on the LP 1815513<https://bugs.launchpad.net/starlingx/+bug/1815513>. Based on my tests, if I unplug the cable of active controller for management network for a long time (for example, 30s), and then plug it, the whole system can recover after some reboots. But if I unplug the cable for a short time, and then plug it. The whole system can't recover. I need to lock/unlock controllers manually to bring the system back. So my questions are: 1. Is the behavior acceptable? (recover the system by manual lock/unlock operations) 2. If the answer is no for #1, we need the system to recover automatically. I am not familiar with internal maintenance logics, could you give me some hints? Thanks. Yi
participants (2)
-
MacDonald, Eric
-
Wang, Yi C