Do you see the same kind of traceback in hwmond_controller-0_thermal_sensor_data ?
Your issue looks like mtcAgent selected RedFish as the BMC access protocol after getting a good response but when the hardware monitor queries sensors from this server it gets a redfishtool python exception.
Please send me the group of logs for controller-0 from mtcAgent containing the 'redfishUtil_get_bmc_info' label so that I can understand more about the server you have.
If you like you can execute the following 2 commands to force the system to use ipmitool instead of redfish.
system service-parameter-modify platform maintenance bmc_access_method=ipmi
system service-parameter-apply platform
Wait a few minutes and then try and read your sensors with the system-sensor-list controller-0
I'd be interested in seeing the raw result of the sensor read commands against this server using redfishtool.
time redfishtool -r <ip> -u <un> -p <pw> Chassis Thermal
time redfishtool -r <ip> -u <un> -p <pw> Chassis Power
Basically the same thing but outside daemon control.
Eric.
-----Original Message-----
From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com]
Sent: Monday, November 11, 2019 12:12 PM
To: MacDonald, Eric; Hu, Yong; Alonso, Juan Carlos; Chen, Yan; starlingx-discuss@lists.starlingx.io
Cc: Cabrales, Ada
Subject: RE: [Starlingx-discuss] Train patch elimination
Importance: High
Hi guys, i checked everything in the way that you suggested.
MtcAgent.log is showing, im working in a Duplex configuration locked and trying to get sensors from controller-0:
p ( 601) redfishUtil_get_bmc_info: Info : controller-0 has 2 Processors ; Ena bled and OK:OK
2019-11-11T10:38:49.554 [1945193.00281] controller-1 mtcAgent --- redfishUtil.cp p ( 624) redfishUtil_get_bmc_info:
Info : controller-0 has 96 GiB Memory ; Ena bled and OK:OK
2019-11-11T10:38:49.554 [1945193.00282] controller-1 mtcAgent |-| mtcNodeHdlrs.c pp (6492) bmc_handler
: Info : controller-0 bmc audit timer started (120 secs)
2019-11-11T10:38:49.554 [1945193.00283] controller-1 mtcAgent |-| mtcNodeHdlrs.c pp (6507) bmc_handler
: Info : controller-0 bmc is accessible using
cd /var/run/bmc/
[sysadmin@controller-1 bmc(keystone_admin)]$ ls
controller-0 controller-1 ipmitool redfishtool
cat /var/run/bmc/
cat controller-0
{"protocol":"redfishtool","power_state":"on"}
cat controller-1
{"protocol":"redfishtool","power_state":"on"}
cd ipmitool
controller-1:/var/run/bmc/ipmitool$
controller-1:/var/run/bmc/ipmitool$
controller-1:/var/run/bmc/ipmitool$ ls
controller-1:/var/run/bmc/ipmitool$
edfishtool$ ls
hwmond_controller-0_power_sensor_data hwmond_controller-1_power_sensor_data mtcAgent_controller-0_bmc_info mtcAgent_controller-1_bmc_info
hwmond_controller-0_thermal_sensor_data hwmond_controller-1_thermal_sensor_data mtcAgent_controller-0_root_query mtcAgent_controller-1_root_query
If I check any of the files I got this:
cat hwmond_controller-0_power_sensor_data
redfishtool:Transport: Fatal timeout waiting for response from rhostTraceback (most recent call last):
File "/usr/bin/redfishtool", line 18, in <module>
main(sys.argv)
File "/usr/lib64/python2.7/site-packages/redfishtool/redfishtoolMain.py", line 372, in main
rc,r,j,d=runSubCmd(rft)
File "/usr/lib64/python2.7/site-packages/redfishtool/redfishtoolMain.py", line 449, in runSubCmd
rc,r,j,d=subCmdTable[rft.subcommand](rft,cmdTop=True)
File "/usr/lib64/python2.7/site-packages/redfishtool/Chassis.py", line 157, in ChassisMain
rc,r,j,d = self.runOperation(rft)
File "/usr/lib64/python2.7/site-packages/redfishtool/Chassis.py", line 113, in runOperation
rc,r,j,d=operationTable[self.operation](self, op, rft, cmdTop=True)
File "/usr/lib64/python2.7/site-packages/redfishtool/Chassis.py", line 428, in getPower
rc,r,j,d=op.get(sc, op, rft, cmdTop, resName)
File "/usr/lib64/python2.7/site-packages/redfishtool/Chassis.py", line 234, in get
sysPath,rc,r,j,d=rft.getPathBy(rft, r, d, prop)
File "/usr/lib64/python2.7/site-packages/redfishtool/redfishtoolTransport.py", line 885, in getPathBy
rc, r, j, d = rft.listCollection(rft, r, coll, prop)
File "/usr/lib64/python2.7/site-packages/redfishtool/redfishtoolTransport.py", line 1025, in listCollection
rc,r,j,d=rft.rftSendRecvRequest(rft.AUTHENTICATED_API, 'GET', r.url, relPath=path)
AttributeError: 'NoneType' object has no attribute 'url'
So , my question should be, is the redfishtool blocking ipmitool to show the values? Because impitool is showing sensors by itself.
BR
Elio
-----Original Message-----
From: MacDonald, Eric <Eric.MacDonald@windriver.com>
Sent: Monday, November 11, 2019 8:33 AM
To: Hu, Yong <yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Chen, Yan <yan.chen@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Cabrales, Ada <ada.cabrales@intel.com>
Subject: Re: [Starlingx-discuss] Train patch elimination
Yong's suggestion to check the server's BMC settings is good.
Some servers such as Dell, have an 'Enable IPMI over LAN setting.
Do you see a BMC access alarm ?
> fm alarm-list
Try to ping the target server BMC.
If that works then I suggest you try and read the sensors manually using ipmitool.
ipmitool -I lanplus -H <ip address> -U <username> -P <password> sensor list
If a list of sensors is displayed and you can't see the sensor list with 'system host-sensor-list' then
1. Inspect recent logs in /var/log/mtcAgent.log and var/log/hwmond.log looking for logs that pertain to BMC accesses.
2. Check that your provisioned username, ip address and password are correct.
Try deprovisioning and reprovisioning the BMC.
> system host-update <hostname> bm_type=none
And then provision again ensuring the credentials are correct.
Wait 5-10 minutes
Tail mtcAgent and hwmond.log files. If there is a failure the logs will help us narrow in on the issue.
3. Provide output of the following commands
cat /var/run/bmc/*
ls -r /var/run/bmc/*
4. Ideally compress and attach the mtcAgent.log file
Eric.
-----Original Message-----
From: Yong Hu [mailto:yong.hu@intel.com]
Sent: Sunday, November 10, 2019 8:27 PM
To: Alonso, Juan Carlos; Chen, Yan; starlingx-discuss@lists.starlingx.io
Cc: Cabrales, Ada
Subject: Re: [Starlingx-discuss] Train patch elimination
JC,
Have you enabled BMC config in your test servers and BIOS?
If so, please use "ipmitool" utility to have a try to read sensor info.
regards,
Yong
On 2019/11/8 11:29 PM, Alonso, Juan Carlos wrote:
> Hello Yan/community,
>
> Any update on this? (:
>
> Regards.
>
> Juan Carlos Alonso
>
> *From:* Alonso, Juan Carlos
> *Sent:* Thursday, November 7, 2019 9:56 AM
> *To:* Chen, Yan <yan.chen@intel.com>; Hu, Yong <yong.hu@intel.com>;
> Gerry.Kopec@windriver.com; starlingx-discuss@lists.starlingx.io
> *Cc:* Cabrales, Ada <ada.cabrales@intel.com>
> *Subject:* RE: Train patch elimination
>
> Hello Yan,
>
> Thank you for your response.
>
> Same question now for:
>
> _Horizon/Custom Confirmation Messages : Custom confirmation message
> for Suppress Sensor Group, Relearn Sensor Model and Suppress Sensor._
>
> I added the bmc_ip, bmc_type, bmc_username and bmc_password for each
> node, but I cannot display any sensors on Horizon nor CLI.
>
> $ system host-update compute-0 bm_username=<user_name>
> bm_password=<password> bm_type=bmc bm_ip=<bmc_ip>
>
> $ system host-lock compute-0
>
> $ system host-sensor-list compute-0
>
> <none>
>
> $ system host-unlock compute-0
>
> $ system host-sensor-list compute-0
>
> <none>
>
> Is there any other configuration needed or is this feature not
> supported anymore?
>
> Regards.
>
> Juan Carlos Alonso
>
> *From:* Chen, Yan <yan.chen@intel.com <mailto:yan.chen@intel.com>>
> *Sent:* Thursday, November 7, 2019 12:15 AM
> *To:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com
> <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com
> <mailto:yong.hu@intel.com>>; Gerry.Kopec@windriver.com
> <mailto:Gerry.Kopec@windriver.com>;
> starlingx-discuss@lists.starlingx.io
> <mailto:starlingx-discuss@lists.starlingx.io>
> *Cc:* Cabrales, Ada <ada.cabrales@intel.com
> <mailto:ada.cabrales@intel.com>>
> *Subject:* RE: Train patch elimination
>
> Carlos,
>
> Please remove _Horizon/CSRF AngularJS Fixes_ related cases in your
> test plan, which are originally planned for STX additional patches, as
> those patches are already removed from our project and not upstreamed.
>
> Yan
>
> *From:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com
> <mailto:juan.carlos.alonso@intel.com>>
> *Sent:* Wednesday, November 6, 2019 23:18
> *To:* Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Chen,
> Yan <yan.chen@intel.com <mailto:yan.chen@intel.com>>;
> Gerry.Kopec@windriver.com <mailto:Gerry.Kopec@windriver.com>;
> starlingx-discuss@lists.starlingx.io
> <mailto:starlingx-discuss@lists.starlingx.io>
> *Cc:* Cabrales, Ada <ada.cabrales@intel.com
> <mailto:ada.cabrales@intel.com>>
> *Subject:* RE: Train patch elimination
>
> Hello,
>
> I hope also the community can help with this questions.
>
> @Chen, Yan <mailto:yan.chen@intel.com>
>
> Can you help me to verify if this feature is still disabled on Train
> version: _Horizon/CSRF AngularJS Fixes_?
>
> On last test plan for Stein this feature was no longer required on
> Horizon and was disabled on SarlingX code.
>
> @Gerry.Kopec@windriver.com <mailto:Gerry.Kopec@windriver.com>
>
> About the Live Migration bugs I mentioned on the last meeting, almost
> all of them are merged, except the last three.
>
> All those patches are related to live migration, so my question is if
> I perform live migrations such patches will be covered or should be
> tested in a different way?
>
> The last three were marked as Unlikely and was not merged on Stein. I
> don’t have more info about such patches.
>
> Enable delete bound trunk for linux bridge agent:
> https://review.opendev.org/#/c/605589
>
> 0040-rpc-removing-timeout-backoff-multiplier.patch (2e4ca00):
> https://review.opendev.org/#/c/623401 and
> https://review.opendev.org/#/c/626109
>
> 0062-dvr-force-admin-state-update-before-distri.patch (edf9731):
> https://review.opendev.org/#/c/625134
>
> 0065-dvr-do-not-create-agent-gateway-ports-unle.patch (2857911):
> https://review.opendev.org/#/c/628071
>
> 0077-api-reject-routes-with-invalid-network-val.patch (6ed8fe6):
> https://review.opendev.org/#/c/623415
>
> 0014-add-support-for-querying-quotas-with-usage.patch (71c07d7):
> _https://review.opendev.org/#/c/621507/_
>
> Prevent DHCP from processing stale RPC messages: ??
>
> 0039-dhcp-handle-concurrent-port-creation-error.patch (a38f89): ??
>
> 0047-wsgi-prevent-accepting-socket-without-a-gr.patch (8e72491): ??
>
> Regards.
>
> Juan Carlos Alonso
>
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss