[Starlingx-discuss] Using NVME Disks instead of SD
Folks, I have been trying to get a NUC based deployment work, with Erich's help we managed to start with a pair of Skull Canyon NUCs running Duplex controllers. This NUC has a pair of Ethernet ports and can be configured with 2 NVME-based disks. I was able to get the devices booted initially with changes to the ISO first by hardcoding /dev/nvme0n1 in metal/bsp-files, later Don's tool helped address alternative boot_device and rootfs_device setting, but I am not sure it went far enough. Erich figured out that we needed to patch the sysinv sqlDB (by hand) in order to unlock and make the second controller usable. I went a little further and tried to hook up both work and storage nodes to create a 2+2+2 Standard lab, I had to hand edit the PXE boot cmdline and also do the sysinv SQL magic to unlock the compute and storage nodes, but the Storage configuration still failed. I am thinking there is still some hardcode "sda" and/or "sdb" in places. Any thoughts on how to enable the Storage nodes properly? I can provide logs, configs, ... I realize this is a problem at initial ISO boot/install to select the drive type, but from there is should be automagically detected on the target, so it would require setting the drive type of the host at the same time as setting the personality so that the anaconda at least knows where to start installing. It may be possible for anaconda to determine the drive time also. Should this be a story, launchpad bug or a Spec to allow use to properly use NVME devices with StarlingX. Or is this a solved problem that lacks proper documentation? Yes, we discussed some of this last March, but I guess it's still a problem and I think it's only going to become more common. Thoughts, direction welcome. Thanks Sau!
For the second controller and other nodes installing from the active controller, you can set the rootfs_device and boot_device installation parameters via sysinv, either with horizon or the "system host-update" command, such as: system host-update 2 personality=controller rootfs_device=/dev/nvme0n1 boot_device=/dev/nvme0n1 You can see these settings via host-show: system host-show controller-1 -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Monday, January 13, 2020 4:03 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Using NVME Disks instead of SD Folks, I have been trying to get a NUC based deployment work, with Erich's help we managed to start with a pair of Skull Canyon NUCs running Duplex controllers. This NUC has a pair of Ethernet ports and can be configured with 2 NVME-based disks. I was able to get the devices booted initially with changes to the ISO first by hardcoding /dev/nvme0n1 in metal/bsp-files, later Don's tool helped address alternative boot_device and rootfs_device setting, but I am not sure it went far enough. Erich figured out that we needed to patch the sysinv sqlDB (by hand) in order to unlock and make the second controller usable. I went a little further and tried to hook up both work and storage nodes to create a 2+2+2 Standard lab, I had to hand edit the PXE boot cmdline and also do the sysinv SQL magic to unlock the compute and storage nodes, but the Storage configuration still failed. I am thinking there is still some hardcode "sda" and/or "sdb" in places. Any thoughts on how to enable the Storage nodes properly? I can provide logs, configs, ... I realize this is a problem at initial ISO boot/install to select the drive type, but from there is should be automagically detected on the target, so it would require setting the drive type of the host at the same time as setting the personality so that the anaconda at least knows where to start installing. It may be possible for anaconda to determine the drive time also. Should this be a story, launchpad bug or a Spec to allow use to properly use NVME devices with StarlingX. Or is this a solved problem that lacks proper documentation? Yes, we discussed some of this last March, but I guess it's still a problem and I think it's only going to become more common. Thoughts, direction welcome. Thanks Sau! _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Ok, so I keep trying some different configurations, I went back to basics and tried a simplex setup and tried to run the set 6 from the AIO simplex docs page [0]. Output of that step:
+ system host-disk-list controller-0 +--------------------------------------+--------------+------------+-------------+----------+---------------+-----+------------------+-------------------------------------------+ | uuid | device_node | device_num | device_type | size_gib | available_gib | rpm | serial_id | device_path | +--------------------------------------+--------------+------------+-------------+----------+---------------+-----+------------------+-------------------------------------------+ | 9e5c01a9-d409-4df2-bc37-b88cbb80f403 | /dev/nvme0n1 | 66304 | NVME | 476.939 | 0.0 | N/A | BTNH907202MD512A | /dev/disk/by-path/pci-0000:72:00.0-nvme-1 | | c34500df-09d2-497c-a90b-dba1f976bfd0 | /dev/nvme1n1 | 66305 | NVME | 476.939 | 476.937 | N/A | BTNH9072074T512A | /dev/disk/by-path/pci-0000:73:00.0-nvme-1 | +--------------------------------------+--------------+------------+-------------+----------+---------------+-----+------------------+-------------------------------------------+ + system host-disk-list controller-0 + awk '/\/dev\/nvme0n1/{print $2}' + xargs -i system host-stor-add controller-0 '{}' Please install storage-0 or configure a Ceph monitor on a worker node before adding storage devices. + system host-stor-list controller-0
So it seems to require a storage-0 or ceph monitor, which is not mentioned in the docs. Is there something I am missing here? [0] https://docs.starlingx.io/deploy_install_guides/r3_release/bare_metal/aio_si... Sau! On 1/14/20 6:22 AM, Penney, Don wrote:
For the second controller and other nodes installing from the active controller, you can set the rootfs_device and boot_device installation parameters via sysinv, either with horizon or the "system host-update" command, such as:
system host-update 2 personality=controller rootfs_device=/dev/nvme0n1 boot_device=/dev/nvme0n1
You can see these settings via host-show: system host-show controller-1
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Monday, January 13, 2020 4:03 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Using NVME Disks instead of SD
Folks,
I have been trying to get a NUC based deployment work, with Erich's help we managed to start with a pair of Skull Canyon NUCs running Duplex controllers. This NUC has a pair of Ethernet ports and can be configured with 2 NVME-based disks.
I was able to get the devices booted initially with changes to the ISO first by hardcoding /dev/nvme0n1 in metal/bsp-files, later Don's tool helped address alternative boot_device and rootfs_device setting, but I am not sure it went far enough. Erich figured out that we needed to patch the sysinv sqlDB (by hand) in order to unlock and make the second controller usable.
I went a little further and tried to hook up both work and storage nodes to create a 2+2+2 Standard lab, I had to hand edit the PXE boot cmdline and also do the sysinv SQL magic to unlock the compute and storage nodes, but the Storage configuration still failed. I am thinking there is still some hardcode "sda" and/or "sdb" in places.
Any thoughts on how to enable the Storage nodes properly? I can provide logs, configs, ...
I realize this is a problem at initial ISO boot/install to select the drive type, but from there is should be automagically detected on the target, so it would require setting the drive type of the host at the same time as setting the personality so that the anaconda at least knows where to start installing. It may be possible for anaconda to determine the drive time also.
Should this be a story, launchpad bug or a Spec to allow use to properly use NVME devices with StarlingX. Or is this a solved problem that lacks proper documentation? Yes, we discussed some of this last March, but I guess it's still a problem and I think it's only going to become more common.
Thoughts, direction welcome.
Thanks Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (2)
-
Penney, Don
-
Saul Wold