We have Raid 5 configuration at our setup on which STX 4.0 is installed.
While extending the XFS filesystem for docker-lv before uploading stx-openstack application
Command:
system host-fs-modify controller-0 docker=60
As we run this command, we get kernel alert logs in /var/log/kern.log/
2020-12-16T13:23:40.467 controller-0 kernel: alert [ 921.174512] XFS (dm-5): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x77fc002 len 1 error 74
2020-12-16T13:23:41.715 controller-0 kernel: alert [ 922.488879] XFS (dm-5): Metadata CRC error detected at xfs_agi_read_verify+0xd0/0xf0 [xfs], xfs_agi block 0x77fc002
2020-12-16T13:23:41.715 controller-0 kernel: alert [ 922.498188] XFS (dm-5): Unmount and run xfs_repair
2020-12-16T13:23:41.715 controller-0 kernel: alert [ 922.507214] XFS (dm-5): First 128 bytes of corrupted metadata buffer:
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516545] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516547] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516548] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516549] 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516551] 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516552] 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516553] 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
2020-12-16T13:23:41.809 controller-0 kernel: alert [ 922.516555] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
In order to avoid this, we thought to initially set the size of docker-lv filesystem to 60 instead of increasing it from 30 to 60
For this, before bootstrap C0, we changed the following:
1. lv_size changed from 30 to 60 in file /usr/share/puppet/modules/platform/manifests/filesystem.pp
class platform::filesystem::docker::params::bootstrap (
$lv_size = '60',
$lv_name = 'docker-lv',
$mountpoint = '/var/lib/docker',
$devmapper = '/dev/mapper/cgts--vg-docker--lv',
$fs_type = 'xfs',
$fs_options = '-n ftype=1',
$fs_use_all = false
) { }
2. Value of
KUBERNETES_DOCKER_STOR_SIZE at line 39 changed to 60 from 30 in file /usr/lib64/python2.7/site-packages/sysinv/common/constants.py
KUBERNETES_DOCKER_STOR_SIZE = 60
With these changes, after bootstrap of Controller-0, the size of docker-lv changes to 60 successfully.
The same is verified running the command "system host-fs-list controller-0"
But once Controller-1 bootstraps and comes up, we still get the size of docker-lv as 30.
Since Controller-0 PXE boots Controller-1, there must be a file at C0 which we could change so that Controller-1's docker-lv gets a size of 60G.
We are unable to find such a file and would require help in order to resolve this.