The test case for StarlingX CEPH upgrade we discussed, I have file link for review:
https://ethercalc.openstack.org/orb83xruwmo8
+ starlingx-discuss for collecting feedback from community…
Thanks,
Tingjie
From: Badea, Daniel [mailto:Daniel.Badea@windriver.com]
Sent: Wednesday, March 27, 2019 5:52 PM
To: Chen, Tingjie <tingjie.chen@intel.com>
Cc: Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Poncea, Ovidiu <Ovidiu.Poncea@windriver.com>;
Jones, Bruce E <bruce.e.jones@intel.com>; Lara, Cesar <cesar.lara@intel.com>
Subject: RE: Discussion about StarlingX test cases in CEPH
Hi Tingjie,
Please note that CEPH_STOR_TIER_04 ("associate services with a new storage tier") will fail because support multiple tiers is currently broken. There is a review
in progress to fix it:
https://review.openstack.org/#/c/632346/3
Best regards,
Daniel B.
From: Chen, Tingjie
Sent: Tuesday, March 26, 2019 4:10 PM
To: Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>
Subject: RE: Discussion about StarlingX test cases in CEPH
Hi Ricardo,
For the test case of IO path, I have setup my environment and dry run, it need network configuration in VM or BM.
Steps:
1/ Make sure your VM/BM can access external network, if yes, ignore the following commands in step 1.
It is needed also in containerized configuration, My VM setting for example, suppose your IP of controller-0 (active controller) is 10.10.10.3
In Host:
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE
Also if you have proxy in VM/BM please don’t forget to set.
2/ Install FIO and related libraries.
[wrsroot@controller-0 ~(keystone_admin)]$ sudo yum update
If you have no repo base list, just find one from network and push into /etc/yum.repos.d/
[wrsroot@controller-0 ~(keystone_admin)]$ sudo yum install fio
3/ Create pool and rbd, start to run fio.
In fio config file (rbd.fio), assign one pool and rbd and you created manually.
[wrsroot@controller-0 ~(keystone_admin)]$ cat rbd.fio
[global]
ioengine=rbd
clientname=admin
pool=test_pool # create pool named test_pool before run fio
rbdname=test_rbd # create rbd named test_rbd (1G in my example) in test_pool before run fio
invalidate=0
rw=randwrite
bs=4k
[rbd_iodepth32]
iodepth=32
Then run the fio:
[wrsroot@controller-0 ~(keystone_admin)]$ fio rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][99.6%][r=0KiB/s,w=4624KiB/s][r=0,w=1156 IOPS][eta 00m:01s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=1694134: Tue Mar 26 00:22:04 2019
write: IOPS=1112, BW=4448KiB/s (4555kB/s)(1024MiB/235722msec)
slat (nsec): min=987, max=13491k, avg=4692.70, stdev=63898.52
clat (usec): min=1688, max=242157, avg=28660.64, stdev=14844.91
lat (usec): min=1705, max=242160, avg=28665.34, stdev=14845.14
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 18],
| 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 29],
| 70.00th=[ 33], 80.00th=[ 38], 90.00th=[ 45], 95.00th=[ 53],
| 99.00th=[ 79], 99.50th=[ 97], 99.90th=[ 169], 99.95th=[ 178],
| 99.99th=[ 213]
bw ( KiB/s): min= 1416, max= 7016, per=99.96%, avg=4446.03, stdev=765.16, samples=471
iops : min= 354, max= 1754, avg=1111.41, stdev=191.28, samples=471
lat (msec) : 2=0.01%, 4=0.01%, 10=1.56%, 20=26.82%, 50=65.29%
lat (msec) : 100=5.89%, 250=0.44%
cpu : usr=0.63%, sys=0.29%, ctx=14440, majf=0, minf=8434
IO depths : 1=1.6%, 2=3.9%, 4=9.6%, 8=23.8%, 16=57.1%, 32=4.1%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=96.3%, 8=0.2%, 16=0.3%, 32=3.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,262144,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=4448KiB/s (4555kB/s), 4448KiB/s-4448KiB/s (4555kB/s-4555kB/s), io=1024MiB (1074MB), run=235722-235722msec
Disk stats (read/write):
sda: ios=2664/10592, merge=0/2020, ticks=2333/40725, in_queue=42859, util=11.94%
And just refer the flow with Read / Write of CEPH internal process.
So it seems the TCs we aligned mostly, what is the next process to final confirm?
J
@Ricardo, will you list the new TCs and dry-run on 2+2+2 configuration firstly?
Thanks,
Tingjie
From: Perez, Ricardo O
Sent: Tuesday, March 26, 2019 5:57 AM
To: Chen, Tingjie <tingjie.chen@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>
Subject: RE: Discussion about StarlingX test cases in CEPH
Hi Tingjie,
Thanks lot for you shared details, please see my embedded answers below.
Regards
-Richo
From: Chen, Tingjie
Sent: Monday, March 25, 2019 8:32 AM
To: Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>
Subject: RE: Discussion about StarlingX test cases in CEPH
Hi Ricardo,
For the test cases we proposed, it seems there are 2 items need to clarify per your comments.
1/ RESTful plugin:
This case is to verify CEPH-MGR via python-client with restful API, we have no automation script and yes the rest API is complex for manual commands.
I can provide the lists can be verified, first step we can go through some GET operations, the case will be passed if information shows normally and correctly.
// get the user and keys.
[wrsroot@controller-0 ~(keystone_admin)]$ ceph restful list-keys
{
"admin": "579b6a27-e019-4887-b0ce-ee2fee6c4134"
}
// get ceph-mgr service endpoint and port
[wrsroot@controller-0 ~(keystone_admin)]$ ceph mgr services
{
"restful": "https://controller-0:5001/"
}
// get the available service link list
[wrsroot@controller-0 ~(keystone_admin)]$ curl -k
https://controller-0:5001/doc
…
// for example, get monitors detail information
[wrsroot@controller-0 ~(keystone_admin)]$
curl -k -u admin:579b6a27-e019-4887-b0ce-ee2fee6c4134 https://controller-0:5001/mon -X GET
[
{
"addr": "192.168.204.3:6789/0",
"in_quorum": true,
"leader": true,
"name": "controller-0",
"public_addr": "192.168.204.3:6789/0",
"rank": 0,
"server": "controller-0"
},
{
"addr": "192.168.204.4:6789/0",
"in_quorum": true,
"leader": false,
"name": "controller-1",
"public_addr": "192.168.204.4:6789/0",
"rank": 1,
"server": "controller-1"
},
{
"addr": "192.168.204.95:6789/0",
"in_quorum": true,
"leader": false,
"name": "storage-0",
"public_addr": "192.168.204.95:6789/0",
"rank": 2,
"server": "storage-0"
}
]
[Perez, Ricardo O] Thanks for sharing the commands and the expected results.
2/ IO path:
The read/write test with fio with rbd engine will go through CEPH full stack, include librados, rbd, osd, mon and messages, this is a comprehensive case and also performance
impact if needed in the future.
I am preparing the FIO environment in StarlingX, since current setup does not support fio by default, more details will provide if have progress.
[Perez, Ricardo O] Then, I believe you are going to share with us when FIO environment is ready as well the steps to be executed right ?
BTW: May I ask your plan for StarlingX deployment?
[Perez, Ricardo O] Sure, by now, when we have to test something related to CEPH, we normally use a 2+2+2 configuration (2 controllers, 2 computes and 2 storage nodes). By now, we are able to use a mix of
network cards, Mellanox / Intel. (in this 2+2+2 config) if more nodes are required we are able just to use the default attached cards in the servers.
Thanks,
Tingjie
From: Perez, Ricardo O
Sent: Saturday, March 23, 2019 5:49 AM
To: Chen, Tingjie <tingjie.chen@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>
Subject: RE: Discussion about StarlingX test cases in CEPH
Hi Tingjie,
See my embedded answers below
From: Chen, Tingjie
Sent: Friday, March 22, 2019 1:58 AM
To: Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com>; Xie, Cindy <cindy.xie@intel.com>;
Zhu, Vivian <vivian.zhu@intel.com>
Subject: Discussion about StarlingX test cases in CEPH
Hi,
Just kick-off a new thread for the discussion about CEPH test case.
1/ Previously Ada has share the link:
https://docs.google.com/spreadsheets/d/1O2zWn-R83Wj1SqmeUtxCP59_DsSM0UNZW0gRALnDn6w/edit?usp=sharing
We have some detailed comments on some cases, as purple color.
also there are questions for the test plan:
a/ We execute test suites, do you have test framework/script or use commands directly?
[Perez, Ricardo O] Currently we execute commands directly.
b/ In case of failure, how to decide whether it is blocking issue, maybe we can define the priority of test cases, how do you think?
[Perez, Ricardo O] As part of the validation conventions, normally a blocking issue its one of these things: Block you to enable / disable some feature of your software, it fails in the way that there
is no way to recover the original state, it’s impossible to perform some step required to enable a further feature. About priority of the tests, sure, we can define it.
|
Test ID |
Test Name |
Test Objective |
Expected behavior |
Comments |
|
1 |
CEPH_STOR_TIER_01 |
The objective of this test is to ensure that a new storage tier can be created. |
Additional storage tier is successfully created. |
Scenario of storage tier is not used frequently, espacially when use SSD since no need to cache pool (Tingjie).
[Perez, Ricardo O] I agree that might not be a widely used scenario, but as the feature is there, we should test it. |
|
2 |
CEPH_STOR_TIER_02 |
The objective of this test is to ensure that a new storage tier can be associated with an OSD. |
Storage tier is successfully associated with OSD |
|
|
3 |
CEPH_STOR_TIER_03 |
The objective of this test is to ensure that a new storage tier can be associated with a backend. |
Storage tier can be successfully associated with a backend |
|
|
4 |
CEPH_STOR_TIER_04 |
The objective of this test is to ensure you can associate services with a new storage tier. |
The new storage tier can be used. |
|
|
5 |
CEPH_STOR_REP_05 |
The objective of this test is to ensure you can provision the system to have replication factor 3. |
After replication factor 3 is enabled, there are 3 copies of the data present on the system. |
Performance can vary wildly amount different Ceph clusters, it all depends on what the replication factor is set to. With a replication factor of 2 you will see roughly half
the write performance compared to a replication factor of 1. The drop in write performance between replication factor 2 and 3 is also pretty dramatic. This is not surprising since replication takes time and you must wait for multiple OSDs to complete a write
instead of just one. [Perez, Ricardo O] At this point, I beleive the intention of the test isn’t to verify the data. Just to see that the system are still functional no matter which replication factor you use. This for sure
will impact in the performance but this is out of the scope of the test. |
|
6 |
CEPH_STOR_SWI_06 |
The objective of this test is to ensure you can enable swift on the system. |
Swift should be successfully enabled at the end of this test. |
|
|
7 |
CEPH_STOR_PROC_07 |
The objective of this test is to repeatedly kill the ceph monitor process and ensure they are restarted by the system. |
The ceph monitor processes should alarm when expected, and should recover when killed. |
|
|
8 |
CEPH_STOR_OSD_08 |
The objective of this test is to repeatedly kill the ceph osd process and ensure they are restarted by the system. |
The ceph osd processes should alarm when expected, and should recover when killed. |
|
|
9 |
CEPH_STOR_SCALABILITY_09 |
The objective of this test is to test the basic provisioning procedure for 8 storage node ceph systems.
|
The system is properly configured and functioning as expected at the end of the test. |
Since currently 2-9x node CEPH storage cluster support, it is fine, but we have not deploy so many yet, maybe dry-run first. (Tingjie).
[Perez, Ricardo O] This is i show the original test was defined, for sure we can adjust, normally we use 2 storage nodes for BM. |
|
10 |
CEPH_STOR_CORE_10 |
The objective of this test is to ensure that host reinstall of nodes running ceph-mon works properly on all supported configs. |
Ceph should be healthy at the end of the test. |
the robustness test should strip the influence of residual data/config, propose one Precondition: clean deployment when each test (Tingjie). [Perez, Ricardo O] This test is just to ensure that a host can be re-installed as many times as required, if you believe such pre-condition is required we can add it. However in the real world this should
work flawlessly despite of the state of the system. |
|
11 |
CEPH_STOR_CORE_11 |
The objective of this test is to ensure that host delete and reprovision of nodes running ceph-mon works properly on all supported configs. |
Ceph should be healthy at the end of the test. |
|
|
12 |
CEPH_STOR_CORE_12 |
The objective of this test is to ensure that semantic checks with respect to node lock, work properly on nodes running ceph monitors. |
Semantic checks should work as expected. |
Not sure the meaning of semantic check about node lock. (Tingjie). [Perez, Ricardo O] Semantic check means basically that if you lock / unlock or perform any action to an specific node, the “system” checks their current state (semantically) and allows / deny such operations
depending on the defined state. Please check the detailed steps for the tests here: https://review.openstack.org/#/c/640546/1/manual-tests/storage/storage_regression_test_plan.rst The original test name is STOR_CORE_016 |
|
13 |
CEPH_STOR_CORE_13 |
The objective of this test is to ensure that the user can provision SSD journals. |
It should be possible to modify the journal configuration on the SSD disks. |
Journals is related on Filestore only and removed in Bluestore. CEPH mimic by default support Bluestore but in StarlingX still use Filestore since puppet issues, maybe switch
to Bluestore in future, just share the information. (Tingjie) |
|
14 |
CEPH_STOR_HW_14 |
The objective of this test is to ensure that the hardware disk replacement procedure for OSDs is accurate. |
The system should be functional and healthy after hardware disk replacement. |
|
|
15 |
CEPH_STOR_HW_15 |
The objective of this test is to ensure that the hardware disk replacement procedure for journal disks is accurate. |
The system should be functional and healthy after hardware disk replacement. |
|
|
16 |
CEPH_STOR_DOR_16 |
To verify the system recovers after a DOR test (dead-office-recovery). |
Storage system recovers after DOR test |
Not sure the context of DOR test: dead-office-recovery means? (Tingjie). [Perez, Ricardo O] The context is, all nodes shutdown ( by a power outage or any other issue), running VMs on compute nodes, after all came back, we have that VMs are still able to resume ping and their
attached storages are still funcional. |
|
17 |
CEPH_STOR_FAULT_17 |
To verify the system can recover when there is a cable pull on the cluster network. |
Storage system recovers after cable pull |
Any operations defined when you pull out the cable? (Tingjie). [Perez, Ricardo O] I’m not quite sure what do you mean with “operations”?, but here what we want is to see if after a physically disconnection and connection , the system are still working in an stable
way, and able to continue providing services. |
|
18 |
CEPH_STOR_FS_18 |
To verify that the sizes of the ceph pools can be modified. |
It should be possible for the user to change the size of the ceph pools |
Do you mean the max size of ceph pools, can set quota. or PGs ? or warning log threshold? (Tingjie). [Perez, Ricardo O] 'ceph osd pool get-quota <poolname>' |
|
19 |
CEPH_STOR_OPROF_19 |
This test validates the creation and application of storage profiles on a system. |
It should be possible to apply an existing storage porfile to a new node |
What is the form of storage profile, do you have any script or tools to apply the profile for creation and application? (Tingjie). [Perez, Ricardo O] A storage profile is a configuration file that is saved using Horizon. So, no script is required. You can use such profile to setup a new node with the same configuration (if required). |
|
20 |
CEPH_STOR_PART_20 |
This test validates that multiple partitions can be created and the partition modification/deletion behaviour is correct |
Partition creation, deletion and semantic checks should work as expected. |
Partition on Disk? You means deploy in bare-metal or Disk in Virtual Machine? (Tingjie). [Perez, Ricardo O] By now, BM. |
|
21 |
CEPH_STOR_FS_21 |
To ensure that the size of ceph-mon can be increased. |
The size should be increased on both controllers |
Actually we have no interface to resize the ceph-mon, or you means the warning threshold percentage? (Tingjie). [Perez, Ricardo O] 'system ceph-mon-modify <node> ceph_mon_gib=<value>' |
2/ Besides we have proposed cases with CEPH functionality interface coverage.
|
TC |
Module |
Commands |
Description |
|
ceph_status |
TOTAL |
ceph -s |
Ceph total status and health check [Perez, Ricardo O] I believe this is already included in the upper list. |
|
io_path |
TOTAL |
fio xxx.conf |
Read/Write test. [Perez, Ricardo O] This is a good one, however, this Will allow us to see the IO of the Disk, not CEPH by itself, however I would like to hear the details about this test. |
|
osd_add/remove/tree |
OSD |
ceph osd add/destroy/tree |
osd common operations and verify after each commands. [Perez, Ricardo O] This are already included in the upper list. |
|
pool_create/modify/list |
Pool |
ceph osd create test-pool 64 3 |
ceph pool common operations and verify after each commands, pool can be increased PGs.. [Perez, Ricardo O] This look ok. |
|
mon_operation/status |
MON |
|
ceph mon operations: increase (add)/decrease (kill) and status check. [Perez, Ricardo O] Already included |
|
radosgw_status |
RADOSGW |
|
merge Ada's cases with swift: case 6 [Perez, Ricardo O] No comment here. |
|
restful_pugin operations |
MGR |
|
restful interface common operations [Perez, Ricardo O] Do you have scripts or tools to do this ? Rest API tests are quite complex. |
|
rbd_create/delete/resize |
RBD |
rbd create --size 10G test_pool/test_rbd; |
create and delete operation, and verify status [Perez, Ricardo O] Looks ok to me. |
|
rbd_snapshot |
rbd snap create test_pool/test_rbd@test_rbd_snap |
snapshot operations. [Perez, Ricardo O] Looks ok to me. |
Thanks,
Tingjie
SSG OTC NST Storage
Tel: +86(21)88216699
Mobile: 15901876439