[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

nicholasyang2022 · 2024-12-13T07:35:41Z

This pull request implements pre-migration checks for SLES 16. These checks are expected to run before migrating to SLES 16.

These checks ensure all the cluster nodes runs the latest version of corosync 2.x and pacemaker 2.1.x, and report used resource agents and fence agents that will be removed in SLES 16. It also provides advice for further actions need to take after migration.

Use Cases

`crm cluster health sles16`

suse@ha-1-1:~> sudo crm cluster health sles16
------ corosync @ ha-1-2 ------
[WARN] Cluster services are not running. Check results may be outdated or inaccurate.
       * corosync
       * pacemaker
[WARN] Corosync transport "udpu" will be deprecated in corosync 3. Please use "knet".

------ corosync @ localhost ------
[INFO] Checking dependency version...
[FAIL] Pacemaker version not supported
       Supported version: 2.1.7 <= Pacemaker < 3
       Actual version:    Pacemaker == 2.1.5
[INFO] Checking service status...
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3. Please use "knet".
------ cib ------
[INFO] Checking used resource agents...
[FAIL] SAPHanaSR Classic will be removed in SLES 16.
       Before migrating to SLES 16, replace it with SAPHanaSR-angi.
[WARN] stonith:external/sbd will be removed. Please use stonith:fence_sbd

****** summary ******
[FAIL] The pacemaker cluster stack can not migrate to SLES 16. Please fix all the "FAIL" problems above before migrating to SLES 16.

`crm cluster health sles16 --fix`

> sudo crm cluster health sles16 --fix
ERROR: "--fix" is only available in SLES 16.

When some of the nodes are offline

------ corosync @ ha-1-2 ------
Cannot create SSH connection to root@ha-1-2: ssh: connect to host ha-1-2 port 22: No route to host

------ corosync @ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3. Please use "knet".
------ cib ------
[INFO] Checking used resource agents...

****** summary ******
[FAIL] The pacemaker cluster stack can not migrate to SLES 16. Please fix all the "FAIL" problems above before migrating to SLES 16.

codecov · 2024-12-16T06:54:24Z

Codecov Report

Attention: Patch coverage is 82.72059% with 47 lines in your changes missing coverage. Please review.

Project coverage is 67.01%. Comparing base (f5b1328) to head (7670957).
Report is 27 commits behind head on 4.6-pre-migration-staging.

Files with missing lines	Patch %	Lines
crmsh/migration.py	83.03%	38 Missing ⚠️
crmsh/ui_cluster.py	75.67%	9 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	`52.97% <82.72%> (+0.31%)`	⬆️
unit	`49.22% <29.04%> (-0.19%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/cibquery.py	`100.00% <100.00%> (ø)`
crmsh/sh.py	`93.22% <ø> (ø)`
crmsh/xmlutil.py	`69.23% <100.00%> (+0.03%)`	⬆️
crmsh/ui_cluster.py	`74.21% <75.67%> (-0.65%)`	⬇️
crmsh/migration.py	`83.03% <83.03%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

liangxin1300 · 2025-01-01T14:01:02Z

crmsh/migration-supported-resource-agents.txt

+stonith:fence_vmware-rest
+stonith:fence_wti
+stonith:fence_xenapi
+stonith:fence_zvm


How has the above data been generated?
Can we automatically do that based on the locally installed resource-agents and fence-agents-* rpm?
Or, can we generate the data at the time crmsh using it (_load_supported_resource_agents)?

How has the above data been generated?

Extracted the file list from resource-agents/fence-agents.

Can we automatically do that based on the locally installed resource-agents and fence-agents-* rpm?

No, because we need to know what is supported by SLES 16 instead of SLES 15 SP6.

liangxin1300 · 2025-01-03T05:41:14Z

Suggestion:

1. Add completer for `crm cluster health`, now `hawk2` can be completed, so do `sles16`.

The same point for #1422

2. Add success info

# crm cluster health sles16
------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[PASS]

How about change the last line as

[PASS] All checks completed successfully.

Or

[INFO] All checks completed successfully.

I prefer this [INFO] one, since [PASS] is not the log level
The same for #1422

3. Add failure error

# crm cluster health sles16
------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[FAIL] Cluster services are not running
       * corosync
       * pacemaker
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[FAIL]

How about change the last line similar:

[ERROR] Some checks failed, please check

And, change [FAIL] Cluster services are not running
as

[ERROR] Cluster services are not running

The same for #1422

4. Add above output into `/var/log/crmsh/crmsh.log` log file, too

The same for #1422

5. Keep the color of node line consistency

Better keep all of node lines white (no color)?

The same for #1422

liangxin1300 · 2025-01-03T06:27:45Z

Suggestion:

6. How about there are Failed Resource Actions?

For some reason, some RA failed. For now, crm cluster health sles16 seems quiet for such an error.
I think we should hint the user to address that failed RA, then do the migration job
The same for #1422

7. crm report collect pre-migration info

I think we also need to collect the output of crm cluster health sles16 in crm report.
How about implementing a collect_pre_migraion function in report.collect.py, and dump the output into pre-migration.txt?

8. Inaccurate usage?

# crm cluster health sles15
usage: crm [-h] [-f] {hawk2,sles16}
crm: error: argument component: invalid choice: 'sles15' (choose from 'hawk2', 'sles16')

Should be

usage: crm cluster health {hawk2,sles16}

The same for #1422

9. Fence agents checking

[FAIL] The following fence agents will be removed in SLES 16.
       * stonith:external/sbd

Better to add what's the alternative?

[ERROR] The following fence agents will be replaced
      - stonith:external/sbd
      + stonith:fence_sbd

crmsh/migration.py

…c#PED-11808)

…jsc#PED-11808)

nicholasyang2022 · 2025-01-13T07:06:30Z

Usages changed to

suse@ha-1-1:~> sudo crm cluster health foo
usage: health [-h] [-f] {hawk2,sles16}
health: error: argument component: invalid choice: 'foo' (choose from 'hawk2', 'sles16')

suse@ha-1-1:~> sudo crm cluster health sles16 foo
usage: sles16 [-h] [--json [{oneline,pretty}]] [--local]
sles16: error: unrecognized arguments: foo

liangxin1300 · 2025-01-13T06:45:23Z

crmsh/migration.py

+    handler.handle_tip(f'Corosync transport "{transport}" will be deprecated in corosync 3.', [
+        'After migrating to SLES 16, run "crm cluster health sles16 --fix" to migrate it to transport "knet".',
+    ])
+    if corosync.get_value("totem.rrp_mode") in {'active', 'passive'}:


Although with low possibility, rrp_mode can be set to none for the one-ring
should include none value here

none means SRP, which is supported and does not needs any actions.

doc/crm.8.adoc

…PED-11808) should use recursive queries

…1808)

…ilesystem_with_fstype

crmsh/migration.py

liangxin1300

Looks good to me, great work!

Thanks!

zzhou1

Great work, some feedback,

Reporting the CIB related check results in "localhost" section is misleading, since they are concepts at the cluster level when the cluster is running.
How to interpret the cluster is down? Since in such case, the CIB check is kind of local. Hence, the related open question is should CIB check run in the remote node?
"Good to migrate to SLES 16." on ha-1-2 is not right, while CIB check is "FAIL" at the cluster level in the summary section.
Sounds the following information is duplicate in my view

------ localhost ------
...
[FAIL] Please fix all the "FAIL" problems above before migrating to SLES 16.

****** summary ******
[FAIL] Please fix all the "FAIL" problems above before migrating to SLES 16.

zzhou1 · 2025-01-16T09:13:47Z

Wonderful, thanks for the great work!
Let's merge this PR into the development branch instead of the official 4.6 branch for the timebeing.

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 70f74a6 to 1a80eee Compare December 16, 2024 06:23

nicholasyang2022 changed the title ~~[crmsh] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252) Dec 16, 2024

zzhou1 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-11808) Dec 17, 2024

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 2 times, most recently from d9038c6 to a2518dd Compare December 24, 2024 05:34

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 7 times, most recently from a740b3f to 2bcad35 Compare December 31, 2024 09:28

liangxin1300 self-requested a review January 1, 2025 13:54

liangxin1300 reviewed Jan 1, 2025

View reviewed changes

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 3 times, most recently from 2932b25 to 8cf8ef3 Compare January 2, 2025 00:15

nicholasyang2022 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-11808)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 and pacemaker 3 (jsc#PED-11808) Jan 2, 2025

nicholasyang2022 marked this pull request as ready for review January 2, 2025 00:57

nicholasyang2022 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 and pacemaker 3 (jsc#PED-11808)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) Jan 2, 2025

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 4 times, most recently from 22ea8a0 to a338bc6 Compare January 2, 2025 08:57

liangxin1300 reviewed Jan 3, 2025

View reviewed changes

crmsh/migration.py Outdated Show resolved Hide resolved

liangxin1300 reviewed Jan 3, 2025

View reviewed changes

crmsh/migration.py Outdated Show resolved Hide resolved

Dev: ui_cluster: add 'crm cluster health sles [--fix]' (jsc#PED-8252)

4488385

nicholasyang2022 added 2 commits January 13, 2025 14:37

Dev: pre-migration: do not colorize hostname header (jsc#PED-11808)

14f5b2e

Dev: pre-migration: add completer for 'crm cluster health sles16' (js…

4dae2e3

…c#PED-11808)

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 2dff378 to d1563aa Compare January 13, 2025 06:41

nicholasyang2022 added 3 commits January 13, 2025 15:05

Fix: ui_cluster: fix do_health usage output

22fd915

Dev: pre-migration: add message about removing stonith:external/sbd (…

abbbc3e

…jsc#PED-11808)

Dev: pre-migration: refine the message for stopped cluster services (…

843d637

…jsc#PED-11808)

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from d1563aa to d9c0449 Compare January 13, 2025 07:05

liangxin1300 reviewed Jan 13, 2025

View reviewed changes

doc/crm.8.adoc Show resolved Hide resolved

nicholasyang2022 added 4 commits January 13, 2025 16:54

Doc: ui_cluster: add document for crm cluster health hawk2|sles16

d8b62b1

Fix: cibquery: grouped primitives is missing from query results (jsc#…

2368846

…PED-11808) should use recursive queries

Dev: behave: add functional tests for pre-migration checks (jsc#PED-1…

b6b8987

…1808)

Refactor: cibquery: has_primitive_filesystem_ocfs2 to has_primitive_f…

26ccee4

…ilesystem_with_fstype

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from d9c0449 to 94926c5 Compare January 13, 2025 08:54

liangxin1300 reviewed Jan 13, 2025

View reviewed changes

crmsh/migration.py Outdated Show resolved Hide resolved

liangxin1300 approved these changes Jan 13, 2025

View reviewed changes

liangxin1300 requested a review from zzhou1 January 13, 2025 12:30

liangxin1300 mentioned this pull request Jan 13, 2025

crm report collect pre-migration info #1664

Closed

Dev: pre-migration: add summary section to output (jsc#PED-11808)

252befd

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 2 times, most recently from 8fc0558 to b51ec24 Compare January 15, 2025 08:45

zzhou1 requested changes Jan 15, 2025

View reviewed changes

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from b51ec24 to abe07c3 Compare January 16, 2025 07:39

Dev: migration: refine message wording (jsc#PED-11808)

7670957

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from abe07c3 to 7670957 Compare January 16, 2025 07:58

nicholasyang2022 changed the base branch from crmsh-4.6 to 4.6-pre-migration-staging January 16, 2025 09:24

nicholasyang2022 merged commit 97fa52b into ClusterLabs:4.6-pre-migration-staging Jan 16, 2025
33 checks passed

nicholasyang2022 deleted the ped-8252-20241204-4.6 branch January 16, 2025 09:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

nicholasyang2022 commented Dec 13, 2024 •

edited

Loading

codecov bot commented Dec 16, 2024 •

edited

Loading

liangxin1300 Jan 1, 2025 •

edited

Loading

nicholasyang2022 Jan 1, 2025

liangxin1300 commented Jan 3, 2025 •

edited

Loading

liangxin1300 commented Jan 3, 2025 •

edited

Loading

nicholasyang2022 commented Jan 13, 2025

liangxin1300 Jan 13, 2025 •

edited

Loading

nicholasyang2022 Jan 13, 2025

liangxin1300 left a comment

zzhou1 left a comment •

edited

Loading

zzhou1 commented Jan 16, 2025

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

Conversation

nicholasyang2022 commented Dec 13, 2024 • edited Loading

Use Cases

crm cluster health sles16

crm cluster health sles16 --fix

When some of the nodes are offline

codecov bot commented Dec 16, 2024 • edited Loading

Codecov Report

liangxin1300 Jan 1, 2025 • edited Loading

Choose a reason for hiding this comment

nicholasyang2022 Jan 1, 2025

Choose a reason for hiding this comment

liangxin1300 commented Jan 3, 2025 • edited Loading

Suggestion:

1. Add completer for crm cluster health, now hawk2 can be completed, so do sles16.

2. Add success info

3. Add failure error

4. Add above output into /var/log/crmsh/crmsh.log log file, too

5. Keep the color of node line consistency

liangxin1300 commented Jan 3, 2025 • edited Loading

6. How about there are Failed Resource Actions?

7. crm report collect pre-migration info

8. Inaccurate usage?

9. Fence agents checking

nicholasyang2022 commented Jan 13, 2025

liangxin1300 Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

nicholasyang2022 Jan 13, 2025

Choose a reason for hiding this comment

liangxin1300 left a comment

Choose a reason for hiding this comment

zzhou1 left a comment • edited Loading

Choose a reason for hiding this comment

zzhou1 commented Jan 16, 2025

nicholasyang2022 commented Dec 13, 2024 •

edited

Loading

`crm cluster health sles16`

`crm cluster health sles16 --fix`

codecov bot commented Dec 16, 2024 •

edited

Loading

liangxin1300 Jan 1, 2025 •

edited

Loading

liangxin1300 commented Jan 3, 2025 •

edited

Loading

1. Add completer for `crm cluster health`, now `hawk2` can be completed, so do `sles16`.

4. Add above output into `/var/log/crmsh/crmsh.log` log file, too

liangxin1300 commented Jan 3, 2025 •

edited

Loading

liangxin1300 Jan 13, 2025 •

edited

Loading

zzhou1 left a comment •

edited

Loading