ci: Add test plan to run CI tests

inventory/group_vars/active_roles.yml

@@ -27,6 +27,8 @@ present_templates:
   - .github/workflows/weekly_ci.yml
   - .github/workflows/woke.yml
   - README-ansible.md
+  - plans/README-plans.md
+  - plans/test_playbooks_parallel.fmf
   - tests/vars/rh_distros_vars.yml
 absent_files:
   - .github/workflows/ansible-plugin-scan.yml
@@ -41,9 +43,6 @@ absent_files:
   - .github/actions/custom-woke-action/testdata/bad.txt
   - .github/actions/custom-woke-action
   - .github/actions
-  - plans/README-plans.md
-  - plans/general.fmf
-  - plans
 github_org: linux-system-roles
 github_url_prefix: "https://github.com/"
 default_ansible_lint:

inventory/host_vars/certificate.yml

@@ -5,3 +5,5 @@ github_actions:
   codeql:
     schedule:
       - cron: "20 17 * * 4"
+tmt_hardware:
+  memory: ">= 4096 MB"

inventory/host_vars/mssql.yml

@@ -6,3 +6,5 @@ ansible_lint:
   skip_list:
     - galaxy[no-changelog]
     - galaxy[no-runtime]
+tmt_hardware:
+  memory: ">= 4096 MB"

inventory/host_vars/network.yml

@@ -21,3 +21,7 @@ lsr_rh_distros_extra: [OracleLinux]
 codespell_skip:
   - ./tests/files/client.key.nocrypt
   - ./tests/files/client.pem
+tmt_hardware:
+  network:
+    - type: eth
+    - type: eth

playbooks/templates/.github/workflows/tft.yml

@@ -164,7 +164,6 @@ jobs:
         uses: sclorg/testing-farm-as-github-action@v3
         if: contains(needs.prepare_vars.outputs.supported_platforms, matrix.platform)
         with:
-          git_url: https://github.com/linux-system-roles/tft-tests
           git_ref: main
           pipeline_settings: '{ "type": "tmt-multihost" }'
           environment_settings: '{ "provisioning": { "tags": { "BusinessUnit": "system_roles" } } }'
@@ -188,22 +187,13 @@ jobs:
           api_key: ${{ secrets.TF_API_KEY_RH }}
           update_pull_request_status: false
 {%- endraw +%}
-{%- if inventory_hostname == "network" %}
+          tmt_plan_filter: "tag:playbooks_parallel,{{ inventory_hostname }}"
 {%- raw %}
-          tmt_hardware: '{ "memory": ">= ${{ needs.prepare_vars.outputs.memory }} MB", "network": [{"type": "eth"},{"type": "eth"}] }'
-{%- endraw +%}
-{% else %}
-{%- raw %}
-          tmt_hardware: '{ "memory": ">= ${{ needs.prepare_vars.outputs.memory }} MB" }'
-{%- endraw +%}
-{%- endif %}
-          tmt_plan_filter: "tag:general,{{ inventory_hostname }}"
 
       - name: Set final commit status
         uses: myrotvorets/set-commit-status-action@master
         if: always() && contains(needs.prepare_vars.outputs.supported_platforms, matrix.platform)
         with:
-{%- raw %}
           sha: ${{ needs.prepare_vars.outputs.head_sha }}
           status: ${{ job.status }}
           context: ${{ matrix.platform }}|ansible-${{ matrix.ansible_version }}

playbooks/templates/plans/README-plans.md

@@ -1,30 +1,55 @@
 # Introduction CI Testing Plans
 
-Linux System Roles CI runs [tmt](https://tmt.readthedocs.io/en/stable/index.html) test plans in [Testing farm](https://docs.testing-farm.io/Testing%20Farm/0.1/index.html) with the [tmt.yml](https://github.com/linux-system-roles/{{ inventory_hostname }}/blob/main/.github/workflows/tmt.yml) GitHub workflow.
+Linux System Roles CI runs [tmt](https://tmt.readthedocs.io/en/stable/index.html) test plans in [Testing farm](https://docs.testing-farm.io/Testing%20Farm/0.1/index.html) with the [tft.yml](https://github.com/linux-system-roles/{{ inventory_hostname }}/blob/main/.github/workflows/tft.yml) GitHub workflow.
 
-The plans/general.fmf plan is a test plan that is general for all roles. It does the following steps:
+The `plans/test_playbooks_parallel.fmf` plan is a test plan that runs test playbooks in parallel on multiple managed nodes.
+`plans/test_playbooks_parallel.fmf` is generated centrally from `https://github.com/linux-system-roles/.github/`.
+The automation calculates the number of managed nodes to provision with this formula:
 
-1. Provisions two machines, one used as an Ansible control node, and second used as a managed node.
-2. Does the required preparation on machines.
+```plain
+number-of-test-playbooks / 10 + 1
+```
+
+The `plans/test_playbooks_parallel.fmf` plan does the following steps:
+
+1. Provisions systems to be used as a control node and as managed nodes.
+2. Does the required preparation on systems.
 3. For the given role and the given PR, runs the general test from [test.sh](https://github.com/linux-system-roles/tft-tests/blob/main/tests/general/test.sh).
 
-The [tmt.yml](https://github.com/linux-system-roles/{{ inventory_hostname }}/blob/main/.github/workflows/tmt.yml) workflow runs the above plan and uploads the results to our Fedora storage for public access.
+The [tft.yml](https://github.com/linux-system-roles/{{ inventory_hostname }}/blob/main/.github/workflows/tft.yml) workflow runs the above plan and uploads the results to our Fedora storage for public access.
 This workflow uses Testing Farm's Github Action [Schedule tests on Testing Farm](https://github.com/marketplace/actions/schedule-tests-on-testing-farm).
 
 ## Running Tests
 
-You can run tests locally with the `tmt try` cli.
-
-### Prerequisites
-
-* Install `tmt` as described in [Installation](https://tmt.readthedocs.io/en/stable/stories/install.html).
+You can run tests locally with the `tmt try` cli or remotely in Testing Far.
 
 ### Running Tests Locally
 
-To run tests locally, in the role repository, enter `tmt try -p plans/general <platform>`.
-
-This command identifies the plans/general plan and provisions two local VMs, one used as an Ansible control node, and second used as a managed node.
-
-tmt try is in development and does not identify tests from URL automatically, so after provisioning the machines, you must type `t`, `p`, `t` from the interactive prompt to identify tests, run preparation steps, and run the tests.
-
-You can modify environment variables in  `plans/general.fmf` to, e.g. run only specified test playbooks by overwriting `SYSTEM_ROLES_ONLY_TESTS`.
+1. Install `tmt` as described in [Installation](https://tmt.readthedocs.io/en/stable/stories/install.html).
+2. Change to the role repository directory.
+3. Modify `plans/test_playbooks_parallel.fmf` to suit your requirements:
+    1. Due to [issue #3138](https://github.com/teemtee/tmt/issues/3138), comment out all managed nodes except for one.
+    2. Optionally modify environment variables to, e.g. run only specified test playbooks by modifying `SYSTEM_ROLES_ONLY_TESTS`.
+4. Enter `tmt try -p plans/test_playbooks_parallel <platform>`.
+    This command identifies the `plans/test_playbooks_parallel.fmf` plan and provisions local VMs, a control node and a managed node.
+5. tmt try is in development and does not identify tests from URL automatically, so after provisioning the machines, you must type `t`, `p`, `t` from the interactive prompt to identify tests, run preparation steps, and run the tests.
+
+### Running in Testing Farm
+
+1. Install `testing-farm` as described in [Installation](https://gitlab.com/testing-farm/cli/-/blob/main/README.adoc#user-content-installation).
+2. Change to the role repository directory.
+3. If you want to run tests with edits in your branch, you need to commit and push changes first to some branch.
+4. You can uncomment "Inject your ssh public key to test systems" discover step in the plan if you want to troubleshoot tests by SSHing into test systems in Testing Farm.
+5. Enter `testing-farm request`.
+    Edit to your needs.
+
+    ```bash
+    $ TESTING_FARM_API_TOKEN=<your_api_token> \
+        testing-farm request --pipeline-type="tmt-multihost" \
+        --plan-filter="tag:playbooks_parallel" \
+        --git-url "https://github.com/<my_user>/{{ inventory_hostname }}" \
+        --git-ref "<my_branch>" \
+        --compose CentOS-Stream-9 \
+        -e "SYSTEM_ROLES_ONLY_TESTS=tests_default.yml" \
+        --no-wait
+    ```

playbooks/templates/plans/test_playbooks_parallel.fmf

@@ -0,0 +1,76 @@
+summary: A general test for a system role
+tag: playbooks_parallel
+provision:
+  # TF uses `how: artemis`, and `tmt try`` uses `how: virtual`.
+  # Hence there is no need to define `how` explicitly.
+  - name: control-node1
+    role: control_node
+    # `connection: system` is required for `how: virtual` to assign VMs a real
+    # IP making SSH configuration easier.
+    # This setting is ignored in `artemis`, so we can leave it as is.
+    connection: system
+{% for node_num in range(managed_nodes_count | int) %}
+  - name: managed-node{{ node_num + 1 }}
+    role: managed_node
+    connection: system
+{% if tmt_hardware is defined %}
+    hardware:
+{# regex_replace is a workaround for https://github.com/yaml/pyyaml/issues/234 #}
+      {{ tmt_hardware | to_nice_yaml | indent(8) | regex_replace("- *","- ") }}
+{%- endif -%}
+{% endfor %}
+environment:
+  ANSIBLE_VER: 2.17
+  REPO_NAME: {{ inventory_hostname }}
+  PYTHON_VERSION: 3.12
+  SYSTEM_ROLES_ONLY_TESTS: ""  # tests_default.yml
+  TEST_LOCAL_CHANGES: true
+  PR_NUM: ""
+  LINUXSYSTEMROLES_USER: ""
+  LINUXSYSTEMROLES_DOMAIN: ""
+  LINUXSYSTEMROLES_SSH_KEY: ""
+  ARTIFACTS_DIR: ""
+  ARTIFACTS_URL: ""
+  LSR_TFT_DEBUG: false
+prepare:
+  - name: Use vault.centos.org repos (CS 7, 8 EOL workaround)
+    script: |
+      if grep -q 'CentOS Stream release 8' /etc/redhat-release; then
+        sed -i '/^mirror/d;s/#\(baseurl=http:\/\/\)mirror/\1vault/' /etc/yum.repos.d/*.repo
+      fi
+      if grep -q 'CentOS Linux release 7.9' /etc/redhat-release; then
+        sed -i '/^mirror/d;s/#\?\(baseurl=http:\/\/\)mirror/\1vault/' /etc/yum.repos.d/*.repo
+      fi
+  # Replace with feature: epel: enabled once https://github.com/teemtee/tmt/pull/3128 is merged
+  - name: Enable epel to install beakerlib
+    script: |
+      # CS 10 and Fedora doesn't require epel
+      if grep -q -e 'CentOS Stream release 10' -e 'Fedora release' /etc/redhat-release; then
+        exit 0
+      fi
+      yum install epel-release yum-utils -y
+      yum-config-manager --enable epel epel-debuginfo epel-source
+discover:
+  - name: Prepare managed node
+    how: fmf
+    where: managed_node
+    filter: tag:prep_managed_node
+    url: https://github.com/linux-system-roles/tft-tests
+    ref: main
+  - name: Run test playbooks from control_node
+    how: fmf
+    where: control_node
+    filter: tag:test_playbooks
+    url: https://github.com/linux-system-roles/tft-tests
+    ref: main
+  # Uncomment this step for troubleshooting
+  # This is required because currently testing-farm cli doesn't support running multi-node plans
+  # You can set ID_RSA_PUB in the environment section above to your public key to distribute it to nodes
+  # - name: Inject your ssh public key to test systems
+  #   how: fmf
+  #   where: control_node
+  #   filter: tag:reserve_system
+  #   url: https://github.com/linux-system-roles/tft-tests
+  #   ref: main
+execute:
+    how: tmt

playbooks/update_files.yml

@@ -54,6 +54,14 @@
           when: git_dir is not exists
           changed_when: false
 
+        - name: Get number of test playbooks
+          shell: ls {{ git_dir }}/tests/tests_*.yml | wc -l
+          register: test_playbooks_count
+
+        - name: Set managed_nodes_count
+          set_fact:
+            managed_nodes_count: "{{ ((test_playbooks_count.stdout | int) / 10 + 1) | int }}"
+
         - name: Set main branch var
           set_fact:
             __main_br: main