Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document all individual projects #113

Merged
merged 2 commits into from
Feb 13, 2025
+304 −869
Merged

Document all individual projects #113

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
624cae0
Document all individual projects
ramperher Feb 12, 2025
822ce31
Replace classical by common
ramperher Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 7 additions & 117 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,13 @@ Example CNF is an OpenShift workload to exercice an SRIOV setup, based on [TestP

![Flow](documentation/network_setup.png)

It is providing the following operators:
It provides the following operators:

* trex-operator
* It provides TRex Traffic Generator, decomposed in three components:
* TRexConfig CR, pinning the config of the TRex server instance. Related pod is `trexconfig-<x>` pod.
* TRexApp CR, setting up the job that launches TRex execution to generate traffic towards the system under test. Related pod is `trex-app` pod.
* TRex Operator, ensuring CR reconciliation via controller-manager pod. Related pod is `trex-operator-controller-manager-<x>` pod.
* Following information can be extracted from pod logs:
* In `trexconfig-<x>` pod logs, you can see the trex statistics printed periodically.
* The summary of the test execution can be seen at the end of the `trex-app` job logs.
* In `trex-operator-controller-manager-<x>` pod, you can see the execution of the Ansible playbooks that ensures the reconciliation loop of the operator.
- [trex-operator](trex-operator)
- [testpmd-operator](testpmd-operator)
- [cnf-app-mac-operator](cnf-app-mac-operator)

![Operator behavior](documentation/trex-operator.png)

* testpmd-operator
* Final application, also known as CNF Application, which is a standard TestPMD instance using the default MAC forwarding module. It uses two components:
* TestPMD CR, which creates a pod to implement the MAC forwarding module as final application. Related pod is `testpmd-app-<x>` pod (only one replica is used).
* TestPMD Operator, ensuring CR reconciliation via controller-manager pod. Related pod is `testpmd-operator-controller-manager-<x>` pod.
* Following information can be extracted from pod logs:
* To see the TestPMD statistics printed periodically for this module, you can rely on `testpmd-app-<x>` pod logs.
* In `testpmd-operator-controller-manager-<x>` pod, you can see the execution of the Ansible playbooks that ensures the reconciliation loop of the operator.

![Operator behavior](documentation/testpmd-operator.png)

* cnf-app-mac-operator
* Auxiliary operator just composed by one component, which is CNFAppMac Operator, a Golang-based operator in charge of ensuring reconciliation for CNFAppMac CR, which is a wrapper created for the `testpmd-app-<x>` pod and linked to it, and that is used to extract the network information of the pods (network, MAC and PCI addresses), to be offered to other components of the solution.

![Operator behavior](documentation/cnf-app-mac-operator.png)

You can use them from the [Example CNF Catalog](https://quay.io/repository/rh-nfv-int/nfv-example-cnf-catalog?tab=tags). Image generation is automated with [Github workflows](.github/workflows) in this repo.
You can use them from the [Example CNF Catalog](https://quay.io/repository/rh-nfv-int/nfv-example-cnf-catalog?tab=tags). Image generation is automated with [Github workflows](.github/workflows) in this repo, using this [Makefile](Makefile).

## Pre-requirements

Expand All @@ -53,78 +30,8 @@ The three operators defined in this repository are built with [Operator SDK tool

We can differentiate between these two cases:

### Ansible-based operators

This is the case of testpmd-operator and trex-operator.

Base structure for each case is achieved with the following commands, then it's just a matter of accommodating the required code for each operator in the corresponding files and folders:

- testpmd-operator

```
$ mkdir testpmd-operator; cd testpmd-operator
$ operator-sdk init --domain openshift.io --plugins ansible
$ operator-sdk create api --version v1 --generate-role --group examplecnf --kind TestPMD
```

- trex-operator

```
$ mkdir trex-operator; cd trex-operator
$ operator-sdk init --domain openshift.io --plugins ansible
$ operator-sdk create api --version v1 --generate-role --group examplecnf --kind TRexApp
$ operator-sdk create api --version v1 --generate-role --group examplecnf --kind TRexConfig
```

### Go-based operators

This is the case of cnf-app-mac-operator.

Base structure for this case is achieved with the following commands, then it's just a matter of accommodating the required code for the operator in the corresponding files and folders:

- cnf-app-mac-operator

For operator-sdk v1.38.0, you need to have installed the same Go version used in operator-sdk, which is at least Go 1.22.5+.

```
$ operator-sdk version
operator-sdk version: "v1.38.0", commit: "0735b20c84e5c33cda4ed87daa3c21dcdc01bb79", kubernetes version: "1.30.0", go version: "go1.22.5", GOOS: "linux", GOARCH: "amd64"
```

Create the project structure and the CNFAppMac API:

```
$ mkdir cnf-app-mac-operator; cd cnf-app-mac-operator
$ operator-sdk init --domain openshift.io --repo github.com/openshift-kni/example-cnf/tree/main/cnf-app-mac-operator
$ operator-sdk create api --version v1 --group examplecnf --kind CNFAppMac --controller --resource
```

At this point, remove RBAC resource creation in Makefile > manifests task. Then, review cmd/main.go and api/v1/cnfappmac_types.go, then run:

```
$ make generate
$ make manifests
```

Create webhook and certmanager:

```
$ operator-sdk create webhook --version v1 --group examplecnf --kind CNFAppMac --defaulting --programmatic-validation
```

Review the generated files properly, then:

```
$ make manifests
```

Comment webhook references in PROJECT and cmd/main.go files (older versions were not using this), review internal/controller/cnfappmac_controller.go and review the rest of files.

To conclude, build the main.go file to check it's working fine:

```
$ go build cmd/main.go
```
- Ansible-based operators: this is the case of [testpmd-operator](testpmd-operator/README.md#how-to-build-the-operator) and [trex-operator](trex-operator/README.md#how-to-build-the-operator).
- Go-based operators: this is the case of [cnf-app-mac-operator](cnf-app-mac-operator/README.md#how-to-build-the-operator).

## Pod affinity rules

Expand Down Expand Up @@ -273,23 +180,6 @@ $ ip -s -d link show dev ens2f0

Under [utils](utils) folder, you can find some utilities included in Example CNF to extend the functionalities offered by the tool.

- [webserver.go](utils/webserver.go): a Golang-based webserver to implement liveness, readiness and startup probes in the container images offered in [testpmd-container-app](testpmd-container-app) and [trex-container-app](trex-container-app) folders. The Makefiles offered in these directories take care of copying the webserver code from the utils directory to each image's directory.
- [required-annotations.yaml](utils/required-annotations.yaml): annotations to be appended to the CSVs to pass Preflight's RequiredAnnotations tests. They are appended automatically thanks to the Makefile tasks from each operator.
- [support-images](support_images): projects where you can find the Dockerfile required to build some of the images used as build images by the Example CNF images. These images can be found on quay.io/rh-nfv-int and they are publicly available, you only need credentials to access quay.io. The images can be built with the following commands (you need to run it in a RHEL host with a valid RHEL subscription to be able to download the packages installed in the images, and you need a valid quay.io credentials to push it to quay.io):

```
# build images
$ cd utils/support-images
$ podman build dpdk-23.11 -f dpdk-23.11/Dockerfile -t "quay.io/rh-nfv-int/dpdk-23.11:v0.0.1"
$ podman build ubi8-base-testpmd -f ubi8-base-testpmd/Dockerfile -t "quay.io/rh-nfv-int/ubi8-base-testpmd:v0.0.1"
$ podman build ubi8-base-trex -f ubi8-base-trex/Dockerfile -t "quay.io/rh-nfv-int/ubi8-base-trex:v0.0.1"

# push images (to quay.io/rh-nfv-int)
$ podman push quay.io/rh-nfv-int/dpdk-23.11:v0.0.1
$ podman push quay.io/rh-nfv-int/ubi8-base-testpmd:v0.0.1
$ podman push quay.io/rh-nfv-int/ubi8-base-trex:v0.0.1
```

## Acknowledgements

Please write to [email protected] in case you need more information for using and testing Example CNF in the scenarios that have been proposed in this repository.
75 changes: 56 additions & 19 deletions cnf-app-mac-operator/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,63 @@
CNF App MAC FETCH
=================
# cnf-app-mac-operator

An operator helps in creating `CNFAppMac` CR for all the pods in a namespace
which has VFs attached to it.
![Operator behavior](../documentation/cnf-app-mac-operator.png)

TODO:
------------------------
* Create CR only for a custom set of pods (based on label or name)
Auxiliary operator just composed by one component, which is CNFAppMac Operator, a Golang-based operator in charge of ensuring reconciliation for CNFAppMac CR, which is a wrapper created for the `testpmd-app-<x>` pod and linked to it, and that is used to extract the network information of the pods (network, MAC and PCI addresses), to be offered to other components of the solution.

License
------------------------
## How to build the operator

Copyright 2024.
Base structure for this case is achieved with the following commands, then it's just a matter of accommodating the required code for the operator in the corresponding files and folders:

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
For operator-sdk v1.38.0, you need to have installed the same Go version used in operator-sdk, which is at least Go 1.22.5+.

http://www.apache.org/licenses/LICENSE-2.0
```
$ operator-sdk version
operator-sdk version: "v1.38.0", commit: "0735b20c84e5c33cda4ed87daa3c21dcdc01bb79", kubernetes version: "1.30.0", go version: "go1.22.5", GOOS: "linux", GOARCH: "amd64"
```

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Create the project structure and the CNFAppMac API:

```
$ mkdir cnf-app-mac-operator; cd cnf-app-mac-operator
$ operator-sdk init --domain openshift.io --repo github.com/openshift-kni/example-cnf/tree/main/cnf-app-mac-operator
$ operator-sdk create api --version v1 --group examplecnf --kind CNFAppMac --controller --resource
```

At this point, remove RBAC resource creation in Makefile > manifests task. Then, review cmd/main.go and api/v1/cnfappmac_types.go, then run:

```
$ make generate
$ make manifests
```

Create webhook and certmanager:

```
$ operator-sdk create webhook --version v1 --group examplecnf --kind CNFAppMac --defaulting --programmatic-validation
```

Review the generated files properly, then:

```
$ make manifests
```

Comment webhook references in PROJECT and cmd/main.go files (older versions were not using this), review internal/controller/cnfappmac_controller.go and review the rest of files.

To conclude, build the main.go file to check it's working fine:

```
$ go build cmd/main.go
```

## What to update if bumping operator version

Apart from the modifications you have to do, you also need to update the operator version in these files:

- [CHANGELOG.md](CHANGELOG.md).
- [Makefile](Makefile).
- [Dockerfile](Dockerfile).

Also, make sure that the operator version is within the interval defined in [required-annotations.yaml](../utils/required-annotations.yaml) file for `olm.skipRange` annotation, else update that file to modify the current range.

A classical change is the update of Operator SDK version used in the operator. Here's an [example](https://github.com/openshift-kni/example-cnf/pull/108) where this is done.
ramperher marked this conversation as resolved.
Show resolved Hide resolved
6 changes: 3 additions & 3 deletions documentation/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,9 @@ We can see these pods:

The three data-plane pods (`testpmd-app`, `trexconfig` and `job-trex-app`) execute different scripts to configure all the software pieces that involves the traffic exchange between TestPMD and TRex:

- `testpmd-app`: it runs `/usr/local/bin/example-cnf/testpmd-wrapper`. [This script](../testpmd-container-app/cnfapp/scripts/testpmd-wrapper) retrieves context information to build the command to run `testpmd` command-line tool. In this baseline scenario, once finished, `testpmd` is invoked in auto-start mode, printing the statistics every minute.
- `trexconfig`: it runs `/usr/local/bin/trex-wrapper`. [This script](../trex-container-app/server/scripts/trex-wrapper) takes also some context information to build a config file for TRex, then it launches `_t-rex-64` binary using the generated config file, and printing statistics regularly. This does not launch the traffic generation, since this is done with the TRexApp job.
- `job-trex-app`: it runs `/usr/local/bin/trex-wrapper` (but a different one compared to `trexconfig`). [This script](../trex-container-app/app/scripts/trex-wrapper) triggers a custom Python script that builds the traffic profile, according to the TRex variables provided in the pipeline (duration of the job, packet size, data rate, etc.), and start sending the traffic. If defining a duration, a timeout will be enabled to stop the execution after the given duration, and then statistics will be printed and packet loss will be calculated. If packet loss is equal or less than 0, this means there's no packet loss, else, packet loss is present and the pod status will be different than Completed.
- `testpmd-app`: check [cnfapp container image docs](../testpmd-container-app/cnfapp/README.md) for more details. In this baseline scenario, once finished, `testpmd` is invoked in auto-start mode, printing the statistics every minute.
- `trexconfig`: check [TRex server container image docs](../trex-container-app/server/README.md) for more details.
- `job-trex-app`: check [TRex app container image docs](../trex-container-app/app/README.md) for more details.

### What logs should I check?

Expand Down
2 changes: 1 addition & 1 deletion nfv-example-cnf-index/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# nfv-example-cnf-index

This repository builds an index image that includes the [Example cnf operators](https://github.com/openshift-kni/example-cnf) used by the [example_cnf_deploy role](https://github.com/redhatci/ansible-collection-redhatci-ocp/blob/main/roles/example_cnf_deploy/README.md).
This repository builds an index image that includes the [Example CNF operators](../README.md) used by the [example_cnf_deploy role](https://github.com/redhatci/ansible-collection-redhatci-ocp/blob/main/roles/example_cnf_deploy/README.md).

The [CHANGELOG](./CHANGELOG.md) includes the information about the version of the image as well as the operators and their versions included in the image.
5 changes: 5 additions & 0 deletions testpmd-container-app/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# testpmd-container-app

Container images used by [testpmd-operator](../testpmd-operator/). We have the following projects:

- [cnfapp](cnfapp).
70 changes: 66 additions & 4 deletions testpmd-container-app/cnfapp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,68 @@
Pushing images to Red Hat registry
----------------------------------
# cnfapp

Follow [partner guide](https://redhat-connect.gitbook.io/partner-guide-for-red-hat-openshift-and-container/certify-your-application/image-upload) to upload the image for certification. The image will be available in the respective partner (fredco) project at https://connect.redhat.com/project.
Container image that allows you to configure TestPMD in a container and prepare it to launch it.

Follow the push images manually steps.
## What is it executed?

This container runs `/usr/local/bin/example-cnf/testpmd-wrapper`. [This script](scripts/testpmd-wrapper) retrieves context information to build the command to run `testpmd` command-line tool. Then, depending on the selected mode to launch Example CNF, `testpmd` will be executed in auto-start mode (baseline scenario), or, if launching the troubleshooting scenario, you can select between that mode and the interactive mode.

The scripts (generated by `testpmd-wrapper`) to deploy each mode are placed in the same directory:

```
sh-4.4$ pwd
/usr/local/bin/example-cnf/run
sh-4.4$ ls
testpmd-interactive testpmd-run
```

They present some differences:

- `testpmd-run`: runs `testpmd` in auto-start mode.

```
sh-4.4$ cat testpmd-run
/usr/local/bin/example-cnf/testpmd -l 3,5,53 --in-memory -a 0000:86:02.2 -a 0000:86:03.0 --socket-mem 0,1024 -n 6 --proc-type auto --file-prefix pg -- --nb-cores=2 --rxq=1 --txq=1 --rxd=2048 --txd=2048 --auto-start --eth-peer 0,20:04:0f:f1:89:01 --eth-peer 1,20:04:0f:f1:89:02 --forward-mode=mac --stats-period 1 2>&1 | tee /var/log/testpmd/app.log
```

The call to `testpmd` receives, as arguments (remember we have to differentiate between [EAL parameters](https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html) and [TestPMD parameters](https://doc.dpdk.org/guides/testpmd_app_ug/run_app.html), they're separated by `--`), the following parameters:

- CPU cores to bind to `testpmd`. This matches with the number of cores assigned for the pod that is launching this container. If using reduced mode, only three cores are used: first CPU for the console, and second core CPU (two siblings) for `testpmd` execution. This modification is managed by `testpmd-wrapper`.
- Option to not create any shared data structures and run entirely in memory.
- PCI addresses.
- Preallocation of specific amounts of memory per socket. Depending on the NUMA node that is used, `testpmd-wrapper` will tune this value.
- Number of memory channels to use.
- Type of current process as `auto`.
- Shared data file prefix for DPDK process named as `pg`
- Number of forwarding cores.
- Number of TX/RX queues.
- Number of descriptors in the TX/RX rings. These values are doubled if using reduced mode, everything managed by `testpmd-wrapper`.
- Enable auto-start mode.
- Provide the pair of port-MAC address for each port.
- Enable MAC forwarding mode.
- Print the statistics every minute.

Finally, redirect the logs to `/var/log/testpmd/app.log`

- `testpmd-interactive`: runs `testpmd` in interactive mode.

```
sh-4.4$ cat testpmd-interactive
/usr/local/bin/example-cnf/testpmd -l 3,5,53 --in-memory -a 0000:86:02.2 -a 0000:86:03.0 --socket-mem 0,1024 -n 6 --proc-type auto --file-prefix pg -- --nb-cores=2 --rxq=1 --txq=1 --rxd=2048 --txd=2048 --i --eth-peer 0,20:04:0f:f1:89:01 --eth-peer 1,20:04:0f:f1:89:02 --forward-mode=mac
```

Parameters are mostly similar than in the previous case, but with the following differences:

- Use interactive mode (`--i`) instead of auto-start mode (`--auto-start`).
- Do not print statistics, since they can be retrieved manually.

You have more details about testing this feature in the [testing docs](../../documentation/testing.md).

## What to update if bumping container version

Apart from the modifications you have to do, you also need to update the container version in these files:

- [Dockerfile](Dockerfile).
- [build.sh](../build.sh) (from parent folder).
- [Makefile](../Makefile) (from parent folder).

Here's an [example](https://github.com/openshift-kni/example-cnf/pull/111) where this is done.
Loading