Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add paygo product version condition to skip useless repositories #1428

Conversation

maximenoel8
Copy link
Contributor

@maximenoel8 maximenoel8 commented Nov 24, 2023

What does this PR change?

Repositories

Paygo instances on cloud ( testing with AWS ) already have their repositories by default.
Using internal product repositories doesn't make sense for paygo testing.
Currently to have access to products repositories we have two solutions :

  • using internal repositories
  • using SCC repositories

In my changes, I introduce a new product version extension call paygo. If you have paygo in your product version name, it will disable salt repo and scc

Nevertheless, we still need the testsuite repositories for the minion, so I only keep respo.testsuite call.

I also removed a redundant repo call in testsuite instead of adding condition around it.
By following the salt paths, I find out that by default we call repo three time.

Server installation

Server packages are already installed and don't need to be install again.

@@ -3,7 +3,6 @@

include:
- scc
- repos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we have here this?

- repos.testsuite

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repos.testsuite will be call during default > minimal. I try to remove useless call to repo.

Copy link
Member

@srbarrios srbarrios Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If one of the salt states of this sls requires another sls, I think it would make sense to include it here.
I would like the opinion of a Salt expert for this doubt.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC what is on minimal should also be present in the general highstate. Somaone can call the script highstate.sh which runs the highstate only, and should have the same behaviour.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do a quick search inside the salt folder, we will notice that for:

- require:
      - sls:

We always include the sls that is required afterward, I think that we should keep this in common unless there is a good reason to do it differently here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please before resolving this, let's finish the discussion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a Salt PoV, IIRC the require works as long as repos.testsuite (in this case) is part of the sls files that are looked at. But from a sumaform developer PoV, I would always include it where I need it. Relying on global state is a general code smell.

@@ -1,6 +1,6 @@
include:
- scc.minion
{% if 'build_image' not in grains.get('product_version') | default('', true) %}
{% if 'build_image' and 'paygo' not in grains.get('product_version') | default('', true) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check two values at same time?
I think it should be:

Suggested change
{% if 'build_image' and 'paygo' not in grains.get('product_version') | default('', true) %}
{% if 'build_image' not in grains.get('product_version') and 'paygo' not in grains.get('product_version') | default('', true) %}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this expression and it works. Do you want to have it in our format ?

Copy link
Member

@srbarrios srbarrios Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works just because the first part it returns always true.

{% if 'build_image' %}
true
{% endif %}

You can test it here: https://j2live.ttl255.com/

@@ -3,7 +3,6 @@

include:
- scc
- repos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC what is on minimal should also be present in the general highstate. Somaone can call the script highstate.sh which runs the highstate only, and should have the same behaviour.

@@ -14,7 +13,7 @@ default_cucumber_requisites:
- milkyway-dummy
- virgo-dummy
- require:
- sls: repos
- sls: repos.testsuite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In pkg.installed, the packages are coming from repos.testsuite. I'm not an salt expert but I find it easier to read if we point directly to the needed resource and not to a folder with 20+ files.
Also, for my paygo minion, I want those repositories but not the other repositories.
Because of the sumaform architecture, it's hard to split those resources. Calling the specific salt file solves the issue without adding extra condition around the requirement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep in mind that require is just for ordering the states. pkg.installed takes a fromrepo argument if you want to specify which repo to actually use when installing things. Of course, if there is only one repo that has a package, that's automatically used by zypper.

salt/repos/default.sls Show resolved Hide resolved
@maximenoel8
Copy link
Contributor Author

maximenoel8 commented Nov 29, 2023

Thanks for the reviews, I still need a new deployment to test the default value. I introduced this condition 2 years ago so I don't remember completely. From my memory, I struggled a bit with this condition.
I think because we use not in, we need this default. In the other example, we don't have default if the condition is in and we have default if the condition is not in

But it requires some extra testing.

@maximenoel8 maximenoel8 force-pushed the add_support_paygo_image_salt_states branch from 78a76c3 to 660bfea Compare November 30, 2023 00:41
@maximenoel8
Copy link
Contributor Author

maximenoel8 commented Nov 30, 2023

The default value is required or you have this error :

14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.hostname
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% endif %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.network
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.firewall
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.avahi
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% if 'build_image' and 'paygo' not in grains.get('product_version')%}    <======================
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - repos
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% else %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - repos.testsuite
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% endif %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.time
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[...]
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m---
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'default/pkgs.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'scc/init.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'default/grub.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'default/sshd.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'default/ids.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'default/testsuite.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'mirror/init.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[INFO    ] Fetching file from saltenv 'base', ** done ** 'mirror/configuration.sls'
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m�[0;1;31mlocal:�[0;0m
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    �[0;1;31mData failed to compile:�[0;0m
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m�[0;1;31m----------
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    Rendering SLS 'base:default.minimal' failed: Jinja error: argument of type 'NoneType' is not iterable
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0mTraceback (most recent call last):
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    output = template.render(**decoded_context)
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    return original_render(self, *args, **kwargs)
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    return self.environment.handle_exception(exc_info, True)
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    reraise(exc_type, exc_value, tb)
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m    raise value.with_traceback(tb)
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  File "<template>", line 8, in top-level template code
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0mTypeError: argument of type 'NoneType' is not iterable
14:53:34  
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m; line 8
14:53:34  
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m---
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[...]
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.hostname
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% endif %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.network
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.firewall
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.avahi
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% if 'build_image' and 'paygo' not in grains.get('product_version')%}    <======================
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - repos
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% else %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - repos.testsuite
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  {% endif %}
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m  - default.time
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m[...]
14:53:34  �[0m�[1mmodule.mirror.module.mirror.module.host.null_resource.provisioning[0] (remote-exec):�[0m �[0m---�[0;0m

salt/default/minimal.sls Outdated Show resolved Hide resolved
@@ -3,7 +3,6 @@

include:
- scc
- repos
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a Salt PoV, IIRC the require works as long as repos.testsuite (in this case) is part of the sls files that are looked at. But from a sumaform developer PoV, I would always include it where I need it. Relying on global state is a general code smell.

@@ -14,7 +13,7 @@ default_cucumber_requisites:
- milkyway-dummy
- virgo-dummy
- require:
- sls: repos
- sls: repos.testsuite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep in mind that require is just for ordering the states. pkg.installed takes a fromrepo argument if you want to specify which repo to actually use when installing things. Of course, if there is only one repo that has a package, that's automatically used by zypper.

salt/server/init.sls Show resolved Hide resolved
@maximenoel8 maximenoel8 merged commit 21fd99f into uyuni-project:master Dec 3, 2023
2 checks passed
@maximenoel8 maximenoel8 deleted the add_support_paygo_image_salt_states branch December 3, 2023 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants