From 64157d8bc3c026dac29109b1f98489005426c930 Mon Sep 17 00:00:00 2001 From: Ben Mares Date: Mon, 16 Sep 2024 11:06:09 +0200 Subject: [PATCH 1/2] Create a first draft pixi migration guide --- docs/pixi-migration.md | 158 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 docs/pixi-migration.md diff --git a/docs/pixi-migration.md b/docs/pixi-migration.md new file mode 100644 index 000000000..b48a20463 --- /dev/null +++ b/docs/pixi-migration.md @@ -0,0 +1,158 @@ +# pixi migration guide + +So you want to try pixi, the new project-centric Conda environment manager? +We will walk you through the steps to get started, whether or not you are a conda-lock user. + +Perhaps "migration" is the wrong word, because pixi is an option, not a replacement. +It will not override existing workflows. +Removing pixi from your project is as simple as deleting the files indicated in step 2 below. + +## The steps to get started + +1. Visit and install pixi if you haven't already. + You can also find information about IDE integrations and other documentation. + *Note*: As of writing, VS Code's Python extension has native support for pixi. + +2. Prepare for the new files and directories related to pixi. + + - `pixi.toml` defines the environment specifications and config options. We will generate this with `conda-lock`. + + - `pixi.lock` is managed by pixi and maintains a particular solution to the environment specifications from `pixi.toml`. + + - `.pixi/` is the environment cache directory where the environments from the lock file `pixi.lock` are installed and updated. + Pixi will recreate this directory if it is deleted. + + Both `pixi.toml` and `pixi.lock` should be version controlled. The `.pixi/` directory should be ignored by `.gitignore`, `.dockerignore`, and/or similar. + + *Note*: For Python projects it is possible to use `pyproject.toml` instead of `pixi.toml`. + For starters we recommend sticking with `pixi.toml` since as of writing there is very little advantage to using `pyproject.toml`. + Using separate files also provides more clear separation between Python project dependency management (`pyproject.toml`) and environment management (`pixi.toml`). Besides, migrating to `pyproject.toml` is mostly a matter of prefixing the table names in `pixi.toml` with `tool.pixi.`. + +3. Install the latest version of `conda-lock` if you don't already have it. + + ```sh + pipx install --force "conda-lock>=3" + ``` + + or + + ```sh + pipx upgrade conda-lock + ``` + +4. Generate a `pixi.toml` file with `conda-lock`. + + ```sh + conda-lock render-lock-spec --kind=pixi.toml --stdout \ + --file=source-file-1 \ + --file=source-file-2 \ + ... + --pixi-project-name=my-project-name + ``` + + Each `--file` argument should include a path to some `environment.yml` or `pyproject.toml`. + + The `--pixi-project-name` allows you to specify the name of the project. + + For Python development, you can optionally include one or more `--editable python-project-name=relative-path` arguments to specify local packages. For example, + + ```sh + --editable python-package=. + ``` + + works like `pip install -e .`. (More generally, `.` can be replaced with the relative path to a Python project root containing a `pyproject.toml` or `setup.py` file.) + + For example for `conda-lock` we use the following command to generate our `pixi.toml` file: + + ```sh + conda-lock render-lock-spec --kind=pixi.toml --stdout \ + --file=environments/dev-environment.yaml \ + --file=pyproject.toml \ + --pixi-project-name=conda-lock \ + --editable conda-lock=. + ``` + + When you are satisfied with the output, append `> pixi.toml` to the command to save it to a file. + + Pay attention to any warnings that `conda-lock` may print. Some specifications may not be translatable, and others might not yet be implemented. + + By default, dependencies from `pyproject.toml` will be converted to conda-forge dependencies. If you wish to keep them as PyPI dependencies, you can add the following table to the `pyproject.toml`: + + ```toml + [tool.conda-lock] + default-non-conda-source = "pip" + ``` + + If you have a `test-dependencies.yml` file and want it to define a feature named `test`, then you should add + + ```yaml + category: test + ``` + + to the top level of `test-dependencies.yml`. + +5. (Optional) Generate a `pixi.lock` file with pixi. + + ```sh + pixi update + ``` + + This step is optional because pixi will automatically generate a lock file as needed from the other commands. + By running `pixi update` we are doing one step at a time. + +6. (Optional) Install the environment from the lock file. + + ```sh + pixi install + ``` + + Optional because pixi automatically installs environments as needed. + +7. Start an activated shell to test the environment. + + ```sh + pixi shell + ``` + + This command replaces `conda activate`. However, it uses a subshell instead of the current shell, so instead of `conda deactivate` you use `exit`. + +8. Commit `pixi.toml` and `pixi.lock` to your version control system. + +## Concepts + +### Features + +Conda-lock has a concept called "categories" for grouping dependencies. +Each dependency belongs to one or more categories. +When conda-lock generates a lock file, it solves for an environment containing all categories and determines which dependencies are needed for each category. +An environment can be created by seleting some subset of categories, and only the necessary dependencies will be installed. + +Pixi has a similar but more clever concept called "features". +Unlike categories, features are groups of dependency specifications that need not be mutually compatible. +Named environments are specified by selecting some mutually compatible features. +Then each named environment is solved for independently. + +There is also a default environment, and it can be overridden. +We override the default environment to include all categories. + +... + +#### The default feature + +Explain the implicit default feature, and how it is used by `pixi.toml` tables that do not specify a feature. + +Also explain the default environment. + +All environments implicitly include the default feature! + +## Infrastructure + +### GitHub Actions + +If you used [setup-micromamba](https://github.com/mamba-org/setup-micromamba), [setup-miniconda](https://github.com/conda-incubator/setup-miniconda) or similar, then for pixi you can use [setup-pixi](https://pixi.sh/latest/advanced/github_actions/). + +For periodically updating the lock file, you can use [pixi-diff-to-markdown](https://pixi.sh/latest/advanced/updates_github_actions/). + +### Docker + +See [pixi-docker](https://github.com/prefix-dev/pixi-docker). From 2c62c8f74a79ea15714f8dcb400c6ebae8756cf2 Mon Sep 17 00:00:00 2001 From: Ben Mares Date: Mon, 16 Sep 2024 11:22:12 +0200 Subject: [PATCH 2/2] Add ibis example --- docs/pixi-migration.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/pixi-migration.md b/docs/pixi-migration.md index b48a20463..15bb048f0 100644 --- a/docs/pixi-migration.md +++ b/docs/pixi-migration.md @@ -7,6 +7,33 @@ Perhaps "migration" is the wrong word, because pixi is an option, not a replacem It will not override existing workflows. Removing pixi from your project is as simple as deleting the files indicated in step 2 below. +## Quick example + +Use conda-lock and pixi to make a development environment for ibis. + +```sh +pipx upgrade conda-lock || pipx install --force "conda-lock>=3" +cd /tmp +git clone https://github.com/ibis-project/ibis.git +cd ibis +git checkout -b add-pixi +echo .pixi >> .gitignore +git add .gitignore +git commit -m "Ignore pixi cache directory" +conda-lock render-lock-spec --kind=pixi.toml --stdout \ + --file=pyproject.toml \ + --channel=conda-forge \ + --pixi-project-name=ibis \ + --editable ibis=. \ + > pixi.toml +git add pixi.toml +git commit -m "Add pixi.toml configuration" +pixi update +git add pixi.lock +git commit -m "Add pixi.lock lock file" +pixi shell +``` + ## The steps to get started 1. Visit and install pixi if you haven't already.