Difficult to specify earth2grid as a dependency in other projects

[oliverwm1]

Because of the requirement to use --no-build-isolation when installing earth2grid, it is difficult to specify it as a requirement in other python packages. Do you have any suggestions of how to get around this issue / install without that flag?

earth2grid/README.md

Line 24 in a5d5a3f

pip install --no-build-isolation ./earth2grid

[nbren12]

--no-build-isolation is used to avoid reinstalling the whole pytorch stack just to build the earth2grid wheel, which is quite time consuming and error prone when pytorch is already installed and linked against the correct CUDA installation.

I think you can put earth2grid in your requirements.txt/setup.py and then do pip install --no-build-isolation -r requirements.txt. This is how I did it in a recent project

# requirements.txt
https://github.com/NVIDIA/modulus/archive/90c8f68ae4a4065109b8af26ae9b097f292f1564.tar.gz

Installation

pip install --no-build-isolation -r requirements.txt

Would this work for you? Unfortunately, pip doesn't work well for compiled packages, and it would take a fair amount of effort for us to distribute our own wheels compiled against the various CUDA versions.

I hope earth2grid is worth some TLC. I've certainly found it easier to install than e.g. cartopy.

[oliverwm1]

Thanks for quick reply.

--no-build-isolation is used to avoid reinstalling the whole pytorch stack just to build the earth2grid wheel

Got it. I believed we tried without this flag and it failed, at least in some settings. It didn't just take longer.

Would this work for you?

Okay, potentially we can have earth2grid as an extra dep so that it's not necessary to use the --no-build-isolation when earth2grid is not required. I don't want this flag to be necessary to install our package in all circumstances. Will try this and report back.

easier to install than e.g. cartopy.

cartopy has improved! At least we've found the pip install possible since v.022 works well.

[nbren12]

Okay, potentially we can have earth2grid as an extra dep so that it's not necessary to use the --no-build-isolation when earth2grid is not required. I don't want this flag to be necessary to install our package in all circumstances. Will try this and report back.

Sounds good.

Ultimately, we could explore JIT or some other techniques to make the library easy to use. We've even had good luck with torch.compile creating optimized CUDA kernels for some HPX operations, so it may be possible to move more of the HPX implementation into python.

[nbren12]

@akshaysubr do you have any thoughts?

[nbren12]

here's a link for JIT: https://pytorch.org/tutorials/advanced/cpp_extension.html#jit-compiling-extensions

[oliverwm1]

Hitting an issue separate from the 'no build dependencies'. Made a PR: #24