Hooks cannot use venv or virtualenv on pre-commit.ci

[takluyver]

Apologies if this is the wrong place for this - it seemed like the most likely location I could find.

I'm looking at a failure of a hook called check-manifest, which does some checks for Python packages. That uses the build package, which uses either venv or virtualenv to make a temporary environment to build the package.

With venv we get:

Error message for venv

['/pc/clone/ATu6JSMGTMCUlky_tF5p3g/py_env-python3/bin/python', '-m', 'venv', '/tmp/check-manifest-sbh6_9e7-sdist/env'] failed (status 1):
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/check-manifest-sbh6_9e7-sdist/env/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

With virtualenv we get:

Error message for virtualenv

['/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/bin/python', '-m', 'virtualenv', '/tmp/check-manifest-jg73uy7n-sdist/env'] failed (status 1):
RuntimeError: failed to build image pip, wheel, setuptools because:
Traceback (most recent call last):
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 56, in _install
    installer.build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/symlink.py", line 54, in build_image
    super(SymlinkPipInstall, self).build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/base.py", line 52, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/usr/lib/python3.8/zipfile.py", line 1647, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/usr/lib/python3.8/zipfile.py", line 1693, in _extract_member
    os.makedirs(upperdirs)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/virtualenv/cache/wheel/3.8/image/1/SymlinkPipInstall/pip-22.0.3-py3-none-any'

Traceback (most recent call last):
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 56, in _install
    installer.build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/symlink.py", line 54, in build_image
    super(SymlinkPipInstall, self).build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/base.py", line 52, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/usr/lib/python3.8/zipfile.py", line 1647, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/usr/lib/python3.8/zipfile.py", line 1693, in _extract_member
    os.makedirs(upperdirs)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/virtualenv/cache/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.37.1-py2.py3-none-any'

Traceback (most recent call last):
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 56, in _install
    installer.build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/symlink.py", line 54, in build_image
    super(SymlinkPipInstall, self).build_image()
  File "/pc/clone/lcQ6xsAPT823lS1bShIQSQ/py_env-python3/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/pip_install/base.py", line 52, in build_image
    zip_ref.extractall(str(self._image_dir))
  File "/usr/lib/python3.8/zipfile.py", line 1647, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/usr/lib/python3.8/zipfile.py", line 1693, in _extract_member
    os.makedirs(upperdirs)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/virtualenv/cache/wheel/3.8/image/1/SymlinkPipInstall/setuptools-60.6.0-py3-none-any'

The venv issue is fairly clear: Debian (frustratingly) splits out part of the standard library as packages like python3.8-venv, and those aren't installed. AFAIK, neither the hook author nor the end user can specify extra apt packages to install on pre-commit.ci?

The virtualenv issue is murkier. I can see that the Dockerfile sets several virtualenv environment variables, but it's not clear to me why it doesn't work when called from a hook.

runner-image/Dockerfile

Lines 88 to 94 in 79a11f6

	# ensure virtualenv appdata cache is populated
	ENV \
	VIRTUALENV_OVERRIDE_APP_DATA=/opt/virtualenv/cache \
	VIRTUALENV_SYMLINK_APP_DATA=1
	COPY build/seed-virtualenv-cache /tmp/seed-virtualenv-cache
	RUN /tmp/seed-virtualenv-cache
	ENV VIRTUALENV_READ_ONLY_APP_DATA=1

[asottile]

network is not available at runtime anyway so even if this were "fixed" it would not work -- this is why --no-build-isolation was added to check-manifest: https://github.com/mgedmin/check-manifest#version-control-integration