From c09b5f718a5a14c420c1649ac5bf117d8fd68d9e Mon Sep 17 00:00:00 2001 From: per1234 Date: Thu, 21 Nov 2024 20:39:25 -0800 Subject: [PATCH] Use Ubuntu 18.10 in Linux build container Background ========== Shared Library Dependencies --------------------------- The Linux build of Arduino IDE has dynamic linkage against the libstdc++ and glibc shared libraries. This results in it having a dependency on the version of the libraries that happens to be present in the environment it is built in. Although newer versions of the shared libraries are compatible with executables linked against an older version, the reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example, if Arduino IDE were built on a machine with version 3.4.33 of libstdc++, then attempting to run it on a machine with an older version of libstdc++ would fail with an error like: ``` Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.33' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/nsfw.node) ``` Likewise, if Arduino IDE were built on a machine with version 2.39 of glibc, then attempting to run it on a machine with an older version of glibc would fail with an error like: ``` Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /home/foo/arduino-ide/resources/app/node_modules/nsfw/build/Release/nsfw.node) ``` Build Machine Requirements -------------------------- The IDE builds distributed by Arduino should be compatible with a reasonable range of Linux distribution versions. In order to achieve this, the builds must be performed in a machine with an older version of the shared libraries. The shared libraries are part of the Linux distro, and installing a different version is not feasible. So this imposes a maximum limit on the build machine's distro version. The distributed builds are generated via a GitHub Actions workflow. The most simple approach is to run the build in the machine of the GitHub-hosted runners provided for each operating system. However, GitHub provides a limited range of operating system versions in their runners, and removes the older versions as newer versions are added. This means that building in the GitHub-hosted runner machine would not allow for the desired range of Linux distro version compatibility. For this reason, the Linux build is performed in a Docker container that provides an older version of Ubuntu. The same situation of incompatibility with Linux distro versions that have a version of the shared library dependencies older than the version present on the build machine occurs for several of the tools and frameworks used by the build process (e.g., Node.js, Python). In this case, the tables are turned as we are now the user rather than the distributor and so are at the mercy of the Linux distro version compatibility range provided by the distributor. So this imposes a minimum limit on the build machine's distro version. Although several of the dependencies used by the standard build system have dependencies on versions of glibc higher than the version 2.27 present in Ubuntu 18.04, it was possible to use this distro version in the Linux build container by using alternative distributions and/or versions of these dependencies. Workflow Artifacts ------------------ The build workflow uses GitHub actions workflow artifacts to transfer the files generated by the build job to subsequent jobs in the workflow. The "actions/upload-artifact" action is used for this purpose. Problem ======= GitHub is dropping support for the workflow artifacts produced by the version 3.x of the "actions/upload-artifact" action that was previously used by the build job. So the action version used in the build workflow was updated to the current version 4.x. This version of the action uses a newer version of the Node.js runtime (20). Unfortunately the the Node.js 20 runtime used by the action has a dependency on glibc version 2.28, which causes the Linux build job to fail after the update of the "actions/upload-artifact" action: ``` Run actions/upload-artifact@v4 /__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node) ``` Unlike the other dependencies of the build process, it is no longer possible to work around this incompatibility by continuing to use the older compatible version of the "actions/upload-artifact" action. It is also impossible to replace the incompatible Node.js 20.x distribution used by the action, since it comes from the read-only file system of the runner image. Likewise, it is not possible to configure or force the action to use a Node.js installation at a different path on the runner machine. Resolution ========== Compatibility with the new version of the "actions/upload-artifact" action is attained by updating the version of Linux in the build container to 18.10, which is the oldest version that has glibc 2.28. The presence of a newer glibc version in the container also makes it compatible with several other dependencies of the build process, meaning the code in the Dockerfile and workflow for working around the incompatibilities of Ubuntu 18.04 can be removed. Consequences ============ Unfortunately this means the loss of compatibility of the Linux Arduino IDE builds with distros that use glibc 2.27 (e.g., Ubuntu 18.04). User of those distros will now find that Arduino IDE fails to start with an error like: ``` Error: node-loader: Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/pty.node) at 85467 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:2766) at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105) at 23571 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3374073) at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105) at 55444 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3369761) at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105) at 24290 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1780542) at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105) at 43416 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1770138) at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105) ``` --- .github/workflows/assets/linux.Dockerfile | 93 ++++++----------------- .github/workflows/build.yml | 39 ++++------ 2 files changed, 37 insertions(+), 95 deletions(-) diff --git a/.github/workflows/assets/linux.Dockerfile b/.github/workflows/assets/linux.Dockerfile index 35d546ca0..9124f0365 100644 --- a/.github/workflows/assets/linux.Dockerfile +++ b/.github/workflows/assets/linux.Dockerfile @@ -1,43 +1,28 @@ # The Arduino IDE Linux build workflow job runs in this container. # syntax=docker/dockerfile:1 -FROM ubuntu:18.04 - -# See: https://unofficial-builds.nodejs.org/download/release/ -ARG node_version="18.17.1" +# See: https://hub.docker.com/_/ubuntu/tags +FROM ubuntu:18.10 +# This is required in order to use the Ubuntu package repositories for EOL Ubuntu versions: +# https://help.ubuntu.com/community/EOLUpgrades#Update_sources.list RUN \ - apt-get \ - --yes \ - update + sed \ + --in-place \ + --regexp-extended \ + --expression='s/([a-z]{2}\.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g' \ + "/etc/apt/sources.list" -# This is required to get add-apt-repository RUN \ apt-get \ --yes \ - install \ - "software-properties-common=0.96.24.32.22" + update -# Install Git -# The PPA is required to get a modern version of Git. The version in the Ubuntu 18.04 package repository is 2.17.1, -# while action/checkout@v3 requires 2.18 or higher. RUN \ - add-apt-repository \ - --yes \ - "ppa:git-core/ppa" && \ - apt-get \ - --yes \ - update && \ - \ apt-get \ --yes \ install \ - "git" && \ - \ - apt-get \ - --yes \ - purge \ - "software-properties-common" + "git" # The repository path must be added to safe.directory, otherwise any Git operations on it would fail with a # "dubious ownership" error. actions/checkout configures this, but it is not applied to containers. @@ -51,18 +36,12 @@ ENV \ # Install Python # The Python installed by actions/setup-python has dependency on a higher version of glibc than available in the -# ubuntu:18.04 container. +# container. RUN \ apt-get \ --yes \ install \ - "python3.8-minimal=3.8.0-3ubuntu1~18.04.2" && \ - \ - ln \ - --symbolic \ - --force \ - "$(which python3.8)" \ - "/usr/bin/python3" + "python3.7-minimal=3.7.3-2~18.10" # Install Theia's package dependencies # These are pre-installed in the GitHub Actions hosted runner machines. @@ -70,43 +49,15 @@ RUN \ apt-get \ --yes \ install \ - "libsecret-1-dev=0.18.6-1" \ - "libx11-dev=2:1.6.4-3ubuntu0.4" \ + "libsecret-1-dev=0.18.6-3" \ + "libx11-dev=2:1.6.7-1" \ "libxkbfile-dev=1:1.0.9-2" -# Install Node.js -# It is necessary to use the "unofficial" linux-x64-glibc-217 build because the official Node.js 18.x is dynamically -# linked against glibc 2.28, while Ubuntu 18.04 has glibc 2.27. -ARG node_installation_path="/tmp/node-installation" -ARG artifact_name="node-v${node_version}-linux-x64-glibc-217" -RUN \ - mkdir "$node_installation_path" && \ - cd "$node_installation_path" && \ - \ - apt-get \ - --yes \ - install \ - "wget=1.19.4-1ubuntu2.2" && \ - \ - archive_name="${artifact_name}.tar.xz" && \ - wget \ - "https://unofficial-builds.nodejs.org/download/release/v${node_version}/${archive_name}" && \ - \ - apt-get \ - --yes \ - purge \ - "wget" && \ - \ - tar \ - --file="$archive_name" \ - --extract && \ - rm "$archive_name" -ENV PATH="${PATH}:${node_installation_path}/${artifact_name}/bin" - -# Install Yarn -# Yarn is pre-installed in the GitHub Actions hosted runner machines. +# Target python3 symlink to Python 3.7 installation. It would otherwise target version 3.6 due to the installation of +# the `python3` package as a transitive dependency. RUN \ - npm \ - install \ - --global \ - "yarn@1.22.19" + ln \ + --symbolic \ + --force \ + "$(which python3.7)" \ + "/usr/bin/python3" diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index cdc2b0f4e..e4e09b73f 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -48,6 +48,7 @@ env: GO_VERSION: '1.21' # See: https://github.com/actions/setup-node/#readme NODE_VERSION: '18.17' + YARN_VERSION: '1.22' JOB_TRANSFER_ARTIFACT_PREFIX: build-artifacts- CHANGELOG_ARTIFACTS: changelog STAGED_CHANNEL_FILE_ARTIFACT_PREFIX: staged-channel-file- @@ -316,21 +317,26 @@ jobs: if not exist "${{ matrix.config.working-directory }}" mklink /d "${{ matrix.config.working-directory }}" "C:\actions-runner\_work\arduino-ide\arduino-ide" - name: Checkout - if: fromJSON(matrix.config.container) == null uses: actions/checkout@v4 - - name: Checkout - # actions/checkout@v4 has dependency on a higher version of glibc than available in the Linux container. - if: fromJSON(matrix.config.container) != null - uses: actions/checkout@v3 - name: Install Node.js - if: fromJSON(matrix.config.container) == null && runner.name != 'WINDOWS-SIGN-PC' + if: runner.name != 'WINDOWS-SIGN-PC' uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} registry-url: 'https://registry.npmjs.org' - cache: 'yarn' + # Yarn is a prerequisite for the action's cache feature, so caching should be disabled when running in the + # container where Yarn is not pre-installed. + cache: ${{ fromJSON(matrix.config.container) == null && 'yarn' || null }} + + - name: Install Yarn + if: runner.name != 'WINDOWS-SIGN-PC' + run: | + npm \ + install \ + --global \ + "yarn@${{ env.YARN_VERSION }}" - name: Install Python 3.x if: fromJSON(matrix.config.container) == null && runner.name != 'WINDOWS-SIGN-PC' @@ -339,33 +345,18 @@ jobs: python-version: '3.11.x' - name: Install Go - if: fromJSON(matrix.config.container) == null && runner.name != 'WINDOWS-SIGN-PC' + if: runner.name != 'WINDOWS-SIGN-PC' uses: actions/setup-go@v5 with: go-version: ${{ env.GO_VERSION }} - - name: Install Go - # actions/setup-go@v5 has dependency on a higher version of glibc than available in the Linux container. - if: fromJSON(matrix.config.container) != null && runner.name != 'WINDOWS-SIGN-PC' - uses: actions/setup-go@v4 - with: - go-version: ${{ env.GO_VERSION }} - - name: Install Taskfile - if: fromJSON(matrix.config.container) == null && runner.name != 'WINDOWS-SIGN-PC' + if: runner.name != 'WINDOWS-SIGN-PC' uses: arduino/setup-task@v2 with: repo-token: ${{ secrets.GITHUB_TOKEN }} version: 3.x - - name: Install Taskfile - # actions/setup-task@v2 has dependency on a higher version of glibc than available in the Linux container. - if: fromJSON(matrix.config.container) != null && runner.name != 'WINDOWS-SIGN-PC' - uses: arduino/setup-task@v1 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - version: 3.x - - name: Package env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}