Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): pnpm support #3822

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

feat(core): pnpm support #3822

wants to merge 37 commits into from

Conversation

erickzhao
Copy link
Member

@erickzhao erickzhao commented Jan 28, 2025

Closes #2633
Closes #3574

ref #3813

This PR aims to provide a basic level of pnpm support for Electron Forge.

Context

Under the hood, Forge's init and import commands use a Node.js package manager to install dependencies and otherwise execute commands. The logic is set up to use yarn if detected, and falls back to npm otherwise.

With the proliferation of alternative package managers in recent years, one of the most popular requests on our issue tracker was to support pnpm (#2633).

However, this required a decent amount of refactoring since we relied on the unmaintained (yarn-or-npm) package to perform this package manager detection. This locked us out of support for other package managers until this utility was completely refactored.

Prior art

Big thanks to @makeryi and @goosewobbler for opening their respective PRs attempting support (#3351 and #3574).

Based on their efforts, I first landed #3813, which replaces yarn-or-npm with the more generalized detect-package-manager package without changing the API signature for our utility APIs.

(This utility has also been replaced. See next section.)

This PR follows that up by replacing our package manager utilities (which still only supported yarn or npm) with a generic package manager detection system that returns the current package manager as well as its associated commands and flags.

Implementation

Package manager detection

The core of this PR refactors the yarn-or-npm file in @electron-forge/core-utils to instead be a generic package-manager util.

Instead of returning just the name of the package manager, resolving which package manager to use also returns its install command as well as the command flags we may want to use.

export type SupportedPackageManager = 'yarn' | 'npm' | 'pnpm';
export type PMDetails = { executable: SupportedPackageManager; install: string; dev: string; exact: string };
const MANAGERS: Record<SupportedPackageManager, PMDetails> = {
yarn: {
executable: 'yarn',
install: 'add',
dev: '--dev',
exact: '--exact',
},
npm: {
executable: 'npm',
install: 'install',
dev: '--save-dev',
exact: '--save-exact',
},
pnpm: {
executable: 'pnpm',
install: 'add',
dev: '--save-dev',
exact: '--save-exact',
},
};

Note

In theory, we could do away with these objects for now because the shorthand <pm> install -E -D works across npm, yarn, and pnpm. However, this setup future-proofs us against future commands/flags that we'd want to add that aren't necessarily compatible.

The behaviour for resolvePackageManager now differs. If an unsupported package manager is detected via NODE_INSTALLER or detect-package-manager, we default to npm instead of whatever the detected system package manager is.

  1. If process.env.NODE_INSTALLER is detected, use its value.
  2. If process.env.npm_config_user_agent is detected, parse the package manager and its version from the user agent.
  3. If a lockfile is detected in an ancestor directory, use that.
  4. Otherwise, fall back to npm.

This solves the case where we'd detect an unsupported package manager (e.g. bun).

node-linker=hoisted

Out of the box, pnpm provides improved disk space efficiency and install speed because it symlinks all dependencies and stores them in a central location on disk (see Motivation section of the pnpm docs for more details).

However, Forge expects node_modules to exist on disk at specific locations because we bundle all production npm dependencies into the Electron app when packaging.

To that end, we expect the config to be node-linker=hoisted when running Electron Forge. I added a clause to check-system.ts to ensure this by checking the value of pnpm config get node-linker.

async function checkPnpmNodeLinker() {
const nodeLinkerValue = await spawnPackageManager(['config', 'get', 'node-linker']);
if (nodeLinkerValue !== 'hoisted') {
throw new Error('When using pnpm, `node-linker` must be set to "hoisted". Run `pnpm config set node-linker hoisted` to set this config value.');
}
}

This setting is added out of the box when initializing Forge templates with pnpm via .npmrc:

node-linker = hoisted

if (pm.executable === 'pnpm') {
rootFiles.push('.npmrc');
}

pnpm workspaces

I think we actually already supported pnpm workspaces via:

async function findAncestorNodeModulesPath(dir: string, packageName: string): Promise<string | undefined> {
d('Looking for a lock file to indicate the root of the repo');
const lockPath = await findUp(['package-lock.json', 'yarn.lock', 'pnpm-lock.yaml'], { cwd: dir, type: 'file' });
if (lockPath) {
d(`Found lock file: ${lockPath}`);
const nodeModulesPath = path.join(path.dirname(lockPath), 'node_modules', packageName);
if (await fs.pathExists(nodeModulesPath)) {
return nodeModulesPath;
}
}
return Promise.resolve(undefined);
}

Supported pnpm version ranges

I'm not sure if v8.0.0 is a correct lower bound for pnpm, but it's the lowest version they have for their documentation so I'm assuming something has to do with their support policy.

Testing

This PR expands the existing test suite to also support pnpm (mostly in api.slow.spec.ts).

Note

Previously, we actually didn't test packaging/making for npm. This PR has the side effect of fully testing the API across all supported package managers, which does bring up the Time to Green for our CI.

@erickzhao erickzhao changed the title Pnpmpnpmpnpmpnpm feat: pnpm support Jan 28, 2025
@erickzhao erickzhao changed the title feat: pnpm support feat(core): pnpm support Jan 28, 2025
@@ -46,7 +46,7 @@ commands:
- run:
name: 'Run fast tests'
command: |
yarn test:fast --reporter=junit --outputFile="./reports/out/test_output.xml"
yarn test:fast --reporter=default --reporter=junit --outputFile="./reports/out/test_output.xml"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the default reporter both gives us a clearer idea of the failures in CI and prevents us from hitting the 10-minute CircleCI no output timeout.

- run:
name: 'Install pnpm'
command: |
npm install -g [email protected]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also do this in corepack in theory, but I found this to be easier to grok.

all: '>= 1.0.0',
},
pnpm: {
all: '>= 8.0.0',
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is necessary, but the Node compatibility matrix in the pnpm docs only lists up to pnpm 8 so I feel like this is a decent lower bound: https://pnpm.io/installation#compatibility

Comment on lines 49 to 65

function warnIfPackageManagerIsntAKnownGoodVersion(packageManager: string, version: string, allowlistedVersions: { [key: string]: string }) {
const osVersions = allowlistedVersions[process.platform];
const versions = osVersions ? `${allowlistedVersions.all} || ${osVersions}` : allowlistedVersions.all;
const versionString = version.toString();
checkValidPackageManagerVersion(packageManager, versionString, versions);
}

async function checkPackageManagerVersion() {
const version = await forgeUtils.yarnOrNpmSpawn(['--version']);
const versionString = version.toString().trim();
if (await forgeUtils.hasYarn()) {
warnIfPackageManagerIsntAKnownGoodVersion('Yarn', versionString, YARN_ALLOWLISTED_VERSIONS);
return `yarn@${versionString}`;
} else {
warnIfPackageManagerIsntAKnownGoodVersion('NPM', versionString, NPM_ALLOWLISTED_VERSIONS);
return `npm@${versionString}`;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These functions were refactored out into a single checkPackageManager utility.

hasYarn = hasYarn;

yarnOrNpmSpawn = yarnOrNpmSpawn;
spawnPackageManager = spawnPackageManager;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a change in API signatures for @electron-forge/core-utils a breaking change? If it is, I can add a temporary shim for these utils and deprecate them.

@erickzhao erickzhao marked this pull request as ready for review January 30, 2025 00:44
@erickzhao erickzhao requested a review from a team as a code owner January 30, 2025 00:44
@erickzhao erickzhao requested a review from BlackHole1 January 30, 2025 00:48
@erickzhao
Copy link
Member Author

Is this PR ready to merge?

@only-issues it hasn't gotten any PR approvals yet, so not yet.

.npmrc Outdated Show resolved Hide resolved
packages/api/cli/src/util/check-system.ts Outdated Show resolved Hide resolved
packages/api/core/spec/slow/api.slow.spec.ts Outdated Show resolved Hide resolved
packages/template/base/tmpl/.npmrc Outdated Show resolved Hide resolved
@@ -15,12 +15,12 @@ describe('ViteTypeScriptTemplate', () => {
let dir: string;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can use context to avoid mutable variables.

This suggestion is beyond the scope of the current PR, just a side note :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely worth exploring in a follow-up PR!

packages/utils/core-utils/spec/package-manager.spec.ts Outdated Show resolved Hide resolved
vi.spyOn(console, 'warn').mockImplementation(() => undefined);
});
afterEach(() => {
if (!nodeInstaller) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand this. Is it a mistake? Based on the context, I feel it should be written as: if (nodeInstaller)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added clarifying comments in 519aa9f (because I got confused again myself when skimming through my implementation), but the gist of it is:

  • Before the test, we store the initial process.env.NODE_INSTALLER value.
  • During the test, process.env.NODE_INSTALLER gets added in the test arrangement code.
  • After the test, we restore the initial value. If process.env.NODE_INSTALLER wasn't present before the test ran, it gets deleted again. Otherwise, it gets restored to the initial value we saved before the test.

packages/utils/core-utils/src/package-manager.ts Outdated Show resolved Hide resolved
erickzhao and others added 2 commits February 4, 2025 22:58
@erickzhao erickzhao requested review from BlackHole1, erikian and a team February 5, 2025 22:51
Copy link
Member

@BlackHole1 BlackHole1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you for your work!

@erickzhao erickzhao requested a review from erikian February 6, 2025 23:14
Copy link
Member

@erikian erikian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, just some final notes about templates + workspaces

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I've been testing how templates interact with workspaces using the script below. I don't recall right now why I gave up on using yarn link:prepare to use the local Forge packages, but something wasn't quite working and I did some controversial overrides so I could at least do some basic testing. Before running this script, I do a clean Forge install/build with git clean -dfx && yarn && yarn build:

#!/usr/bin/env bash

PNPM_WORKSPACE_ROOT=forge-pnpm
FORGE_ROOT=~/Documents/electron/forge

# optional cleanup
# rm -rf $PNPM_WORKSPACE_ROOT || true

mkdir $PNPM_WORKSPACE_ROOT
cd $PNPM_WORKSPACE_ROOT

# Create a basic `package.json`
pnpm init

# Use local Forge packages - all of them
npm pkg set pnpm.overrides."@electron-forge/cli"=$FORGE_ROOT/packages/api/cli
npm pkg set pnpm.overrides."@electron-forge/core"=$FORGE_ROOT/packages/api/core
npm pkg set pnpm.overrides."@electron-forge/maker-appx"=$FORGE_ROOT/packages/maker/appx
npm pkg set pnpm.overrides."@electron-forge/maker-base"=$FORGE_ROOT/packages/maker/base
npm pkg set pnpm.overrides."@electron-forge/maker-deb"=$FORGE_ROOT/packages/maker/deb
npm pkg set pnpm.overrides."@electron-forge/maker-dmg"=$FORGE_ROOT/packages/maker/dmg
npm pkg set pnpm.overrides."@electron-forge/maker-flatpak"=$FORGE_ROOT/packages/maker/flatpak
npm pkg set pnpm.overrides."@electron-forge/maker-pkg"=$FORGE_ROOT/packages/maker/pkg
npm pkg set pnpm.overrides."@electron-forge/maker-rpm"=$FORGE_ROOT/packages/maker/rpm
npm pkg set pnpm.overrides."@electron-forge/maker-snap"=$FORGE_ROOT/packages/maker/snap
npm pkg set pnpm.overrides."@electron-forge/maker-squirrel"=$FORGE_ROOT/packages/maker/squirrel
npm pkg set pnpm.overrides."@electron-forge/maker-wix"=$FORGE_ROOT/packages/maker/wix
npm pkg set pnpm.overrides."@electron-forge/maker-zip"=$FORGE_ROOT/packages/maker/zip
npm pkg set pnpm.overrides."@electron-forge/plugin-auto-unpack-natives"=$FORGE_ROOT/packages/plugin/auto-unpack-natives
npm pkg set pnpm.overrides."@electron-forge/plugin-base"=$FORGE_ROOT/packages/plugin/base
npm pkg set pnpm.overrides."@electron-forge/plugin-electronegativity"=$FORGE_ROOT/packages/plugin/electronegativity
npm pkg set pnpm.overrides."@electron-forge/plugin-fuses"=$FORGE_ROOT/packages/plugin/fuses
npm pkg set pnpm.overrides."@electron-forge/plugin-local-electron"=$FORGE_ROOT/packages/plugin/local-electron
npm pkg set pnpm.overrides."@electron-forge/plugin-vite"=$FORGE_ROOT/packages/plugin/vite
npm pkg set pnpm.overrides."@electron-forge/plugin-webpack"=$FORGE_ROOT/packages/plugin/webpack
npm pkg set pnpm.overrides."@electron-forge/publisher-base-static"=$FORGE_ROOT/packages/publisher/base-static
npm pkg set pnpm.overrides."@electron-forge/publisher-base"=$FORGE_ROOT/packages/publisher/base
npm pkg set pnpm.overrides."@electron-forge/publisher-bitbucket"=$FORGE_ROOT/packages/publisher/bitbucket
npm pkg set pnpm.overrides."@electron-forge/publisher-electron-release-server"=$FORGE_ROOT/packages/publisher/electron-release-server
npm pkg set pnpm.overrides."@electron-forge/publisher-gcs"=$FORGE_ROOT/packages/publisher/gcs
npm pkg set pnpm.overrides."@electron-forge/publisher-github"=$FORGE_ROOT/packages/publisher/github
npm pkg set pnpm.overrides."@electron-forge/publisher-nucleus"=$FORGE_ROOT/packages/publisher/nucleus
npm pkg set pnpm.overrides."@electron-forge/publisher-s3"=$FORGE_ROOT/packages/publisher/s3
npm pkg set pnpm.overrides."@electron-forge/publisher-snapcraft"=$FORGE_ROOT/packages/publisher/snapcraft
npm pkg set pnpm.overrides."@electron-forge/template-base"=$FORGE_ROOT/packages/template/base
npm pkg set pnpm.overrides."@electron-forge/template-vite-typescript"=$FORGE_ROOT/packages/template/vite-typescript
npm pkg set pnpm.overrides."@electron-forge/template-vite"=$FORGE_ROOT/packages/template/vite
npm pkg set pnpm.overrides."@electron-forge/template-webpack-typescript"=$FORGE_ROOT/packages/template/webpack-typescript
npm pkg set pnpm.overrides."@electron-forge/template-webpack"=$FORGE_ROOT/packages/template/webpack
npm pkg set pnpm.overrides."@electron-forge/core-utils"=$FORGE_ROOT/packages/utils/core-utils
npm pkg set pnpm.overrides."@electron-forge/test-utils"=$FORGE_ROOT/packages/utils/test-utils
npm pkg set pnpm.overrides."@electron-forge/tracer"=$FORGE_ROOT/packages/utils/tracer
npm pkg set pnpm.overrides."@electron-forge/shared-types"=$FORGE_ROOT/packages/utils/types
npm pkg set pnpm.overrides."@electron-forge/web-multi-logger"=$FORGE_ROOT/packages/utils/web-multi-logger
npm pkg set pnpm.overrides."@electron-forge/create-electron-app"=$FORGE_ROOT/packages/external/create-electron-app

# Set up a pnpm workspace
echo "packages:
  - 'packages/*'
" > pnpm-workspace.yaml

# Uncomment to create the `pnpm-lock.yaml` file before initializing the template
# pnpm install

# Initialize a Forge template in `./packages/desktop` using the local `create-electron-app` package
pnpm create electron-app@$FORGE_ROOT/packages/external/create-electron-app packages/desktop

Some test findings:

  • running pnpm install in the workspace root prints WARN The field "pnpm.onlyBuiltDependencies" was found in /Users/ian/Documents/electron/repros/forge-pnpm/packages/desktop/package.json. This will not take effect. You should configure "pnpm.onlyBuiltDependencies" at the root of the workspace instead., so we'd need to set up that field in the workspace root when initializing templates.
  • similarly, running pnpm start in packages/desktop prints the ✖ When using pnpm, `node-linker` must be set to "hoisted". Run `pnpm config set node-linker hoisted` to set this config value. error, so I think the .npmrc file should also live in the workspace root.
  • running pnpm package in packages/desktop fails with Error: Failed to locate module "electron-squirrel-startup" from "/Users/ian/Documents/electron/repros/forge-pnpm/packages/desktop". This might be related to my weird local setup (or the fact that I'm testing on macOS), but could also be a side-effect of not having node-linker = hoisted in the workspace root.

@goosewobbler goosewobbler mentioned this pull request Feb 7, 2025
3 tasks
@goosewobbler
Copy link

goosewobbler commented Feb 7, 2025

I have a question about use of hoist-pattern vs node-linker=hoisted. My usecase is a monorepo, some packages in the repo use Forge but most don't. I had some issue with using node-linker=hoisted for the repo. Will this PR allow for the use of hoist-pattern to target specific dependencies rather than hoisting everything with node-linker=hoisted?

Presumably another option for this is to try and get PNPM support for hoisting dependencies on a per-workspace / package basis.

@erickzhao
Copy link
Member Author

Hey @goosewobbler, thanks for bringing that up.

I have a question about use of hoist-pattern vs node-linker=hoisted. My usecase is a monorepo, some packages in the repo use Forge but most don't. I had some issue with using node-linker=hoisted for the repo. Will this PR allow for the use of hoist-pattern to target specific dependencies rather than hoisting everything with node-linker=hoisted?

I think I can amend the PR to change that!

@goosewobbler
Copy link

goosewobbler commented Feb 7, 2025

Nice, there's also public-hoist-pattern, not sure if that one will be useful here. Admittedly I have used neither of these, just trying to ensure we have options in case node-linker=hoisted causes issues again.

@erickzhao
Copy link
Member Author

Added exceptions for hoist-pattern and public-hoist-pattern, although the PR doesn't validate whether or not the actual pattern works (use at your own risk!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support pnpm
5 participants