Skip to main content

Npm Q&A

Npm is a package manager for JavaScript projects. What this means is that it enables easy installation of packages (published to the public registry, https://www.npmjs.com, or private registries) while also installing nested dependencies between packages. I received a bunch of questions about npm, and here are the answers! Note that I haven't looked at implementation details or any specs on how npm works, these answers are merely based on my observations after nearly a decade of using npm.

Questions​

Why is there a .package-lock.json file in node_modules?​

  • I'd guess that it's to keep track of exactly the package-lock.json file that generated the current installations. But idk for sure, and that file being there has never mattered to me.

How does npm handle a mono-repo?​

  • In a mono-repo, you define a "workspaces" field in the root-level package.json. Typically it's just ["./packages/*"],. Each matched directory with a package.json file then turns in a "workspace", or a sub-package. For installation purposes (npm i), these workspaces are treated just like normal npm dependencies: they and all their dependencies get installed as part of the mono-repo's npm i command.

Why did you make mono-vir if npm handles mono-repos already?​

  • npm allows running npm scripts in each workspace with the --workspaces flag (like npm test --workspaces) but they are not run in dependency order.
  • mono-vir handles running scripts within workspaces in the order of their dependency graph. For example, if you have backend and frontend workspaces that depend on a common workspace, the common workspace will be executed first when run inside of mono-vir. This is critical for some workflows, like compiling TypeScript to JavaScript. (mono-vir for-each npm run compile)
  • mono-vir also allows running scripts within workspaces in parallel. If the order of execution doesn't matter, this can be used to easily run tests in all packages all in parallel (mono-vir for-each-async npm test).

Why are some dependencies installed into a mono-repo's root node_modules and others are stored in individual workspace node_modules?​

  • This likely has to do with version mismatches. If mono-repo workspace A depends on TypeScript v3 but a different mono-repo workspace B depends on TypeScript v4, a shared top-level dependency can't satisfy both requirements. TypeScript in this case will be installed inside a workspace node_modules for at least one of the workspaces.

If two mono-repo workspaces depend on the same package with the same version, will that package be downloaded twice?​

  • No.
  • It is possible to get into a state where it could be downloaded twice. Like if they original were different versions but then changed to be the same version, the dependency might be duplicated in different workspace node_modules folders (I've noticed behavior like this). But with a clean install (delete all node_modules folders), there won't be duplicates like that.

Why are some folders in node_modules prefixed with an @?​

  • Because those are part of the package names.
  • Packages can be scoped within a specific user or "organization" in npm. The @ signifies this.
  • Examples:
    • virmator is not scoped within a user or organization.
    • @virmator/lint is scoped within the organization virmator.

What is node_modules/.bin?​

  • This is populated by packages that you've installed that have self-defined "bin" scripts.
  • These scripts are intended to be run directly in the CLI, just like you use cd, git, or grep. These are all appended to PATH whenever you enter an npm context, such as by using npx or npm run with a package.json script.
  • Any globally installed packages (which I usually discourage doing) will be populated into your entire shell PATH. Those won't need npx or npm run.

What about other node_modules/.* folders? Like node_modules/.cache or node_modules/.prisma?​

  • There are populated by packages themselves. node_modules is just a folder, so anything can save anything in there.
  • Prisma uses node_modules/.prisma as a place to dump its generated JS client.
  • I've seen various packages stick outputs into node_modules/.cache, like Prettier.
  • One interesting thing that I've noticed is that node_modules/.* folders are not automatically overwritten when running npm i, so it's a good place to dump generated outputs.

What happens if you depend on different versions of a package?​

  • A single package can't directly depend on different versions of the same package.
  • It is, however, common for nested dependencies to depend on incompatible versions of the same package. In this case, a separate node_modules folder is created within the dependency in node_modules.
  • For example, if dependency A depends on package Cv3 and dependency B depends on package Cv5, the generated folder structure might look like this: 
    node_modules/
        A/
            node_modules/
                C/
            package.json
        B/
            node_modules/
                C/
            package.json

Why do I frequently run into issues on Windows with sometimes needing npx and sometimes not needing npx?​

  • Bugs. Something to do with how sub-shells are launched that causes them to not inherit the parent's npx context. Probably because Windows is less tested by developers.
  • It could also be related to interactions between Git Bash, WSL, cmd, PowerShell, etc.
  • I don't use Windows, so I have no personal experience debugging this. Maybe there's a config somewhere that would fix this permanently.

In package.json, what do the following fields mean?​

  • version: the version that the respective package was downloaded as (for package.jsons within node_modules) or the version that you're about to push for your own package (for a root package.json).
  • license: the license name that the package has licensed itself with.
  • dependencies: the packages that this package depends on. These will be downloaded through npm i.
  • peerDependencies: this is the same as dependencies except that it is meant for listing compatibility with other packages that this package is meant to be used with. It is typically more lenient.
    • For example, if you're building a Prettier plugin, you would want to list the Prettier package as a peer dependency because your plugin won't be using Prettier directly but will actually be used by Prettier. This field is then used to list the versions of Prettier that your plugin is compatible with.
  • engines: the runtime engines that your package is compatible with. For example, you could list a set of Node.js versions here.
  • bin: this lists the entry point(s) for your package as a CLI script.
  • all package.json fields are explained in the docs: https://docs.npmjs.com/cli/configuring-npm/package-json

What about fields in package-lock.json?​

How do you determine whether an npm package should or should not be used in a given app?​

Here are the steps:

  1. read the package's documentation
  2. look at what export fields the package's package.json defines:
    • "main": this is meant for CJS exports but can also be used for ESM exports
    • "module": this is meant for ESM export
  3. look at the source code itself to determine which module format it's using
    • any use of require means it's CJS
    • any use of top level import (not the dynamic import function import()) means it's ESM
  4. determine if your app's build process supports the module system that the package uses
    • For example, without an expensive conversion process, frontend builds only support ESM.
  5. check if the package is using any built-in Node.js modules (like node:path).
    • These won't work in a browser unless you have polyfills for them (which I discourage).

This will usually be good enough. In the end, though, the best way to check is to simply try importing and running the package in your app see if it works!

How do you publish package to npm?​

  1. Create a package.json within a given directory
  2. Populate the "name" and "version" field with valid values
  3. run npm publish

That's all you need! There's no requirement for code or directory structure, beyond the required package.json file.

To ship something with exports, you'll need to also populate the "module" and/or "main" package.json fields.

My personal flow, which has been heavily abstracted, goes as follows:

  1. Create a new repo in GitHub.
  2. Clone the repo locally.
  3. Run npm init -y in that new repo.
  4. Run npm i -D virmator and npx virmator init
  5. Write code and commit it.
  6. Run virmator publish npm run test:all
    • this command does a bunch of stuff, including:
      • automatic version incrementing
      • running the provided command (npm run test:all in this case)
      • npm publish
      • git push

Steps 1-4 are only needed for new packages.

How do you publish a CLI command package?​

  1. create an entry point file (I usually name mine cli.ts)
  2. make sure to include shebang at the top of the entry point
    • #!/usr/bin/env -S npx tsx for TypeScript
    • #!/usr/bin/env bash for Bash
    • #!/usr/bin/env node for raw (or compiled) JavaScript
  3. set "bin" in your package.json to point to the entry point file
  4. publish your package as usual

What is package-lock.json?​

  • package.json dependencies frequently use version prefixes (like ^ or ~), which I highly recommend using. This can result in different versions installed than the explicitly listed version in your package.json file. package-lock.json keeps track of the exact version you installed so you can have reproducible builds in testing, staging, production, etc.

What do the different version prefixes mean?​

What is ncu?​

  • ncu is the CLI command for the npm-check-updates package. It is used to update your package.json dependency versions to match the latest.
  • It's an easy script for bulk updating dependencies.

Can you have nested workspaces?​

  • Meaning, can you have a workspaces field in a package.json that is itself a workspace (aka nested within a mono-repo)?
  • No. From personal testing, npm simply ignores the nested workspaces field.

What's the difference between npm ci and npm i?​

  • npm i: (or npm install)
    • installs all dependencies
    • installs the latest version for each dependency that matches the used prefix (if any)
    • updates package-lock.json
    • can be used to add individual dependencies (like npm i @augment-vir/common)
  • npm ci:
    • removes node_modules
    • installs all dependencies exactly as defined within package-lock.json (and does not update package-lock.json)
    • cannot be used to add a new individual dependency
    • should only be used (never npm i) in automated or deploy environments
    • is good to use in local dev to make sure your dependencies are up to date if you don't want to update package-lock.json
  • Full docs: https://docs.npmjs.com/cli/commands/npm-ci