How to refer to package files during npx execution

I’m writing here as per the suggestion at www.npmjs.com/support – please let me know if there’s a better place for this question. This is in regards to this project:

I’m writing a bash script intended to be run via npx. The script automates the process of setting up a “starter site” for me – runs snowpack’s version of CRA, sets up tailwind, puts some config files and a snippets file in place. From this script, I’d like to be able to directly access all the files in the NPM published tarball, so that I can copy them (e.g. config files, snippets, etc) to the right place.

I spent a lot of time on google trying to find a good way to accomplish this, but I gave up, and figured out a not-good way of accomplishing this (via the special bash variable $0). See the script in its entirety here by following the link to my project in the first paragraph:

# If we're running locally, $BASE is CWD:
BASE="$(dirname "$0")"

# If we're running via npx, $BASE
# is $CWD/../lib/node_modules/[package name]:
if [ ! -f "$BASE"/package.json ]; then
  BASE="$(dirname "$BASE")"/lib/node_modules/"$(basename "$0")"
fi

# Make sure we have absolute path
BASE="$(cd "$BASE" && pwd)"

The idea is, at least on my mac, when I run npx, it seems to unpack the tarball under ~/.npm/_npx/[xxxxx]/lib/node_modules/[package name]
and the main file indicated by bin in package.json is copied to and invoked via
~/.npm/_npx/[xxxxx]/bin/[package_name]

But this is fragile, given

  1. I can’t find this ~npx/lib/bin directory scheme documented anywhere
  2. The relative locations of the running bin and the rest of the files changes based on whether I’m running the file locally or via npx.

The script works just fine in its current form, but, can anyone tell me a more sane way to refer to the path of the extracted tarball contents of a running npx script/package, from inside the running script?

Edit: I had a pretty reasonable four or so inline links for ease of reading/reference, but then am told “Sorry, new users can only put 2 links in a post.”

Edit:

But this is fragile, given

  1. I can’t find this _npx/lib/bin directory scheme documented anywhere
    _____________^

The issue here is that, unlike Node.js, bash does not realpath the script argument.

So, you can’t just do BASE=$(dirname $0) and expect it to work reliably.

Here’s an example that shows how you can accomplish this using readlink and path resolution in bash.

{
  "name": "@isaacs/npx-find-files",
  "version": "1.0.1",
  "bin": "find-my-files.sh",
  "license": "ISC"
}
#!/usr/bin/env bash

# find-my-files.sh

resolve () {
  local symlink=$1
  local target=$2
  if ! [ -n "$target" ]; then
    echo $symlink
    exit 0
  fi
  local symdir=$(dirname $symlink)
  local targetdir=$(dirname $target)
  local resolveddir=$(cd $symdir &>/dev/null; cd $targetdir &>/dev/null; pwd)
  local resolved="$resolveddir/$(basename $target)"
  resolve "$resolved" "$(readlink $resolved)"
}

target=$(readlink $0)
REALPATH=$(resolve $0 $target)
echo "main is: $REALPATH"
node $(dirname $REALPATH)/lib/index.js
// lib/index.js
console.log('you found me!')
$ npx @isaacs/npx-find-files
main is: /Users/isaacs/.npm/_npx/15b1f1df3db163ec/node_modules/@isaacs/npx-find-files/find-my-files.sh
you found me!

EDIT: realized the symlink walking technique needs to be recursive to avoid bash var scoping, or it won’t work if you have multiple levels of symlinks :wink:

1 Like

Also: note that on Windows (assuming it’s a Windows that has bash installed), using dirname $0 actually will work, because npm installs bins on Windows using a .cmd shim file, rather than a symbolic link (since Windows does not support the shebang line in scripts).

1 Like

Hi Isaac,

I really appreciate the detailed response. I’d like to try to avoid having that much extra code, if possible. What do you think of as an alternative, going into a temp directory and doing either:

INVOKED_PACKAGE="$(ps -p "$(echo "$PPID")" -o command= | cut -d" " -f3)"
echo {} > package.json
npm install "$INVOKED_PACKAGE"
PACKAGE_FILES=node_modules/"$INVOKED_PACKAGE"

or

INVOKED_PACKAGE="$(ps -p $(echo "$PPID") -o command= | cut -d" " -f3)"
PACKAGE="$(npm pack "$INVOKED_PACKAGE")"
tar xzf "$PACKAGE"
PACKAGE_FILES=package

And doing the INVOKED_PACKAGE bit so as not to have to specialize for unscoped-version vs testing/scoped-version vs local-fs-version.

It’s a bit redundant to wind up downloading the package via npx and then downloading again, but I think the trade is worth the simplicity(?)

Also, is there a less roundabout way of coming up with INVOKED_PACKAGE?

Best,
Justin

1 Like

Ok, so rather than just reading the chain of symlinks (which is a chain of 1 in the normal case), you’re going to:

  • spawn ps to list all processes related to the current process
  • pipe that output to cut to pull out the package name via string output (flaky, as you note!)
  • fetch the tarball for that package from the registry (and hope you get the same version)
  • unpack that tarball
  • run your code from the tarball you just fetched

Vs:

  • calling readlink on the argument that bash received
  • resolving the relative directories using cd and pwd (both bash builtins, very fast!)
  • running the code that shipped with the actual script being run.

It may be more lines of bash, but omg, just reading the links and resolving the paths is way less code overall, several orders of magnitude more performant, and portable to every system that has bash, cd, pwd, and readlink (so, every system that has bash). You could make it even more portable by writing as a sh script, of course.

Out of curiosity, why a bash script anyway? Why not just write your bin in node, and have it realpath-ed from the start?

1 Like

I’m embarrassed to say that until just now, I didn’t realize that the file inside the _npx/xxx/bin directory was a symlink. With this new knowledge, I do see that accomplishing this any other way than resolving the symlink would be completely bonkers (and not the good kind of bonkers). Thanks for talking some sense into me.

Re: why bash –
Earlier versions of the script simply echoed strings into files instead of using template files, so I didn’t need to find package files. And bash seemed more natural with all the execa and fs calls I’d have to use in node. But I will reassess and probably rewrite this in node, or some combination of both.