What's new in esy 0.4.x

This is the first public blog post on the esy dev blog. We've been writing lots and lots of code to make esy work well, and until now we haven't communicated much about esy.

What Is Esy

If you've stumbled upon this and don't know what esy is: esy is a "package.json"-like workflow with first class support for native development.

Esy started as part of the reason effort with the goal of implementing isolated and fast native Reason/OCaml project builds that were familiar to JavaScript developers. Esy itself is compiled natively, and can manage packages for most compiled languages (we use it to package/distribute C/C++ packages in addition to Reason and OCaml).

Esy should be familiar to anyone with experience with Yarn, or npm (just run esy inside a directory with a package.json).

• esy provides a unified package management workflow that can install/build packages from opam as well as native packages published to npm

• esy is not tied to any particular choice of a language/platform. Though we are focusing on native Reason/OCaml first.

• esy is build-system agnostic: you don't have to port a project to some specific build system to make it work with esy.

• esy tries to provide "hermetic" builds so that builds of packages are unaffected by the system software installed at global paths: if it works on my machine then it should work on yours.

• esy caches built packages across projects: with a warm cache, new projects are cheap to initialise and build.

This is not an exhaustive list of esy features but we think these are the most important points. Read more about esy's motivations in the esy docs

esy 0.4.x

We recently promoted the 0.4.9 release of esy as latest. That means if you execute:

% npm install -g esy

You'll get esy@0.4.9 which is packed with new features. Below we discuss some of those.

Plug'n'play Installations

TL;DR: esy won't populate node_modules directory with package sources anymore, and esy now supports installing Plug'n'play(pnp) JavaScript dependencies.

In its first iteration, esy initially installed all dependency sources into node_modules. The benefit of this is that approach is that it was compatible with popular JS tooling that relied on node_modules directory. The downside was that for native packages (the main focus of esy) copying sources over from cache to a node_modules directory was unnecessary and a waste of time and disk space.

Furthermore, esy already builds projects purely out-of-source to ensure reproducibility - installing into node_modules was merely done to adhere to JavaScript's conventions, and it actually risked compromising reproducibility.

So how could esy achieve the best of both worlds (JS runtime compatibility) and (maintaining reproducible package builds)?

Fortunately, Yarn team figured out how to ditch node_modules for JS packages. They designed a new convention called Plug'n'play installations ("pnp"). Pnp is a way to run JS packages directly from the global package cache without copying them to node_modules, while having Node, Webpack and other runtimes to be able to resolve code from there.

As of 0.4.x, esy now places a copy of Yarn's pnp.js runtime at installation time into your project when installing JavaScript dependencies. That pnp.js allows node's module resolution to work even if dependencies are not in node_modules. That makes esy JavaScript dependencies more like native dependencies — they don't have to be copied into node_modules. Now, even if your project has JavaScript dependencies, installations with a warm cache are fast. Like, really fast (timings are for esy-ocaml/hello-reason project):

% time esy install
info install 0.4.7
info fetching: done
info installing: done
esy install  0.08s user 0.06s system 93% cpu 0.142 total

This also means that esy is now compatible with the most important parts of the JS ecosystem: webpack, jest, flow, react-scripts, rollup, prettier and others which have all been made pnp compatible thanks to the efforts of Yarn. A few npm packages have still not made themselves pnp compatible — you should file issues on those projects requesting that they support pnp, so that they can be used with Yarn pnp, esy, and any other package manager that adopts the pnp standard.

The workflow for working with JS (non-native) packages with esy looks like this:

• After installing dependencies as usual, execute pnp-enabled NodeJs interpreter:

  % esy node

• Execute npm binaries installed with packages like webpack, flow, jest and similar:

  % esy webpack
  % esy flow
  % esy jest

Alpha preview of Windows support

Another huge feature which shipped in 0.4.x is preliminary native Windows support! Install and use esy directly from native Windows command prompt without needing to install anything else on your system. Yes — it produces pure native Windows binaries that run on any Windows machine.

Thanks to heroic effots of Bryan Phelps and foundational work by the OCaml community developing Reason/OCaml, native project management on Windows are now as easy as on macOS/Linux:

C:\Users\Andrey> git clone https://github.com/facebook/reason
C:\Users\Andrey> cd reason
C:\Users\Andrey> esy
info install 0.4.9
info fetching: done
info installing: done

There is more to say about how Windows support is implemented in esy and we will make sure there's a dedicated post on this in the near future where we will describe some of the foundational compiler and tooling work for Windows that the OCaml community has invested in.

Note though that Windows support is still considered alpha — there are rough edges which needs to be fixed. If you are a developer who works on Windows and want to help — jump into esy/esy issues labelled "Windows" and help us! The good thing is that we have CI running on Windows too (a big thank you to Ulrik Strid for making sure a large part of our test suite can run on Windows).

Other 0.4.x goodies

There are lots and lots of changes in the 0.4.x release line, which could have arguably been versioned 0.5.x. In the future, we are aiming for more incremental releases.

There are too many new features to list, but to highlight a couple of entries in the CHANGELOG:

• Improved workflow for linked packages: working on multiple packages in development is now more efficient and esy allows more flexibility in how you organize your project.

• Support for multiple isolated environments constructed on the fly from package configs. Now you may have multiple .json configurations in your project root (similar to a monorepo) and install/build them in total isolation. If you have a package.dev.json file you can use it explicitly:

  % esy @package.dev.json build

• Flexible package override mechanism which allows turning any source code distribution into an esy package, bringing it into your project with all the benefits of the esy workflow: cached builds, isolated environments, etc. This lets you turn any git hash or URL into an esy package without forking it, even if that package doesn't have a package.json file.

  {
    "resolutions": {
      "pkg-config": {
        "source": "https://...",
        "override": {
          "build": [
            "./configure --prefix #{self.install}",
            "make"
          ],
          "install": [
            "make install"
          ]
        }
      }
    }

• Numerous improvements to esy's user interface: new commands (esy show and esy status), faster esy x ... command invocations, a new set of low level plumbing commands for "scriptable" esy workflows, ...

• New esy.lock format which is easier to review on updates.

• Bug fixes, bug fixes, bug fixes, ...

Some of these features are not documented yet properly but we'll make sure we do this and then post updates on this blog.

Stay tuned!