Using Webpacker with Rails Engines

November 26, 2021


⚠️ This post or page is under development.

Hey. Just a warning: this page is a bit rough yet! I try to release my work while it's still rough and then take multiple passes to refine it. Your patience and gentle kindness is appreciated. :)


NB: The following post is duplicated on this thread on Github issues.

There are four primary options in terms of using webpacker with a Rails Engine / Gem.

The first point of decision is whether you need your modules to be isolated from each other or if you're okay sharing between the host app and the engine. If you choose isolation, then your engine assets are compiled on their own and loaded separately. This is probably the right choice if you're worried about conflicts between the two sides, or if you can't realistically specify what version of a module the host app must use (e.g. a FOSS engine that has to operate as a black box and can't guarantee that the host app won't have a conflicting module version installed).

(If it's not quite clear what problem can occur here, imagine this situation: the engine's javascript is written using `foo.js` version 1, and it's installed into the host app which already uses `foo.js` version 2. Because there's not isolation, the engine uses whatever module `import {} from "foo"` loads, which could potentially (? -- I'm not a webpack expert by any means) be version 2, causing many errors because of an incompatible API.)

The problem is that isolation is tricky to achieve. There's no consensus among those who have attempted to install a separate instance of webpacker within their engine as to whether it works, let alone whether it works well. If you _can_ force the host app to use the engine's module versions, i.e. you can say by dint of controlling both that they _will_ both use `foo.js` version 1, then _you should go that route because it's much easier_.

Options 1 and 2: Shared Modules

What this should look like when it's all working:

```js

// host: /app/javascript/packs/application.js

import Rails from "@rails/ujs"

//...

import "my_engine"

```

There are three primary obstacles to overcome with shared modules:

1. Get webpacker to load the engine's assets;

2. Get webpack-dev-server to watch the engine's assets and rebuild when they change;

3. Make sure that `yarn install` also triggers `yarn install` for engines (i.e. for ease of Heroku deployment).

In both cases, you'll need to set the engine up as follows. We'll assume that your engine is called `my_engine`.

1. `cd /path/to/my_engine`

2. `mkdir -p app/javascript/my_engine`;

2. Edit `app/javascript/my_engine/index.js` and insert:

```js

// Require your modules here, but you'll run into conflicts if you

// try to let the host app AND the engine start Rails-UJS /

// Turbolinks etc... I generally let the host / dummy app start them.

console.log("Called from my_engine.");

```

There are two different options here depending on whether the source for your engine lives within the host app.

Option One: Engine source within host app (e.g. `/vendor/my_engine`)

This option is largely cribbed from @brendon's [answer above](https://github.com/rails/webpacker/issues/348#issuecomment-719151447). Caveat lector that I haven't tried this option out, but from a theoretical basis it should work well. (Please let me know if it doesn't work).

Get webpacker to load the engine's assets

1. In `config/webpacker.yml`, change `additional_paths` to read `additional_paths: ["/vendor/my_engine/app/javascript"]`

Get webpack-dev-server to watch the engine's assets and rebuild when they change;

This is already taken care of by step 1.

Make sure that `yarn install` also triggers `yarn install` for engines

This particular answer is largely borrowed from @valterkraemer's [answer](https://github.com/rails/webpacker/issues/348#issuecomment-635480949), but I've simplified it to not need a separate script file at the cost of having to explicitly state which engines to `yarn install`. I'm also using an approach [inspired by](https://github.com/rails/webpacker/issues/348#issuecomment-653284703) @tubbo but updated to use a non-deprecated command.

1. Edit the host application's `package.json`, and append the following before the final brace:

```json

"scripts": {

"postinstall": "./scripts/yarn_postinstall.js"

}

```

This tells yarn to execute `scripts/yarn_postinstall.js` after it finishes installing in the host app.

2. In the host app, create `scripts/yarn_postinstall.js`:

```js

#!/usr/bin/env node

const { spawn, execSync } = require("child_process");

const { resolve } = require("path");

function yarn_install_engine(name) {

const engineRoot = execSync(`bundle info --path ${name}`).toString().trim()

spawn("yarn", ["install", "--ignore-scripts"], {

env: process.env,

cwd: engineRoot,

stdio: "inherit"

});

}

// Add your engines here.

yarn_install_engine("my_engine");

```

This script gets the path to the root of the engine and runs `yarn install --ignore-scripts`. You need the flag or this subsequent `yarn install` will trigger the post-install script we specified above, creating an endless loop.

3. `chmod u+x scripts/yarn_postinstall.js`.

You'll have to set the script executable or yarn will give you a permission error.

Option 2: Engine Source Not Vendored inside Host App

This will work whether you're using a local path for the engine or whether it's hosted somewhere else (i.e. you pull it down from Rubygems).

This is the solution that I wound up using and I can vouch for its completeness. I definitely need to credit @tubbo's [answer here](https://github.com/rails/webpacker/issues/348#issuecomment-653284703) and need to praise his work on the `gem()` functionality, which should by all means be adopted into webpacker properly. In fact, this is basically a direct adaption of what he wrote but done within the host app, not webpacker proper.

Get webpacker to load the engine's assets

1. Within the host app's `config/webpack/environment.js`, right after `const { environment} = require("@rails/webpacker")`, insert:

```js

const { execSync } = require('child_process')

function gem(name) {

const root = execSync(`bundle info --path ${name}`).toString().trim()

return `${root}/app/client`

}

environment.config.resolve.modules = [

gem("my_engine"),

];

```

Get webpack-dev-server to watch the engine's assets and rebuild when they change;

1. Within the host app, create `lib/webpacker/engine_extension.rb`:

```ruby

# frozen_string_literal: true

require "webpacker/configuration"

class Webpacker::Configuration

def additional_paths

fetch(:additional_paths) + resolved_paths + engine_paths

end

def engine_paths

engines.map {|engine| "#{engine.root}/app/javascript" }

end

def engines

defined?(::Rails) ? ObjectSpace.each_object(::Rails::Engine) : []

end

end

```

When loaded after the engines, this looks up each Rails Engine and appends `#{engine_root}/app/javascript` to the `additional_paths` webpacker option.

2. Load the extension by editing the host's `bin/webpack` and `bin/webpack-dev-server`:

```ruby

# Find the part that says...

Dir.chdir(APP_ROOT) do

# And insert this next line:

require "webpacker/engine_extension"

Webpacker::WebpackRunner.run(ARGV) # or Webpacker::DevServerRunner.run(ARGV)

end

```

This loads the extension after the Rails engine has been loaded and required by bundler.

Make sure that `yarn install` also triggers `yarn install` for engines

Use the same answer as defined in Option 1 for this one :)

Isolated Modules

I include this section for completeness. This was the first thing that I tried and I certainly find this to be a much more challenging option. I wanted to be able to have my engine assets isolated from my host app, but I also wanted to make sure that I wasn't double-including heavy modules like Bootstrap, Tailwind, or even light things like Stimulus. That meant I needed to go the way of `splitChunks` and `javascript_packs_with_chunks_tag`, and I found that to be downright impossible to get working within a day or two. That's why I'd recommend you skip over option 3 and go to option 4 if this is the path that you're required to go.

Option 3: Install Webpacker in the Engine

Simply follow the [guide](https://github.com/rails/webpacker/blob/5-x-stable/docs/engines.md) in the 5-x-stable branch of this repository.

Option 4: Extract Assets to an NPM package

I did not go this route, but based on [discussions](https://github.com/rails/webpacker/issues/348#issuecomment-633248393) in the thread above, particularly by @jrochkind, @tvdeyen, and @thebravoman, it seems like full isolation requires extracting your assets into a separate package for yarn. Prior art mentioned was the [Ahoy project](https://github.com/ankane/ahoy#javascript).

Wrap-up

Rails engines are absolutely critical pieces of functionality not just because they're the best way bar none to package up additional application-level functionality from the community, but also because they're excellent transitional structures that enable encapsulation of domain-differentiated functionality within an application. Their first-class support means that there's an easier path for a growing/groaning monolith to manage multiple bounded domains without having to either pollute the host app namespace or to move directly to some sort of service-oriented architecture. All the same, once there's appropriate first-class support for engines, then it's dead simple to move to SOA if necessary, since everything -- including assets -- is neatly packaged up.

I personally find their _concept_ to be hugely enticing for managing bloat and technical debt. But until this point it's been an incomplete solution for me because of the missing asset piece. There's Sprockets, sure, but having to include Sprockets inside a webpacker-only host application complicates build steps and it sucks to go without the benefits of the modern build tooling.

I'm very happy that I managed to get this to work for me and I'm crossing my fingers that as I continue to build out my engine I don't discover some awful technical hurdle I've overlooked. And I hope that going forward there are simpler solutions for engines, as well as other first-class abstractions for maintaining strongly separated bounded domains within Rails apps. I wish I had the time to build it myself, but I'm hopeful that what I've compiled here can at minimum serve as a much more convenient and well-compiled jumping-off point for others that wind up with the same problem I have.