Webpack 5 upgrade #7351

na9da · 2024-12-05T01:01:35Z

What this PR does

Fixes #6998
Depends on TerriaJS/TerriaMap#718 and #7311

Upgrades webpack and other build dependencies to the latest version.

Note: bulk of the changes are from running the saas-migrator that upgrades .scss files to modern api.

Changes summary

Packages upgraded
webpack, ...
Babel transforms
- Removed transformation to CJS for the default build. We still use it for node builds.
- Other transforms removed plugin-proposal-class-properties, proposal-object-rest-spread, plugin-syntax-dynamic-import. These must now be well supported by browsers.
Cleaned up configureWebpack.js
- Removed rules that are no longer necessary
- Replaced raw-loader file-loader url-loader etc with builtin asset modules
- Removed use of webpack specific qualifiers for imports like raw-loader!file-loader etc to make it more ESM compliant. Added extra config in webpack to correctly handle imports of static files (like csv and xml).
- Added rules to polyfill nodejs modules that are referenced from some of our dependencies.
- Removed hot reload config. AFAICT this was not being used and required maintaining two different code paths in webpack configuration. If required we can add it back later after testing that it works.
- Replaced multiple configureWebpack parameters with a single options parameter.
- Removed string-replace-webpack-plugin as it was outdated and has a few vulnerabilities.
  - TODO: Move rewrite of default Cesium API key warning to the app
Upgraded SASS
- Migrated SASS files to use modern API (by running the sass-migrator script)
- Replaced webpack aliases ~terriajs-variables and ~terriajs-mixins with respective relative path. This was necessary to run the migrator. Also it means simpler webpack config.
- css-modules-typescript-loader that generates TS types for SASS modules is outdated and no longer works with webpack 5. I have made a repo from a fork of another similar package that can generate types. The other alternative would be to use this TS plugin - but not sure if it'll slow down compilation/editing.
Modified TIFFImageryProvider to work with webpack5 build.
TODO: Find a replacement for svg-sprite-loader. It appears to be not maintained and has a bunch of audit warnings.
Rewrote a couple of specs that depended on some CJS feature
TODO: Test in Windows.

I'll annotate the changes to make review easier.

Test me

Specs should pass
Test everything! - http://ci.terria.io/webpack5

Checklist

There are unit tests to verify my changes are correct or unit tests aren't applicable (if so, write quick reason why unit tests don't exist)
I've updated relevant documentation in doc/.
I've updated CHANGES.md with what I changed.
I've provided instructions in the PR description on how to test this PR.

- Rewrite webpack aliased paths like ~terriajs-variables and ~terriajs-mixins to plain relative paths so that we can run the auto migrator. This was done using the following shell command: ``` find lib/ -name '*.scss' -exec bash -c 'sed -i "s|~terriajs-variables|$(realpath -s --relative-to=$(dirname "$1") lib/Sass/common/_variables)|" $1' bash {} \; ``` - Run auto migrator ``` find lib/ -name '*.scss' -exec npx sass-migrator module {} \; ```

Instead of using 'raw-loader!' etc in the import path, we now handle these import in webpack config. See webpack.config.make.js.

Removes use of 'raw-loader!' etc in import paths. Instead handle the imports in webpack config.

Webpack literally works only for 'process.env.NODE_ENV'.

pjonsson · 2024-12-05T10:46:00Z

package.json

    "bottleneck": "^2.19.5",
    "catalog-converter": "^0.0.9",
    "classnames": "^2.3.1",
    "clipboard": "^2.0.8",
-    "commander": "^11.1.0 ",
+    "commander": "^12.1.0 ",


Commit 0a451d1 downgraded commander to 11.1.0 so it would work with node 16.

I'm not against bumping the engine version though, I would prefer to bump engines to node >= 20 so we could "easily" get rid of the security issues in terriajs-server as well while we're at it.

Agree that we should bump the engine version, will look into it. In this case though commander is only used for a couple of CLI things and the CI hasn't tripped on it. So it should be fine i guess.

pjonsson · 2024-12-05T10:47:59Z

package.json

@@ -165,25 +161,22 @@
    "react-virtual": "~2.3.2",
    "resolve-url-loader": "^5.0.0",
    "retry": "^0.12.0",
-    "sass-loader": "^10",
+    "sass-loader": "^16.0.3",


I'm fairly sure this also requires bumping the engine version.

pjonsson · 2024-12-05T11:01:42Z

I looked through this and besides the things I commented on, the things that are there looks good to me. Would love to get the (approved) CJS to ESM PR merged soon so this PR doesn't end up in limbo.

pjonsson · 2024-12-09T13:08:30Z

package.json

-    "css-loader": "^2.1.0",
-    "css-modules-typescript-loader": "^2.0.4",
+    "css-loader": "^7.1.2",
+    "terriajs-typings-for-css-modules-loader": "^2.5.2",


Non-alphabetical sorting, so this package will be shuffled down to the other terriajs-packages the next time someone runs yarn add.

pjonsson · 2024-12-09T21:39:44Z

@na9da just noticed that the late revert of polyfill.js in CJS-to-ESM PR changed the export defaults .. back into the previous module.exports = ....

ljowen · 2024-12-10T04:24:57Z

buildprocess/configureWebpack.js

 const ForkTsCheckerWebpackPlugin = require("fork-ts-checker-webpack-plugin");
 const ForkTsCheckerNotifierWebpackPlugin = require("fork-ts-checker-notifier-webpack-plugin");
 const webpack = require("webpack");

-function configureWebpack(
+/**


pjonsson · 2024-12-10T15:03:01Z

Since this repository was just converted to ESM, should the other terriajs-repositories (that aren't forks of something else) also be converted to ESM?

I had a look at terriajs-server and it seems quite easy, but I had to upgrade jasmine and yargs.

na9da · 2024-12-10T21:21:30Z

@na9da just noticed that the late revert of polyfill.js in CJS-to-ESM PR changed the export defaults .. back into the previous module.exports = ....

Yep, i'll make those changes in this PR. I didn't want to create two successive breaking changes.

na9da added 15 commits December 4, 2024 15:28

Initial changes for webpack 5 upgrade.

055f80e

Convert spec fixtures imports to valid ESM style.

5bf5b22

Instead of using 'raw-loader!' etc in the import path, we now handle these import in webpack config. See webpack.config.make.js.

Convert asset imports to valid ESM style.

8ce997a

Removes use of 'raw-loader!' etc in import paths. Instead handle the imports in webpack config.

Remove nullish checks.

ed0ddbd

Webpack literally works only for 'process.env.NODE_ENV'.

Update parameters.

1ec85e7

Update .babelrc.

ceced0c

Add types for csv, dac and xml imports.

3edacc0

Use webpack5 version of tiff imagery provider.

018a6aa

Temporarily use webpack5 branch for CI.

913b35e

Disable lint temporarily until we fix the import problem.

f9c6a68

Prettier scss files.

56f96b3

Remove cesium debug pragma from dev mode (for now).

d3736be

Rewrite search provider specs to not depend on CJS module feature.

212044f

Fix SASS import for timeline component.

18609cf

na9da changed the base branch from main to cjs-to-esm December 5, 2024 01:28

na9da mentioned this pull request Dec 5, 2024

Webpack 5 upgrade TerriaJS/TerriaMap#718

Draft

na9da added 3 commits December 5, 2024 17:53

Fix tools build script.

95d9bb2

Update terriajs-tiff-imagery-provider.

3d63532

Fix TIFF imagery provider import.

7ce0d00

pjonsson reviewed Dec 5, 2024

View reviewed changes

na9da added 3 commits December 6, 2024 16:47

Replace require() for images with import.

02309d0

Replace asset require() in lib with import.

77cf215

Convert remaining require() in specs to import.

0de2466

na9da mentioned this pull request Dec 8, 2024

CJS to ESM #7311

Merged

5 tasks

Base automatically changed from cjs-to-esm to main December 8, 2024 23:36

na9da marked this pull request as ready for review December 8, 2024 23:53

pjonsson reviewed Dec 9, 2024

View reviewed changes

ljowen reviewed Dec 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Webpack 5 upgrade #7351

Webpack 5 upgrade #7351

na9da commented Dec 5, 2024 •

edited

Loading

pjonsson Dec 5, 2024

na9da Dec 5, 2024 •

edited

Loading

pjonsson Dec 5, 2024

pjonsson commented Dec 5, 2024

pjonsson Dec 9, 2024

pjonsson commented Dec 9, 2024

ljowen Dec 10, 2024

pjonsson commented Dec 10, 2024

na9da commented Dec 10, 2024

Webpack 5 upgrade #7351

Are you sure you want to change the base?

Webpack 5 upgrade #7351

Conversation

na9da commented Dec 5, 2024 • edited Loading

What this PR does

Test me

Checklist

pjonsson Dec 5, 2024

Choose a reason for hiding this comment

na9da Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

pjonsson Dec 5, 2024

Choose a reason for hiding this comment

pjonsson commented Dec 5, 2024

pjonsson Dec 9, 2024

Choose a reason for hiding this comment

pjonsson commented Dec 9, 2024

ljowen Dec 10, 2024

Choose a reason for hiding this comment

pjonsson commented Dec 10, 2024

na9da commented Dec 10, 2024

na9da commented Dec 5, 2024 •

edited

Loading

na9da Dec 5, 2024 •

edited

Loading