-
-
Notifications
You must be signed in to change notification settings - Fork 36
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Switch from Karma to Web Test Runner, and ES6 module exports (#97)
* devcontainer setup * replace commonjs with es6 modules * replace karma with web test runner * remove examples folder, since the tests serve as examples * v0.0.13
- Loading branch information
1 parent
42a4245
commit 7236b20
Showing
37 changed files
with
6,811 additions
and
2,550 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "postCreateCommand": "/workspaces/wasm-git/.devcontainer/post-create.sh", | ||
| "customizations": { | ||
| "vscode": { | ||
| "extensions": [ | ||
| "dtsvet.vscode-wasm", | ||
| "ms-vscode.cpptools-extension-pack", | ||
| "github.vscode-github-actions" | ||
| ] | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| #!/bin/bash | ||
| curl https://wasmtime.dev/install.sh -sSf | bash | ||
| npm install | ||
| sh setup.sh | ||
|
|
||
| git clone https://github.com/emscripten-core/emsdk.git | ||
| cd emsdk | ||
| git pull | ||
| ./emsdk install latest | ||
| ./emsdk activate latest | ||
| cd .. | ||
|
|
||
| npx playwright install-deps | ||
| npx playwright install |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,10 +2,12 @@ Wasm-git | |
| ======== | ||
| (Wasm should be pronounced like `awesome` starting with a `W` ). | ||
|
|
||
|  | ||
|  | ||
|
|
||
| GIT for nodejs and the browser using [libgit2](https://libgit2.org/) compiled to WebAssembly with [Emscripten](https://emscripten.org). | ||
|
|
||
| The main purpose of bringing git to the browser, is to enable storage of web application data locally in the users web browser, with the option to synchronize with a remote server. | ||
|
|
||
| # Demo in the browser | ||
|
|
||
| A simple demo in the browser can be found at: | ||
|
|
@@ -14,150 +16,37 @@ https://wasm-git.petersalomonsen.com/ | |
|
|
||
| **Please do not abuse, this is open for you to test and see the proof of concept** | ||
|
|
||
| The sources for the demo can be found in the [githttpserver](https://github.com/petersalomonsen/githttpserver) project, which is a simple git server deployable to [kubernetes](https://github.com/kubernetes/kubernetes). Showing basic operations like cloning, edit files, add and commit, push and pull. | ||
|
|
||
| # Example WebWorker with pre built binaries | ||
|
|
||
| For running in the browser you should have your git interaction code in a [webworker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers). This is because of the use of synchronous http requests and long running operations that would block if running on the main thread. | ||
|
|
||
| Here's an example of a simple webworker that uses pre-built binaries from https://unpkg.com/[email protected]/ | ||
|
|
||
| ```js | ||
| var Module = { | ||
| locateFile: function(s) { | ||
| return 'https://unpkg.com/[email protected]/' + s; | ||
| } | ||
| }; | ||
|
|
||
| importScripts('https://unpkg.com/[email protected]/lg2.js'); | ||
|
|
||
| Module.onRuntimeInitialized = () => { | ||
| const lg = Module; | ||
|
|
||
| FS.mkdir('/working'); | ||
| FS.mount(MEMFS, { }, '/working'); | ||
| FS.chdir('/working'); | ||
|
|
||
| FS.writeFile('/home/web_user/.gitconfig', '[user]\n' + | ||
| 'name = Test User\n' + | ||
| 'email = [email protected]'); | ||
|
|
||
| // clone a local git repository and make some commits | ||
|
|
||
| lg.callMain(['clone',`http://localhost:5000/test.git`, 'testrepo']); | ||
|
|
||
| FS.readdir('testrepo'); | ||
| } | ||
| ``` | ||
|
|
||
| You'll start the worker from your html with the tag: | ||
| The sources for the demo can be found in the [githttpserver](https://github.com/petersalomonsen/githttpserver) project. It shows basic operations like cloning, edit files, add and commit, push and pull. | ||
|
|
||
| `<script>new Worker('yourworker.js')</script>;` | ||
| # Demo videos | ||
|
|
||
| The example above expects to find a git repository at `http://localhost:5000/test.git`. If you want to clone from github you'd need a proxy running locally because of [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) restrictions that would prevent you | ||
| accessing github directly. For testing you can use the proxy found in [examples/webserverwithgithubproxy.js](examples/webserverwithgithubproxy.js) | ||
| Videos showing example applications using wasm-git can bee seen in [this playlist](https://www.youtube.com/watch?v=1Hqy7cVkygU&list=PLv5wm4YuO4Iyx00ifs6xUwIRSFnBI8GZh). Wasm-git is used for local and offline storage of web application data, and for syncing with a remote server. | ||
|
|
||
| # Use in Browser without a WebWorker | ||
| A webworker is more performant but for cases where you really need to work in the browser, the http requests must be asynchronous and not synchronous as in the default builds. | ||
| # Examples | ||
|
|
||
| If you use the `emscriptenbuilds/build.sh` you can build `async` versions with: | ||
| ``` | ||
| ./build.sh Release-async | ||
| ``` | ||
| and | ||
| ``` | ||
| ./build.sh Debug-async | ||
| ``` | ||
| To use the async wasm-git build you need to load `lg2.js` using a `<script>` tag in HTML, and set up a method to wait for initialisation of the lg2 module. For example, your `index.html` can include the following, and set up a promise for your JavaScript implementation to `await`: | ||
| ```html | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <script src='/build/lg2.js'></script> | ||
|
|
||
| <script> | ||
| window.lg2Ready = false; | ||
| window.lg2 = Module; | ||
| window.lg2ReadyPromise = new Promise((resolve) => { | ||
| Module.onRuntimeInitialized = () => { | ||
| window.lg2Ready = true; | ||
| resolve(true); | ||
| }; | ||
| }); | ||
| </script> | ||
| Wasm-git packages are built in two variants: Synchronuous and Asynchronuous. To run the sync version in the browser, a [webworker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) is needed. This is because of the use of synchronous http requests and long running operations that would block if running on the main thread. The sync version has the smallest binary, but need extra client code to communicate with the web worker. When using the sync version in nodejs [worker_threads](https://nodejs.org/api/worker_threads.html) are used, with [Atomics](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics) to exchange data between threads. | ||
|
|
||
| <script defer src='/build/bundle.js'></script> | ||
| </head> | ||
| The async version use [Emscripten Asyncify](https://emscripten.org/docs/porting/asyncify.html), which allows calling the Wasm-git functions with `async` / `await`. It can also run from the main thread in the browser. Asyncify increase the binary size because of instrumentation to unwind and rewind WebAssembly state, but makes it possible to have simple client code without exchanging data with worker threads like in the sync version. | ||
|
|
||
| <body> | ||
| </body> | ||
| </html> | ||
| ``` | ||
| In your bundle, you will have JavaScript to `await window.lg2ReadyPromise` and can then use `await callMain()` to invoke `libgit2` features exposed by `wasm-git`. Notice you MUST use `await` on every `callMain()` which interacts with a remote repository (e.g. 'clone', 'push' and so on). Example: | ||
| ```javascript | ||
| async function initApp() { | ||
| await window.lg2ReadyPromise; | ||
| await test(); | ||
| } | ||
| Examples of using Wasm-git can be found in the tests: | ||
|
|
||
| async function test() { | ||
| const lg = window.lg2; | ||
| const FS = lg.FS; | ||
| - [test](./test/) for NodeJS | ||
| - [test-browser](./test-browser/) for the sync version in the browser with a web worker | ||
| - [test-browser-async](./test-browser-async/)] for the async version in the browser | ||
|
|
||
| const lg = Module; | ||
| The examples shows importing the `lg2.js` / `lg2-async.js` modules from the local build, but you may also access these from releases available at public CDNs. | ||
|
|
||
| FS.mkdir('/working'); | ||
| FS.mount(MEMFS, { }, '/working'); | ||
| FS.chdir('/working'); | ||
| # Building and developing | ||
|
|
||
| FS.writeFile('/home/web_user/.gitconfig', '[user]\n' + | ||
| 'name = Test User\n' + | ||
| 'email = [email protected]'); | ||
| The [Github actions test pipeline](./.github/workflows/main.yml) shows all the commands needed to install dependencies, build the packages and run the tests. | ||
|
|
||
| // clone a local git repository and make some commits | ||
| Another option is loading the repository into a github codespace, where the configuration in [.devcontainer](./.devcontainer) folder will be used to install dependencies and set up a full development environment. | ||
|
|
||
| await lg.callMain(['clone',`http://localhost:5000/test.git`, 'testrepo']); | ||
| # Emscripten fixes that were needed for making Wasm-git work | ||
|
|
||
| FS.readdir('testrepo'); | ||
| } | ||
| As part of being able to compile libgit2 to WebAssembly and run it in a Javascript environment, some fixes to [Emscripten](https://emscripten.org/) were needed. | ||
|
|
||
| ``` | ||
|
|
||
| Note that the compiled output is about twice the size for non-async builds, and that git operations will take place on the main thread which can affect reponsiveness of your web UI. | ||
|
|
||
| See below for more details on building using `build.sh`. | ||
| # Use from nodejs with pre built binaries | ||
|
|
||
| You may install the npm package containing the binaries: | ||
|
|
||
| `npm install wasm-git` | ||
|
|
||
| example source code for cloning a repository from github: | ||
|
|
||
| ```js | ||
| const lg = require('./node_modules/wasm-git/lg2.js'); | ||
|
|
||
| lg.onRuntimeInitialized = () => { | ||
| const FS = lg.FS; | ||
| const MEMFS = FS.filesystems.MEMFS; | ||
|
|
||
| FS.mkdir('/working'); | ||
| FS.mount(MEMFS, { }, '/working'); | ||
| FS.chdir('/working'); | ||
|
|
||
| FS.writeFile('/home/web_user/.gitconfig', '[user]\n' + | ||
| 'name = Test User\n' + | ||
| 'email = [email protected]'); | ||
|
|
||
| // clone a repository from github | ||
| lg.callMain(['clone','https://github.com/torch2424/made-with-webassembly.git', 'made-with-webassembly']); | ||
| FS.readdir('made-with-webassembly'); | ||
| } | ||
| ``` | ||
|
|
||
| # Building | ||
|
|
||
| **IMPORTANT: This depends on the following mmap fixes for emscripten:** | ||
| Here are the Pull Requests that resolved the issues identified when the first version was developed: | ||
|
|
||
| - https://github.com/emscripten-core/emscripten/pull/10095 | ||
| - https://github.com/emscripten-core/emscripten/pull/10526 | ||
|
|
@@ -167,12 +56,3 @@ for using with `NODEFS` you'll also need https://github.com/emscripten-core/emsc | |
|
|
||
| All of these pull requests are merged to emscripten master as of 2020-03-29. | ||
|
|
||
| See [.github/workflows/main.yml](./.github/workflows/main.yml) for a full build and test pipeline including installing emscripten. | ||
|
|
||
| Run [setup.sh](setup.sh) first to download libgit2 and apply patches. | ||
|
|
||
| Given you have installed and activated emscripten, you can use the script in [emscriptenbuild/build.sh](emscriptenbuild/build.sh) to configure and build, and you'll find the resulting `lg2.js` / `lg2.wasm` under the generated `emscriptenbuild/examples` folder. | ||
|
|
||
| An example of interacting with libgit2 from nodejs can be found in [examples/example_node.js](examples/example_node.js). | ||
|
|
||
| An example for the browser (using webworkers) can be found in [examples/example_webworker.js](examples/example_webworker.js). You can start a webserver for this by running the [examples/webserverwithgithubproxy.js](examples/webserverwithgithubproxy.js) script, which will launch a http server at http://localhost:5000 with a proxy to github. Proxy instead of direct calls is needed because of CORS restrictions in a browser environment. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.