Rust and Wasm Side-by-Side

Background

At my work, we're considering using WebAssembly (hereafter abbreviated as WASM) because it allows us to cross-compile just about any language for use on the web. WebAssembly is a "binary instruction format for a stack-based virtual machine". Basically, this means it's a binary language designed to be run anywhere, but generally speaking, it's used right now within web browsers as a replacement for Javascript modules. One of the languages we are considering using as a source language is Rust, a new(ish) language designed for type safety, performance, and concurrency.

As such, I've been reading up the Rust programming language, and, more importantly, how to compile Rust to WebAssembly. After learning from the example module given in the RustWasm book, I decided to convert one of my personal npm libraries from JS -> Rust ^[1]. The first step in this endeavor was to convert over the existing tests that I had in mocha/chai to Rust's native test format, so that as I added code, I could verify that it was working as I expected it to work ^[2]. However, the ability to run these tests became an issue when I moved to a WASM target.

The Problem

One of the big downsides to compiling Rust to WASM is that there isn't a good way to debug code within the browser, and have it link back to the Rust source. The Rust-WASM book notes this in the section on debugging:

Unfortunately, the debugging story for WebAssembly is still immature. On most Unix systems, DWARF is used to encode the information that a debugger needs to provide source-level inspection of a running program. There is an alternative format that encodes similar information on Windows. Currently, there is no equivalent for WebAssembly.

Instead, they recommend using testing (specifically, automated testing) to identify regressions before they make it into the build. This is a great idea, and one I obviously support, given my original writing of tests prior to implementing my library. However, there's one big downside to writing tests in Rust when compiling to WASM, as detailed in the next subsection of the Rust-WASM book:

Note that in order to run the #[test]s without compiler and linker errors, you will need to comment out the crate-type = "cdylib" bits in wasm-game-of-life/Cargo.toml.

Wait... what? In order to run my Rust tests, I need to make changes to my source repository just to get it to build? This is a bit of a non-starter for me, because I want to run on a continuous integration system, where I'm able to build the Rust package, run the tests, build the WASM binary, and then deploy to NPM, all automatically.

In other words, what I want to be able to do is something like this:

# These commands compile (and subsequently test) the native Rust code
$ cargo build
$ cargo test

# This command builds the wasm module
$ cargo build --target=wasm32-unknown-unknown

The key here, though, is that I want to be able to do this without making any source file or build-file changes.

Possible Solutions^[3]

I spent quite a bit of time trying to determine what I could do here. One thought that came to mind was that I could have a pre-build script that runs that performs this commenting-out of #[wasm_bindgen] attributes manually. How this would work is, before building, the script would make copies of everything in the local directory to a sub-directory. It would then switch the crate-type in subdirectory/Cargo.toml to lib, instead of cdylib and comment out any instances of #[wasm_bindgen]. Needless to say, I didn't want to go this route - it seemed incredibly fragile and error-prone.

Next, a coworker of mine suggested I separate the native Rust code from the WASM aspect of the code, and make the WASM library utilize the code from the native library, similar to how geotoy handles this^[4]. This is a great solution, but, when I set up my library, I couldn't for the life of me figure out how to import an enum or struct into the WASM portion of the library.

Basically, geotoy has a setup where it uses the WASM library to expose a set of functions that serve as the API for the library. These functions then utilize the data structures in src/lib.rs. However, the data structures themselves aren't exposed. With cratchit, what I want to expose is the Account and AccountsChart data structures, as well as the Currency and AccountType enumerations directly, instead of a set of gateway functions that utilize these data structures ^[5]. I suspect there probably is a way to get it to expose the structs, but I wasn't able to figure it out.

Final Solution

The solution I eventually arrived at doesn't separate the libraries per se. Instead, it makes the #[wasm_bindgen] attributes conditional on whether or not you're building for a WASM target.

The first thing you need to do is make sure you're using both cdylib and rlib (or lib) in your Cargo.toml:

[lib]
crate-type = ["cdylib", "rlib"]

Next, (this might be specific to me), I had to add #![feature(custom_attribute)] to the top of my crate attributes (i.e. at the top of src/lib.rs:

#![feature(custom_attribute)]

extern crate cfg_if;
extern crate json;
extern crate wasm_bindgen;

use wasm_bindgen::prelude::*;
use cfg_if::cfg_if;
use std::collections::HashMap;

...

Next, wherever you previously used #[wasm_bindgen], make it conditional on the target architecture:

/// Use this instead of #[wasm_bindgen]
#[cfg_attr(target_arch = "wasm32", wasm_bindgen)]

Finally, and this might, again, be something that is specific to my use case, you need to pull types out of submodules and into the root crate src/lib.rs. In other words, in my src/lib.rs, I had the following:

pub mod accounts;
pub mod currency;

And I had two files, src/currency.rs and src/accounts.rs, which defined types related to accounts and currencies, respectively. wasm_bindgen didn't appear to like this, so I moved this code into src/lib.rs and removed references to the modules in the tests.

What I did to test was not use wasm-pack initially. Instead, I ran cargo +nightly build --target=wasm32-unknown-unknown to ensure that no errors were present, and that it created a .wasm file in the appropriate target directory:

$ cargo +nightly build --target=wasm32-unknown-unknown
...
$ ls -al target/wasm32-unknown-unknown/debug
total 9520
drwxr-xr-x@ 13 scottj staff 416 Oct 26 10:15 .
drwxr-xr-x 3 scottj staff 96 Oct 26 10:14 ..
-rw-r--r-- 1 scottj staff 0 Oct 26 10:14 .cargo-lock
drwxr-xr-x 8 scottj staff 256 Oct 26 10:14 .fingerprint
drwxr-xr-x 3 scottj staff 96 Oct 26 10:14 build
-rw-r--r-- 1 scottj staff 122 Oct 26 10:15 cratchit.d
-rwxr-xr-x 2 scottj staff 3549051 Oct 26 10:15 cratchit.wasm
drwxr-xr-x 13 scottj staff 416 Oct 26 10:15 deps
drwxr-xr-x 2 scottj staff 64 Oct 26 10:14 examples
drwxr-xr-x 3 scottj staff 96 Oct 26 10:15 incremental
-rw-r--r-- 1 scottj staff 125 Oct 26 10:15 libcratchit.d
-rw-r--r-- 2 scottj staff 1311774 Oct 26 10:15 libcratchit.rlib
drwxr-xr-x 2 scottj staff 64 Oct 26 10:14 native

And, of course, I made sure the tests ran without any changes:

$ cargo test
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/test_accounts-54475044840749bd

running 6 tests
test account_type_from_integer ... ok
test account_creation ... ok
test account_type_from_string ... ok
test adding_top_level_accounts_to_accounts_chart ... ok
test getting_all_account_ids_in_a_chart ... ok
test creating_accounts_chart_from_json ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/test_currency-d2d59a6d3436c103

running 1 test
test currency_translation_from_string ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests cratchit

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Now, when you want to build your WASM module, you can run:

wasm-pack build

And, it will create the packaged WASM library for you in pkg.

Bonus: Run tests on Travis and deploy WASM to NPM

This was another area that took a bit of figuring out for me. The problem I ran into with Travis-CI was that it doesn't install wasm-pack by default. As such, you need to install it manually in a script. However, if you have cache: cargo enabled, it will hang if wasm-pack was previously installed. Thus, I added the following to my .travis.yml file to conditionally install wasm-pack:

before_script: |
  if hash wasm-pack 2>/dev/null; then
      echo "Wasm-pack already is installed"
  else
      curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
  fi

Now, you can add the following to your script section build both the WASM module and the native Rust module:

script:
- cargo clean
- cargo build
- cargo test
- wasm-pack build --target=nodejs

Note the section that calls wasm-pack build has an additional argument: --target=nodejs. If you want to test locally using npm link, you will need this argument, since it packages the WASM module with a main parameter in the package.json.

And, finally, add the deployment logic^[6]:

before_deploy:
- cd pkg
deploy:
  provider: npm
  email: <your email address>
  on:
    tags: true
  skip_cleanup: true
  api_key:
    secure: <YOUR_API_KEY>

Note that this deployment logic will only deploy on tagged releases, so you may want to change that if you want different behavior.

One last thing to realize: this currently requires nightly rust. So, you'll need to make sure you're using nightly Rust locally, and you need to make sure that, if you're building on travis, that you allow stable and beta Rust to fail:

rust:
- stable
- beta
- nightly
matrix:
  allow_failures:
    - rust: stable
    - rust: beta

1. The library in question is Cratchit, which is a library designed to encapsulate accounting data for an accounting application running in electron. It's not important that you have familiarity with this library, but if you want to follow along, feel free. ↩︎

2. I'm going to gloss over the rest of the adventures regarding me porting to Rust, since it's not super interesting, and involves me fighting a lot with the borrow checker to get things to work. ↩︎

3. If you're not interested in my musings about the problem and how I came to the solution, feel free to skip to the section on the solution I used, later on in this post. ↩︎

4. The coworker in question is Johannes Hoff who also is a contributer on the geotoy repo. Interestingly enough, the original author of geotoy, Nick Fitzgerald, was an intern at Mozilla when I first started there. It's amazing to me sometimes how small this world is, given that I haven't seen fitzgen in years, and one of my coworkers happens to have a solution to a problem I have, in collaboration with this other person I worked closely with. Sometimes the wheel of fate astounds me. ↩︎

5. There probably is a good reason for exposing gateway functions, rather than the underlying data structures themselves, which I will discover at some future point, but for now, my chosen solution seems to be a little cleaner. ↩︎

6. Make sure that you encrypt your api key. You can do this by retrieving an api key from travis-ci.org and then running: travis encrypt <api_key> --add deploy.api_key. I waited until I had the configuration otherwise the way I wanted it before doing this. ↩︎