diff options
| -rw-r--r-- | BENCHMARKS.md | 35 | ||||
| -rw-r--r-- | Cargo.lock | 138 | ||||
| -rw-r--r-- | README.md | 28 |
3 files changed, 103 insertions, 98 deletions
diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 13cbda9..78f05d7 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1,9 +1,11 @@ # Benchmark results -All benchmarks are run on a Ryzen 7 5800X, with 32GB of RAM, running Linux 6.6. -WebAssembly files are optimized using [wasm-opt](https://github.com/WebAssembly/binaryen) +All benchmarks are run on a Ryzen 7 5800X with 32GB of RAM, running Linux 6.6. +WebAssembly files are optimized using [wasm-opt](https://github.com/WebAssembly/binaryen), and the benchmark code is available in the `benches` folder. +These are mainly preliminary benchmarks, and I will be adding more in the future that are also looking into memory usage and other metrics. + ## WebAssembly Settings All WebAssembly files are compiled with the following settings: @@ -15,45 +17,50 @@ All WebAssembly files are compiled with the following settings: All runtimes are compiled with the following settings: -- `unsafe` features are enabled +- `unsafe` features are enabled. - `opt-level` is set to 3, `lto` is set to `thin`, `codegen-units` is set to 1. +## Versions + +- `tinywasm`: `0.4.0` +- `wasmi`: `0.31.0` +- `wasmer`: `4.2.0` + ## Results | Benchmark | Native | TinyWasm | Wasmi | Wasmer (Single Pass) | | ------------ | ------ | -------- | -------- | -------------------- | -| `argon2id` | 0.52ms | 110.08ms | 44.408ms | 4.76ms | | `fib` | 6ns | 44.76µs | 48.96µs | 52µs | | `fib-rec` | 284ns | 25.565ms | 5.11ms | 0.50ms | +| `argon2id` | 0.52ms | 110.08ms | 44.408ms | 4.76ms | | `selfhosted` | 45µs | 2.18ms | 4.25ms | 258.87ms | -### Argon2id - -This benchmark runs the Argon2id hashing algorithm, with 2 iterations, 1KB of memory, and 1 parallel lane. -I had to decrease the memory usage from the default to 1KB, because especially the interpreters were struggling to finish in a reasonable amount of time. -This is something where `simd` instructions would be really useful, and it also highlights some of the issues with the current implementation of TinyWasm's Value Stack and Memory Instances. - ### Fib The first benchmark is a simple optimized Fibonacci function, which is a good way to show the overhead of calling functions and parsing the bytecode. -TinyWasm is slightly faster then Wasmi here, but that's probably because of the overhead of parsing the bytecode as TinyWasm uses a custom bytecode to pre-process the WebAssembly bytecode. +TinyWasm is slightly faster than Wasmi here, but that's probably because of the overhead of parsing the bytecode, as TinyWasm uses a custom bytecode to pre-process the WebAssembly bytecode. ### Fib-Rec This benchmark is a recursive Fibonacci function, which highlights some of the issues with the current implementation of TinyWasm's Call Stack. TinyWasm is a lot slower here, but that's because there's currently no way to reuse the same Call Frame for recursive calls, so a new Call Frame is allocated for every call. This is not a problem for most programs, and the upcoming `tail-call` proposal will make this a lot easier to implement. +### Argon2id + +This benchmark runs the Argon2id hashing algorithm, with 2 iterations, 1KB of memory, and 1 parallel lane. +I had to decrease the memory usage from the default to 1KB, because especially the interpreters were struggling to finish in a reasonable amount of time. +This is where `simd` instructions would be really useful, and it also highlights some of the issues with the current implementation of TinyWasm's Value Stack and Memory Instances. + ### Selfhosted This benchmark runs TinyWasm itself in the VM, and parses and executes the `print.wasm` example from the `examples` folder. -This is a godd way to show some of TinyWasm's strengths - the code is pretty large at 702KB and Wasmer struggles massively with it, even with the Single Pass compiler. I think it's a decent real-world performance benchmark, but definitely favors TinyWasm a bit. +This is a good way to show some of TinyWasm's strengths - the code is quite large at 702KB and Wasmer struggles massively with it, even with the Single Pass compiler. I think it's a decent real-world performance benchmark, but it definitely favors TinyWasm a bit. Wasmer also offers a pre-parsed module format, so keep in mind that this number could be a bit lower if that was used (but probably still on the same order of magnitude). This number seems so high that I'm not sure if I'm doing something wrong, so I will be looking into this in the future. ### Conclusion -After profiling and fixing some low hanging fruits, I found the biggest bottleneck to be Vector operations, especially for the Value Stack, and having shared access to Memory Instances using RefCell. These are the two areas I will be focusing on improving in the future, trying out to use -Arena Allocation and other data structures to improve performance. Still, I'm quite happy with the results, especially considering the use of standard Rust data structures. Additionally, typed FuncHandles have a significant overhead over the untyped ones, so I will be looking into improving that as well. +After profiling and fixing some low-hanging fruits, I found the biggest bottleneck to be Vector operations, especially for the Value Stack, and having shared access to Memory Instances using RefCell. These are the two areas I will be focusing on improving in the future, trying out Arena Allocation and other data structures to improve performance. Additionally, typed FuncHandles have a significant overhead over the untyped ones, so I will be looking into improving that as well. Still, I'm quite happy with the results, especially considering the use of standard Rust data structures. # Running benchmarks @@ -60,9 +60,9 @@ checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" [[package]] name = "anstyle" -version = "1.0.4" +version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7079075b41f533b8c61d2a4d073c4676e1f8b249ff94a393b0595db304e0dd87" +checksum = "2faccea4cc4ab4a667ce676a30e8ec13922a692c99bb8f5b11f1502c72e04220" [[package]] name = "argh" @@ -244,9 +244,9 @@ dependencies = [ [[package]] name = "bytemuck" -version = "1.14.0" +version = "1.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "374d28ec25809ee0e23827c2ab573d729e293f281dfe393500e7ad618baa61c6" +checksum = "ed2490600f404f2b94c167e31d3ed1d5f3c225a0f3b80230053b3e0b7b962bd9" [[package]] name = "byteorder" @@ -283,9 +283,9 @@ checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" [[package]] name = "chrono" -version = "0.4.32" +version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41daef31d7a747c5c847246f36de49ced6f7403b4cdabc807a97b5cc184cda7a" +checksum = "9f13690e35a5e4ace198e7beea2895d29f3a9cc55015fcebe6336bd2010af9eb" dependencies = [ "android-tzdata", "iana-time-zone", @@ -640,9 +640,9 @@ dependencies = [ [[package]] name = "darling" -version = "0.20.3" +version = "0.20.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0209d94da627ab5605dcccf08bb18afa5009cfbef48d8a8b7d7bdbc79be25c5e" +checksum = "da01daa5f6d41c91358398e8db4dde38e292378da1f28300b59ef4732b879454" dependencies = [ "darling_core", "darling_macro", @@ -650,9 +650,9 @@ dependencies = [ [[package]] name = "darling_core" -version = "0.20.3" +version = "0.20.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "177e3443818124b357d8e76f53be906d60937f0d3a90773a664fa63fa253e621" +checksum = "f44f6238b948a3c6c3073cdf53bb0c2d5e024ee27e0f35bfe9d556a12395808a" dependencies = [ "fnv", "ident_case", @@ -663,9 +663,9 @@ dependencies = [ [[package]] name = "darling_macro" -version = "0.20.3" +version = "0.20.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "836a9bbc7ad63342d6d6e7b815ccab164bc77a2d95d84bc3117a8c0d5c98e2d5" +checksum = "0d2d88bd93979b1feb760a6b5c531ac5ba06bd63e74894c377af02faee07b9cd" dependencies = [ "darling_core", "quote", @@ -1299,9 +1299,9 @@ dependencies = [ [[package]] name = "memoffset" -version = "0.8.0" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d61c719bcfbcf5d62b3a09efa6088de8c54bc0bfcd3ea7ae39fcc186108b8de1" +checksum = "5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c" dependencies = [ "autocfg", ] @@ -1640,9 +1640,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.4" +version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b7fa1134405e2ec9353fd416b17f8dacd46c473d7d3fd1cf202706a14eb792a" +checksum = "5bb987efffd3c6d0d8f5f89510bb458559eab11e4f869acb20bf845e016259cd" dependencies = [ "aho-corasick", "memchr", @@ -1810,9 +1810,9 @@ checksum = "b97ed7a9823b74f99c7742f5336af7be5ecd3eeafcb1507d1fa93347b1d589b0" [[package]] name = "serde" -version = "1.0.195" +version = "1.0.196" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63261df402c67811e9ac6def069e4786148c4563f4b50fd4bf30aa370d626b02" +checksum = "870026e60fa08c69f064aa766c10f10b1d62db9ccd4d0abb206472bee0ce3b32" dependencies = [ "serde_derive", ] @@ -1830,9 +1830,9 @@ dependencies = [ [[package]] name = "serde_derive" -version = "1.0.195" +version = "1.0.196" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46fe8f8603d81ba86327b23a2e9cdf49e1255fb94a4c5f297f6ee0547178ea2c" +checksum = "33c85360c95e7d137454dc81d9a4ed2b8efd8fbe19cee57357b32b9771fccb67" dependencies = [ "proc-macro2", "quote", @@ -1841,9 +1841,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.111" +version = "1.0.113" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "176e46fa42316f18edd598015a5166857fc835ec732f5215eac6b7bdbf0a84f4" +checksum = "69801b70b1c3dac963ecb03a364ba0ceda9cf60c71cfe475e99864759c8b8a79" dependencies = [ "itoa", "ryu", @@ -2035,7 +2035,7 @@ dependencies = [ "tinywasm-parser", "tinywasm-types", "wasm-testsuite", - "wast", + "wast 70.0.2", ] [[package]] @@ -2047,7 +2047,7 @@ dependencies = [ "log", "pretty_env_logger", "tinywasm", - "wast", + "wast 70.0.2", ] [[package]] @@ -2244,29 +2244,6 @@ dependencies = [ ] [[package]] -name = "wasm-bindgen-downcast" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5dac026d43bcca6e7ce1c0956ba68f59edf6403e8e930a5d891be72c31a44340" -dependencies = [ - "js-sys", - "once_cell", - "wasm-bindgen", - "wasm-bindgen-downcast-macros", -] - -[[package]] -name = "wasm-bindgen-downcast-macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5020cfa87c7cecefef118055d44e3c1fc122c7ec25701d528ee458a0b45f38f" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] name = "wasm-bindgen-macro" version = "0.2.90" source = "registry+https://github.com/rust-lang/crates.io-index" @@ -2297,9 +2274,18 @@ checksum = "4d91413b1c31d7539ba5ef2451af3f0b833a005eb27a631cec32bc0635a8602b" [[package]] name = "wasm-encoder" -version = "0.39.0" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ba64e81215916eaeb48fee292f29401d69235d62d8b8fd92a7b2844ec5ae5f7" +dependencies = [ + "leb128", +] + +[[package]] +name = "wasm-encoder" +version = "0.41.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "111495d6204760238512f57a9af162f45086504da332af210f2f75dd80b34f1d" +checksum = "e09bca7d6388637d27fb5edbeab11f56bfabcef8743c55ae34370e1e5030a071" dependencies = [ "leb128", ] @@ -2313,9 +2299,9 @@ dependencies = [ [[package]] name = "wasmer" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e626f958755a90a6552b9528f59b58a62ae288e6c17fcf40e99495bc33c60f0" +checksum = "5467c7a23f9be04d5691590bea509dbea27e5ba5810d0020bef908456a495f33" dependencies = [ "bytes", "cfg-if", @@ -2330,7 +2316,6 @@ dependencies = [ "target-lexicon", "thiserror", "wasm-bindgen", - "wasm-bindgen-downcast", "wasmer-compiler", "wasmer-compiler-cranelift", "wasmer-compiler-singlepass", @@ -2343,9 +2328,9 @@ dependencies = [ [[package]] name = "wasmer-compiler" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "848e1922694cf97f4df680a0534c9d72c836378b5eb2313c1708fe1a75b40044" +checksum = "510ad01a668d774f3a103a7c219bbc0970be93e8f1b27e2fdb48d1f4ccd1deff" dependencies = [ "backtrace", "bytes", @@ -2370,9 +2355,9 @@ dependencies = [ [[package]] name = "wasmer-compiler-cranelift" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d96bce6fad15a954edcfc2749b59e47ea7de524b6ef3df392035636491a40b4" +checksum = "54bf93078990d83960d798de3c5935bddaba771fc2fefb9ed6bab9c0bbdea5c1" dependencies = [ "cranelift-codegen", "cranelift-entity", @@ -2389,9 +2374,9 @@ dependencies = [ [[package]] name = "wasmer-compiler-singlepass" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebaa865b40ffb3351b03dab9fe9930a5248c25daebd55b464b79b862d9b55ccd" +checksum = "8f4d6359d66a8bcefac26d48fcb0f3f0882bdf122b52121a1ae21f918706e040" dependencies = [ "byteorder", "dynasm", @@ -2408,9 +2393,9 @@ dependencies = [ [[package]] name = "wasmer-derive" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f08f80d166a9279671b7af7a09409c28ede2e0b4e3acabbf0e3cb22c8038ba7" +checksum = "1b374fd34d97b1c091d8675f9bc472df52dc6787d139d3762d42c7dc84813a9b" dependencies = [ "proc-macro-error", "proc-macro2", @@ -2420,9 +2405,9 @@ dependencies = [ [[package]] name = "wasmer-types" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae2c892882f0b416783fb4310e5697f5c30587f6f9555f9d4f2be85ab39d5d3d" +checksum = "0caf1c87937b52aba8e9f920a278e1beda282f7439612c0b48f51a58e7a87bab" dependencies = [ "bytecheck 0.6.11", "enum-iterator", @@ -2436,9 +2421,9 @@ dependencies = [ [[package]] name = "wasmer-vm" -version = "4.2.2" +version = "4.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c0a9a57b627fb39e5a491058d4365f099bc9b140031c000fded24a3306d9480" +checksum = "58315c25492bc72a33f47a7d7fb0869a0106fc0164ec051e349a9e1eddba9a01" dependencies = [ "backtrace", "cc", @@ -2514,23 +2499,36 @@ dependencies = [ [[package]] name = "wast" -version = "70.0.0" +version = "64.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ee4bc54bbe1c6924160b9f75e374a1d07532e7580eb632c0ee6cdd109bb217e" +checksum = "a259b226fd6910225aa7baeba82f9d9933b6d00f2ce1b49b80fa4214328237cc" dependencies = [ "leb128", "memchr", "unicode-width", - "wasm-encoder", + "wasm-encoder 0.32.0", +] + +[[package]] +name = "wast" +version = "70.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d5061300042ff5065123dae1e27d00c03f567d34a2937c8472255148a216dc" +dependencies = [ + "bumpalo", + "leb128", + "memchr", + "unicode-width", + "wasm-encoder 0.41.0", ] [[package]] name = "wat" -version = "1.0.83" +version = "1.0.71" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f0dce8cdc288c717cf01e461a1e451a7b8445d53451123536ba576e423a101a" +checksum = "53253d920ab413fca1c7dc2161d601c79b4fdf631d0ba51dd4343bf9b556c3f6" dependencies = [ - "wast", + "wast 64.0.0", ] [[package]] @@ -12,27 +12,27 @@ ## Why TinyWasm? -- **Tiny** - Designed to be as small as possible without sacrificing too much performance or functionality. -- **Fast enough** - TinyWasm is reasonably fast, especially when compared to other interpreters. See [Benchmarks](./BENCHMARKS.md) for more details. -- **Portable** - Runs on any platform llvm supports, including WebAssembly. Minimal external dependencies. +- **Tiny**: TinyWasm is designed to be as small as possible without significantly compromising performance or functionality. +- **Portable**: TinyWasm runs on any platform that LLVM supports, including WebAssembly itself, with minimal external dependencies. +- **Lightweight**: TinyWasm is easy to integrate and has a low call overhead, making it suitable for scripting and embedding. ## Status -TinyWasm, starting from version `0.3.0`, passes all the WebAssembly 1.0 tests in the [WebAssembly Test Suite](https://github.com/WebAssembly/testsuite). The 2.0 tests are in progress. This is enough to run most WebAssembly programs, including TinyWasm itself compiled to WebAssembly (see [examples/wasm-rust.rs](./examples/wasm-rust.rs)). Results of the testsuite can be found [here](https://github.com/explodingcamera/tinywasm/tree/main/crates/tinywasm/tests/generated). +As of version `0.3.0`, TinyWasm successfully passes all the WebAssembly 1.0 tests in the [WebAssembly Test Suite](https://github.com/WebAssembly/testsuite). Work on the 2.0 tests is ongoing. This achievement ensures that TinyWasm can run most WebAssembly programs, including versions of TinyWasm itself compiled to WebAssembly (see [examples/wasm-rust.rs](./examples/wasm-rust.rs)). The results of the testsuite are available [here](https://github.com/explodingcamera/tinywasm/tree/main/crates/tinywasm/tests/generated). -Some APIs to interact with the runtime are not yet exposed, and the existing ones are subject to change, but the core functionality is mostly complete. - -TinyWasm is not (yet) designed for performance, but rather for simplicity, size and portability. See [Performance](#performance) for more details. +The API is still unstable and may change at any time, so don't use it in production _yet_. Note that TinyWasm isn't primarily designed for high performance; its focus lies more on simplicity, size, and portability. More details on its performance aspects can be found in [BENCHMARKS.md](./BENCHMARKS.md). ## Supported Proposals -- [**Mutable Globals**](https://github.com/WebAssembly/mutable-global/blob/master/proposals/mutable-global/Overview.md) - **Fully implemented** -- [**Multi-value**](https://github.com/WebAssembly/spec/blob/master/proposals/multi-value/Overview.md) - **Fully implemented** -- [**Sign-extension operators**](https://github.com/WebAssembly/spec/blob/master/proposals/sign-extension-ops/Overview.md) - **Fully implemented** -- [**Bulk Memory Operations**](https://github.com/WebAssembly/spec/blob/master/proposals/bulk-memory-operations/Overview.md) - **Fully implemented** (as of version `0.4.0`) -- [**Reference Types**](https://github.com/WebAssembly/reference-types/blob/master/proposals/reference-types/Overview.md) - **_Partially implemented_** -- [**Multiple Memories**](https://github.com/WebAssembly/multi-memory/blob/master/proposals/multi-memory/Overview.md) - **_Partially implemented_** -- [**Memory64**](https://github.com/WebAssembly/memory64/blob/master/proposals/memory64/Overview.md) - **_Partially implemented_** +| Proposal | Implementation Status | Version | +| -------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------- | +| [**Mutable Globals**](https://github.com/WebAssembly/mutable-global/blob/master/proposals/mutable-global/Overview.md) | Fully implemented | 0.2.0 | +| [**Multi-value**](https://github.com/WebAssembly/spec/blob/master/proposals/multi-value/Overview.md) | Fully implemented | 0.2.0 | +| [**Sign-extension operators**](https://github.com/WebAssembly/spec/blob/master/proposals/sign-extension-ops/Overview.md) | Fully implemented | 0.2.0 | +| [**Bulk Memory Operations**](https://github.com/WebAssembly/spec/blob/master/proposals/bulk-memory-operations/Overview.md) | Fully implemented | 0.4.0 | +| [**Reference Types**](https://github.com/WebAssembly/reference-types/blob/master/proposals/reference-types/Overview.md) | Partially implemented | N/A | +| [**Multiple Memories**](https://github.com/WebAssembly/multi-memory/blob/master/proposals/multi-memory/Overview.md) | Partially implemented | N/A | +| [**Memory64**](https://github.com/WebAssembly/memory64/blob/master/proposals/memory64/Overview.md) | Partially implemented | N/A | ## Usage |