This drops the text format (%.wat) and renames
InstantiateModuleFromCode to InstantiateModuleFromBinary as it is no
longer ambiguous.
We decided to stop supporting the text format as it isn't typically used
in production, yet costs a lot of work to develop. Given the resources
available and the increased work added with WebAssembly 2.0 and soon
WASI 2, we can't afford to spend the time on it.
The old parser is used only internally and will eventually be moved to
its own repository named watzero, possibly towards archival.
See #59
Signed-off-by: Adrian Cole <adrian@tetrate.io>
The componentized successor to wasi_snapshot_preview1 is not compatible
with the prior imports or even error numbers. Before releasing wazero
1.0 we need to change this package to reflect that WASI 2 is effectively
a different API.
Fixes#263
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This consolidates the pattern used for context overrides, notably
replacing clock overrides via experimental.WithTimeNowUnixNano
and making all context keys internal.
This also makes sure experimental example tests are handled the same
way, notably backfilling one for WithFS
This removes tedium in our examples and docs by using `Runtime.Close`
instead of tracking everything. Internal tests still track too much, but
anyway at least this stops suggesting others should do it.
This also changes our examples to use log.PanicXX so that the line
number goes into the console output.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This performs several changes to allow compilation config to be
centralized and scoped properly. The immediate effects are that we can
now process external types during `Runtime.CompileModule` instead of
doing so later during `Runtime.InstantiateModule`. Another nice side
effect is memory size problems can err at a source line instead of
having to be handled in several places.
There are some API effects to this, and to pay for them, some less used
APIs were removed. The "easy APIs" are left alone. For example, the APIs
to compile and instantiate a module from Go or Wasm in one step are left
alone.
Here are the changes, some of which are only for consistency. Rationale
is summarized in each point.
* ModuleBuilder.Build -> ModuleBuilder.Compile
* The result of this is similar to `CompileModule`, and pairs better
with `ModuleBuilder.Instantiate` which is like `InstantiateModule`.
* CompiledCode -> CompiledModule
* We punted on this name, the result is more than just code. This is
better I think and more consistent as it introduces less terms.
* Adds CompileConfig param to Runtime.CompileModule.
* This holds existing features and will have future ones, such as
mapping externtypes to uint64 for wasm that doesn't yet support it.
* Merges Runtime.InstantiateModuleWithConfig with Runtime.InstantiateModule
* This allows us to explain APIs in terms of implicit or explicit
compilation and config, vs implicit, kindof implicit, and explicit.
* Removes Runtime.InstantiateModuleFromCodeWithConfig
* Similar to above, this API only saves the compilation step and also
difficult to reason with from a name POV.
* RuntimeConfig.WithMemory(CapacityPages|LimitPages) -> CompileConfig.WithMemorySizer
* This allows all error handling to be attached to the source line
* This also allows someone to reduce unbounded memory while knowing
what its minimum is.
* ModuleConfig.With(Import|ImportModule) -> CompileConfig.WithImportRenamer
* This allows more types of import manipulation, also without
conflating functions with globals.
* Adds api.ExternType
* Needed for ImportRenamer and will be needed later for ExportRenamer.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We get undefined behavior, at least in code that re-slices, unless
capacity is set. This ensures the buffer under the string has a capacity
set to the same length as its size. This also shows more explicitly the
problems in TinyGo (ex some type mismatch you have to ignore until
fixed).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
WebAssembly Core Working Draft 1 recently came out. Before that, we had
a toe-hold feature bucked called FinishedFeatures. This replaces
`RuntimeConfig.WithFinishedFeatures` with `RuntimeConfig.WithWasmCore2`.
This also adds `WithWasmCore1` for those who want to lock into 1.0
features as opposed to relying on defaults.
This also fixes some design debt where we hadn't finished migrating
public types that require constructor functions (NewXxx) to interfaces.
By using interfaces, we prevent people from accidentally initializing
key configuration directly (via &Xxx), causing nil field refs. This also
helps prevent confusion about how to use the type (ex pointer or not) as
there's only one way (as an interface).
See https://github.com/tetratelabs/wazero/issues/516
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This prepares for exposing operations like Memory.Grow while keeping the
ability to trace what did that, by adding a `context.Context` initial
parameter. This adds this to all API methods that mutate or return
mutated data.
Before, we made a change to trace functions and general lifecycle
commands, but we missed this part. Ex. We track functions, but can't
track what closed the module, changed memory or a mutable constant.
Changing to do this now is not only more consistent, but helps us
optimize at least the interpreter to help users identify otherwise
opaque code that can cause harm. This is critical before we add more
functions that can cause harm, such as Memory.Grow.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We don't currently read DWARF, and are checking in Wasm binaries to
source that include it. This changes the settings to not include that,
so that our git pull performance doesn't degrade over time.
Note: This keeps the name section, so our current panics will be able to
show the source functions same as before.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This is an API breaking change that does a few things:
* Stop encouraging practice that can break context propagation:
* Stops caching `context.Context` in `wazero.RuntimeConfig`
* Stops caching `context.Context` in `api.Module`
* Fixes context propagation in function calls:
* Changes `api.Function`'s arg0 from `api.Module` to `context.Context`
* Adds `context.Context` parameter in instantiation (propagates to
.start)
* Allows context propagation for heavy operations like compile:
* Adds `context.Context` as the initial parameter of `CompileModule`
The design we had earlier was a good start, but this is the only way to
ensure coherence when users start correlating or tracing. While adding a
`context.Context` parameter may seem difficult, wazero is a low-level
library and WebAssembly is notoriously difficult to troubleshoot. In
other words, it will be easier to explain to users to pass (even nil) as
the context parameter vs try to figure out things without coherent
context.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Before, we tested the examples/ directory using "ExampleXX", but this is
not ideal because it literally embeds the call to `main` into example
godoc output. This stops doing that for a different infrastructure.
This also makes sure there's a godoc example for both the main package
and wasi, so that people looking at https://pkg.go.dev see something and
also a link to our real examples directory.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This shows how to define, export and import functions written in Go.
Fixes#464
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Thanks to #454, now the compiled binary (code segment) can be reused for
multiple module instances originating from the same source (wasm.Module).
This commit introduces the caching mechanism on engine where it caches
compiled functions keyed on `wasm.Module`. As a result, this allows us to
do the fast module instantiation from the same *CompiledCode.
In order to release the cache properly, this also adds `Close` method
on CompiledCode.
Here's some bench result for instantiating multiple modules from the same CompiledCode:
```
name old time/op new time/op delta
Initialization/interpreter-32 2.84ms ± 3% 0.06ms ± 1% -97.73% (p=0.008 n=5+5)
Initialization/jit-32 10.7ms ±18% 0.1ms ± 1% -99.52% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
Initialization/interpreter-32 1.25MB ± 0% 0.15MB ± 0% -88.41% (p=0.008 n=5+5)
Initialization/jit-32 4.46MB ± 0% 0.15MB ± 0% -96.69% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
Initialization/interpreter-32 35.2k ± 0% 0.3k ± 0% -99.29% (p=0.008 n=5+5)
Initialization/jit-32 94.1k ± 0% 0.2k ± 0% -99.74% (p=0.008 n=5+5)
```
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
This deduplicates examples, leaving only the most maintained or targeted
ones. Notably, this makes each runnable, avoiding test dependencies.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Before, complicated wasm could be hard to implement, particularly as it
might have cyclic imports. This change allows users to re-map imports to
untangle any cycles or to break up monolithic modules like "env".
Signed-off-by: Adrian Cole <adrian@tetrate.io>
During #425, @neilalexander gave constructive feedback that the API is
both moving fast, and not good enough yet. This attempts to reduce the
incidental complexity at the cost of a little conflation.
### odd presence of `wasm` and `wasi` packages -> `api` package
We had public API packages in wasm and wasi, which helped us avoid
leaking too many internals as public. That these had names that look
like there should be implementations in them cause unnecessary
confusion. This squashes both into one package "api" which has no
package collission with anything.
We've long struggled with the poorly specified and non-uniformly
implemented WASI specification. Trying to bring visibility to its
constraints knowing they are routinely invalid taints our API for no
good reason. This removes all `WASI` commands for a default to invoke
the function `_start` if it exists. In doing so, there's only one path
to start a module.
Moreover, this puts all wasi code in a top-level package "wasi" as it
isn't re-imported by any internal types.
### Reuse of Module for pre and post instantiation to `Binary` -> `Module`
Module is defined by WebAssembly in many phases, from decoded to
instantiated. However, using the same noun in multiple packages is very
confusing. We at one point tried a name "DecodedModule" or
"InstantiatedModule", but this is a fools errand. By deviating slightly
from the spec we can make it unambiguous what a module is.
This make a result of compilation a `Binary`, retaining `Module` for an
instantiated one. In doing so, there's no longer any name conflicts
whatsoever.
### Confusion about config -> `ModuleConfig`
Also caused by splitting wasm into wasm+wasi is configuration. This
conflates both into the same type `ModuleConfig` as it is simpler than
trying to explain a "will never be finished" api of wasi snapshot-01 in
routine use of WebAssembly. In other words, this further moves WASI out
of the foreground as it has been nothing but burden.
```diff
--- a/README.md
+++ b/README.md
@@ -49,8 +49,8 @@ For example, here's how you can allow WebAssembly modules to read
-wm, err := r.InstantiateModule(wazero.WASISnapshotPreview1())
-defer wm.Close()
+wm, err := wasi.InstantiateSnapshotPreview1(r)
+defer wm.Close()
-sysConfig := wazero.NewSysConfig().WithFS(os.DirFS("/work/home"))
-module, err := wazero.StartWASICommandWithConfig(r, compiled, sysConfig)
+config := wazero.ModuleConfig().WithFS(os.DirFS("/work/home"))
+module, err := r.InstantiateModule(binary, config)
defer module.Close()
...
```
This introduces `SysConfig` to replace `WASIConfig` and formalize documentation around system calls.
The only incompatible change planned after this is to switch from wasi.FS to fs.FS
Implementation Notes:
Defaulting to os.Stdin os.Stdout and os.Stderr doesn't make sense for
the same reasons as why we don't propagate ENV or ARGV: it violates
sand-boxing. Moreover, these are worse as they prevent concurrency and
can also lead to console overload if accidentally not overridden.
This also changes default stdin to read EOF as that is safer than reading
from os.DevNull, which can run the host out of file descriptors.
Finally, this removes "WithPreopens" for "WithFS" and "WithWorkDirFS",
to focus on the intended result. Similar Docker, if the WorkDir isn't set, it
defaults to the same as root.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
The prior design had a problem, where multiple imports of WASI would end
up having different file descriptors for the same file. Moreover, there
was no means to close any of these when a module importing WASI was
closed.
This moves all existing functionality to a new type SystemContext, which
is owned by the importing module, similar to how it owns its memory.
While this PR doesn't fix some problems such as unclosed files, the code
is now organized in a way it can be, and these issues will be resolved
by #394.
In order to fix scope, `WASISnapshotPreview1WithConfig` had to be
removed.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
StartWASICommandWithConfig is like StartWASICommand, except you can override configuration based on the importing
module. For example, you can use this to define different args depending on the importing module.
```go
// Initialize base configuration:
r := wazero.NewRuntime()
config := wazero.NewWASIConfig().WithStdout(buf)
wasi, _ := r.NewHostModule(wazero.WASISnapshotPreview1WithConfig(config))
decoded, _ := r.CompileModule(source)
// Assign configuration only when ready to instantiate.
module, _ := StartWASICommandWithConfig(r, decoded, config.WithArgs("rotate", "angle=90", "dir=cw"))
```
This also changes the config to be immutable since it may be reused many
times. Instead of assigning fields directly, use `WithXXX`.
Finally, this changes the examples to encourage using `.Close()`
Fixes#364
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This reverts `Runtime.NewModule` back to `InstantiateModule` as it calls
more attention to the registration aspect of it, and also makes a chain
of `NewXX` more clear. This is particularly helpful as this change
introduces `ModuleBuilder` which is created by `NewModuleBuilder`.
`ModuleBuilder` is a way to define a WebAssembly 1.0 (20191205) in Go.
The first iteration allows setting the module name and exported
functions. The next PR will add globals.
Ex. Below defines and instantiates a module named "env" with one function:
```go
hello := func() {
fmt.Fprintln(stdout, "hello!")
}
_, err := r.NewModuleBuilder("env").ExportFunction("hello", hello).InstantiateModule()
```
If the same module may be instantiated multiple times, it is more efficient to separate steps. Ex.
```go
env, err := r.NewModuleBuilder("env").ExportFunction("get_random_string", getRandomString).Build()
_, err := r.InstantiateModule(env.WithName("env.1"))
_, err := r.InstantiateModule(env.WithName("env.2"))
```
Note: Builder methods do not return errors, to allow chaining. Any validation errors are deferred until Build.
Note: Insertion order is not retained. Anything defined by this builder is sorted lexicographically on Build.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This converges host-defined modules with Wasm defined modules by
introducing a custom section for host-defined functions. The net result
are far less types and consistent initialization.
* HostModule is removed for Module
* HostFunction is removed for Function
* ModuleContext is removed for Module
Note: One impact of this is that the low-level API no longer accepts a
go context (context.Context), rather a `wasm.Module` which the function
is called in context of. This meant exposing `wasm.Module.WithContext`
to override the default.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This simplifies state management and the amount of terminology end-users
need to learn by using one concept `Runtime` instead of two: `Engine`
and `Store`. This bridges the concepts to the specification by still
having `wazero.Runtime` implement `wasm.Store`.
The net result is that we can know for sure which "engine" is used when
decoding. This allows us a lot of flexibility especially pre-compilation
when JIT is possible.
This also changes the default to JIT based on compiler flags so that
downstream projects like wapc-go don't have to do this individually
(including tracking of which OS+Arch have JIT).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This moves code for `ValueType` constants public, so that we can
centralize discussion on how values are encoded. We'll need this when
re-introducing a Globals API.
To keep complexity down in consideration that there are only 4 types, I
copied the constants into the internalwasm package. This reduces the
import complexity otherwise would have caused.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takaya Saeki <takaya@tetrate.io>
This makes instantiating host or wasm-defined modules similar, by using
XXXConfig types. Doing so also allows configuration engines to seed
properties.
Since decoded modules are only usable during instantiation, this pushes
decoding inside those functions. By doing so, the API is easier to use
as it has less choices and less errors to catch. Detection is done
internally by peeking at the magic number.
See https://github.com/tetratelabs/wazero/issues/279
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Before, we emulated the map api by returning `fn, ok` to tell if a
function was there or not. This made a lot of cruft vs returning `nil`
instead, especially as many times configuration of functions is near
where they are called in the source (so they aren't likely to be nil).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This adds this interface `wasm.Store` which gives access to functions in
a store without leaking an API to change the store. This is primarily to
support configuration use cases where post-initialization, there's no
need or desire to mutate the store. This also backfills codecs needed to
handle float results.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
This allows users to decouple from wazero code when authoring host
functions. Notably, this allows them to opt out of using a context, or
only using a Go context instead of HostFunctionCallContext.
This backfills docs on how to write host functions (in simple terms).
Finally, this does not optimize engines to avoid propagating context or
looking up memory if it would never be used. That could be done later.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This moves to a new end-user API under the root package `wazero`. This
simplifies call sites while hardening function calls to their known
return value. Most importantly, this moves most logic internal, as
noted in the RATIONALE.md.
Ex.
```go
// Read WebAssembly binary containing an exported "fac" function.
source, _ := os.ReadFile("./tests/engine/testdata/fac.wasm")
// Decode the binary as WebAssembly module.
mod, _ := wazero.DecodeModuleBinary(source)
// Initialize the execution environment called "store" with Interpreter-based engine.
store := wazero.NewStore()
// Instantiate the module, which returns its exported functions
functions, _ := store.Instantiate(mod)
// Get the factorial function
fac, _ := functions.GetFunctionI64Return("fac")
// Discover 7! is 5040
fmt.Println(fac(context.Background(), 7))
```
PS I changed the README to factorial because the wat version of
fibonacci is not consistent with the TinyGo one!
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takaya Saeki <takaya@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
This adds two interfaces with only one implementation each, used to
decouple end-user code from internals.
* wasm.HostFunctionCallContext - allows access to go Context and Memory
* wasm.Memory - read/write standard types and arbitrary data
This currently ports existing code to use the interfaces. It is true
that over time we can use structs instead of interfaces internally to
save a pointer reference in WASI call sites. However, I'm choosing not
to do this right now, as it is more important to see the interfaces
work. We should defer optimizing towards structs until after we
repackage implementations under the "internal" directory.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Before, we wrote WASI functions with jargon that is not typically used
in go. As this project is in go and implements a text compiler, we can
relay points more directly and specifically. This is the first example
to move to this.
This centralizes documentation through a new interface `wasi.API`
currently instantiated in `wasi.RegisterAPI(store, opts)`. Later, we
can figure out how to export the context needed to use it properly.
Co-authored-by: Takaya Saeki <takaya@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This PR generalizes e2e tests in jit/engine_test.go and moves them
tests/adhoc directory to run these tests against interpreter as well.
Notably this removes all the testdata directory from wasm/*
and helps isolates the usage of raw binary or texts into examples,
bench and tests directory.
During the course of generalization, I found the inconsistency on the
panic handling between JIT and Interpreter which seems a bug of latter
so I fixed it.
This adds function parsing, consolidating type use code in order to
reduce replication.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
