This introduces the new API wazero.Cache interface which can be passed to wazero.RuntimeConfig.
Users can configure this to share the underlying compilation cache across multiple wazero.Runtime.
And along the way, this deletes the experimental file cache API as it's replaced by this new API.
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
This makes DWARF enabled by default, with an opt-out flag. This didn't
need to be experimental as the feature was requested for a long time and
there's no API impact to using it.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
While most compilers will only read args/environ once, tools like WAGI
make heavy use of environment, possibly dozens of long variables. This
optimizes both args and environ for this reason and also to setup for
optimizing other functions.
Here are the notable changes:
* eagerly coerce to byte slices instead of strings
* re-use null terminated length for writing values
* avoid loops that call mem.WriteXXX internally
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This switches to gofumpt and applies changes, as I've noticed working
in dapr (who uses this) that it finds some things that are annoying,
such as inconsistent block formatting in test tables.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This restores the ability to leave out the initial context parameter
when defining functions with reflection. This is important because some
projects are porting from a different library to wazero, and all the
alternatives are not contextualized.
For example, this project is porting envoy host functions, and the
original definitions (in mosn) don't have a context parameter. By being
lenient, they can migrate easier.
See 6b813482b6/pkg/proxywasm/wazero/imports_v1.go
Signed-off-by: Adrian Cole <adrian@tetrate.io>
It is more often the case that projects are enabling a freestanding
target, and that may or may not have an exporting memory depending on
how that's interpreted. This adds the ability to inspect memories
similar to how you can already inspect compiled code prior to
instantiation. For example, you can enforce an ABI constraint that
"memory" must be exported even if WASI is not in use.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We formerly introduced `MemorySizer` as a way to control capacity independently of size. This was the first and only feature in `CompileConfig`. While possibly used privately, `MemorySizer` has never been used in public GitHub code.
These APIs interfere with how we do caching of compiled modules. Notably, they can change the min or max defined in wasm, which invalidates some constants. This has also had a bad experience, forcing everyone to boilerplate`wazero.NewCompileConfig()` despite that API never being used in open source.
This addresses the use cases in a different way, by moving configuration to `RuntimeConfig` instead. This allows us to remove `MemorySizer` and `CompileConfig`, and the problems with them, yet still retaining functionality in case someone uses it.
* `RuntimeConfig.WithMemoryLimitPages(uint32)`: Prevents memory from growing to 4GB (spec limit) per instance.
* This works regardless of whether the wasm encodes max or not. If there is no max, it becomes effectively this value.
* `RuntimeConfig.WithMemoryCapacityFromMax(bool)`: Prevents reallocations (when growing).
* Wasm that never sets max will grow from min to the limit above.
Note: Those who want to change their wasm (ex insert a max where there was none), have to do that externally, ex via compiler settings or post-build transformations such as [wabin](https://github.com/tetratelabs/wabin)
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We at one point considered making `ModuleBuilder` create complete
WebAssembly binaries. However, we recently spun out
[wabin](https://github.com/tetratelabs/wabin), which allows this.
Meanwhile, the features in `ModuleBuilder` were confusing and misused.
For example, the only two cases memory was exported on GitHub were done
by accident. This is because host functions act on the guest's memory,
not their own.
Hence, this removes memory and globals from host side definitions, and
renames the type to HostModuleBuilder to clarify this is not ever going
to be used to construct normal Wasm binaries.
Most importantly, this simplifies the API and reduces a lot of code. It
is important to make changes like this, particularly deleting any
experimental things that didn't end up useful.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Anuraag Agrawal <anuraaga@gmail.com>
While compilers should be conservative when targeting WebAssembly Core
features, runtimes should be lenient as otherwise people need to
constantly turn on all features. Currently, most examples have to turn
on 2.0 features because compilers such as AssemblyScript and TinyGo use
them by default. This matches the policy with the reality, and should
make first time use easier.
This top-levels an internal type as `api.CoreFeatures` and defaults to
2.0 as opposed to 1.0, our previous default. This is less cluttered than
the excess of `WithXXX` methods we had prior to implementing all
planned WebAssembly Core Specification 1.0 features.
Finally, this backfills rationale as flat config types were a distinct
decision even if feature set selection muddied the topic.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
staticcheck linters broke until recent golangci-lint. Now, normal
behaviour of enforcing no nil context works again. Ex.
```
assemblyscript/assemblyscript_example_test.go:16:25: SA1012: do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (staticcheck)
r := wazero.NewRuntime(nil)
```
Since default lint already checks for nil context, this removes our
permission of nil context args. The original reason we permitted nil is
no longer valid: we once allowed context to be stashed in config, and
removed that as it caused bugs. We forgot to undo allowing nil
explicitly.
Note: this doesn't particularly check in our code for nil context,
similar as we don't particularly check in our code for nil anything
else. End users should use linters as none of our parameters should be
nil anyway.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This changes the default random source to provide deterministic values
similar to how nanotime and walltime do. This also prevents any worries
about if wasm can deplete the host's underlying source of entropy.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This simplifies FunctionListener definition by making it possible to
implement both interfaces without intermediate state. Passing the
function definition to the before/after callbacks is the key.
This also continues efforts towards Go 1.19 doc formatting.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This improves the experimental logging listener to show parameter name
and values like so:
```
--> ._start.command_export()
--> .__wasm_call_ctors()
--> .__wasilibc_initialize_environ()
==> wasi_snapshot_preview1.environ_sizes_get(result.environc=1048572,result.environBufSize=1048568)
<== ESUCCESS
<-- ()
==> wasi_snapshot_preview1.fd_prestat_get(fd=3,result.prestat=1048568)
<== ESUCCESS
--> .dlmalloc(2)
--> .sbrk(0)
<-- (1114112)
<-- (1060080)
--snip--
```
The convention `==>` implies it was a host function call
(def.IsHostFunction). This also improves the lifecycle by creating
listeners during compile. Finally, this backfills param names for
assemblyscript and wasi.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This top-levels `api.FunctionDefinition` which was formerly
experimental, and also adds import metadata to it. Now, it holds all
metadata known at compile time.
Here are the public API visible changes:
* api.ExportedFunction - replaced with api.FunctionDefinition as it is
usable for all types of functions.
* api.Function - `.ParamTypes/ResultTypes()` are replaced with
`.Definition().
* api.FunctionDefinition - extracted from experimental and adds
`.Import()` to get the any imported module and function name.
* experimental.FunctionDefinition - replaced with
api.FunctionDefinition.
* experimental.FunctionListenerFactory - adds first arg of the
instantiated module name, as it can be different than compiled.
* wazero.CompiledModule - Adds `.ImportedFunctions()` and changes result
type of `.ExportedFunctions()` to api.FunctionDefinition.
Internally, logic to create function definition are consolidated between
host and wasm-defined functions, notably wasm.Module now includes
`.BuildFunctionDefinitions()` which reduces duplication in
wasm.ModuleInstance `.BuildFunctions()`,
This obviates #681 by deleting the `ExportedFunction` type which
overlaps with this information.
This fixes#637 as it includes more metadata including imports.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Before, we introduced a type `api.ImportRenamer` to resolve conflicts
where the "env" module was shared between AssemblyScript and
user-defined functions. This API was never used in GitHub, and is
complicated.
AssemblyScript also isn't the only ABI to share the "env" module, as
other web APIs like Emscripten do also. The less complicated approach is
to have packages that need to share "env" use
`ModuleBuilder.ExportFunctions` instead, and use namespaces as needed if
there is overlap.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This changes the AssemblyScript abort handler and WASI proc_exit
implementation to panic the caller which eventually invoked close.
This ensures no code executes afterwards, For example, LLVM inserts
unreachable instructions after calls to exit.
See https://github.com/emscripten-core/emscripten/issues/12322
See #601
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This removes WithWorkDirFS and any other attempts to resolve the current directory (".") in host functions. This is a reaction to reality of compilers who track this inside wasm (not via host functions). One nice side effect is substantially simpler internal implementation of file-systems.
This also allows experimental.WithFS to block file access via passing nil.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This implements wasi_snapshot_preview1.poll_oneoff for relative clock events,
and in doing so stubs `Nanosleep` which defaults to noop, but can be configured
to `time.Sleep`.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This ensures fake clocks increment so that compilers that implement
sleep with them don't spin.
This also fixes a mutability bug in config where we weren't really doing
clone properly because map references are shared.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This adds two clock interfaces: sys.Walltime and sys.Nanotime to
allow implementations to override readings for purposes of security or
determinism.
The default values of both are a fake timestamp, to avoid the sandbox
break we formerly had by returning the real time. This is similar to how
we don't inherit OS Env values.
This drops the text format (%.wat) and renames
InstantiateModuleFromCode to InstantiateModuleFromBinary as it is no
longer ambiguous.
We decided to stop supporting the text format as it isn't typically used
in production, yet costs a lot of work to develop. Given the resources
available and the increased work added with WebAssembly 2.0 and soon
WASI 2, we can't afford to spend the time on it.
The old parser is used only internally and will eventually be moved to
its own repository named watzero, possibly towards archival.
See #59
Signed-off-by: Adrian Cole <adrian@tetrate.io>
I researched what is available for improving godoc. Notably, this does
the following on every non-internal go file (tests excluded).
* consolidates links where there's a better single one
* wraps See where there are multiple relevant links
* reduces use of Note: where possible
* Uses preformatting for bullets similar to gob
* Uses titles where appropriate
I looked into it and while there are many ways to re-render godoc into
markdown, there are limited options on what to do in godoc itself. There
are tricks implicitly used where preformatting (tab indent) is used to
avoid having to add extra lines everywhere. This is done in gob, for
example.
See https://go.dev/src/encoding/gob/doc.go
See https://github.com/fluhus/godoc-tricks/blob/master/doc.go
See #426
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This consolidates the pattern used for context overrides, notably
replacing clock overrides via experimental.WithTimeNowUnixNano
and making all context keys internal.
This also makes sure experimental example tests are handled the same
way, notably backfilling one for WithFS
This notably changes NewRuntimeJIT to NewRuntimeCompiler as well renames
packages from jit to compiler.
This clarifies the implementation is AOT, not JIT, at least when
clarified to where it occurs (Runtime.CompileModule). In doing so, we
reduce any concern that compilation will happen during function
execution. We also free ourselves to create a JIT option without
confusion in the future via CompileConfig or otherwise.
Fixes#560
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This removes tedium in our examples and docs by using `Runtime.Close`
instead of tracking everything. Internal tests still track too much, but
anyway at least this stops suggesting others should do it.
This also changes our examples to use log.PanicXX so that the line
number goes into the console output.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This performs several changes to allow compilation config to be
centralized and scoped properly. The immediate effects are that we can
now process external types during `Runtime.CompileModule` instead of
doing so later during `Runtime.InstantiateModule`. Another nice side
effect is memory size problems can err at a source line instead of
having to be handled in several places.
There are some API effects to this, and to pay for them, some less used
APIs were removed. The "easy APIs" are left alone. For example, the APIs
to compile and instantiate a module from Go or Wasm in one step are left
alone.
Here are the changes, some of which are only for consistency. Rationale
is summarized in each point.
* ModuleBuilder.Build -> ModuleBuilder.Compile
* The result of this is similar to `CompileModule`, and pairs better
with `ModuleBuilder.Instantiate` which is like `InstantiateModule`.
* CompiledCode -> CompiledModule
* We punted on this name, the result is more than just code. This is
better I think and more consistent as it introduces less terms.
* Adds CompileConfig param to Runtime.CompileModule.
* This holds existing features and will have future ones, such as
mapping externtypes to uint64 for wasm that doesn't yet support it.
* Merges Runtime.InstantiateModuleWithConfig with Runtime.InstantiateModule
* This allows us to explain APIs in terms of implicit or explicit
compilation and config, vs implicit, kindof implicit, and explicit.
* Removes Runtime.InstantiateModuleFromCodeWithConfig
* Similar to above, this API only saves the compilation step and also
difficult to reason with from a name POV.
* RuntimeConfig.WithMemory(CapacityPages|LimitPages) -> CompileConfig.WithMemorySizer
* This allows all error handling to be attached to the source line
* This also allows someone to reduce unbounded memory while knowing
what its minimum is.
* ModuleConfig.With(Import|ImportModule) -> CompileConfig.WithImportRenamer
* This allows more types of import manipulation, also without
conflating functions with globals.
* Adds api.ExternType
* Needed for ImportRenamer and will be needed later for ExportRenamer.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This commit adds support for multiple tables per module.
Notably, if the WithFeatureReferenceTypes is enabled,
call_indirect, table.init and table.copy instructions
can reference non-zero indexed tables.
part of #484
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
This makes wazero.ModuleConfig an interface instead of a struct to
prevent it from being used incorrectly. For example, even though the
fields are not exported, someone can mistakenly instantiate this
when it is a struct, and in doing so violate internal assumptions.
This also fixes a mutability bug in the implementation.
Follow-up from #519 and the last in this series
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This makes wazero.CompiledCode an interface instead of a struct to
prevent it from being used incorrectly. For example, even though the
fields are not exported, someone can mistakenly instantiate this
when it is a struct, and in doing so violate internal assumptions.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
WebAssembly Core Working Draft 1 recently came out. Before that, we had
a toe-hold feature bucked called FinishedFeatures. This replaces
`RuntimeConfig.WithFinishedFeatures` with `RuntimeConfig.WithWasmCore2`.
This also adds `WithWasmCore1` for those who want to lock into 1.0
features as opposed to relying on defaults.
This also fixes some design debt where we hadn't finished migrating
public types that require constructor functions (NewXxx) to interfaces.
By using interfaces, we prevent people from accidentally initializing
key configuration directly (via &Xxx), causing nil field refs. This also
helps prevent confusion about how to use the type (ex pointer or not) as
there's only one way (as an interface).
See https://github.com/tetratelabs/wazero/issues/516
Signed-off-by: Adrian Cole <adrian@tetrate.io>
`Runtime.WithMemoryCapacityPages` is a function that determines memory
capacity in pages (65536 bytes per page). The inputs are the min and
possibly nil max defined by the module, and the default is to return
the min.
Ex. To set capacity to max when exists:
```golang
c.WithMemoryCapacityPages(func(minPages uint32, maxPages *uint32) uint32 {
if maxPages != nil {
return *maxPages
}
return minPages
})
```
Note: This applies at compile time, ModuleBuilder.Build or Runtime.CompileModule.
Fixes#500
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This prepares for exposing operations like Memory.Grow while keeping the
ability to trace what did that, by adding a `context.Context` initial
parameter. This adds this to all API methods that mutate or return
mutated data.
Before, we made a change to trace functions and general lifecycle
commands, but we missed this part. Ex. We track functions, but can't
track what closed the module, changed memory or a mutable constant.
Changing to do this now is not only more consistent, but helps us
optimize at least the interpreter to help users identify otherwise
opaque code that can cause harm. This is critical before we add more
functions that can cause harm, such as Memory.Grow.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This is an API breaking change that does a few things:
* Stop encouraging practice that can break context propagation:
* Stops caching `context.Context` in `wazero.RuntimeConfig`
* Stops caching `context.Context` in `api.Module`
* Fixes context propagation in function calls:
* Changes `api.Function`'s arg0 from `api.Module` to `context.Context`
* Adds `context.Context` parameter in instantiation (propagates to
.start)
* Allows context propagation for heavy operations like compile:
* Adds `context.Context` as the initial parameter of `CompileModule`
The design we had earlier was a good start, but this is the only way to
ensure coherence when users start correlating or tracing. While adding a
`context.Context` parameter may seem difficult, wazero is a low-level
library and WebAssembly is notoriously difficult to troubleshoot. In
other words, it will be easier to explain to users to pass (even nil) as
the context parameter vs try to figure out things without coherent
context.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This commit allows CompiledCode to be re-used regardless of
the existence of import replacement configs for instantiation.
In order to achieve this, we introduce ModuleID, which is sha256
checksum calculated on source bytes, as a key for module compilation
cache. Previously, we used*wasm.Module as keys for caches which
differ before/after import replacement.
Signed-off-by: Takeshi Yoneda takeshi@tetrate.io
