This extracts a utility `syscallfs.ReaderAtOffset()` to allow WASI and
gojs to re-use the same logic to implement `syscall.Pread`.
What's different than before is that if WASI passes multiple iovecs an
emulated `ReaderAt` will seek to the read position on each call to
`Read` vs once per loop. This was a design decision to keep the call
sites compatible between files that implement ReaderAt and those that
emulate them with Seeker (e.g. avoid the need for a read-scoped closer/
defer function). The main use case for emulation is `embed.file`, whose
seek function is cheap, so there's little performance impact to this.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This rewrites compositeFS to syscallfs.FS following wasi-sdk preopen
rules. Notably, this allows use of read-only mounts now.
For example,
```bash
$ GOOS=js GOARCH=wasm bin/go test -c -o template.wasm text/template
$ wazero run -mount=src/text/template:/ -mount=/tmp:/tmp template.wasm -test.v
=== RUN TestExecute
--- PASS: TestExecute (0.07s)
--snip--
```
This is the first step to native WASI handling of multiple pre-opens.
After this change, it is still the case that there's only one pre-open
FD visible to wasm. A later change will make it possible for WASI to see
multiple pre-opens while `GOOS=js` which doesn't use preopens, remains
on a rootFS.
A future PR may need to add a CLI flag to disable escaping directories,
(e.g. make ../.. EINVAL), similar to `fs.FS` in Go. The simplest way to
allow this is to use a host-side RootFS even in WASI, and wrap that with
a `syscallfs` filename filter.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This works around a known glitch in windows where directory entry stat
doesn't match its corresponding file stat (namely times don't). It
consolidates more test files, in the process, to ensure we are more
likely to trigger issues like this earlier.
Future work will finish the last couple places where we still use go's
fstest.MapFS internally, as well introduce stat tests at the syscallfs
abstraction: right now, most tests are still only defined in WASI.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Formerly, we introduced `wazero.Namespace` to help avoid module name or import conflicts while still sharing the runtime's compilation cache. Now that we've introduced `CompilationCache` `wazero.Namespace` is no longer necessary. By removing it, we reduce the conceptual load on end users as well internal complexity. Since most users don't use namespace, the change isn't very impactful.
Users who are only trying to avoid module name conflict can generate a name like below instead of using multiple runtimes:
```go
moduleName := fmt.Sprintf("%d", atomic.AddUint64(&m.instanceCounter, 1))
module, err := runtime.InstantiateModule(ctx, compiled, config.WithName(moduleName))
```
For `HostModuleBuilder` users, we no longer take `Namespace` as the last parameter of `Instantiate` method:
```diff
// log to the console.
_, err := r.NewHostModuleBuilder("env").
NewFunctionBuilder().WithFunc(logString).Export("log").
- Instantiate(ctx, r)
+ Instantiate(ctx)
if err != nil {
log.Panicln(err)
}
```
The following is an example diff a use of namespace can use to keep compilation cache while also ensuring their modules don't conflict:
```diff
func useMultipleRuntimes(ctx context.Context, cache) {
- r := wazero.NewRuntime(ctx)
+ cache := wazero.NewCompilationCache()
for i := 0; i < N; i++ {
- // Create a new namespace to instantiate modules into.
- ns := r.NewNamespace(ctx) // Note: this is closed when the Runtime is
+ r := wazero.NewRuntimeWithConfig(ctx, wazero.NewRuntimeConfig().WithCompilationCache(cache))
// Instantiate a new "env" module which exports a stateful function.
_, err := r.NewHostModuleBuilder("env").
```
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
@achille-roussel mentioned on chat that we have an accident waiting to
happen. This fixes it and backfills the missing test.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This introduces the new API wazero.Cache interface which can be passed to wazero.RuntimeConfig.
Users can configure this to share the underlying compilation cache across multiple wazero.Runtime.
And along the way, this deletes the experimental file cache API as it's replaced by this new API.
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
This consolidates test files and ensures our various implementations of
`syscallfs.FS` pass `fstest.TestFS`.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This adds FS.Path which holds the pre-open path currently only used in
WASI. It also fixes a TODO where we didn't know for sure if the FD
parameter for `path_` functions must always be a pre-open. The TL;DR; is
that usually it is, but it may not be (e.g. in our zig-cc example we can
see any directory FD, not just pre-opens).
Finally, this fixes a bug in our path resolution where we mistook paths
like "foo/foo" for "foo" because we only considered basenames instead of
the full path from the pre-open root.
This also makes pre-open directory lookup lazy because I noticed in
Trivy specifically, this is unnecessary for us to do eagerly, as they
change the FS at runtime per-call. In other words, any value from init
time is invalid later.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This avoids logging activity on stdio file descriptors, in order to help
make troubleshooting easier. Usually, there isn't an issue in these, yet
wasm panics are harder to read if there is also logging of the ..
logging.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This consolidates internal code to syscallfs, which removes the fs.FS
specific path rules, except when adapting one to syscallfs. For example,
this allows the underlying filesystem to decide if relative paths are
supported or not, as well any EINVAL related concerns.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This stubs all remaining syscalls for `GOARCH=wasm GOOS=js` to return
ENOSYS, instead of panic'ing. This allows us to see the parameters it
receives.
For example:
```
==> go.syscall/js.valueCall(fs.truncate(path=/tmp/_Go_TestTruncate135754730,length=0))
<== (err=function not implemented,ok=false)
```
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This masks interfaces returned by `fs.File` so that we don't
accidentally allow writes when opened for reading. This also adds
`syscall.NewReadFS` which can enforce read-only access in general, such
as would be ideal for tests that try to read files from the host root
filesystem (e.g. /etc/passwd).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
As noted in slack, we are unlikley to long term use fs.FS internally.
This ensures we attempt to cast to syscallfs.FS for all I/O by panicing
on fs.Open.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This adds ExportedFunctionDefinitions and ExportedMemoryDefinitions to
api.Module so that those who can't access CompileModule can see them.
Fixes#839
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This makes the version, arch, os tuple into a subdirectory to help
troubleshooting and cache management in general. The version is left
inside the binary key regardless.
Note: when installing via `go install ./cmd/wazero/...` the version ends
up as `dev`. This helps make that obvious. For example.
```bash
$ wazero version
dev
$ ./build/tinygo test -target wasi -c -o os.wasm os
$ wazero run -cachedir=$HOME/.wazero -mount=.:/ -env=HOME=. os.wasm -test.v
$ find $HOME/.wazero
/Users/adrian/.wazero
/Users/adrian/.wazero/wazero-dev-amd64-darwin
/Users/adrian/.wazero/wazero-dev-amd64-darwin/1f149f4bf475a33023ce33302780bee29ec08e89bd57cfbdf639c65c6009f1a4
```
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This fixes and backfills tests for open flags, notably O_CREAT. I
verified this with TinyGo tests, which now proceed when attempting to
create new files.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This renames the internal writefs package to syscallfs as it is largely
dependent on syscall signatures. This also implements utimes in gojs.
WASI will be a follow-up change as it requires more infrastructure.
Notably, we also need non-TinyGo tests because TinyGo doesn't yet
support os.Chtimes or corresponding syscalls.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This implements path_(create|remove)_directory path_unlink_file in wasi, particularly needed to use TinyGo tests to verify our interpretation of WASI. Use of this requires the experimental `writefs.DirFS`.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This splits unlink and rmdir from remove, as it is not only more precise
in GOOS=js, but it is also needed to implement wasi. I verified this
works by running go unit tests with logging.
```
==> go.syscall/js.valueCall(fs.open(name=/tmp/TestErrIsNotExist1062486353,flags=,perm=----------))
<== (err=<nil>,fd=10)
==> go.syscall/js.valueCall(fs.fstat(fd=10))
<== (err=<nil>,stat={isDir=true,mode=-rwx------,size=96,mtimeMs=1672285985206})
==> go.syscall/js.valueCall(fs.readdir(name=/tmp/TestErrIsNotExist1062486353))
<== (err=<nil>,dirents=&{[001]})
==> go.syscall/js.valueCall(fs.unlink(path=/tmp/TestErrIsNotExist1062486353/001))
<== (err=is a directory,ok=true)
==> go.syscall/js.valueCall(fs.rmdir(path=/tmp/TestErrIsNotExist1062486353/001))
<== (err=<nil>,ok=false)
```
Signed-off-by: Adrian Cole <adrian@tetrate.io>
The type we use to expose write operations is still evolving. It might
be a single writefs.FS interface, or similar to go where we have an
interface per feature (e.g. writefs.MkdirFS). These choices are all
implementation details for DirFS and won't be settled before the end of
the month version cutoff. Instead, this only exposes the ability to
create a DirFS, not an arbitrary implementation of writefs.FS. This does
so by making `writefs.FS` an internal type.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We originally exported WASI errno as we originally supported invoking
host functions directly (e.g. without a guest). This was an invalid call
pattern, and we removed that. However, we left the errnos exported even
though the caller of a guest won't ever see them. This prevented us from
re-using them cleanly in features such as logging.
This moves all constants including function names and flag enums
internal so that there is less duplication between logging and
implementation of wasi functions. This also helps in reference searches,
as we can analyze uses of a particular function name.
The only constant left exported is the module name, as there's a use
case for that (overriding implementations via FunctionBuilder).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This adds writefs.FS, allowing functions to create and delete files.
This begins by implementing them on `GOARCH=js GOOS=wasm`. The current
status is a lot farther than before, even if completing write on WASI is
left for a later PR (possibly by another volunteer).
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This uses logging infrastructure introduced in #938 to show
paths and certain result parameters. While further tuning is expected,
the main thing this does is show which paths are affected by syscalls.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This changes the listener signature to accept context and calling
module, so that all possible parameters and results can be logged. This
also changes the logging listener to make parameters visible when
logging results.
This infrastructure supports some helpful use cases, such as logging
WASI result parameters, such as the prestat path, which is only knowable
after the function has been called. The context parameter supposed
reading results of gojs functions, which are stored host-side in a go
context object.
Future pull requests will complete this as well backfill unit tests.
This is raised independently mainly to keep the PR size down of the
upcoming filesystem logger.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
`GOARCH=wasm GOOS=js` defines parameter names in go source, and they are
indirectly related to the wasm parameter "sp". This creates a pseudo
name section so that we can access the parameter names. The alternative
would be adding a hack to normal FunctionDefinition, only used for gojs.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
This refactors GOOS and GOARCH specific code into their own packages.
This allows logging interceptors to be built without cyclic package
dependencies.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
