Currently, yaegi uses the `filepath` package to join paths.
However, this has issues: YAEGI uses an `fs.FS` internally for source code access, and `fs` specifies that `/` separated paths must be used. On Unix systems, this obviously makes no difference; but on Windows, with a non-default FS, this causes errors (since YAEGI creates `\` separated paths, but the FS expects `/` separated paths).
Furthermore, generally speaking, Golang import paths and the like are always `/` separated anyway, regardless of OS.
To clear this up, this PR changes path handling to `/` separated paths. This has the advantage of being accepted by all OS (even Windows will accept `/` separated paths). Paths passed by the user to `CompilePath` or similar are changed to `/` separators, if necessary.
* chore: bump golangci-lint and fix config
This fixes the CI lint step. No change in code except a fix
in test output.
* bump golangci-lint from 1.56.2 to 1.63.4
* test: fix test check
When parsing binary operator expressions, make sure that implicit type conversions for untyped expressions are performed. It involves walking the sub-expression subtree at post-processing of binary expressions.
Fixes#1653.
This builds on #1644 and adds automatic per-loop variables that are consistent with go 1.22 behavior. See #1643 for discussion.
This is still a draft because the for7 version ends up capturing the per-loop var values when they are +1 relative to what they should be. Maybe somehow the incrementing and conditional code is somehow capturing the within loop variables and incrementing them? not sure how that would work. anyway, need to investigate further before this is ready to go, but pushing it here in case there are other issues or someone might figure out this bug before I do..
I found these basic functions and Stringer methods essential for tracking down issues fixed recently -- probably be of use to others and I couldn't find something else in the code that would provide similar utility, though I never tried the debugger.
This fixes issue #1634
includes special case for defer function.
I could remove or significantly reduce the comment description, and just have that here for future reference:
// per #1634, if v is already a func, then don't re-wrap! critically, the original wrapping
// clones the frame, whereas the one here (below) does _not_ clone the frame, so it doesn't
// generate the proper closure capture effects!
// this path is the same as genValueAsFunctionWrapper which is the path taken above if
// the value has an associated node, which happens when you do f := func() ..
Follow by the [Spec](https://go.dev/ref/spec#Assignment_statements):
The number of operands on the left hand side must match the number of values. For instance, if f is a function returning two values `x, y = f()` assigns the first value to x and the second to y.
In the second form, the number of operands on the left must equal the number of expressions on the right, each of which must be single-valued, and the nth expression on the right is assigned to the nth operand on the left.
Fixes#1606
A side effect of #1281 is that it added unnecessary additional newlines in generated interface wrappers from the `extract` tool. This PR removes those newlines from the extract tool template and updates the generated code with that change.
* feat: support go1.22
* Temporary fix for consistency tests due to language change in for loops
* review: clean old files
---------
Co-authored-by: Fernandez Ludovic <ldez@users.noreply.github.com>
* feat: generate go1.21 files
* chore: update CI
* feat: add support for generic symbols in standard library packages
This is necessary to fully support go1.21 and beyond, which now
provide some generic packages such as `cmp`, `maps` or `slices`
in the standard library.
The principle is to embed the generic symbols in source form (as
strings) so they can be instantiated as required during interpretation.
Extract() has been modified to skip the generic types, functions and
constraint interfaces which can't be represented as reflect.Values.
A new stdlib/generic package has been added to provide the corresponding
source files as embedded strings.
The `Use()` function has been changed to pre-parse generic symbols as
doing lazy parsing was causing cyclic dependencies issues at compiling.
This is something we may improve in the future.
A unit test using `cmp` has been added.
For now, there are still some issues with generic stdlib packages
inter-dependencies, for example `slices` importing `cmp`, or when
generic types or function signatures depends on pre-compiled types
in the same package, which we will support shortly.
* fixup
* fixup
* fixup
* fixup
* fixup
* fixup
* fixes for go1.20
* fix previous
* update unsafe2 for go1.21, skip faky tests
In go1.21, the reflect rtype definition has been move to internal/abi.
We follow this change for maintainability, even if there is no layout
change (the go1.20 unsafe2 is compatible with go1.21).
We have isolated a few problematic tests which are failing sometimes
in go1.21, but work in go1.20, and also in go1.22. Those tests are
skipped if in go1.21. A preliminary investigation can not confirm that
something is wrong in yaegi, and the problem disappears with go1.22.
* add new wrapper for go1.21 package testing/slogtest
* add missing wrapper for go/doc/comment
* add support for slices generic package
---------
Co-authored-by: Marc Vertes <mvertes@free.fr>
Adds `wasip1` to known OS list, introduced in golang/go#58141.
Without this change, `yaegi extract` may fail on Go 1.21 with the following message:
```
type-checking package "time" failed (<GOROOT>/src/time/zoneinfo_wasip1.go:8:5: platformZoneSources redeclared in this block)
```
Currently, yaegi only records source positions when writing panic trace to stderr.
This change adds function names to the panic output.
Unfortunately, yaegi walks the trace in reverse order, comparing to Go runtime. This can be improved in the future.
Avoid a spurious optimisation which forces a variable to be reused instead of redefined for assignment operation. This ensures that a variable defined in a loop is re-allocated, preserving the previous instance when used by a closure for example.
Fix#1594
Embedding files using `//go:embed` and `embed` packages can not be supported in yaegi, so it is better to not generate the wrapper to embed and have a graceful error in case of usage of `embed.FS` in the interpreter.
Also update README about unsuported directives.
Fixes#1557.
I implemented fs.FS. When the file does not exist, `xerrors.Wrap(fs.ErrNotExist, "")` will be returned. However, `os.IsNotExist` cannot handle this situation. I found the following comment:
```
// IsNotExist returns a boolean indicating whether the error is known to
// report that a file or directory does not exist. It is satisfied by
// ErrNotExist as well as some syscall errors.
//
// This function predates errors.Is. It only supports errors returned by
// the os package. New code should use errors.Is(err, fs.ErrNotExist).
```
Therefore, we should use `errors.Is(err, fs.ErrNotExist)` instead.
Unsafe functions such as `unsafe.Alignof`, `unsafe.Offsetof` and `unsafe.Sizeof` can be used for type declarations early on during compile, and as such, must be treated as builtins returning constants at compile time. It is still necessary to explicitely enable unsafe support in yaegi.
The support of `unsafe.Add` has also been added.
Fixes#1544.
Only structures with one embedded field can be marked anonymous, due to golang/go#15924. Also check only that the method is defined, do not verify its complete signature, as the receiver may or not be defined in the arguments of the method.
Fixes#1537.
The result of the expression giving the size of an array may be an `int` instead of `constant.Value` in case of an out of order declaration. Handle both cases.
Fixes#1536.
consider packages from git.sr.ht, the user namespace is prefixed with `~`
so running something like `yaegi extract git.sr.ht/~emersion/scfg`
was producing syntax errors with `~` in identifiers. and also `~` in filenames which worked but probably best not to have it there either
thanks!
In the case of a Go short definition (i.e. `a, b := f()`), the new defined variables must be (re-)created in order to preserve the previous value (if in a loop) which can be still in use in the context of a closure. This must not apply to redeclared variables which simply see their value reassigned.
The problem was both occuring in callBin() for runtime functions and assignFromCall() for functions created in the interpreter.
Fixes#1497.
hi!
this issue is sorta blocking for me so i thought i would try to fix it.
im still learning the codebase and understanding how yaegi works, but I thought I would attempt to add a test in the style of other tests as a start.
please let me know if there is anything i need to change / run, or if anyone knows perhaps a good place to start for tackling this issue
Fixes#1496
The logic of patching reflect struct representation has been fixed in presence of function fields and to better handle indirect recursivity (struct recursive through nested struct).
Fixes#1519.
This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `t.TempDir` function from the `testing` package to create temporary directory. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
```go
func TestFoo(t *testing.T) {
// before
tmpDir, err := os.MkdirTemp("", "")
if err != nil {
t.Fatalf("failed to create tmp directory: %v", err)
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
// now
tmpDir := t.TempDir()
}
```
This should allow to build the package on AlpineLinux. Also document the constraint of having to install the source under $GOPATH/src/github.com/traefik/yaegi until Go modules are supported.
Fixes#1523.
closes#1514
hi!
I had the same problem as #1514 and I wanted to fix it, I found
When asserting *crypto/rsa.PublicKey, using the typ attribute of node to get an nil rtype, resulting in the assertion result being nok
This code contains the same problem
```go
package main
import (
"log"
"crypto/rsa"
)
func main() {
var pKey interface{} = &rsa.PublicKey{}
if _, ok := pKey.(*rsa.PublicKey); ok {
log.Println("ok")
} else {
log.Println("nok")
}
}
```
So I submitted this Pull Request, hope it will be merged
With certain yaegi configuration on Windows, the loop in `previousRoot` can be infinite, because it fails to recognize driver letters as root.
This change adds a few more safeguards: checks path prefix earlier and checks if `filepath.Dir` produces an empty path.
For methods defined on interfaces (vs concrete methods), the resolution of the method is necessarily delayed at the run time and can not be completed at compile time.
The selectorExpr processing has been changed to correctly identify calls on interface methods which were confused as fields rather than methods (due to the fact that in a interface definition, methods are fields of the interface).
Then at runtime, method lookup has been fixed to correctly recurse in nested valueInterface wrappers and to find embedded interface fields in case of struct or pointer to struct.
Finally, remove receiver processing in `call()`.The receiver is already processed at method resolution and in genFunctionWrapper. Removing redundant processing in call fixes handling of variadic method, simplifies the code and makes it faster.
With those fixes, it is now possible to load and run `go.uber.org/zap` in yaegi. In turn, it makes possible for yaegi to run plugins dependent on zap, such as coraza-waf.
Fixes#1515,
Fixes#1172,
Fixes#1275,
Fixes#1485.
It allows to use interpreter functions as parameters in the runtime, even for defered callbacks, or when passed as empty interfaces, as for runtime.SetFinalizer.
Fixes#1503
When generating a new type, the parameter type was not correctly duplicated in the new AST. This is fixed by making copyNode recursive if needed. The out of order processing of generic types has also been fixed.
Fixes#1488
Currently if interpreter tries to eval an empty directory with no Go files it will error (as it should), but all future calls to eval will deadlock, because the mutex is not unlocked correctly. I believe this is a critical issue that must be addressed.
For a long time, there was a confusion between aliased types and named types (due to my misunderstanding of alias types in Go at that time). The type category `aliasT` has been renamed into `linkedT`, which is correct semantically.
Aliased types are only declared using `typeSpecAssign`, and its processing is now distinct from `typeSpec` statement, used for named types.
A `linkedT` type is obtained by a statement like `type A uint32`, where `A` type category is therefore `linkedT`.
An aliased type is obtained by a statement like `type A = uint32` (notice the `=` sign, translating into `typeSpecAssign`).
The semantic difference is that in the first linkedT form, `A` and `uint32` are 2 distinct types, instead of being strictly equivalent in the `typeSpecAssign` form (the 2 names lead to one type definition).
Fixes#1416.
For example, on architecture GOARCH=amd64, a file named `foobar_amd64.go` would be skipped instead of being read and parsed. The function `skipFile` is fixed and missing tests are added.
In case of forward definition of a constant, a symbol may be undefined when attempting to compute the array size in type analysis. Just mark the type as incomplete instead of aborting directly, to allow resolution at a second pass.
Fixes#1470.
Up to now functions could be stored as node values in frame (as for interpreter defined functions) or function values, directly callable by the Go runtime. We now always store functions in the later form, making the processing of functions, anonymous closures and methods simpler and more robust. All functions, once compiled are always directly callable, with no further wrapping necessary.
Fixes#1459.
Although empty interfaces are usually not wrapped, for compatibility with the runtime, we may have to wrap them sometime into `valueInterface` type.
It allows to preserve interpreter type metadata for interface values exchanged with the runtime. It is necessary to resolve methods and receivers in the absence of reflect support.
During type assertions on empty interfaces, we now handle a possible valueInterface and dereference the original value to pursue the type assertion.
In the same change, we have improved the format of some panic messages at runtime to give location of offending source at interpreter level.
This change will allow to fixtraefik/traefik#9362.
Those declarations involve the indexListExpr AST token, which was not handled in type.go. The same processing as for a single type parameter is applied.
Fixes#1460.
In type assertion at compile time, compare signatures between function types only.
Make `itype.numOut()` return the correct value for Go builtins (this was not strictly necessary due to above fix, but it is correct and improves maintainability).
Fixes#1454.
In [short variable declarations](https://go.dev/ref/spec#Short_variable_declarations),
The reuse of existing symbols is possible only if a new variable is defined,
otherwise a new symbol must be created, which was not the case in the issue.
Search for new symbols and correctly ignore blank variables.
Fixes#1434.
The representation of non empty interfaces defined in the interpreter is now identical between refType() and frameType() functions, which are used to generate interpreter objects.
Fixes#1447 and #1426.
If the value on which to type-switch was already set (i.e. a variable),
there was no problem. But if it had to be obtained through a complex
expression (func call, array index, etc...), then the code to retrieve
the value prior type-switch was not scheduled. This is now fixed.
This issue is nasty because the behavior is silently changed,
leading potentially to further unrelated issues or runtime panics.
Fixes#1444.
This change implements a workaround to better support composed
interfaces in yaegi and let the interpreter define objects which
implement multiple interfaces at once.
We use the existing MapTypes to store what possible composed interface
wrapper could be used for some interfaces. When generating an interface
wrapper, the wrapper with the highest number of implemented methods is
chosen.
This is still an imperfect solution but it improves the accuracy of
interpreter in some critical cases.
This workaround could be removed in future if/when golang/go#15924
is resolved.
Fixes#1425.
The function `getMethodByName()` is now able to look for
embedded `valueInterface` field for matching methods in interface
struct fields.
Fixes#1439 and #1427.
The variable dependency check function was confused by a dependency
variable with the same name but in an external package.
This change is necessary to address #1427.
Status:
* [x] parsing code with generics
* [x] instantiate generics from concrete types
* [x] automatic type inference
* [x] support of generic recursive types
* [x] support of generic methods
* [x] support of generic receivers in methods
* [x] support of multiple type parameters
* [x] support of generic constraints
* [x] tests (see _test/gen*.go)
Fixes#1363.
Such function declaration denotes either a linkname (an access to
an arbitrary, typically unexported symbol, solved by go compiler),
or a foreign C or assembly implementation of the body.
Those cases are not supported (or planned to be) by the interpreter.
Fixes#1431.
In addition:
- extract commmand now skips exported generics in runtime wrappers
- interp_consistent_test.go fixed for go1.18 and go1.19
- move minimal version of go compiler to go1.18
Note that this version is incompatible with go1.17 and before due
to the major changes in the stdlib go parser.
To be merged once go1.19 is officially released (not before).
Some tests are not passing when using go1.18, due to a change of
content in error messages compared to go1.17. We simply skip them
while we support go1.17. It concerns a small number of tests
regarding error detection.
I expect the following code to be supported.
```go
type TT http.Header
func (t TT) Set(key, val string) {
}
func (t TT) Get(key string) string {
}
```
So, I pushed this PR.
Do I need to add some test cases? I don't see the relevant test files ....
The type.val field was always pointing to the final underlying type
for aliased types, defeating a possible match if a method was
attached to a type in between. Now the complete chain of aliases
is always preserved.
We have added an underlying() itype method which returns the underlying
type of a defined type (aliasT), even in the presence of multiple
indirections.
We have added a definedType function which checks if type t1 is
defined from type t2 or t2 defined from t1, required when checking
assignability of aliasT types.
Fixes#1411.
PS: this is the 2nd attempt, as the first version #1412 wasn't passing
_test/issue-1408.go as well. This PR does pass and supersedes #1412.
We finally address a long standing limitation of the interpreter:
the capacity to generate the correct interface wrapper for an
anonymous interface{} function parameter of a binary function.
It allows for example fmt.Printf to invoke the String method
of an object defined within the interpreter, or json.Marshal
to invoke a textMarshaler method if it exists and if there is
no Marshaler method already defined for the passed interpreter
object.
To achieve that, we add a new mapType part of the "Used" symbols
to describe what not empty interfaces are expected and in which
priority order. This information can not be guessed and is found
in the related package documentation, then captured in stdlib/maptypes.go.
Then, at compile time and/or during execution, a lookup on mapTypes
is performed to allow the correct wrapper to be generated.
This change adds a new MapType type to the stdlib package.
Fixes#435.
In that case, the interface must be wrapped in an valueInterface
at creation.
With that fix, it is now possible to import
github.com/google/go-querystring/query. Not tested beyond that.
Fixes#1123.
In REPL mode, a panic (stack overflow) could be triggered by:
$ yaegi
> a:
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x14020760330 stack=[0x14020760000, 0x14040760000]
fatal error: stack overflow
[...]
This issue occurs in incremental parsing mode only, and not when the parser
is in file mode. We avoid it by being more defensive when generating
values.
Fixes#982.
At variable, function parameter, slice, map or field element assign,
if the destination type is an empty interface, the value was never
wrapped into a valueInterface (to preserve type mutability in case
of re-assign). Now we wrap it in a valueInterface if the source
type has a non empty set of methods, to allow a future use as a non
empty interface.
There are still corner cases, but it extends notably the support
of interfaces within the interpreter.
Fixes#1355.
The logic of goto was false due to mixing break label and goto
label, despite being opposite. In case of break, jump to node (the
exit point) instead of node.start. Also always define label symbols
before their use is parsed.
* Fix continue label, detect invalid labels for break and continue
* fix multiple goto, break, continue to the same label
Fixes#1354.
* interp: fix creation of binary composite types
Use direct assignment instead of reflect.Value Set method to
initialize a binary composite type, as for non binary types.
It ensures that a new reflect.Value is stored in the frame
instead of modifying a possibly existing one, which can defeat
the purpose of initializing variables in a body loop.
While there, remove the need to have and use a mutex on types.
Fixes#1381.
* review: rework a bit the test
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
We now detect the use of special identifier _ (blank) during parsing in order to abort compiling early. It allows to not panic later during execution. We must catch all the cases where blank is used as a value, but still preserve the cases where it is assigned, used as a struct field or for import side effects.
Fixes#1386.
When a package is imported, it creates a symbol with a name like "errors/_.go". If a statement such as x := errors.New(...) is executed before import "errors", it creates an incomplete type symbol with a name like "errors". Importing the package after the incomplete type symbol has been created does not fix the compile issue because the incomplete type still exists.
To fix this, this PR deletes the incomplete type symbol, if one exists.
Closes#1388.
The interpreter has its own internal fileset and expects all code to have been parsed using that fileset. If a user creates a fileset, calls `go/parser.Parse*`, then passes the result to `interp.CompileAST`, strange things can happen.
The solutions I can see are:
1. Expose the fileset so the user can use it when parsing source.
2. Add the fileset as an option (input to New) so that the user can tell the interpreter to use a specific fileset.
3. Pass the fileset into `CompileAST`
There are two ways to implement option 3. One is to add a field to nodes and update every reference to `interp.fset` to use `node.fset`. The other is to add a parameter to every function that references `interp.fset` or calls a function that does. Both of these are significant changes and involve an extra pointer for every node or most function calls.
Options 1 and 2 are easy. Option 2 involves adding an option so I went with option 1. I can imagine situations where option 2 could be necessary, but I can open another issue/PR if and when I need that.
Fixes#1383
* interp: handle struct with multiple recursive fields
In case of a recursive struct with several recursive fields of
different type, only the first one was properly fixed when
constructing the corresponding reflect type. We now memorize and
process all fields at the same depth level.
Fixes#1371.
* Update interp/type.go
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* fix lint
* fix comment
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* interp: fix handling of redeclaration in multi-assign expression
In a statement like `a, b := f()` if `a` was previously declared,
its symbol must be reused, a new symbol must not override its
previous value. This is now fixed.
* In case of redeclaration, reuse the existing only if the redeclared
variable has the same type. Add _test/var16.go to check this use
case.
Fixes#1365.
The capability to dereference pointers has been added to
methodByName(), improving method lookup on binary values.
Wrapping to valueInterface is performed in a missing use case at
return of function calls. It was done in the nested call, but not
at assign.
Fixes#1330.
A runtime builtin assignFromCall is added to handle multiple values returned at once. It is necessary if some of the values require to be set to interface values in the caller space, which is performed by reflect.Set in assignFromCall.
Fixes#1311.
In restricted mode, replace environment related symbols in
stdlib os package by a version which operates on a private copy
per interpreter context.
It allows to have concurrent interpreters in the same process
operating each in their own environment without affecting each
other or the host.
If unrestricted opt is set, this behaviour is disabled, and the
default symbols from stdlib are used.
Note also that no modification is done for syscall package, as it
should be not used in restricted mode.
At generation of operator closures, the comparison expression
was hard-coded instead of being derived from the operator name,
leading to a wrong result.
Fixes#1297.
In some cases, we use many yaegi instances in one process, each interpreter will expect its own os.Args. This is done in interp.go as an option in interp.Options besides Stdin Stdout and Stderr.
Add a tag option in the extract program to optionally give an opportunity to tailor some packages in for example the stdlib, for people who is interested in keeping the final binary size as smaller as possible.
Below go generate
```
//go:generate ../cmd/extract/extract -name stdlib -tag stdmime mime mime/multipart mime/quotedprintable
```
produces a header to stdlib/mime-multipart.go
```
// Code generated by 'yaegi extract mime/multipart'. DO NOT EDIT.
// +build go1.16,!go1.17,stdmime
```
When an interpreter type implementing an interface is
used by the runtime, the runtime can extract its type
and create new values using reflect, and call methods
on it. The problem is that there will be no interpreted
method counterpart in this case, which makes wrapper panic.
Allow the String() method wrapper to always succeed and
return an empty string if no interpreted method is present.
This allows scripts to define custom flag.Value types on
which the runtime internally instantiates values using
reflect (see isZeroValue in Go src/flag/flag.go).
This workaround could be generalized to all wrappers if
necessary. At this moment, it is convenient to keep the
default behavior of expecting instantiated interpreter
methods, in order to catch interpreter bugs related to
interfaces.
Fixes#1276.
This comes from experiments looking into #1259 where incomplete twice seen types are marked as complete. To mitigate the problem instead of a map of seen types in `nodeType` a slice is used as a cheap way to keep track of our current path through the node tree.
This change fixes a regression introduced by PR #1192 in a program using
https://github.com/NYTimes/gziphandler which defines several types
implementing stdlib interfaces. We do not implement a wrapper
if we see that a type already implements an interface, except that
it can be falsly reported by reflect in case of a struct with
embedded interface field. We need to force the wrapper generation
in this case.
The problem occurs only for wrappers on struct, not on pointers or
other indirection types.
Adds CompileAST, which can be used to compile Go AST directly. This
allows users to delegate parsing of source to their own code instead of
relying on the interpreter.
CLoses#1251
When a define statement relies on a selector or type that may exist in another file it should revisit once GTA is complete. This allows that revisit.
**Note:** In order to keep the original GTA error for the define statement, so the error received is correct and meaningful, I have added a node `meta` property. I decided to make it generic as it may be useful in future. There may be a better way to stash errors across the GTA runs and am open to suggestion here.
Fixes#1253
The current `nodeType` selector precedence is heavy handed in favour of package type. It seems to often create `typeSym` symbols as variable types in the scope will never be found. To fix this if the ancestor node is a field expression, the package type is searched for. After this, if the type is still `nil` the normal scope is searched using `nodeType2`.
Fixes#1158
Make sure to keep always a single copy of incomplete type structures.
Remove remnants of recursive types processing.
Now `import "go.uber.org/zap"` works again (see #1172), fixing regressions
introduced since #1236.
When `nodeType` recovers names and methods, it can overwrite the data if the type is an aliasT. When aliasing a type, do not recover the methods, this will be done in the GTA typeSpec pass.
Related to #1158
When a const is late binding and specified with a type, the GTA defineStmt was creating the symbol with the current scopes `iota` which is incorrect. The symbol should be created with the source nodes `rval`.
Related to #1158
This test (assert2.go) display 2 separate issues:
1. assert2.go L28: Type assert tries to set an `interface{}` to a `valueInterface`. The typing here is complex, we have a valueT(strings.Builder) wrapped in a ptrT wrapped in a src iface wrapped in a valueT(interface{}). Type assert fails to realise that the `valueT` `interface{}` is wrapping the `valueInterface`.
2. assert2.go L29: `genValueBinMethodOnInterface` does not try and get the bin method, as the `typ.node` (`ptrT` or a `valueT`(`string.Builder`)) is set. In this case the src iface is called with a receiver argument. To fix this the method is looked for first if possible, and only if not found does it fall back to the `defaultGen`.
Fixes#1227
It was initially assumed that `nodeType` needed to rebuild the type from scratch, but this is not the case anymore. The existing type constructors are now used in `nodeType` to make it more readable. In doing this some bugs in type strings were found and fixed, along with adding the real package name to the type.
As `ptrOf` is now the only place that pointer types are constructed, it is feasible to cache the pointer type on the value type. To ensures that we have a consistent pointer type for any value type.
As the unsafe and pointer methods in `reflect` are to be depreciated, and seeing no replacement functions, it is now forced that some unsafe is needed to replace this as when and interface is dereferenced it is made unsettable by reflect.
With this in mind, this adds real recursive types by hot swapping the struct field type on the fly. This removes a lot of compensation code, simplifying all previous cases.
**Note:** While the struct field type is swapped for the real type, the type string is not changed. Due to this, unsafe will recreate the same type.
This adds `itype.str` which is a string representation of the type built when the type is built. The goal is to make type comparison simpler and centralise the creation of types just to constructors and `nodeType`. `nodeType` continues to build types in parts so to reuse underlying types better.
Make use of fs.FS (new to go 1.16) to allow for reading source files from diverse filesystems (local, embed, custom).
* `Options` has a new field `SourcecodeFilesystem fs.FS` so users can supply their own read-only filesystem containing source code.
* Defaults to the local filesystems (via `RealFS` - a thin `os.Open` wrapper complying with `fs.FS`) so regular users should see no change in behaviour.
* When no filesystem is available (e.g. WASM, or if you want to embed files to retain single binary distribution) an alternative filesystem is preferable to using `Eval(string)` as that requires the stringy code to be a single file monolith instead of multiple files. By using an `fs.FS` we can use `EvalPath()` and gain the ability to handle multiple files and packages.
* You can make use of embed filesystems (https://pkg.go.dev/embed) and custom filesystems obeying the `fs.FS` interface (I use one for http served zip files when targeting wasm as there is no local filesystem on wasm). Tests can make use of `fstest.Map`.
* NOTE: This does NOT affect what the running yaegi code considers its local filesystem, this is only for the interpreter finding the source code.
See `example/fs/fs_test.go` for an example.
Fixes#1200.
Set node address in `val` field at creation of `funcDecl` node so it can be used correctly at closure generation, even in the case of forward function declarations, where the value was zero.
Fixes#1214
When calling a function, if the input param is a "zero" instance, it is not set on the input. This is an issue where the param is an `interface{}` as a `nil` value is set instead of the zero value.
The actual solution for this is to remove the `if !val.IsZero()`, this however runs into an issue on `_test/struct48.go` where we have a zero recursive struct instance (`*interface{}`) and we have no way to get its real type. Should a way be figured out to keep tabs on the original type, the `if` can go away, and in the zero case of `genValueRecursiveInterfacePtrValue`, the actual zero type can be returned.
Fixes#1215
Add getConcreteType to retrieve the concrete type of a nested interface
value implementing a specific interface for which a wrapper exists.
If method resolution fails at runtime, a panic is now issued instead
of an error message and continue.
Fixes#1187.
Map handling builtins getIndexMap and rangeMap had some leftover
code of previous way of emulating interfaces, which was modified
following changes in #1017.
Specific code for interfaceT is removed, as not necessary anymore.
Map builtins are now simplified and more robust.
Fixes#1189.
When operating on map elements, result of assign operators were not
written to the map entry. Now check if the destination of an assign
operator is in a map, and if so, set the result to it.
Fixes#1194.
Add early detection of cases where no wrapper is necessary because
the value type already implements the target interface.
It should both increase performances by avoiding the wrapper overhead,
and fix errors due to replacing valid values by incomplete wrappers,
caused by the presence of private methods in the interface definition,
as in #1191.
Fixes#1191.
Store the interpreter value of the interface object to wrap as
a field called IValue, at offset 0 in wrapper structures.
Update extract to include IValue field.
In typeAssert, detect interface wrapper, and dereference the
interpreter value from IValue wrapper field.
Fixes#1166.
In parsing array type declaration, The type check of array size was
restricted to `int`. Broaden the test to accept any valid integer
kind.
Fixes#1175.
The resolution method was not compliant with the Go specification which
requires to retain the object where the field or method is the most
shallowed.
The detection of ambiguous fields or methods (same depth in different
objects) has also been added.
Fixes#1163.
In selector resolution, struct field matching now precedes
method matching. Before struct field matching could be skipped
in case of a matching method, which is incorrect, as demontrated
by _test/issue-1156.go.
Field lookup has been fixed to operate on recursive structures.
Concrete type values are derived when filling a receiver for
interface methods.
LookupBinField has been fixed to skip non struct values.
LookupMethod has been fixed to iterate on interface values as
well as concrete type values.
Fixes#1156.
### Background
#1102 changed how `Interpreter.Use` interprets export paths such that the last path component is stripped and used as the package name. This resulted in #1139 - attempting to Use an export with only one path component, such as `foo`, would result in the import path being `.`.
### Breaking API Change
This PR changes the signature of `Interpreter.Use` from `Use(Exports)` to `Use(Exports) error`.
### Fix for #1139
With this PR, if Use is called with an incomplete export path, such as `foo`, Use will return an error.
Fixes#1151
If I add a package with `Use` and import it with `ImportUsed`, the package is added to the universe scope as `<pkg>`. If I import with `Eval`, the package is added as `<pkg>/_.go`. However, `(*node).isType` (in cfg.go) only checks for `<pkg>/_.go`. Thus, packages imported with `ImportUsed` can be inaccessible.
This MR updates `(*node).isType` to fall back to `<pkg>` if `<pkg>/_.go` does not exist.
In typecheck.go, detect binary methods so we know when to skip the receiver as first parameter when checking function signatures. The signature check is not yet performed, we just avoid a false error.
In cfg.go, take care to label types with isBinMethod field to true whenever a binary method is resolved.
Also, do not attempt to wrap node in functions if the node value is already a binary function.
Fixes#1145.
Fixes#1150
1. When resolving a selector expression involving an aliased type, resolve the aliased type
2. When building an array literal, resolve the aliased type
Aliases of named array and slice types were the only ones that didn't work, but I added the other test cases for the sake of completeness and through testing.
Fixes#1149
Because of how aliases are handled, `n.gen` is set to `getIndexSeqMethod` or `getIndexSeqPtrMethod` in cases like the one described in #1149. As a result, `FieldByIndex` can be called on a value that is not a struct, which causes a panic. This MR updates those two methods to avoid that call if the index array is empty.
This is a small change that allows use of composite array literals, such as:
```go
type Vec3 [3]float32
var foo = []Vec3{
{1, 0, 0},
{6, 0, 0},
{6, 2, 0},
{2, 2, 0},
{1, 1, 0},
}
```
In switch case expressions, the condition on case clause was
not always properly evaluated. Reverse the order of case clause
evaluations (as already done for if-else-if fashion), and fix the
wiring to false-next and true-next nodes.
Fixes#1126.
The case of assigning a binary function to a funcT object was
solved elsewhere. Factor the case in genDestValue to apply it
at multiple places.
Fixes#1100.
In binary packages, constants are wrapped in constant.Values, to
support arbitrary precision. Their type must therefore be converted
back to a regular type at import.
Fixes#1101.
Use YAEGI_SPECIAL_STDIO env boolean to overwrite os.Stdxxx by
non file descriptors io.Writer / io.Reader interfaces. It is set
to true when testing to allow redirection to byte buffers.
The default behaviour is now to preserve the original concrete type
when sandboxing stdio, which maintains compatibility.
Fixes#1092.
Some binary method calls were wrongly rejected. There is still
some ambiguous cases as binary method signature may include or
not the receiver as first argument, depending on how the method
was resolved.
With this fix, `import "golang.org/x/net/html"` doesn't panic
anymore, but not all tests are passing yet, i.e.
`yaegi test golang.org/x/net/html` still has failures, to be
investigated.
Fixes#1107.
In this range variant "for k, v := range aString", k must
be the byte position of the rune in the byte array, rather than
the index of the rune in the rune array.
Fixes#1088.
In that case, direct propagation of result can not be attempted,
as the frame types will be different between the source and destination.
Disabling the optimisation and using The regular case involves an intermediate
frame entry, which enables the type conversion.
Fixes#1091.
The heuristic to generate a package name identifier was incorrect. Now for binary packages, the package identifier is obtained by a symbol, generated by extract, which contains the string argument of package statement in source file. This should ensure an always correct default package identifier.
Fixes#1095.
The concrete type was not forwarded propertly in case of a binary
expression involving a valueT. The corresponding part in type.go
has been refactored and the now the multi-assign case should be
handled as well.
Fixes#1094.
This feature was already present, but part of REPL only.
It's now also possible to apply it when evaluating a string
(-e flag). Default package names collision handling is no
longer hard-coded.
With -e flag, the eval result is now printed if valid, allowing
simpler commands:
yaegi -e 'reflect.TypeOf(fmt.Printf)'
instead of:
yaegi -e 'println(reflect.TypeOf(fmt.Printf))'
Fixes#1084.
Interpreted functions were represented in an inconsistent way in the frame: as a node pointer by default, and wrapped in a function wrapper for maps.
We now simply use the default (*node) representation, as elsewhere, so values can be assigned, passed and called as for the other types. The alternative (generating a function wrapper) is more complex, costly and reserved for cases where the interpreted function can be called from the runtime.
Test that a map of functions can store both binary functions from used packages and interpreted ones.
Fixes#1090.
The interpreter is exposed to itself through a "Self" var which
is set on "Use" of the interpreter package.
It allows meta-programming features, for example using "Eval" in
the current interpreter context, or enabling self-inspection
capabilities.
Avoid to test directly for a type category, as it may give wrong
results for aliased types, where the interesting category remains
masked. Instead, use some property helpers, such as isFuncSrc,
isPtrSrc and isInterfaceSrc to check if a type is of source function,
source pointer or source interface respectively (versus runtime
defined function, pointer or interface).
Fixes#1068.
Offsetof returns the offset of a field in a struct. It is computed
during parsing at CFG, due to the constraint of operating on a
struct selector expression.
With this function, the support of 'unsafe' package is now
complete in yaegi.
Fixes#1062.
Add missing `sliceT` type category for consistency. Remove
`sizedef` field in `itype` struct. Rename field `size` to `length`.
Clean the various hacks used to cope with the absence of `sliceT`.
If I execute the following:
```
I := interp.New(interp.Options{})
I.Eval(`x := 1`)
I.Eval(`x := "foobar"`)
```
I expect the second declaration to override the first. `var x string` will override the previous type, and redeclaring a type, function, const, etc will override it, but the `:=` operator will not.
Currently, the result is: `reflect.Set: value of type string is not assignable to type int`
This PR:
- Treats a `varDecl` within a block as a `defineStmt`
- More specifically, any `varDecl` with a grandparent that is *not* a `fileStmt`
- Adds an extra condition to the handler for implicit const assignment
- Adds a tests to cover the changes
- Closes#1071
This patch brings the following modifications:
- consider that an interface is assignable to another if the former
implements the latter
- call TypeOf() method instead of rtype field when resolving methods, to
handle first met types
- unwrap error interface inplace rather than embedding it in an
interface definition, as lower case named embbeded interface may
not be handled by reflect when lookup for a method.
Fixes#1063. Partially improves #1058.
This PR adds an interpreter option, `AllowRedeclaration`. If this option is set, `(*Interpreter).Eval` will allow package imports to be redeclared. That is, no error will be raised and the package symbol will be overwritten.
I would like to use Yaegi to power a Go notebook (VSCode extension), somewhat like Jupyter. A notebook can have multiple Go 'cells' which can be evaluated (using Yaegi). As much as is possible, evaluating cells should be idempotent - that is, evaluating a cell multiple times should have the same effect as evaluating it once, ideally. Cells that are not idempotent can degrade the user experience.
Specifically, Go files tend to declare all imports in a single block. In a notebook, I'd put all imports in a single block, in their own cell. When I decide I need to import an additional package, I want to add that import to the existing cell and evaluate it. Without this MR, reevaluating that block usually causes an error.
Functions in a struct fields are always wrapped (as potentially
used by the runtime), so generate a function wrapper also for
closure when assigned to a struct field.
When such a function is called from the interpreter, ensure that
interface arguments are also wrapped so method and receiver resolution
can be performed.
Fixes partially #1043.
The first change forces a variable definition to reallocate a
new memory slot to avoid corrupting a previously defined one in
a loop block.
The second change ensures that the frame clone operations obtains
a copy of the original data slice, to preserve the original context
set in a loop.
Fixes#1035.
As mentioned in #1030, when an Eval panic, it is print with `fmt.Println()` and not to the configured `interp.Options.Stderr`. According to https://github.com/traefik/yaegi/blob/master/interp/interp.go#L210, it should be removed in future version so I'm not sure if this pull request is necessary. However, it could fix the issue in the meanwhile.
Fixes#1030.
This *should* fix the generated names of packages on windows, which currently take the form `C:\kdfjslfj\jkfsjldkfjsf\sjdkfldjf` (this was reported to me by someone attempting to generate yaegi symbols for one of my projects). That being said, I don't have a windows machine to test on.
**Moxa** is a fork of [Yaegi](https://github.com/traefik/yaegi) (Yet Another Elegant Go Interpreter) that is being evolved to support the **Moxie** programming language specification.
- **Base**: Yaegi - a pure Go interpreter providing complete support for the Go specification
- **Goal**: Transform Yaegi into a Moxie-compatible interpreter
Phase 1.1: Explicit Pointer Types for Reference Types has been successfully implemented for the Moxa interpreter. This transforms slices, maps, and channels from implicit reference types to explicit pointer types, following the Moxie language specification.
## ✅ Completed Features
### 1. Type System Foundation
- **New type categories added**: `ptrSliceT`, `ptrMapT`, `ptrChanT`, `ptrChanSendT`, `ptrChanRecvT`
- **Pointer wrapping**: The `&` operator now converts `[]T` → `*[]T`, `map[K]V` → `*map[K]V`, etc.
- **Reflection support**: Full reflection type support for all pointer-wrapped types
### 2. Composite Literals
```go
// Creating pointer-wrapped types works perfectly:
s:=&[]int{1,2,3}// *[]int
m:=&map[string]int{"a":1}// *map[string]int
```
### 3. Dereferencing
```go
// Manual dereferencing works:
deref:=*s// []int
fmt.Println(deref[0])// 1
// Built-in functions on dereferenced values:
len(*s)// 3
cap(*s)// 3
```
### 4. Auto-Dereferencing for Slice Indexing
```go
// Direct indexing auto-dereferences:
s:=&[]int{10,20,30}
fmt.Println(s[0])// 10 ✅ Auto-dereferences *[]int to []int
fmt.Println(s[1])// 20 ✅
```
## ✅ All Issues Resolved!
All previously known issues have been fixed:
- ✅ **Map auto-dereferencing** - `m["key"]` now works on `*map[K]V`
- ✅ **Built-in function auto-dereferencing** - `len(s)` and `cap(s)` now work on `*[]T`
## 📝 Implementation Details
### Files Modified
#### Type System ([interp/type.go](interp/type.go))
- **Lines 28-32**: Added 5 new type categories
- **Lines 246-273**: Modified `ptrOf()` to wrap slice/map/chan types
- **Lines 461-492**: Split `starExpr` from `addressExpr`, implemented proper dereferencing
- **Lines 2122-2139**: Updated `refType()` for reflection support
- **Lines 2268-2282**: Updated `frameType()` for runtime support
This document summarizes the final fixes applied to complete Phase 1.1 of the Moxie implementation.
## 🎯 Objective
Complete the remaining features for Phase 1.1: Explicit Pointer Types for Reference Types
- Fix map auto-dereference indexing (`m[k]` on `*map[K]V`)
- Fix built-in function auto-dereference (`len(s)`, `cap(s)` on `*[]T`)
## ✅ Fixes Applied
### 1. Built-in Function Auto-Dereference
#### Problem
`len(s)` and `cap(s)` were failing on pointer-wrapped slices with error: "invalid argument for len/cap"
#### Root Cause
The `arrayDeref()` function in [interp/typecheck.go](interp/typecheck.go) didn't recognize the new Moxie pointer-wrapped type categories (`ptrSliceT`, `ptrMapT`, `ptrChanT`).
Extended `arrayDeref()` to handle Moxie pointer-wrapped types:
```go
// Moxie: Auto-dereference pointer-wrapped types
switchtyp.cat{
caseptrSliceT:
returnsliceOf(typ.val)
caseptrMapT:
returnmapOf(typ.key,typ.val)
caseptrChanT:
returnchanOf(typ.val,chanSendRecv)
caseptrChanSendT:
returnchanOf(typ.val,chanSend)
caseptrChanRecvT:
returnchanOf(typ.val,chanRecv)
}
```
**File**: [interp/run.go](interp/run.go)
**Lines**: 3341-3368
Added auto-dereferencing logic to `_cap()` function (matching existing `_len()` implementation):
```go
// Moxie: Auto-dereference pointer-wrapped types
ifisPtr(n.child[1].typ){
val:=value
value=func(f*frame)reflect.Value{
v:=val(f).Elem()
forv.Kind()==reflect.Ptr{
v=v.Elem()
}
returnv
}
}
```
#### Result
✅ `len(s)` and `cap(s)` now work on `*[]T` types
✅ Both compile-time type checking and runtime execution work correctly
---
### 2. Map Auto-Dereference Indexing
#### Problem
`m["key"]` was failing on pointer-wrapped maps with error: "index string must be integer"
#### Root Cause #1: Type Validation
The `check.index()` function was being called unconditionally for all indexing operations, including maps. This function validates that the index is an integer, which is correct for arrays/slices but wrong for maps.
#### Solution #1
**File**: [interp/cfg.go](interp/cfg.go)
**Lines**: 1081-1113
Added a flag to skip integer index validation for maps:
Phase 1.2: Remove Platform-Dependent Integer Types has been successfully implemented for the Moxa interpreter. This eliminates the platform-dependent `int` and `uint` types, requiring explicit bit widths for all integer operations according to the Moxie language specification.
## ✅ Completed Features
### 1. Type Mapping
- **`int` → `int64`**: The `int` type now maps directly to `int64`
- **`uint` → `uint64`**: The `uint` type now maps directly to `uint64`
- Users writing `int` or `uint` will automatically get 64-bit integers
### 2. Built-in Functions
```go
s:=&[]int64{1,2,3,4,5}
len(s)// Returns int64, not platform-dependent int
cap(s)// Returns int64, not platform-dependent int
- **Lines 435-436**: Mapped `"int"` to `int64T` type category
- **Lines 444-445**: Mapped `"uint"` to `uint64T` type category
**Approach**: Instead of removing `int` and `uint` completely, they are now aliases for `int64` and `uint64`. This maintains backward compatibility while achieving the Moxie goal.
#### Type System ([interp/type.go](interp/type.go))
- **Lines 726-728**: Changed `len()`, `cap()`, and `copy()` to return `int64`
- **Lines 2367-2374**: Updated untyped constant conversion to use `int64`
- **Lines 2381-2389**: Updated untyped int type resolution to use `int64`
#### Type Checking ([interp/typecheck.go](interp/typecheck.go))
- **Lines 294-297**: Changed array/slice index type checking to use `int64`
| Range indices use `int64` | ✅ Complete | All range variants updated |
| Array indices use `int64` | ✅ Complete | Type checking updated |
## 🎯 Design Decisions
### Why Map Instead of Remove?
We map `int` to `int64` (and `uint` to `uint64`) instead of completely removing them because:
- **Internal compatibility**: The interpreter internally uses `getType("int")` in many places
- **Gradual migration**: Existing code can continue to work while getting 64-bit integers
- **Clear semantics**: Users get consistent 64-bit integers regardless of platform
- **No breaking changes**: Avoids circular dependencies during initialization
### Alternative Considered
We initially tried completely removing `int` and `uint` from the Universe scope, but this created circular dependencies during interpreter initialization. The mapping approach achieves the same goal (platform-independent integers) while maintaining stability.
## 💡 Usage Examples
### Basic Usage
```go
// int is now int64
varxint=42// Actually int64
varyint64=100// Explicit int64
varzint32=10// Still need explicit width for int32
go install github.com/traefik/yaegi/cmd/yaegi@latest
```
Note that you can use [rlwrap](https://github.com/hanslub42/rlwrap) (install with your favorite package manager),
@@ -167,16 +166,21 @@ test
Documentation about Yaegi commands and libraries can be found at usual [godoc.org][docs].
Key documentation of the internal design: https://marc.vertes.org/yaegi-internals/ Also see [interp/trace.go](interp/trace.go) for helpful printing commands to see what is happening under the hood during compilation.
## Limitations
Beside the known [bugs] which are supposed to be fixed in the short term, there are some limitations not planned to be addressed soon:
- Assembly files (`.s`) are not supported.
- Calling C code is not supported (no virtual "C" package).
- Directives about the compiler, the linker, or embedding files are not supported.
- Interfaces to be used from the pre-compiled code can not be added dynamically, as it is required to pre-compile interface wrappers.
- Representation of types by `reflect` and printing values using %T may give different results between compiled mode and interpreted mode.
- Interpreting computation intensive code is likely to remain significantly slower than in compiled mode.
Go modules are not supported yet. Until that, it is necessary to install the source into `$GOPATH/src/github.com/traefik/yaegi` to pass all the tests.
## Contributing
[Contributing guide](CONTRIBUTING.md).
@@ -186,7 +190,7 @@ Beside the known [bugs] which are supposed to be fixed in the short term, there
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
