Currently, yaegi uses the `filepath` package to join paths.
However, this has issues: YAEGI uses an `fs.FS` internally for source code access, and `fs` specifies that `/` separated paths must be used. On Unix systems, this obviously makes no difference; but on Windows, with a non-default FS, this causes errors (since YAEGI creates `\` separated paths, but the FS expects `/` separated paths).
Furthermore, generally speaking, Golang import paths and the like are always `/` separated anyway, regardless of OS.
To clear this up, this PR changes path handling to `/` separated paths. This has the advantage of being accepted by all OS (even Windows will accept `/` separated paths). Paths passed by the user to `CompilePath` or similar are changed to `/` separators, if necessary.
* chore: bump golangci-lint and fix config
This fixes the CI lint step. No change in code except a fix
in test output.
* bump golangci-lint from 1.56.2 to 1.63.4
* test: fix test check
When parsing binary operator expressions, make sure that implicit type conversions for untyped expressions are performed. It involves walking the sub-expression subtree at post-processing of binary expressions.
Fixes#1653.
This builds on #1644 and adds automatic per-loop variables that are consistent with go 1.22 behavior. See #1643 for discussion.
This is still a draft because the for7 version ends up capturing the per-loop var values when they are +1 relative to what they should be. Maybe somehow the incrementing and conditional code is somehow capturing the within loop variables and incrementing them? not sure how that would work. anyway, need to investigate further before this is ready to go, but pushing it here in case there are other issues or someone might figure out this bug before I do..
I found these basic functions and Stringer methods essential for tracking down issues fixed recently -- probably be of use to others and I couldn't find something else in the code that would provide similar utility, though I never tried the debugger.
This fixes issue #1634
includes special case for defer function.
I could remove or significantly reduce the comment description, and just have that here for future reference:
// per #1634, if v is already a func, then don't re-wrap! critically, the original wrapping
// clones the frame, whereas the one here (below) does _not_ clone the frame, so it doesn't
// generate the proper closure capture effects!
// this path is the same as genValueAsFunctionWrapper which is the path taken above if
// the value has an associated node, which happens when you do f := func() ..
Follow by the [Spec](https://go.dev/ref/spec#Assignment_statements):
The number of operands on the left hand side must match the number of values. For instance, if f is a function returning two values `x, y = f()` assigns the first value to x and the second to y.
In the second form, the number of operands on the left must equal the number of expressions on the right, each of which must be single-valued, and the nth expression on the right is assigned to the nth operand on the left.
Fixes#1606
A side effect of #1281 is that it added unnecessary additional newlines in generated interface wrappers from the `extract` tool. This PR removes those newlines from the extract tool template and updates the generated code with that change.
* feat: support go1.22
* Temporary fix for consistency tests due to language change in for loops
* review: clean old files
---------
Co-authored-by: Fernandez Ludovic <ldez@users.noreply.github.com>
* feat: generate go1.21 files
* chore: update CI
* feat: add support for generic symbols in standard library packages
This is necessary to fully support go1.21 and beyond, which now
provide some generic packages such as `cmp`, `maps` or `slices`
in the standard library.
The principle is to embed the generic symbols in source form (as
strings) so they can be instantiated as required during interpretation.
Extract() has been modified to skip the generic types, functions and
constraint interfaces which can't be represented as reflect.Values.
A new stdlib/generic package has been added to provide the corresponding
source files as embedded strings.
The `Use()` function has been changed to pre-parse generic symbols as
doing lazy parsing was causing cyclic dependencies issues at compiling.
This is something we may improve in the future.
A unit test using `cmp` has been added.
For now, there are still some issues with generic stdlib packages
inter-dependencies, for example `slices` importing `cmp`, or when
generic types or function signatures depends on pre-compiled types
in the same package, which we will support shortly.
* fixup
* fixup
* fixup
* fixup
* fixup
* fixup
* fixes for go1.20
* fix previous
* update unsafe2 for go1.21, skip faky tests
In go1.21, the reflect rtype definition has been move to internal/abi.
We follow this change for maintainability, even if there is no layout
change (the go1.20 unsafe2 is compatible with go1.21).
We have isolated a few problematic tests which are failing sometimes
in go1.21, but work in go1.20, and also in go1.22. Those tests are
skipped if in go1.21. A preliminary investigation can not confirm that
something is wrong in yaegi, and the problem disappears with go1.22.
* add new wrapper for go1.21 package testing/slogtest
* add missing wrapper for go/doc/comment
* add support for slices generic package
---------
Co-authored-by: Marc Vertes <mvertes@free.fr>
Adds `wasip1` to known OS list, introduced in golang/go#58141.
Without this change, `yaegi extract` may fail on Go 1.21 with the following message:
```
type-checking package "time" failed (<GOROOT>/src/time/zoneinfo_wasip1.go:8:5: platformZoneSources redeclared in this block)
```
Currently, yaegi only records source positions when writing panic trace to stderr.
This change adds function names to the panic output.
Unfortunately, yaegi walks the trace in reverse order, comparing to Go runtime. This can be improved in the future.
Avoid a spurious optimisation which forces a variable to be reused instead of redefined for assignment operation. This ensures that a variable defined in a loop is re-allocated, preserving the previous instance when used by a closure for example.
Fix#1594
Embedding files using `//go:embed` and `embed` packages can not be supported in yaegi, so it is better to not generate the wrapper to embed and have a graceful error in case of usage of `embed.FS` in the interpreter.
Also update README about unsuported directives.
Fixes#1557.
I implemented fs.FS. When the file does not exist, `xerrors.Wrap(fs.ErrNotExist, "")` will be returned. However, `os.IsNotExist` cannot handle this situation. I found the following comment:
```
// IsNotExist returns a boolean indicating whether the error is known to
// report that a file or directory does not exist. It is satisfied by
// ErrNotExist as well as some syscall errors.
//
// This function predates errors.Is. It only supports errors returned by
// the os package. New code should use errors.Is(err, fs.ErrNotExist).
```
Therefore, we should use `errors.Is(err, fs.ErrNotExist)` instead.
Unsafe functions such as `unsafe.Alignof`, `unsafe.Offsetof` and `unsafe.Sizeof` can be used for type declarations early on during compile, and as such, must be treated as builtins returning constants at compile time. It is still necessary to explicitely enable unsafe support in yaegi.
The support of `unsafe.Add` has also been added.
Fixes#1544.
Only structures with one embedded field can be marked anonymous, due to golang/go#15924. Also check only that the method is defined, do not verify its complete signature, as the receiver may or not be defined in the arguments of the method.
Fixes#1537.
The result of the expression giving the size of an array may be an `int` instead of `constant.Value` in case of an out of order declaration. Handle both cases.
Fixes#1536.
consider packages from git.sr.ht, the user namespace is prefixed with `~`
so running something like `yaegi extract git.sr.ht/~emersion/scfg`
was producing syntax errors with `~` in identifiers. and also `~` in filenames which worked but probably best not to have it there either
thanks!
In the case of a Go short definition (i.e. `a, b := f()`), the new defined variables must be (re-)created in order to preserve the previous value (if in a loop) which can be still in use in the context of a closure. This must not apply to redeclared variables which simply see their value reassigned.
The problem was both occuring in callBin() for runtime functions and assignFromCall() for functions created in the interpreter.
Fixes#1497.
hi!
this issue is sorta blocking for me so i thought i would try to fix it.
im still learning the codebase and understanding how yaegi works, but I thought I would attempt to add a test in the style of other tests as a start.
please let me know if there is anything i need to change / run, or if anyone knows perhaps a good place to start for tackling this issue
Fixes#1496
The logic of patching reflect struct representation has been fixed in presence of function fields and to better handle indirect recursivity (struct recursive through nested struct).
Fixes#1519.
This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `t.TempDir` function from the `testing` package to create temporary directory. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
```go
func TestFoo(t *testing.T) {
// before
tmpDir, err := os.MkdirTemp("", "")
if err != nil {
t.Fatalf("failed to create tmp directory: %v", err)
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
// now
tmpDir := t.TempDir()
}
```
This should allow to build the package on AlpineLinux. Also document the constraint of having to install the source under $GOPATH/src/github.com/traefik/yaegi until Go modules are supported.
Fixes#1523.
closes#1514
hi!
I had the same problem as #1514 and I wanted to fix it, I found
When asserting *crypto/rsa.PublicKey, using the typ attribute of node to get an nil rtype, resulting in the assertion result being nok
This code contains the same problem
```go
package main
import (
"log"
"crypto/rsa"
)
func main() {
var pKey interface{} = &rsa.PublicKey{}
if _, ok := pKey.(*rsa.PublicKey); ok {
log.Println("ok")
} else {
log.Println("nok")
}
}
```
So I submitted this Pull Request, hope it will be merged
With certain yaegi configuration on Windows, the loop in `previousRoot` can be infinite, because it fails to recognize driver letters as root.
This change adds a few more safeguards: checks path prefix earlier and checks if `filepath.Dir` produces an empty path.
For methods defined on interfaces (vs concrete methods), the resolution of the method is necessarily delayed at the run time and can not be completed at compile time.
The selectorExpr processing has been changed to correctly identify calls on interface methods which were confused as fields rather than methods (due to the fact that in a interface definition, methods are fields of the interface).
Then at runtime, method lookup has been fixed to correctly recurse in nested valueInterface wrappers and to find embedded interface fields in case of struct or pointer to struct.
Finally, remove receiver processing in `call()`.The receiver is already processed at method resolution and in genFunctionWrapper. Removing redundant processing in call fixes handling of variadic method, simplifies the code and makes it faster.
With those fixes, it is now possible to load and run `go.uber.org/zap` in yaegi. In turn, it makes possible for yaegi to run plugins dependent on zap, such as coraza-waf.
Fixes#1515,
Fixes#1172,
Fixes#1275,
Fixes#1485.
It allows to use interpreter functions as parameters in the runtime, even for defered callbacks, or when passed as empty interfaces, as for runtime.SetFinalizer.
Fixes#1503
When generating a new type, the parameter type was not correctly duplicated in the new AST. This is fixed by making copyNode recursive if needed. The out of order processing of generic types has also been fixed.
Fixes#1488
Currently if interpreter tries to eval an empty directory with no Go files it will error (as it should), but all future calls to eval will deadlock, because the mutex is not unlocked correctly. I believe this is a critical issue that must be addressed.
For a long time, there was a confusion between aliased types and named types (due to my misunderstanding of alias types in Go at that time). The type category `aliasT` has been renamed into `linkedT`, which is correct semantically.
Aliased types are only declared using `typeSpecAssign`, and its processing is now distinct from `typeSpec` statement, used for named types.
A `linkedT` type is obtained by a statement like `type A uint32`, where `A` type category is therefore `linkedT`.
An aliased type is obtained by a statement like `type A = uint32` (notice the `=` sign, translating into `typeSpecAssign`).
The semantic difference is that in the first linkedT form, `A` and `uint32` are 2 distinct types, instead of being strictly equivalent in the `typeSpecAssign` form (the 2 names lead to one type definition).
Fixes#1416.
For example, on architecture GOARCH=amd64, a file named `foobar_amd64.go` would be skipped instead of being read and parsed. The function `skipFile` is fixed and missing tests are added.
In case of forward definition of a constant, a symbol may be undefined when attempting to compute the array size in type analysis. Just mark the type as incomplete instead of aborting directly, to allow resolution at a second pass.
Fixes#1470.
Up to now functions could be stored as node values in frame (as for interpreter defined functions) or function values, directly callable by the Go runtime. We now always store functions in the later form, making the processing of functions, anonymous closures and methods simpler and more robust. All functions, once compiled are always directly callable, with no further wrapping necessary.
Fixes#1459.
Although empty interfaces are usually not wrapped, for compatibility with the runtime, we may have to wrap them sometime into `valueInterface` type.
It allows to preserve interpreter type metadata for interface values exchanged with the runtime. It is necessary to resolve methods and receivers in the absence of reflect support.
During type assertions on empty interfaces, we now handle a possible valueInterface and dereference the original value to pursue the type assertion.
In the same change, we have improved the format of some panic messages at runtime to give location of offending source at interpreter level.
This change will allow to fixtraefik/traefik#9362.
Those declarations involve the indexListExpr AST token, which was not handled in type.go. The same processing as for a single type parameter is applied.
Fixes#1460.
In type assertion at compile time, compare signatures between function types only.
Make `itype.numOut()` return the correct value for Go builtins (this was not strictly necessary due to above fix, but it is correct and improves maintainability).
Fixes#1454.
In [short variable declarations](https://go.dev/ref/spec#Short_variable_declarations),
The reuse of existing symbols is possible only if a new variable is defined,
otherwise a new symbol must be created, which was not the case in the issue.
Search for new symbols and correctly ignore blank variables.
Fixes#1434.
The representation of non empty interfaces defined in the interpreter is now identical between refType() and frameType() functions, which are used to generate interpreter objects.
Fixes#1447 and #1426.
If the value on which to type-switch was already set (i.e. a variable),
there was no problem. But if it had to be obtained through a complex
expression (func call, array index, etc...), then the code to retrieve
the value prior type-switch was not scheduled. This is now fixed.
This issue is nasty because the behavior is silently changed,
leading potentially to further unrelated issues or runtime panics.
Fixes#1444.
This change implements a workaround to better support composed
interfaces in yaegi and let the interpreter define objects which
implement multiple interfaces at once.
We use the existing MapTypes to store what possible composed interface
wrapper could be used for some interfaces. When generating an interface
wrapper, the wrapper with the highest number of implemented methods is
chosen.
This is still an imperfect solution but it improves the accuracy of
interpreter in some critical cases.
This workaround could be removed in future if/when golang/go#15924
is resolved.
Fixes#1425.
The function `getMethodByName()` is now able to look for
embedded `valueInterface` field for matching methods in interface
struct fields.
Fixes#1439 and #1427.
The variable dependency check function was confused by a dependency
variable with the same name but in an external package.
This change is necessary to address #1427.
Status:
* [x] parsing code with generics
* [x] instantiate generics from concrete types
* [x] automatic type inference
* [x] support of generic recursive types
* [x] support of generic methods
* [x] support of generic receivers in methods
* [x] support of multiple type parameters
* [x] support of generic constraints
* [x] tests (see _test/gen*.go)
Fixes#1363.
Such function declaration denotes either a linkname (an access to
an arbitrary, typically unexported symbol, solved by go compiler),
or a foreign C or assembly implementation of the body.
Those cases are not supported (or planned to be) by the interpreter.
Fixes#1431.
In addition:
- extract commmand now skips exported generics in runtime wrappers
- interp_consistent_test.go fixed for go1.18 and go1.19
- move minimal version of go compiler to go1.18
Note that this version is incompatible with go1.17 and before due
to the major changes in the stdlib go parser.
To be merged once go1.19 is officially released (not before).
Some tests are not passing when using go1.18, due to a change of
content in error messages compared to go1.17. We simply skip them
while we support go1.17. It concerns a small number of tests
regarding error detection.
I expect the following code to be supported.
```go
type TT http.Header
func (t TT) Set(key, val string) {
}
func (t TT) Get(key string) string {
}
```
So, I pushed this PR.
Do I need to add some test cases? I don't see the relevant test files ....
The type.val field was always pointing to the final underlying type
for aliased types, defeating a possible match if a method was
attached to a type in between. Now the complete chain of aliases
is always preserved.
We have added an underlying() itype method which returns the underlying
type of a defined type (aliasT), even in the presence of multiple
indirections.
We have added a definedType function which checks if type t1 is
defined from type t2 or t2 defined from t1, required when checking
assignability of aliasT types.
Fixes#1411.
PS: this is the 2nd attempt, as the first version #1412 wasn't passing
_test/issue-1408.go as well. This PR does pass and supersedes #1412.
We finally address a long standing limitation of the interpreter:
the capacity to generate the correct interface wrapper for an
anonymous interface{} function parameter of a binary function.
It allows for example fmt.Printf to invoke the String method
of an object defined within the interpreter, or json.Marshal
to invoke a textMarshaler method if it exists and if there is
no Marshaler method already defined for the passed interpreter
object.
To achieve that, we add a new mapType part of the "Used" symbols
to describe what not empty interfaces are expected and in which
priority order. This information can not be guessed and is found
in the related package documentation, then captured in stdlib/maptypes.go.
Then, at compile time and/or during execution, a lookup on mapTypes
is performed to allow the correct wrapper to be generated.
This change adds a new MapType type to the stdlib package.
Fixes#435.
In that case, the interface must be wrapped in an valueInterface
at creation.
With that fix, it is now possible to import
github.com/google/go-querystring/query. Not tested beyond that.
Fixes#1123.
In REPL mode, a panic (stack overflow) could be triggered by:
$ yaegi
> a:
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x14020760330 stack=[0x14020760000, 0x14040760000]
fatal error: stack overflow
[...]
This issue occurs in incremental parsing mode only, and not when the parser
is in file mode. We avoid it by being more defensive when generating
values.
Fixes#982.
At variable, function parameter, slice, map or field element assign,
if the destination type is an empty interface, the value was never
wrapped into a valueInterface (to preserve type mutability in case
of re-assign). Now we wrap it in a valueInterface if the source
type has a non empty set of methods, to allow a future use as a non
empty interface.
There are still corner cases, but it extends notably the support
of interfaces within the interpreter.
Fixes#1355.
The logic of goto was false due to mixing break label and goto
label, despite being opposite. In case of break, jump to node (the
exit point) instead of node.start. Also always define label symbols
before their use is parsed.
* Fix continue label, detect invalid labels for break and continue
* fix multiple goto, break, continue to the same label
Fixes#1354.
* interp: fix creation of binary composite types
Use direct assignment instead of reflect.Value Set method to
initialize a binary composite type, as for non binary types.
It ensures that a new reflect.Value is stored in the frame
instead of modifying a possibly existing one, which can defeat
the purpose of initializing variables in a body loop.
While there, remove the need to have and use a mutex on types.
Fixes#1381.
* review: rework a bit the test
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
We now detect the use of special identifier _ (blank) during parsing in order to abort compiling early. It allows to not panic later during execution. We must catch all the cases where blank is used as a value, but still preserve the cases where it is assigned, used as a struct field or for import side effects.
Fixes#1386.
When a package is imported, it creates a symbol with a name like "errors/_.go". If a statement such as x := errors.New(...) is executed before import "errors", it creates an incomplete type symbol with a name like "errors". Importing the package after the incomplete type symbol has been created does not fix the compile issue because the incomplete type still exists.
To fix this, this PR deletes the incomplete type symbol, if one exists.
Closes#1388.
The interpreter has its own internal fileset and expects all code to have been parsed using that fileset. If a user creates a fileset, calls `go/parser.Parse*`, then passes the result to `interp.CompileAST`, strange things can happen.
The solutions I can see are:
1. Expose the fileset so the user can use it when parsing source.
2. Add the fileset as an option (input to New) so that the user can tell the interpreter to use a specific fileset.
3. Pass the fileset into `CompileAST`
There are two ways to implement option 3. One is to add a field to nodes and update every reference to `interp.fset` to use `node.fset`. The other is to add a parameter to every function that references `interp.fset` or calls a function that does. Both of these are significant changes and involve an extra pointer for every node or most function calls.
Options 1 and 2 are easy. Option 2 involves adding an option so I went with option 1. I can imagine situations where option 2 could be necessary, but I can open another issue/PR if and when I need that.
Fixes#1383
* interp: handle struct with multiple recursive fields
In case of a recursive struct with several recursive fields of
different type, only the first one was properly fixed when
constructing the corresponding reflect type. We now memorize and
process all fields at the same depth level.
Fixes#1371.
* Update interp/type.go
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* fix lint
* fix comment
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* interp: fix handling of redeclaration in multi-assign expression
In a statement like `a, b := f()` if `a` was previously declared,
its symbol must be reused, a new symbol must not override its
previous value. This is now fixed.
* In case of redeclaration, reuse the existing only if the redeclared
variable has the same type. Add _test/var16.go to check this use
case.
Fixes#1365.
The capability to dereference pointers has been added to
methodByName(), improving method lookup on binary values.
Wrapping to valueInterface is performed in a missing use case at
return of function calls. It was done in the nested call, but not
at assign.
Fixes#1330.
A runtime builtin assignFromCall is added to handle multiple values returned at once. It is necessary if some of the values require to be set to interface values in the caller space, which is performed by reflect.Set in assignFromCall.
Fixes#1311.
In restricted mode, replace environment related symbols in
stdlib os package by a version which operates on a private copy
per interpreter context.
It allows to have concurrent interpreters in the same process
operating each in their own environment without affecting each
other or the host.
If unrestricted opt is set, this behaviour is disabled, and the
default symbols from stdlib are used.
Note also that no modification is done for syscall package, as it
should be not used in restricted mode.
At generation of operator closures, the comparison expression
was hard-coded instead of being derived from the operator name,
leading to a wrong result.
Fixes#1297.
In some cases, we use many yaegi instances in one process, each interpreter will expect its own os.Args. This is done in interp.go as an option in interp.Options besides Stdin Stdout and Stderr.
Add a tag option in the extract program to optionally give an opportunity to tailor some packages in for example the stdlib, for people who is interested in keeping the final binary size as smaller as possible.
Below go generate
```
//go:generate ../cmd/extract/extract -name stdlib -tag stdmime mime mime/multipart mime/quotedprintable
```
produces a header to stdlib/mime-multipart.go
```
// Code generated by 'yaegi extract mime/multipart'. DO NOT EDIT.
// +build go1.16,!go1.17,stdmime
```
When an interpreter type implementing an interface is
used by the runtime, the runtime can extract its type
and create new values using reflect, and call methods
on it. The problem is that there will be no interpreted
method counterpart in this case, which makes wrapper panic.
Allow the String() method wrapper to always succeed and
return an empty string if no interpreted method is present.
This allows scripts to define custom flag.Value types on
which the runtime internally instantiates values using
reflect (see isZeroValue in Go src/flag/flag.go).
This workaround could be generalized to all wrappers if
necessary. At this moment, it is convenient to keep the
default behavior of expecting instantiated interpreter
methods, in order to catch interpreter bugs related to
interfaces.
Fixes#1276.
This comes from experiments looking into #1259 where incomplete twice seen types are marked as complete. To mitigate the problem instead of a map of seen types in `nodeType` a slice is used as a cheap way to keep track of our current path through the node tree.
This change fixes a regression introduced by PR #1192 in a program using
https://github.com/NYTimes/gziphandler which defines several types
implementing stdlib interfaces. We do not implement a wrapper
if we see that a type already implements an interface, except that
it can be falsly reported by reflect in case of a struct with
embedded interface field. We need to force the wrapper generation
in this case.
The problem occurs only for wrappers on struct, not on pointers or
other indirection types.
Adds CompileAST, which can be used to compile Go AST directly. This
allows users to delegate parsing of source to their own code instead of
relying on the interpreter.
CLoses#1251
When a define statement relies on a selector or type that may exist in another file it should revisit once GTA is complete. This allows that revisit.
**Note:** In order to keep the original GTA error for the define statement, so the error received is correct and meaningful, I have added a node `meta` property. I decided to make it generic as it may be useful in future. There may be a better way to stash errors across the GTA runs and am open to suggestion here.
Fixes#1253
The current `nodeType` selector precedence is heavy handed in favour of package type. It seems to often create `typeSym` symbols as variable types in the scope will never be found. To fix this if the ancestor node is a field expression, the package type is searched for. After this, if the type is still `nil` the normal scope is searched using `nodeType2`.
Fixes#1158
Make sure to keep always a single copy of incomplete type structures.
Remove remnants of recursive types processing.
Now `import "go.uber.org/zap"` works again (see #1172), fixing regressions
introduced since #1236.
When `nodeType` recovers names and methods, it can overwrite the data if the type is an aliasT. When aliasing a type, do not recover the methods, this will be done in the GTA typeSpec pass.
Related to #1158
When a const is late binding and specified with a type, the GTA defineStmt was creating the symbol with the current scopes `iota` which is incorrect. The symbol should be created with the source nodes `rval`.
Related to #1158
This test (assert2.go) display 2 separate issues:
1. assert2.go L28: Type assert tries to set an `interface{}` to a `valueInterface`. The typing here is complex, we have a valueT(strings.Builder) wrapped in a ptrT wrapped in a src iface wrapped in a valueT(interface{}). Type assert fails to realise that the `valueT` `interface{}` is wrapping the `valueInterface`.
2. assert2.go L29: `genValueBinMethodOnInterface` does not try and get the bin method, as the `typ.node` (`ptrT` or a `valueT`(`string.Builder`)) is set. In this case the src iface is called with a receiver argument. To fix this the method is looked for first if possible, and only if not found does it fall back to the `defaultGen`.
Fixes#1227
It was initially assumed that `nodeType` needed to rebuild the type from scratch, but this is not the case anymore. The existing type constructors are now used in `nodeType` to make it more readable. In doing this some bugs in type strings were found and fixed, along with adding the real package name to the type.
As `ptrOf` is now the only place that pointer types are constructed, it is feasible to cache the pointer type on the value type. To ensures that we have a consistent pointer type for any value type.
As the unsafe and pointer methods in `reflect` are to be depreciated, and seeing no replacement functions, it is now forced that some unsafe is needed to replace this as when and interface is dereferenced it is made unsettable by reflect.
With this in mind, this adds real recursive types by hot swapping the struct field type on the fly. This removes a lot of compensation code, simplifying all previous cases.
**Note:** While the struct field type is swapped for the real type, the type string is not changed. Due to this, unsafe will recreate the same type.
This adds `itype.str` which is a string representation of the type built when the type is built. The goal is to make type comparison simpler and centralise the creation of types just to constructors and `nodeType`. `nodeType` continues to build types in parts so to reuse underlying types better.
Make use of fs.FS (new to go 1.16) to allow for reading source files from diverse filesystems (local, embed, custom).
* `Options` has a new field `SourcecodeFilesystem fs.FS` so users can supply their own read-only filesystem containing source code.
* Defaults to the local filesystems (via `RealFS` - a thin `os.Open` wrapper complying with `fs.FS`) so regular users should see no change in behaviour.
* When no filesystem is available (e.g. WASM, or if you want to embed files to retain single binary distribution) an alternative filesystem is preferable to using `Eval(string)` as that requires the stringy code to be a single file monolith instead of multiple files. By using an `fs.FS` we can use `EvalPath()` and gain the ability to handle multiple files and packages.
* You can make use of embed filesystems (https://pkg.go.dev/embed) and custom filesystems obeying the `fs.FS` interface (I use one for http served zip files when targeting wasm as there is no local filesystem on wasm). Tests can make use of `fstest.Map`.
* NOTE: This does NOT affect what the running yaegi code considers its local filesystem, this is only for the interpreter finding the source code.
See `example/fs/fs_test.go` for an example.
Fixes#1200.
Set node address in `val` field at creation of `funcDecl` node so it can be used correctly at closure generation, even in the case of forward function declarations, where the value was zero.
Fixes#1214
When calling a function, if the input param is a "zero" instance, it is not set on the input. This is an issue where the param is an `interface{}` as a `nil` value is set instead of the zero value.
The actual solution for this is to remove the `if !val.IsZero()`, this however runs into an issue on `_test/struct48.go` where we have a zero recursive struct instance (`*interface{}`) and we have no way to get its real type. Should a way be figured out to keep tabs on the original type, the `if` can go away, and in the zero case of `genValueRecursiveInterfacePtrValue`, the actual zero type can be returned.
Fixes#1215
Add getConcreteType to retrieve the concrete type of a nested interface
value implementing a specific interface for which a wrapper exists.
If method resolution fails at runtime, a panic is now issued instead
of an error message and continue.
Fixes#1187.
Map handling builtins getIndexMap and rangeMap had some leftover
code of previous way of emulating interfaces, which was modified
following changes in #1017.
Specific code for interfaceT is removed, as not necessary anymore.
Map builtins are now simplified and more robust.
Fixes#1189.
When operating on map elements, result of assign operators were not
written to the map entry. Now check if the destination of an assign
operator is in a map, and if so, set the result to it.
Fixes#1194.
Add early detection of cases where no wrapper is necessary because
the value type already implements the target interface.
It should both increase performances by avoiding the wrapper overhead,
and fix errors due to replacing valid values by incomplete wrappers,
caused by the presence of private methods in the interface definition,
as in #1191.
Fixes#1191.
Store the interpreter value of the interface object to wrap as
a field called IValue, at offset 0 in wrapper structures.
Update extract to include IValue field.
In typeAssert, detect interface wrapper, and dereference the
interpreter value from IValue wrapper field.
Fixes#1166.
In parsing array type declaration, The type check of array size was
restricted to `int`. Broaden the test to accept any valid integer
kind.
Fixes#1175.
The resolution method was not compliant with the Go specification which
requires to retain the object where the field or method is the most
shallowed.
The detection of ambiguous fields or methods (same depth in different
objects) has also been added.
Fixes#1163.
In selector resolution, struct field matching now precedes
method matching. Before struct field matching could be skipped
in case of a matching method, which is incorrect, as demontrated
by _test/issue-1156.go.
Field lookup has been fixed to operate on recursive structures.
Concrete type values are derived when filling a receiver for
interface methods.
LookupBinField has been fixed to skip non struct values.
LookupMethod has been fixed to iterate on interface values as
well as concrete type values.
Fixes#1156.
### Background
#1102 changed how `Interpreter.Use` interprets export paths such that the last path component is stripped and used as the package name. This resulted in #1139 - attempting to Use an export with only one path component, such as `foo`, would result in the import path being `.`.
### Breaking API Change
This PR changes the signature of `Interpreter.Use` from `Use(Exports)` to `Use(Exports) error`.
### Fix for #1139
With this PR, if Use is called with an incomplete export path, such as `foo`, Use will return an error.
Fixes#1151
If I add a package with `Use` and import it with `ImportUsed`, the package is added to the universe scope as `<pkg>`. If I import with `Eval`, the package is added as `<pkg>/_.go`. However, `(*node).isType` (in cfg.go) only checks for `<pkg>/_.go`. Thus, packages imported with `ImportUsed` can be inaccessible.
This MR updates `(*node).isType` to fall back to `<pkg>` if `<pkg>/_.go` does not exist.
In typecheck.go, detect binary methods so we know when to skip the receiver as first parameter when checking function signatures. The signature check is not yet performed, we just avoid a false error.
In cfg.go, take care to label types with isBinMethod field to true whenever a binary method is resolved.
Also, do not attempt to wrap node in functions if the node value is already a binary function.
Fixes#1145.
Fixes#1150
1. When resolving a selector expression involving an aliased type, resolve the aliased type
2. When building an array literal, resolve the aliased type
Aliases of named array and slice types were the only ones that didn't work, but I added the other test cases for the sake of completeness and through testing.
Fixes#1149
Because of how aliases are handled, `n.gen` is set to `getIndexSeqMethod` or `getIndexSeqPtrMethod` in cases like the one described in #1149. As a result, `FieldByIndex` can be called on a value that is not a struct, which causes a panic. This MR updates those two methods to avoid that call if the index array is empty.
This is a small change that allows use of composite array literals, such as:
```go
type Vec3 [3]float32
var foo = []Vec3{
{1, 0, 0},
{6, 0, 0},
{6, 2, 0},
{2, 2, 0},
{1, 1, 0},
}
```
In switch case expressions, the condition on case clause was
not always properly evaluated. Reverse the order of case clause
evaluations (as already done for if-else-if fashion), and fix the
wiring to false-next and true-next nodes.
Fixes#1126.
The case of assigning a binary function to a funcT object was
solved elsewhere. Factor the case in genDestValue to apply it
at multiple places.
Fixes#1100.
In binary packages, constants are wrapped in constant.Values, to
support arbitrary precision. Their type must therefore be converted
back to a regular type at import.
Fixes#1101.
Use YAEGI_SPECIAL_STDIO env boolean to overwrite os.Stdxxx by
non file descriptors io.Writer / io.Reader interfaces. It is set
to true when testing to allow redirection to byte buffers.
The default behaviour is now to preserve the original concrete type
when sandboxing stdio, which maintains compatibility.
Fixes#1092.
Some binary method calls were wrongly rejected. There is still
some ambiguous cases as binary method signature may include or
not the receiver as first argument, depending on how the method
was resolved.
With this fix, `import "golang.org/x/net/html"` doesn't panic
anymore, but not all tests are passing yet, i.e.
`yaegi test golang.org/x/net/html` still has failures, to be
investigated.
Fixes#1107.
In this range variant "for k, v := range aString", k must
be the byte position of the rune in the byte array, rather than
the index of the rune in the rune array.
Fixes#1088.
In that case, direct propagation of result can not be attempted,
as the frame types will be different between the source and destination.
Disabling the optimisation and using The regular case involves an intermediate
frame entry, which enables the type conversion.
Fixes#1091.
The heuristic to generate a package name identifier was incorrect. Now for binary packages, the package identifier is obtained by a symbol, generated by extract, which contains the string argument of package statement in source file. This should ensure an always correct default package identifier.
Fixes#1095.
The concrete type was not forwarded propertly in case of a binary
expression involving a valueT. The corresponding part in type.go
has been refactored and the now the multi-assign case should be
handled as well.
Fixes#1094.
This feature was already present, but part of REPL only.
It's now also possible to apply it when evaluating a string
(-e flag). Default package names collision handling is no
longer hard-coded.
With -e flag, the eval result is now printed if valid, allowing
simpler commands:
yaegi -e 'reflect.TypeOf(fmt.Printf)'
instead of:
yaegi -e 'println(reflect.TypeOf(fmt.Printf))'
Fixes#1084.
Interpreted functions were represented in an inconsistent way in the frame: as a node pointer by default, and wrapped in a function wrapper for maps.
We now simply use the default (*node) representation, as elsewhere, so values can be assigned, passed and called as for the other types. The alternative (generating a function wrapper) is more complex, costly and reserved for cases where the interpreted function can be called from the runtime.
Test that a map of functions can store both binary functions from used packages and interpreted ones.
Fixes#1090.
The interpreter is exposed to itself through a "Self" var which
is set on "Use" of the interpreter package.
It allows meta-programming features, for example using "Eval" in
the current interpreter context, or enabling self-inspection
capabilities.
Avoid to test directly for a type category, as it may give wrong
results for aliased types, where the interesting category remains
masked. Instead, use some property helpers, such as isFuncSrc,
isPtrSrc and isInterfaceSrc to check if a type is of source function,
source pointer or source interface respectively (versus runtime
defined function, pointer or interface).
Fixes#1068.
Offsetof returns the offset of a field in a struct. It is computed
during parsing at CFG, due to the constraint of operating on a
struct selector expression.
With this function, the support of 'unsafe' package is now
complete in yaegi.
Fixes#1062.
Add missing `sliceT` type category for consistency. Remove
`sizedef` field in `itype` struct. Rename field `size` to `length`.
Clean the various hacks used to cope with the absence of `sliceT`.
If I execute the following:
```
I := interp.New(interp.Options{})
I.Eval(`x := 1`)
I.Eval(`x := "foobar"`)
```
I expect the second declaration to override the first. `var x string` will override the previous type, and redeclaring a type, function, const, etc will override it, but the `:=` operator will not.
Currently, the result is: `reflect.Set: value of type string is not assignable to type int`
This PR:
- Treats a `varDecl` within a block as a `defineStmt`
- More specifically, any `varDecl` with a grandparent that is *not* a `fileStmt`
- Adds an extra condition to the handler for implicit const assignment
- Adds a tests to cover the changes
- Closes#1071
This patch brings the following modifications:
- consider that an interface is assignable to another if the former
implements the latter
- call TypeOf() method instead of rtype field when resolving methods, to
handle first met types
- unwrap error interface inplace rather than embedding it in an
interface definition, as lower case named embbeded interface may
not be handled by reflect when lookup for a method.
Fixes#1063. Partially improves #1058.
This PR adds an interpreter option, `AllowRedeclaration`. If this option is set, `(*Interpreter).Eval` will allow package imports to be redeclared. That is, no error will be raised and the package symbol will be overwritten.
I would like to use Yaegi to power a Go notebook (VSCode extension), somewhat like Jupyter. A notebook can have multiple Go 'cells' which can be evaluated (using Yaegi). As much as is possible, evaluating cells should be idempotent - that is, evaluating a cell multiple times should have the same effect as evaluating it once, ideally. Cells that are not idempotent can degrade the user experience.
Specifically, Go files tend to declare all imports in a single block. In a notebook, I'd put all imports in a single block, in their own cell. When I decide I need to import an additional package, I want to add that import to the existing cell and evaluate it. Without this MR, reevaluating that block usually causes an error.
Functions in a struct fields are always wrapped (as potentially
used by the runtime), so generate a function wrapper also for
closure when assigned to a struct field.
When such a function is called from the interpreter, ensure that
interface arguments are also wrapped so method and receiver resolution
can be performed.
Fixes partially #1043.
The first change forces a variable definition to reallocate a
new memory slot to avoid corrupting a previously defined one in
a loop block.
The second change ensures that the frame clone operations obtains
a copy of the original data slice, to preserve the original context
set in a loop.
Fixes#1035.
As mentioned in #1030, when an Eval panic, it is print with `fmt.Println()` and not to the configured `interp.Options.Stderr`. According to https://github.com/traefik/yaegi/blob/master/interp/interp.go#L210, it should be removed in future version so I'm not sure if this pull request is necessary. However, it could fix the issue in the meanwhile.
Fixes#1030.
This *should* fix the generated names of packages on windows, which currently take the form `C:\kdfjslfj\jkfsjldkfjsf\sjdkfldjf` (this was reported to me by someone attempting to generate yaegi symbols for one of my projects). That being said, I don't have a windows machine to test on.
This is a follow-up of #1017, generalizing the use of reflect.Set
method to set, and possibly overwrite, the concrete value of
interface objects all accross the implementation of operators.
Previous optimized implementation for non-interface objects is
preserved.
The empty interface (interface{}), and its variants (such as []interface{} and map[string]interface{}), are commonly used in Go to (json) Unmarshal arbitrary data. Within Yaegi, all interface types are wrapped in a valueInterface struct in order to retain all the information needed for a consistent internal state (as reflect is not enough to achieve that). However, this wrapping ends up being problematic when it comes to the type assertions related to the aforementioned Unmarshaling.
Therefore, this PR is an attempt to consider the empty interface (and its variants) as an exception within Yaegi, that should never be wrapped within a valueInterface, and to treat it similarly to the other basic Go types. The assumption is that the wrapping should not be needed, as there is no information about implemented methods to maintain.
Fixes#984Fixes#829Fixes#1015
When checking for untyped values, we can be sure at this stage that they must be a const value or already untyped. Checking for type string equality is no longer a good measure.
Fixes#1000
When passing a function reference as an interface in a composite binary map, the case should be handled to not take the value of the the node.
Related to #886
This is a follow-up of #1014, where an invalid constant definition involving a builtin is now checked at CFG. In addition, some missing arithmetic operators are now detected for assign optimization.
A channel can be used to interchange data with the pre-compiled
runtime and therefore objects impletementing interfaces must be
wrapped if necessary, using genInterfaceWrapper.
A similar treatment could be applied when sending interpreted
functions over a channel, to be provided in a new PR.
Fixes#1010.
Fix the logic to detect recursive struct types, which was giving a false positive.
We now use the local type name as key in tracker map.
A non-regression test case is included (_test/struct49.go).
This completes #1008.
If a struct contains several fields of the same temporary incomplete
type, it could be detected incorrectly as a recursive struct. Pass
a copy of defined types map to avoid this issue.
Fixes#1007.
Always attempt to obtain an integer constant value for operators
expecting so. It allows to use '/' in integer constant defintions,
instead of default big.Rat.
Fixes#1005
An undefined type detection function has been added to better diagnose
incomplete type definitions. Implicit type names in interface or struct
declarations are now better handled. The incomplete status is not
fowarded to aliased type declarations to handle circular definitions.
Fixes#999 and #995. Improves #260 (goes farther, but still fails).
In some cases, the global character of a value was lost, leading to
undefined behaviour. Now a node level field of -1 means that the value
is global, and that it should be accessed from the root data frame.
Fixes#993.
The interpreter interface type was replaced by a reflect.Value in
objects passed or return to function wrappers, losing the ability
to retrieve methods.
The valueInterface is now preserved, and correctly accessed if
wrapped multiple times.
Fixes#977.
typeAssertStatus deals with the 3rd form of type assertion ("_, ok"), for
when one does not care about the result of the assertion itself.
Some cases for it, which are already fixed for the two other forms of
type assertions, had not been fixed for this form yet.
Therefore, this change fixes such cases for this form, while integrating
typeAssertStatus to the same code path as for the two other forms.
The type check was generating false negatives. A correct test to check the
adressable status of an array is more complex to implement, and will
be done later.
Fixes#973.
The long-form (with comma-ok) ones were already fixed but the short-form
ones were not because they were in a completely different code path.
This PR also refactors the code so that both short-form and long-form
are now merged in the same function.
N.B: even though most (all?) cases seem to now be supported, one of them
still yields a result that does not satisfy reflect's Implements method
yet. It does not prevent the resulting assertion to be usable though.
N.B2: the code path for the third-form (_, ok) hasn't been fixed and/or
refactored yet.
Fixes#919
In aliased type declarations, when the target type was imported from
an external package rather than declared locally, the aliased type was
overwritten by target, loosing ability to lookup methods on the aliased
type. Aliasing on imported types is now properly detected and handled.
Fixes#971.
As opposed to other symbols, goto labels must be searched in included
scopes, not upper ones. Implement scope.lookdown to perform this,
to allow calls to goto to be embedded in included scopes where label
is defined.
Fixes#953.
Perform function declaration type check from the upper level scope (the scope where the
function is declared), to avoid possible collisions of local variables with package names.
Fixes#957.
in `callBin`, call arguments are converted to the corresponding
parameter type. In a case of an interface, the original concrete type
should be preserved instead, and only wrapped to an interface type for
internal interpreter types, as runtime values should already implement the
interface.
This change removes the interface wrapping when parameter is a runtime
value (valueT or ptrT to valueT).
This removes some overhead when handling runtime values, and keep a
similar behavior between interpreted and pre-compiled code. For
example, `io.Copy` preserves its internal optimisations when passed a
`bytes.Buffer`.
The function vUint, used to get the unsigned integer value of a value,
variable (frame) or constant, was broken for constant.Value expression.
Fixes#948.
When running GTA, the type `path` was set to `rpath`. This equates to the package path (`github.com/traefik/yaegi`) in most cases. In the vendored case the `rpath` is the sub package path `something/vendor/github.com/traefik/yaegi` causing issues in typecheck and likely further down the line. By using the `importPath` it makes this consistent.
**Note:** I have no clue how to test this decently. I am open to options here.
Potentially Fixes#916
Interpreter function types are represented internally by the AST node
of their definition. The conversion operation creates a new node with
the type field pointing to the target type.
Fixes#936.
A non-constant shift expression can be untyped, requiring to apply a
type from inherited context. This change insures that such context is
propagated during CFG pre-order walk, to be used if necessary.
Fixes#927.
The execution flow is such that a node can end up running several chained exec
funcs, some of which actually originate from other nodes. For example, in:
var m []int // L0
println("hello world") // L1
m[0] = 1 // L2
the offending code is on a node on line 2 (out of range error). However, since
the assignment to m is part of the execution flow of the variable m, we'll get
the panic when running all the chained exec funcs attached to the node for m on
line 0.
Which is why, when that happens, we need to look for the actual node (the one on
L2) where the offending instruction originates from, in order to
properly report the origin of the panic.
Fixes#546
The case of a constant arithmetic expression being of float kind because
of quotient was not handled correctly. Simplify constant extraction to
reflect.Value first, then conversion to target type, and let reflect Convert
method panic if conversion is not valid.
Fixes#920.
Type checking on shift operands was failing for untyped variable values.
Fix propagation of type in assignment. Optimize assignment of arithmetic
operations on variables by skipping the assign and writing directly to
destination frame value in the operator function.
Skip some slow tests when given -short test option.
Fixes#917.
Type check was failing for expression such as: `&(*tree)[node:][0]`, as in
```go
tree, _ := huffmanTreePool.Get().(*[]huffmanTree)
...
initHuffmanTree(&(*tree)[node:][0], histogram[l], -1, int16(l))
```
see c3da72aa01/brotli_bit_stream.go (L469)
The assignable check used to be too strict as it lacked the property that
if an untyped const can be represented as a T, then it is assignable to T.
And we can now use that fixed check to add a missing check: in a return
statement, we now make sure that any of the returned elements are
assignable to what the signature tells us they should be.
In unary constant operations, the test for unsigned was defeated by
testing for int first, which is true also for unsigned. Make sure that
testing for unsigned precedes testing for int.
Fixes#907.
This applies to -syscall, -unsafe and -unrestricted flags with the
corresponding env variables YAEGI_SYSCALL, YAEGI_UNSAFE and
YAEGI_UNRESTRICTED, already used in the same way for the run
sub-command.
Often enough when debugging, one does not know exactly what argument
should be given to Symbols, as it's not always clear in which
scope/namespace the symbol one is looking for is stored in.
Therefore, this change enables the Symbols method to now take the empty
string as an argument, which results in all the known symbols to be
returned (keyed by import path).
As a consequence, when an non-empty argument is given, the returned
result should be similar to what we had before, except it is now
returned as the sole entry of an encompassing map.
In addition, the "binary" symbols (i.e. the ones ingested through a
Use call), are now also taken into account.
There was several issues:
- access to field on pointer to struct from runtime: fix in
lookupBinField
- assign operation was skipped when performed in a comm clause
- the direction of comm clause was wrong if a channel send operation was
performed in a body of a receive comm clause
Fixes#884.
Check first for runtime defined array (typ.cat of valueT)
to avoid checking inexisting properties (an panic), when
deciding to use appendSlice or not.
Fixes#880.
Since the introduction of restricted stdlib and syscall symbols, the
capability of yaegi to interpret itself was broken.
The use of unrestricted symbols is now also controlled by environment
variables, to allow propagation accross nested interpreters.
The interpreter Panic symbol was not wrapped, this is fixed now.
the import path resolution was failing if the working directory was
outside of GOPATH.
The documentation and readme have been ajusted.
Fixes#890.
When the type is implicit, the first element in the list of children is
not the type of the composite literal, but it is one of the actual
children, so it should not be discarded.
Fixes#862
When working with an untyped const expression involving a division, if
the default type of the result should be an int (for example because the
default types of all the operands are ints as well), then we should make
sure that the operation that is applied is indeed an integer division,
and that the type of the result is not a float.
This is achieved by using the QUO_ASSIGN operator, instead of the QUO
operator.
This should fix several problems lurking around, and it notably fixes
one of the visible consequences, which is a systematic panic when using
the REPL as a "calculator".
This incidentally also allows us to revert what was done in
5dfc3b86dc since it now turns out it was
just a hack to fix one of the symptoms.
Fixes#864
As per https://golang.org/cmd/go/#hdr-Build_constraints,
using GOOS=android also matches tags and files for GOOS=linux,
so exclude it explicetly to avoid collisions.
Also using GOOS=illumos matches tags and files for GOOS=solaris.
Fixes#843.
This change fixes two data races related to composite literal creation.
The first one isn't controversial as it is just about initializing the
variable that contains the values in the right place, i.e. within the
n.exec, so that this variable is local to each potential goroutine,
instead of being racily shared by all goroutines.
The second one is more worrying, i.e. having to protect the node typ
with a mutex, because a call to func (t *itype) refType actually
modifies the itype itself, which means it is not concurrent safe.
The change seems to work, and does not seem to introduce regression, but
it is still a concern as it probably is a sign that more similar
guarding has to be done in several other places.
This change allows the interpreter to execute tests and benchmarks
functions provided by packages.
The test subcommand is similar to the "go test" command and
all the relevant flags have been kept.
The ability to evaluate a directory or a package has also been added.
A new method Symbol to access exported symbol values of an interpreted
package has been added. This method is used by the test subcommand.
An EvalTest method has been added to evaluate all Go files, including "*_test.go".
The testing packages from the standard library have been added to stdlib used
symbols.
Struct fields of type interface must be converted in wrapper
values to be reachable by the runtime. Call genInterfaceWrapper
on such values.
Fixes#832.
When searching for a binary method on structures, look up on embedded
fields first, otherwise the resulting index is incorrect, as
reflect.Type.MethodByName succeeds also on container struct.
Fixes#834.
This change fixes two distinct data races:
1) some global vars of type *itype of the interp package are actually
mutated during the lifecycle of an Interpreter. Even worse: if more than
one Interpreter instance are created and used at a given time, they are
actually racing each other for these global vars.
Therefore, this change replaces these global vars with generator
functions that create the needed type on the fly.
2) the symbols given as argument of Interpreter.Use were directly copied
as reference (since they're maps) when mapped inside an Interpreter
instance. Since the usual case is to give the symbols from the stdlib
package, it means when the interpreter mutates its own symbols in
fixStdio, it would actually mutate the corresponding global vars of the
stdlib package. Again, this is at least racy as soon as several
instances of an Intepreter are concurrently running.
This change fixes the race by making sure Interpreter.Use actually
copies the symbol values instead of copying the references.
The goal is to provide greater control of input, output and error
streams of the interpreter. It is now possible to specify those
as options when creating a new interpreter. The provided values
are propagated to relevant stdlib symbols (i.e fmt.Print, etc).
Care is taken to not update the global variables os.Stdout, os.Stdin
and os.Stderr, as to not interfere with the host process.
The REPL function is now simplified. The deprecated version is removed.
The tests are updated to take advantage of the simplified access
to the interpreter output and errors.
Fixes#752.
When running TestEvalScanner with -race=true, one can observe a data race such as:
```
WARNING: DATA RACE
Read at 0x0000029f3d68 by goroutine 52:
github.com/containous/yaegi/interp.(*itype).defaultType()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1466 +0x572
...
github.com/containous/yaegi/interp.(*Interpreter).EvalWithContext.func1()
/Users/mpl/src/github.com/containous/yaegi/interp/interp.go:501 +0xf0
Previous write at 0x0000029f3d68 by goroutine 43:
github.com/containous/yaegi/interp.(*itype).refType()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1419 +0x854
github.com/containous/yaegi/interp.(*itype).TypeOf()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1427 +0xa6
...
github.com/containous/yaegi/interp.(*Interpreter).EvalWithContext.func1()
/Users/mpl/src/github.com/containous/yaegi/interp/interp.go:501 +0xf0
```
Before this change, since closing the pipe to the REPL is done in a defer, it
means that all the i.REPL calls (and hence each goroutine for each of these
calls) are kept running and alive until the very end of the test. It should not
matter, since a new interpreter is created for each test case, and thus all the
i.REPL calls should be completely independent from each other.
And yet, by wrapping each test case in a function call, and thus making each
i.REPL call terminate as soon as the test case is over, the data race seems to
be fixed. This could suggest that the separate i.REPL calls from separate
interpreter instances are somehow sharing some memory, but I do not know how to
explain that.
The problem has yet to be fully understood, but at least this change restores
the test, without making the CI fail again.
The Go parser is able to detect some (but not all) (re-)decleration errors,
if the DeclarationErrors flag is enabled, which was not done so far.
This PR therefore enables that flag, which allows the removal of some of
the now unneeded code that was recently added to support redeclarations.
Fixes#811
In interactive mode, a line starting with the "func" keyword is usually
"wrapped", by prepending to it a "package main" statement, to make it
a valid piece of Go source code.
However, when the line is actually an anonymous function call, such as:
func() { println(3) }()
then this wrapping is not enough, as this is not valid Go in the global
context. Therefore, this kind of of expression must also be wrapped
inside a main func (as is the default case for most REPL inputs).
Since the detection and handling of such a case turned out to be quite
unelegant, this PR instead introduces a retrying phase when a parsing
error occurs for a particular class of cases. That is to say, when a
"func expression" wrapped in a main package fails to be parsed, it is
then wrapped in a main func before parsing is retried.
N.B. TestEvalScanner has been disabled for this change, because the additional test cases revealed a (most-likely already existing) data race.
Fixes#721
The input scanning is now performed in a sub goroutine and
the interrupt is listened in another goroutine, either to cancel Eval
or to cancel the current line scan.
Entering a '\n' after a 'Ctrl-C` to get the prompt is
not necessary anymore.
When determining the size of an array and a symbol is found, the symbol must be a const for the type to be valid.
While it makes sense for this check to be done in type checking, the type can be determined by GTA which would then fail. For now this check is done when getting the node type.
Fixes#825
The trick is that in reflect, the field is called Anonymous, which is
actually a different notion in the Go spec/vocable.
n.b. only works for non-recursive types for now.
Fixes#781
The recent changes that added some redeclaration checks implicitly added more
strictness related to namespaces and scopes which, among other things, broke
some uses that "accidentally" used to work.
For example, given
const script1 = `
import "fmt"
// more code
`
const script2 = `
import "fmt"
// some other code
`
If one Evals script1, then script2, with the same interpreter, without
specifying any scope, as the two fragments would be considered part of the same
(.go file) scope by default, a redeclaration error would be triggered because
import "fmt" is seen twice.
A work-around would have been to specify (a different) i.Name before each Eval
call, so that each script is considered as coming from a different .go file, and
hence are respectively in different scopes with respect to imports.
That lead us to realize we had to make specifying things such as file-scope, and
"incremental mode" (aka REPL), more obvious in the context of an Eval call.
In addition, we want to lay down the foundations for Yaegi being able to behave
more like the go tool wrt to various inputs, i.e. it should be able to take a
package directory, or an import path, as input, instead of just a .go file.
Hence the introduction of a new kind of Eval method (whose signature is not fixed yet):
func (interp *Interpreter) EvalPath(path string) (res reflect.Value, err error)
It partially solves the problem described above because:
1. the path given to EvalPath can be used as the file-scope hint mentioned
above, for now (even though the related implementation details might change).
2. Eval always runs in incremental mode, whereas EvalPath always runs in
non-incremental mode, hence clarifying the situation in that respect.
And to avoid confusion, the Name field of Interpreter is now non-exported,
since it is somewhat redundant with the path argument of EvalPath.
Note that #731 is not fully fixed (and might never be), as a requirement of the
proposed solution is to move the input bits of code into respective files
(instead of leaving them as strings).
Finally, some related bugfixes, documention changes, and some refactoring have
been included. Notably, there is no "empty scope" anymore, i.e. name defaults
to "_.go" when it is not specified.
Updates #731Fixes#778Fixes#798Fixes#789
Co-authored-by: Marc Vertes <mvertes@free.fr>
This adds type checking to SliceExpr as well as handling any required constant type conversion.
It should be noted that the check that `high` and `max` are not nil in a 3-index slice has been left out as this check is handled before type checking. Tests have been included for these cases.
In a couple of occurences, tests with enabled race detector exposed
some concurrent accesses to the cancelation callback used in select and
channel operations send and recv for EvalWithContext. This change ensure
that all accesses to this object are protected by mutex.
Fixes#815.
In the case of a pointer or alias composite literal expression, `compositeGenerator` changes the type to remove the pointer or alias. This causes a nested composite literal to have the wrong type.
Instead of changing the node type, the removal of the pointer or alias is moved to the runtime, allowing the node type to remain unchanged. This fixes potential issues in the type checking.
This adds type checking to CallExpr (excluding builtin type checking, as that is a PR in its own right) as well as handling any required constant type conversion.
This also changes constant strings and runes to be represented as `constant.Value`. Runes change `rval` type at CFG typing time to avoid having to type at AST time. There are also changes to importSpecs and `stdlib` to account for the string change. With this all `untyped` types should now be `constant.Value`s, although errors are still not returned if this is not the case to be sure we do not break things.
This also fixed a bug in `itype.methods` that would panic if the type was recursive.
Avoid goroutines leak, accumulation of defered functions and
spurious resets of signal handlers. Effectively catch interrupt
signal (Ctrl-C) to cancel current eval.
Fixes#713.
So far the REPL loop was treating any parsing error coming from
go/parser to generate the AST, as having occurred because the source
code was not yet complete (unfinished block). And it was therefore
ignoring all of them.
However, some of these errors are legitimate, and must be caught as soon
as they occur, otherwise the REPL cycle would stay in an errored state
forever (even when the block terminates), without the user getting any
feedback about it.
Therefore, this change adds an extra check when a parsing error occurs,
i.e. it verifies that it looks like an "EOF" error (unfinished block)
before it ignores it (as the user is supposed to terminate the block
eventually). Otherwise the error is treated just like a "non-parsing"
(cfg, gta, ...) error and printed out.
Fixes#637
This adds type checking to both `IndexExpr` and `CompositeLitExpr` as well as handling any required constant type conversion.
This includes a change to the type propagation to the children of a composite literal. Previously in most cases the composite literal type was propagated to its children. This does not work with type checking as the actual child type is needed.
Fix#787 changes how interfaces are set on a struct (compositeSparce). This change however makes calling the interface panic.
This PR reverts part of the change in #787 and adds a test to ensure it does not break again.
The Yaegi command line has been changed to provide subcommands.
The following sub-commands are provided:
- extract (formerly goexports)
- help
- run
- test
The previous behaviour is now implemented in run command which
is the default, so the change should be transparent.
In run command, prepare the ability to run a package or a directory
in addition to a file. Not implemented yet
The test command is not implemented yet.
The extract command is meant to generate wrappers to non stdlib
packages.
Fixes#639
The detection of special cases of interpreter functions and interfaces
is more precise. It allows at least to complete parsing of
code where type is derived from a type assertion expression.
Fixes#770.
* interp: support another vendoring case
Namely, when the vendor dir is a sibling (or an uncle) relative to the
current pkg
Fixes#758
* make linter happier
* address review comments
* fix, cleanup, add unit tests
* add dummy files to force dirs into git
The range init AST execution was skipped, and range could work
only over variables or direct function calls. By setting the
start node to the start of init and not init itself, we ensure
that the init AST is always taken into account.
Fixes#775.
* fix: constant definition loop on out of order vars
* fix: do not wire global varDecl
* fix: wire and execute global vars
* chore: add tests
* fix: refactor and lint
By returning early in case of incomplete type in CFG, we avoid
a panic, and let a chance to a new attempt after the missing
type has been parsed.
Fixes 763.
* feature: restrict symbols which can exit the interpreter process
Some symbols such as os.Exit or log.Fatal, which make the current process
to exit, are now restricted. They are replaced by a version which panics
instead of exiting, as panics are recovered by Eval.
The restricted os.FindProcess version is identical to the original
except it errors when trying to return the self process, in order to
forbid killing or signaling the interpreter process from script.
The os/exec symbols are available only through unrestricted package.
The original symbols are stored in an unrestricted package, which
requires an explicit Use, as for unsafe and syscall packages.
The Use() interpreter method has been slightly modified to allow inplace
updating of package symbols, allowing to replace some symbols but not
the entire imported package.
A command line option -unrestricted has been added to yaegi CLI to use
the unrestricted symbols.
Fixes#486.
* fix: lint
* fix: switch type from valueT in struct case
In a struct case in type assertion, if the source is a valueT, we still need to take the struct type to allow method and field resolution.
* fix: handle all ptr structs as well
In all situations where the results are set directly
to the frame, and not using a value helper, the right level of
indirections must be applied, otherwise we may end-up writing
in the wrong frame (the local one, instead of a caller or global).
Fixes#735.
* fix: copy the type in recursion
In more advanced recursive cases, setting the rtype to interface may interfear with typeing. To stop this from happening, instead of setting t.val.rtype to interface in the hope it will be set correctly later, a copy if the type is made, and the rtype of the copy is set to interface{}.
* fix: detect intermediate recursive structs
In the case of a nested recussion, each symbol can have
a different perspective on the recursion. In this case,
it is impossible to move from one struct to the next.
To keep the perspectives the same, any intermediate struct
that contains a recursion should also be set to interface{}.
so that all perspectives are the same.
* fix: handle arb recursion
* chore: refactor dref to be consistent
* fix: invalid recursive struct issue
* fix: handle checkptr issue
* fix: move unsafe into function to stop ptr check
* fix: handle deref in assign
When operations write their result to a non-local frame, the node
level field must be set accordingly, otherwise they attempt to write
in the wrong frame.
Fixes#730.
* fix: untyped constant cconverson to default type
In definition assign expression, the source type is propagated to
the assigned value. If the source is an untyped constant, the
destination type must be set to the default type of the constant
definition. A fixType function is provided to perform this.
In addition, the type conversion and check of constants is
refactored for simplifications.
Fixes#727.
* test: fix _test/const14.go
The standard library syscall package for Solaris defines unimplemented
symbols Syscall6 and RawSyscall6 which makes the build fails on
Solaris platform, now that yaegi command imports syscall symbols.
As the standard library package is locked down, this will remain
unchanged. We just skip those symbols.
Fixes#725.
2020-06-30 22:50:44 +02:00
1068 changed files with 67593 additions and 18416 deletions
**Moxa** is a fork of [Yaegi](https://github.com/traefik/yaegi) (Yet Another Elegant Go Interpreter) that is being evolved to support the **Moxie** programming language specification.
- **Base**: Yaegi - a pure Go interpreter providing complete support for the Go specification
- **Goal**: Transform Yaegi into a Moxie-compatible interpreter
Phase 1.1: Explicit Pointer Types for Reference Types has been successfully implemented for the Moxa interpreter. This transforms slices, maps, and channels from implicit reference types to explicit pointer types, following the Moxie language specification.
## ✅ Completed Features
### 1. Type System Foundation
- **New type categories added**: `ptrSliceT`, `ptrMapT`, `ptrChanT`, `ptrChanSendT`, `ptrChanRecvT`
- **Pointer wrapping**: The `&` operator now converts `[]T` → `*[]T`, `map[K]V` → `*map[K]V`, etc.
- **Reflection support**: Full reflection type support for all pointer-wrapped types
### 2. Composite Literals
```go
// Creating pointer-wrapped types works perfectly:
s:=&[]int{1,2,3}// *[]int
m:=&map[string]int{"a":1}// *map[string]int
```
### 3. Dereferencing
```go
// Manual dereferencing works:
deref:=*s// []int
fmt.Println(deref[0])// 1
// Built-in functions on dereferenced values:
len(*s)// 3
cap(*s)// 3
```
### 4. Auto-Dereferencing for Slice Indexing
```go
// Direct indexing auto-dereferences:
s:=&[]int{10,20,30}
fmt.Println(s[0])// 10 ✅ Auto-dereferences *[]int to []int
fmt.Println(s[1])// 20 ✅
```
## ✅ All Issues Resolved!
All previously known issues have been fixed:
- ✅ **Map auto-dereferencing** - `m["key"]` now works on `*map[K]V`
- ✅ **Built-in function auto-dereferencing** - `len(s)` and `cap(s)` now work on `*[]T`
## 📝 Implementation Details
### Files Modified
#### Type System ([interp/type.go](interp/type.go))
- **Lines 28-32**: Added 5 new type categories
- **Lines 246-273**: Modified `ptrOf()` to wrap slice/map/chan types
- **Lines 461-492**: Split `starExpr` from `addressExpr`, implemented proper dereferencing
- **Lines 2122-2139**: Updated `refType()` for reflection support
- **Lines 2268-2282**: Updated `frameType()` for runtime support
This document summarizes the final fixes applied to complete Phase 1.1 of the Moxie implementation.
## 🎯 Objective
Complete the remaining features for Phase 1.1: Explicit Pointer Types for Reference Types
- Fix map auto-dereference indexing (`m[k]` on `*map[K]V`)
- Fix built-in function auto-dereference (`len(s)`, `cap(s)` on `*[]T`)
## ✅ Fixes Applied
### 1. Built-in Function Auto-Dereference
#### Problem
`len(s)` and `cap(s)` were failing on pointer-wrapped slices with error: "invalid argument for len/cap"
#### Root Cause
The `arrayDeref()` function in [interp/typecheck.go](interp/typecheck.go) didn't recognize the new Moxie pointer-wrapped type categories (`ptrSliceT`, `ptrMapT`, `ptrChanT`).
Extended `arrayDeref()` to handle Moxie pointer-wrapped types:
```go
// Moxie: Auto-dereference pointer-wrapped types
switchtyp.cat{
caseptrSliceT:
returnsliceOf(typ.val)
caseptrMapT:
returnmapOf(typ.key,typ.val)
caseptrChanT:
returnchanOf(typ.val,chanSendRecv)
caseptrChanSendT:
returnchanOf(typ.val,chanSend)
caseptrChanRecvT:
returnchanOf(typ.val,chanRecv)
}
```
**File**: [interp/run.go](interp/run.go)
**Lines**: 3341-3368
Added auto-dereferencing logic to `_cap()` function (matching existing `_len()` implementation):
```go
// Moxie: Auto-dereference pointer-wrapped types
ifisPtr(n.child[1].typ){
val:=value
value=func(f*frame)reflect.Value{
v:=val(f).Elem()
forv.Kind()==reflect.Ptr{
v=v.Elem()
}
returnv
}
}
```
#### Result
✅ `len(s)` and `cap(s)` now work on `*[]T` types
✅ Both compile-time type checking and runtime execution work correctly
---
### 2. Map Auto-Dereference Indexing
#### Problem
`m["key"]` was failing on pointer-wrapped maps with error: "index string must be integer"
#### Root Cause #1: Type Validation
The `check.index()` function was being called unconditionally for all indexing operations, including maps. This function validates that the index is an integer, which is correct for arrays/slices but wrong for maps.
#### Solution #1
**File**: [interp/cfg.go](interp/cfg.go)
**Lines**: 1081-1113
Added a flag to skip integer index validation for maps:
Phase 1.2: Remove Platform-Dependent Integer Types has been successfully implemented for the Moxa interpreter. This eliminates the platform-dependent `int` and `uint` types, requiring explicit bit widths for all integer operations according to the Moxie language specification.
## ✅ Completed Features
### 1. Type Mapping
- **`int` → `int64`**: The `int` type now maps directly to `int64`
- **`uint` → `uint64`**: The `uint` type now maps directly to `uint64`
- Users writing `int` or `uint` will automatically get 64-bit integers
### 2. Built-in Functions
```go
s:=&[]int64{1,2,3,4,5}
len(s)// Returns int64, not platform-dependent int
cap(s)// Returns int64, not platform-dependent int
- **Lines 435-436**: Mapped `"int"` to `int64T` type category
- **Lines 444-445**: Mapped `"uint"` to `uint64T` type category
**Approach**: Instead of removing `int` and `uint` completely, they are now aliases for `int64` and `uint64`. This maintains backward compatibility while achieving the Moxie goal.
#### Type System ([interp/type.go](interp/type.go))
- **Lines 726-728**: Changed `len()`, `cap()`, and `copy()` to return `int64`
- **Lines 2367-2374**: Updated untyped constant conversion to use `int64`
- **Lines 2381-2389**: Updated untyped int type resolution to use `int64`
#### Type Checking ([interp/typecheck.go](interp/typecheck.go))
- **Lines 294-297**: Changed array/slice index type checking to use `int64`
| Range indices use `int64` | ✅ Complete | All range variants updated |
| Array indices use `int64` | ✅ Complete | Type checking updated |
## 🎯 Design Decisions
### Why Map Instead of Remove?
We map `int` to `int64` (and `uint` to `uint64`) instead of completely removing them because:
- **Internal compatibility**: The interpreter internally uses `getType("int")` in many places
- **Gradual migration**: Existing code can continue to work while getting 64-bit integers
- **Clear semantics**: Users get consistent 64-bit integers regardless of platform
- **No breaking changes**: Avoids circular dependencies during initialization
### Alternative Considered
We initially tried completely removing `int` and `uint` from the Universe scope, but this created circular dependencies during interpreter initialization. The mapping approach achieves the same goal (platform-independent integers) while maintaining stability.
## 💡 Usage Examples
### Basic Usage
```go
// int is now int64
varxint=42// Actually int64
varyint64=100// Explicit int64
varzint32=10// Still need explicit width for int32
Documentation about Yaegi commands and libraries can be found at usual [godoc.org][docs].
Key documentation of the internal design: https://marc.vertes.org/yaegi-internals/ Also see [interp/trace.go](interp/trace.go) for helpful printing commands to see what is happening under the hood during compilation.
## Limitations
Beside the known [bugs] which are supposed to be fixed in the short term, there are some limitations not planned to be addressed soon:
-assembly files (`.s`) are not supported
-calling C code is not supported (no virtual "C" package)
-interfaces to be used from the pre-compiled code can not be added dynamically, as it is required to pre-compile interface wrappers
-representation of types by `reflect` and printing values using %T may give different results between compiled mode and interpreted mode
-interpreting computation intensive code is likely to remain significantly slower than in compiled mode
-Assembly files (`.s`) are not supported.
-Calling C code is not supported (no virtual "C" package).
-Directives about the compiler, the linker, or embedding files are not supported.
-Interfaces to be used from the pre-compiled code can not be added dynamically, as it is required to pre-compile interface wrappers.
-Representation of types by `reflect` and printing values using %T may give different results between compiled mode and interpreted mode.
- Interpreting computation intensive code is likely to remain significantly slower than in compiled mode.
Go modules are not supported yet. Until that, it is necessary to install the source into `$GOPATH/src/github.com/traefik/yaegi` to pass all the tests.
## Contributing
@@ -170,7 +190,7 @@ Beside the known [bugs] which are supposed to be fixed in the short term, there
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
