At variable, function parameter, slice, map or field element assign,
if the destination type is an empty interface, the value was never
wrapped into a valueInterface (to preserve type mutability in case
of re-assign). Now we wrap it in a valueInterface if the source
type has a non empty set of methods, to allow a future use as a non
empty interface.
There are still corner cases, but it extends notably the support
of interfaces within the interpreter.
Fixes#1355.
The logic of goto was false due to mixing break label and goto
label, despite being opposite. In case of break, jump to node (the
exit point) instead of node.start. Also always define label symbols
before their use is parsed.
* Fix continue label, detect invalid labels for break and continue
* fix multiple goto, break, continue to the same label
Fixes#1354.
* interp: fix creation of binary composite types
Use direct assignment instead of reflect.Value Set method to
initialize a binary composite type, as for non binary types.
It ensures that a new reflect.Value is stored in the frame
instead of modifying a possibly existing one, which can defeat
the purpose of initializing variables in a body loop.
While there, remove the need to have and use a mutex on types.
Fixes#1381.
* review: rework a bit the test
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
We now detect the use of special identifier _ (blank) during parsing in order to abort compiling early. It allows to not panic later during execution. We must catch all the cases where blank is used as a value, but still preserve the cases where it is assigned, used as a struct field or for import side effects.
Fixes#1386.
When a package is imported, it creates a symbol with a name like "errors/_.go". If a statement such as x := errors.New(...) is executed before import "errors", it creates an incomplete type symbol with a name like "errors". Importing the package after the incomplete type symbol has been created does not fix the compile issue because the incomplete type still exists.
To fix this, this PR deletes the incomplete type symbol, if one exists.
Closes#1388.
The interpreter has its own internal fileset and expects all code to have been parsed using that fileset. If a user creates a fileset, calls `go/parser.Parse*`, then passes the result to `interp.CompileAST`, strange things can happen.
The solutions I can see are:
1. Expose the fileset so the user can use it when parsing source.
2. Add the fileset as an option (input to New) so that the user can tell the interpreter to use a specific fileset.
3. Pass the fileset into `CompileAST`
There are two ways to implement option 3. One is to add a field to nodes and update every reference to `interp.fset` to use `node.fset`. The other is to add a parameter to every function that references `interp.fset` or calls a function that does. Both of these are significant changes and involve an extra pointer for every node or most function calls.
Options 1 and 2 are easy. Option 2 involves adding an option so I went with option 1. I can imagine situations where option 2 could be necessary, but I can open another issue/PR if and when I need that.
Fixes#1383
* interp: handle struct with multiple recursive fields
In case of a recursive struct with several recursive fields of
different type, only the first one was properly fixed when
constructing the corresponding reflect type. We now memorize and
process all fields at the same depth level.
Fixes#1371.
* Update interp/type.go
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* fix lint
* fix comment
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* interp: fix handling of redeclaration in multi-assign expression
In a statement like `a, b := f()` if `a` was previously declared,
its symbol must be reused, a new symbol must not override its
previous value. This is now fixed.
* In case of redeclaration, reuse the existing only if the redeclared
variable has the same type. Add _test/var16.go to check this use
case.
Fixes#1365.
The capability to dereference pointers has been added to
methodByName(), improving method lookup on binary values.
Wrapping to valueInterface is performed in a missing use case at
return of function calls. It was done in the nested call, but not
at assign.
Fixes#1330.
A runtime builtin assignFromCall is added to handle multiple values returned at once. It is necessary if some of the values require to be set to interface values in the caller space, which is performed by reflect.Set in assignFromCall.
Fixes#1311.
In restricted mode, replace environment related symbols in
stdlib os package by a version which operates on a private copy
per interpreter context.
It allows to have concurrent interpreters in the same process
operating each in their own environment without affecting each
other or the host.
If unrestricted opt is set, this behaviour is disabled, and the
default symbols from stdlib are used.
Note also that no modification is done for syscall package, as it
should be not used in restricted mode.
At generation of operator closures, the comparison expression
was hard-coded instead of being derived from the operator name,
leading to a wrong result.
Fixes#1297.
In some cases, we use many yaegi instances in one process, each interpreter will expect its own os.Args. This is done in interp.go as an option in interp.Options besides Stdin Stdout and Stderr.
Add a tag option in the extract program to optionally give an opportunity to tailor some packages in for example the stdlib, for people who is interested in keeping the final binary size as smaller as possible.
Below go generate
```
//go:generate ../cmd/extract/extract -name stdlib -tag stdmime mime mime/multipart mime/quotedprintable
```
produces a header to stdlib/mime-multipart.go
```
// Code generated by 'yaegi extract mime/multipart'. DO NOT EDIT.
// +build go1.16,!go1.17,stdmime
```
When an interpreter type implementing an interface is
used by the runtime, the runtime can extract its type
and create new values using reflect, and call methods
on it. The problem is that there will be no interpreted
method counterpart in this case, which makes wrapper panic.
Allow the String() method wrapper to always succeed and
return an empty string if no interpreted method is present.
This allows scripts to define custom flag.Value types on
which the runtime internally instantiates values using
reflect (see isZeroValue in Go src/flag/flag.go).
This workaround could be generalized to all wrappers if
necessary. At this moment, it is convenient to keep the
default behavior of expecting instantiated interpreter
methods, in order to catch interpreter bugs related to
interfaces.
Fixes#1276.
This comes from experiments looking into #1259 where incomplete twice seen types are marked as complete. To mitigate the problem instead of a map of seen types in `nodeType` a slice is used as a cheap way to keep track of our current path through the node tree.
This change fixes a regression introduced by PR #1192 in a program using
https://github.com/NYTimes/gziphandler which defines several types
implementing stdlib interfaces. We do not implement a wrapper
if we see that a type already implements an interface, except that
it can be falsly reported by reflect in case of a struct with
embedded interface field. We need to force the wrapper generation
in this case.
The problem occurs only for wrappers on struct, not on pointers or
other indirection types.
Adds CompileAST, which can be used to compile Go AST directly. This
allows users to delegate parsing of source to their own code instead of
relying on the interpreter.
CLoses#1251
When a define statement relies on a selector or type that may exist in another file it should revisit once GTA is complete. This allows that revisit.
**Note:** In order to keep the original GTA error for the define statement, so the error received is correct and meaningful, I have added a node `meta` property. I decided to make it generic as it may be useful in future. There may be a better way to stash errors across the GTA runs and am open to suggestion here.
Fixes#1253
The current `nodeType` selector precedence is heavy handed in favour of package type. It seems to often create `typeSym` symbols as variable types in the scope will never be found. To fix this if the ancestor node is a field expression, the package type is searched for. After this, if the type is still `nil` the normal scope is searched using `nodeType2`.
Fixes#1158
Make sure to keep always a single copy of incomplete type structures.
Remove remnants of recursive types processing.
Now `import "go.uber.org/zap"` works again (see #1172), fixing regressions
introduced since #1236.
When `nodeType` recovers names and methods, it can overwrite the data if the type is an aliasT. When aliasing a type, do not recover the methods, this will be done in the GTA typeSpec pass.
Related to #1158
When a const is late binding and specified with a type, the GTA defineStmt was creating the symbol with the current scopes `iota` which is incorrect. The symbol should be created with the source nodes `rval`.
Related to #1158
This test (assert2.go) display 2 separate issues:
1. assert2.go L28: Type assert tries to set an `interface{}` to a `valueInterface`. The typing here is complex, we have a valueT(strings.Builder) wrapped in a ptrT wrapped in a src iface wrapped in a valueT(interface{}). Type assert fails to realise that the `valueT` `interface{}` is wrapping the `valueInterface`.
2. assert2.go L29: `genValueBinMethodOnInterface` does not try and get the bin method, as the `typ.node` (`ptrT` or a `valueT`(`string.Builder`)) is set. In this case the src iface is called with a receiver argument. To fix this the method is looked for first if possible, and only if not found does it fall back to the `defaultGen`.
Fixes#1227
It was initially assumed that `nodeType` needed to rebuild the type from scratch, but this is not the case anymore. The existing type constructors are now used in `nodeType` to make it more readable. In doing this some bugs in type strings were found and fixed, along with adding the real package name to the type.
As `ptrOf` is now the only place that pointer types are constructed, it is feasible to cache the pointer type on the value type. To ensures that we have a consistent pointer type for any value type.
As the unsafe and pointer methods in `reflect` are to be depreciated, and seeing no replacement functions, it is now forced that some unsafe is needed to replace this as when and interface is dereferenced it is made unsettable by reflect.
With this in mind, this adds real recursive types by hot swapping the struct field type on the fly. This removes a lot of compensation code, simplifying all previous cases.
**Note:** While the struct field type is swapped for the real type, the type string is not changed. Due to this, unsafe will recreate the same type.
This adds `itype.str` which is a string representation of the type built when the type is built. The goal is to make type comparison simpler and centralise the creation of types just to constructors and `nodeType`. `nodeType` continues to build types in parts so to reuse underlying types better.
Make use of fs.FS (new to go 1.16) to allow for reading source files from diverse filesystems (local, embed, custom).
* `Options` has a new field `SourcecodeFilesystem fs.FS` so users can supply their own read-only filesystem containing source code.
* Defaults to the local filesystems (via `RealFS` - a thin `os.Open` wrapper complying with `fs.FS`) so regular users should see no change in behaviour.
* When no filesystem is available (e.g. WASM, or if you want to embed files to retain single binary distribution) an alternative filesystem is preferable to using `Eval(string)` as that requires the stringy code to be a single file monolith instead of multiple files. By using an `fs.FS` we can use `EvalPath()` and gain the ability to handle multiple files and packages.
* You can make use of embed filesystems (https://pkg.go.dev/embed) and custom filesystems obeying the `fs.FS` interface (I use one for http served zip files when targeting wasm as there is no local filesystem on wasm). Tests can make use of `fstest.Map`.
* NOTE: This does NOT affect what the running yaegi code considers its local filesystem, this is only for the interpreter finding the source code.
See `example/fs/fs_test.go` for an example.
Fixes#1200.
Set node address in `val` field at creation of `funcDecl` node so it can be used correctly at closure generation, even in the case of forward function declarations, where the value was zero.
Fixes#1214
When calling a function, if the input param is a "zero" instance, it is not set on the input. This is an issue where the param is an `interface{}` as a `nil` value is set instead of the zero value.
The actual solution for this is to remove the `if !val.IsZero()`, this however runs into an issue on `_test/struct48.go` where we have a zero recursive struct instance (`*interface{}`) and we have no way to get its real type. Should a way be figured out to keep tabs on the original type, the `if` can go away, and in the zero case of `genValueRecursiveInterfacePtrValue`, the actual zero type can be returned.
Fixes#1215
Add getConcreteType to retrieve the concrete type of a nested interface
value implementing a specific interface for which a wrapper exists.
If method resolution fails at runtime, a panic is now issued instead
of an error message and continue.
Fixes#1187.
Map handling builtins getIndexMap and rangeMap had some leftover
code of previous way of emulating interfaces, which was modified
following changes in #1017.
Specific code for interfaceT is removed, as not necessary anymore.
Map builtins are now simplified and more robust.
Fixes#1189.
When operating on map elements, result of assign operators were not
written to the map entry. Now check if the destination of an assign
operator is in a map, and if so, set the result to it.
Fixes#1194.
Add early detection of cases where no wrapper is necessary because
the value type already implements the target interface.
It should both increase performances by avoiding the wrapper overhead,
and fix errors due to replacing valid values by incomplete wrappers,
caused by the presence of private methods in the interface definition,
as in #1191.
Fixes#1191.
Store the interpreter value of the interface object to wrap as
a field called IValue, at offset 0 in wrapper structures.
Update extract to include IValue field.
In typeAssert, detect interface wrapper, and dereference the
interpreter value from IValue wrapper field.
Fixes#1166.
In parsing array type declaration, The type check of array size was
restricted to `int`. Broaden the test to accept any valid integer
kind.
Fixes#1175.
The resolution method was not compliant with the Go specification which
requires to retain the object where the field or method is the most
shallowed.
The detection of ambiguous fields or methods (same depth in different
objects) has also been added.
Fixes#1163.
In selector resolution, struct field matching now precedes
method matching. Before struct field matching could be skipped
in case of a matching method, which is incorrect, as demontrated
by _test/issue-1156.go.
Field lookup has been fixed to operate on recursive structures.
Concrete type values are derived when filling a receiver for
interface methods.
LookupBinField has been fixed to skip non struct values.
LookupMethod has been fixed to iterate on interface values as
well as concrete type values.
Fixes#1156.
### Background
#1102 changed how `Interpreter.Use` interprets export paths such that the last path component is stripped and used as the package name. This resulted in #1139 - attempting to Use an export with only one path component, such as `foo`, would result in the import path being `.`.
### Breaking API Change
This PR changes the signature of `Interpreter.Use` from `Use(Exports)` to `Use(Exports) error`.
### Fix for #1139
With this PR, if Use is called with an incomplete export path, such as `foo`, Use will return an error.
Fixes#1151
If I add a package with `Use` and import it with `ImportUsed`, the package is added to the universe scope as `<pkg>`. If I import with `Eval`, the package is added as `<pkg>/_.go`. However, `(*node).isType` (in cfg.go) only checks for `<pkg>/_.go`. Thus, packages imported with `ImportUsed` can be inaccessible.
This MR updates `(*node).isType` to fall back to `<pkg>` if `<pkg>/_.go` does not exist.
In typecheck.go, detect binary methods so we know when to skip the receiver as first parameter when checking function signatures. The signature check is not yet performed, we just avoid a false error.
In cfg.go, take care to label types with isBinMethod field to true whenever a binary method is resolved.
Also, do not attempt to wrap node in functions if the node value is already a binary function.
Fixes#1145.
Fixes#1150
1. When resolving a selector expression involving an aliased type, resolve the aliased type
2. When building an array literal, resolve the aliased type
Aliases of named array and slice types were the only ones that didn't work, but I added the other test cases for the sake of completeness and through testing.
Fixes#1149
Because of how aliases are handled, `n.gen` is set to `getIndexSeqMethod` or `getIndexSeqPtrMethod` in cases like the one described in #1149. As a result, `FieldByIndex` can be called on a value that is not a struct, which causes a panic. This MR updates those two methods to avoid that call if the index array is empty.
This is a small change that allows use of composite array literals, such as:
```go
type Vec3 [3]float32
var foo = []Vec3{
{1, 0, 0},
{6, 0, 0},
{6, 2, 0},
{2, 2, 0},
{1, 1, 0},
}
```
In switch case expressions, the condition on case clause was
not always properly evaluated. Reverse the order of case clause
evaluations (as already done for if-else-if fashion), and fix the
wiring to false-next and true-next nodes.
Fixes#1126.
The case of assigning a binary function to a funcT object was
solved elsewhere. Factor the case in genDestValue to apply it
at multiple places.
Fixes#1100.
In binary packages, constants are wrapped in constant.Values, to
support arbitrary precision. Their type must therefore be converted
back to a regular type at import.
Fixes#1101.
Use YAEGI_SPECIAL_STDIO env boolean to overwrite os.Stdxxx by
non file descriptors io.Writer / io.Reader interfaces. It is set
to true when testing to allow redirection to byte buffers.
The default behaviour is now to preserve the original concrete type
when sandboxing stdio, which maintains compatibility.
Fixes#1092.
Some binary method calls were wrongly rejected. There is still
some ambiguous cases as binary method signature may include or
not the receiver as first argument, depending on how the method
was resolved.
With this fix, `import "golang.org/x/net/html"` doesn't panic
anymore, but not all tests are passing yet, i.e.
`yaegi test golang.org/x/net/html` still has failures, to be
investigated.
Fixes#1107.
In this range variant "for k, v := range aString", k must
be the byte position of the rune in the byte array, rather than
the index of the rune in the rune array.
Fixes#1088.
In that case, direct propagation of result can not be attempted,
as the frame types will be different between the source and destination.
Disabling the optimisation and using The regular case involves an intermediate
frame entry, which enables the type conversion.
Fixes#1091.
The heuristic to generate a package name identifier was incorrect. Now for binary packages, the package identifier is obtained by a symbol, generated by extract, which contains the string argument of package statement in source file. This should ensure an always correct default package identifier.
Fixes#1095.
The concrete type was not forwarded propertly in case of a binary
expression involving a valueT. The corresponding part in type.go
has been refactored and the now the multi-assign case should be
handled as well.
Fixes#1094.
This feature was already present, but part of REPL only.
It's now also possible to apply it when evaluating a string
(-e flag). Default package names collision handling is no
longer hard-coded.
With -e flag, the eval result is now printed if valid, allowing
simpler commands:
yaegi -e 'reflect.TypeOf(fmt.Printf)'
instead of:
yaegi -e 'println(reflect.TypeOf(fmt.Printf))'
Fixes#1084.
Interpreted functions were represented in an inconsistent way in the frame: as a node pointer by default, and wrapped in a function wrapper for maps.
We now simply use the default (*node) representation, as elsewhere, so values can be assigned, passed and called as for the other types. The alternative (generating a function wrapper) is more complex, costly and reserved for cases where the interpreted function can be called from the runtime.
Test that a map of functions can store both binary functions from used packages and interpreted ones.
Fixes#1090.
The interpreter is exposed to itself through a "Self" var which
is set on "Use" of the interpreter package.
It allows meta-programming features, for example using "Eval" in
the current interpreter context, or enabling self-inspection
capabilities.
Avoid to test directly for a type category, as it may give wrong
results for aliased types, where the interesting category remains
masked. Instead, use some property helpers, such as isFuncSrc,
isPtrSrc and isInterfaceSrc to check if a type is of source function,
source pointer or source interface respectively (versus runtime
defined function, pointer or interface).
Fixes#1068.
Offsetof returns the offset of a field in a struct. It is computed
during parsing at CFG, due to the constraint of operating on a
struct selector expression.
With this function, the support of 'unsafe' package is now
complete in yaegi.
Fixes#1062.
Add missing `sliceT` type category for consistency. Remove
`sizedef` field in `itype` struct. Rename field `size` to `length`.
Clean the various hacks used to cope with the absence of `sliceT`.
If I execute the following:
```
I := interp.New(interp.Options{})
I.Eval(`x := 1`)
I.Eval(`x := "foobar"`)
```
I expect the second declaration to override the first. `var x string` will override the previous type, and redeclaring a type, function, const, etc will override it, but the `:=` operator will not.
Currently, the result is: `reflect.Set: value of type string is not assignable to type int`
This PR:
- Treats a `varDecl` within a block as a `defineStmt`
- More specifically, any `varDecl` with a grandparent that is *not* a `fileStmt`
- Adds an extra condition to the handler for implicit const assignment
- Adds a tests to cover the changes
- Closes#1071
This patch brings the following modifications:
- consider that an interface is assignable to another if the former
implements the latter
- call TypeOf() method instead of rtype field when resolving methods, to
handle first met types
- unwrap error interface inplace rather than embedding it in an
interface definition, as lower case named embbeded interface may
not be handled by reflect when lookup for a method.
Fixes#1063. Partially improves #1058.
This PR adds an interpreter option, `AllowRedeclaration`. If this option is set, `(*Interpreter).Eval` will allow package imports to be redeclared. That is, no error will be raised and the package symbol will be overwritten.
I would like to use Yaegi to power a Go notebook (VSCode extension), somewhat like Jupyter. A notebook can have multiple Go 'cells' which can be evaluated (using Yaegi). As much as is possible, evaluating cells should be idempotent - that is, evaluating a cell multiple times should have the same effect as evaluating it once, ideally. Cells that are not idempotent can degrade the user experience.
Specifically, Go files tend to declare all imports in a single block. In a notebook, I'd put all imports in a single block, in their own cell. When I decide I need to import an additional package, I want to add that import to the existing cell and evaluate it. Without this MR, reevaluating that block usually causes an error.
Functions in a struct fields are always wrapped (as potentially
used by the runtime), so generate a function wrapper also for
closure when assigned to a struct field.
When such a function is called from the interpreter, ensure that
interface arguments are also wrapped so method and receiver resolution
can be performed.
Fixes partially #1043.
The first change forces a variable definition to reallocate a
new memory slot to avoid corrupting a previously defined one in
a loop block.
The second change ensures that the frame clone operations obtains
a copy of the original data slice, to preserve the original context
set in a loop.
Fixes#1035.
As mentioned in #1030, when an Eval panic, it is print with `fmt.Println()` and not to the configured `interp.Options.Stderr`. According to https://github.com/traefik/yaegi/blob/master/interp/interp.go#L210, it should be removed in future version so I'm not sure if this pull request is necessary. However, it could fix the issue in the meanwhile.
Fixes#1030.
This *should* fix the generated names of packages on windows, which currently take the form `C:\kdfjslfj\jkfsjldkfjsf\sjdkfldjf` (this was reported to me by someone attempting to generate yaegi symbols for one of my projects). That being said, I don't have a windows machine to test on.
This is a follow-up of #1017, generalizing the use of reflect.Set
method to set, and possibly overwrite, the concrete value of
interface objects all accross the implementation of operators.
Previous optimized implementation for non-interface objects is
preserved.
The empty interface (interface{}), and its variants (such as []interface{} and map[string]interface{}), are commonly used in Go to (json) Unmarshal arbitrary data. Within Yaegi, all interface types are wrapped in a valueInterface struct in order to retain all the information needed for a consistent internal state (as reflect is not enough to achieve that). However, this wrapping ends up being problematic when it comes to the type assertions related to the aforementioned Unmarshaling.
Therefore, this PR is an attempt to consider the empty interface (and its variants) as an exception within Yaegi, that should never be wrapped within a valueInterface, and to treat it similarly to the other basic Go types. The assumption is that the wrapping should not be needed, as there is no information about implemented methods to maintain.
Fixes#984Fixes#829Fixes#1015
When checking for untyped values, we can be sure at this stage that they must be a const value or already untyped. Checking for type string equality is no longer a good measure.
Fixes#1000
When passing a function reference as an interface in a composite binary map, the case should be handled to not take the value of the the node.
Related to #886
This is a follow-up of #1014, where an invalid constant definition involving a builtin is now checked at CFG. In addition, some missing arithmetic operators are now detected for assign optimization.
A channel can be used to interchange data with the pre-compiled
runtime and therefore objects impletementing interfaces must be
wrapped if necessary, using genInterfaceWrapper.
A similar treatment could be applied when sending interpreted
functions over a channel, to be provided in a new PR.
Fixes#1010.
Fix the logic to detect recursive struct types, which was giving a false positive.
We now use the local type name as key in tracker map.
A non-regression test case is included (_test/struct49.go).
This completes #1008.
If a struct contains several fields of the same temporary incomplete
type, it could be detected incorrectly as a recursive struct. Pass
a copy of defined types map to avoid this issue.
Fixes#1007.
Always attempt to obtain an integer constant value for operators
expecting so. It allows to use '/' in integer constant defintions,
instead of default big.Rat.
Fixes#1005
An undefined type detection function has been added to better diagnose
incomplete type definitions. Implicit type names in interface or struct
declarations are now better handled. The incomplete status is not
fowarded to aliased type declarations to handle circular definitions.
Fixes#999 and #995. Improves #260 (goes farther, but still fails).
In some cases, the global character of a value was lost, leading to
undefined behaviour. Now a node level field of -1 means that the value
is global, and that it should be accessed from the root data frame.
Fixes#993.
The interpreter interface type was replaced by a reflect.Value in
objects passed or return to function wrappers, losing the ability
to retrieve methods.
The valueInterface is now preserved, and correctly accessed if
wrapped multiple times.
Fixes#977.
typeAssertStatus deals with the 3rd form of type assertion ("_, ok"), for
when one does not care about the result of the assertion itself.
Some cases for it, which are already fixed for the two other forms of
type assertions, had not been fixed for this form yet.
Therefore, this change fixes such cases for this form, while integrating
typeAssertStatus to the same code path as for the two other forms.
The type check was generating false negatives. A correct test to check the
adressable status of an array is more complex to implement, and will
be done later.
Fixes#973.
The long-form (with comma-ok) ones were already fixed but the short-form
ones were not because they were in a completely different code path.
This PR also refactors the code so that both short-form and long-form
are now merged in the same function.
N.B: even though most (all?) cases seem to now be supported, one of them
still yields a result that does not satisfy reflect's Implements method
yet. It does not prevent the resulting assertion to be usable though.
N.B2: the code path for the third-form (_, ok) hasn't been fixed and/or
refactored yet.
Fixes#919
In aliased type declarations, when the target type was imported from
an external package rather than declared locally, the aliased type was
overwritten by target, loosing ability to lookup methods on the aliased
type. Aliasing on imported types is now properly detected and handled.
Fixes#971.
As opposed to other symbols, goto labels must be searched in included
scopes, not upper ones. Implement scope.lookdown to perform this,
to allow calls to goto to be embedded in included scopes where label
is defined.
Fixes#953.
Perform function declaration type check from the upper level scope (the scope where the
function is declared), to avoid possible collisions of local variables with package names.
Fixes#957.
in `callBin`, call arguments are converted to the corresponding
parameter type. In a case of an interface, the original concrete type
should be preserved instead, and only wrapped to an interface type for
internal interpreter types, as runtime values should already implement the
interface.
This change removes the interface wrapping when parameter is a runtime
value (valueT or ptrT to valueT).
This removes some overhead when handling runtime values, and keep a
similar behavior between interpreted and pre-compiled code. For
example, `io.Copy` preserves its internal optimisations when passed a
`bytes.Buffer`.
The function vUint, used to get the unsigned integer value of a value,
variable (frame) or constant, was broken for constant.Value expression.
Fixes#948.
When running GTA, the type `path` was set to `rpath`. This equates to the package path (`github.com/traefik/yaegi`) in most cases. In the vendored case the `rpath` is the sub package path `something/vendor/github.com/traefik/yaegi` causing issues in typecheck and likely further down the line. By using the `importPath` it makes this consistent.
**Note:** I have no clue how to test this decently. I am open to options here.
Potentially Fixes#916
Interpreter function types are represented internally by the AST node
of their definition. The conversion operation creates a new node with
the type field pointing to the target type.
Fixes#936.
A non-constant shift expression can be untyped, requiring to apply a
type from inherited context. This change insures that such context is
propagated during CFG pre-order walk, to be used if necessary.
Fixes#927.
The execution flow is such that a node can end up running several chained exec
funcs, some of which actually originate from other nodes. For example, in:
var m []int // L0
println("hello world") // L1
m[0] = 1 // L2
the offending code is on a node on line 2 (out of range error). However, since
the assignment to m is part of the execution flow of the variable m, we'll get
the panic when running all the chained exec funcs attached to the node for m on
line 0.
Which is why, when that happens, we need to look for the actual node (the one on
L2) where the offending instruction originates from, in order to
properly report the origin of the panic.
Fixes#546
The case of a constant arithmetic expression being of float kind because
of quotient was not handled correctly. Simplify constant extraction to
reflect.Value first, then conversion to target type, and let reflect Convert
method panic if conversion is not valid.
Fixes#920.
Type checking on shift operands was failing for untyped variable values.
Fix propagation of type in assignment. Optimize assignment of arithmetic
operations on variables by skipping the assign and writing directly to
destination frame value in the operator function.
Skip some slow tests when given -short test option.
Fixes#917.
Type check was failing for expression such as: `&(*tree)[node:][0]`, as in
```go
tree, _ := huffmanTreePool.Get().(*[]huffmanTree)
...
initHuffmanTree(&(*tree)[node:][0], histogram[l], -1, int16(l))
```
see c3da72aa01/brotli_bit_stream.go (L469)
The assignable check used to be too strict as it lacked the property that
if an untyped const can be represented as a T, then it is assignable to T.
And we can now use that fixed check to add a missing check: in a return
statement, we now make sure that any of the returned elements are
assignable to what the signature tells us they should be.
In unary constant operations, the test for unsigned was defeated by
testing for int first, which is true also for unsigned. Make sure that
testing for unsigned precedes testing for int.
Fixes#907.
This applies to -syscall, -unsafe and -unrestricted flags with the
corresponding env variables YAEGI_SYSCALL, YAEGI_UNSAFE and
YAEGI_UNRESTRICTED, already used in the same way for the run
sub-command.
Often enough when debugging, one does not know exactly what argument
should be given to Symbols, as it's not always clear in which
scope/namespace the symbol one is looking for is stored in.
Therefore, this change enables the Symbols method to now take the empty
string as an argument, which results in all the known symbols to be
returned (keyed by import path).
As a consequence, when an non-empty argument is given, the returned
result should be similar to what we had before, except it is now
returned as the sole entry of an encompassing map.
In addition, the "binary" symbols (i.e. the ones ingested through a
Use call), are now also taken into account.
There was several issues:
- access to field on pointer to struct from runtime: fix in
lookupBinField
- assign operation was skipped when performed in a comm clause
- the direction of comm clause was wrong if a channel send operation was
performed in a body of a receive comm clause
Fixes#884.
Check first for runtime defined array (typ.cat of valueT)
to avoid checking inexisting properties (an panic), when
deciding to use appendSlice or not.
Fixes#880.
Since the introduction of restricted stdlib and syscall symbols, the
capability of yaegi to interpret itself was broken.
The use of unrestricted symbols is now also controlled by environment
variables, to allow propagation accross nested interpreters.
The interpreter Panic symbol was not wrapped, this is fixed now.
the import path resolution was failing if the working directory was
outside of GOPATH.
The documentation and readme have been ajusted.
Fixes#890.
When the type is implicit, the first element in the list of children is
not the type of the composite literal, but it is one of the actual
children, so it should not be discarded.
Fixes#862
When working with an untyped const expression involving a division, if
the default type of the result should be an int (for example because the
default types of all the operands are ints as well), then we should make
sure that the operation that is applied is indeed an integer division,
and that the type of the result is not a float.
This is achieved by using the QUO_ASSIGN operator, instead of the QUO
operator.
This should fix several problems lurking around, and it notably fixes
one of the visible consequences, which is a systematic panic when using
the REPL as a "calculator".
This incidentally also allows us to revert what was done in
5dfc3b86dc since it now turns out it was
just a hack to fix one of the symptoms.
Fixes#864
As per https://golang.org/cmd/go/#hdr-Build_constraints,
using GOOS=android also matches tags and files for GOOS=linux,
so exclude it explicetly to avoid collisions.
Also using GOOS=illumos matches tags and files for GOOS=solaris.
Fixes#843.
This change fixes two data races related to composite literal creation.
The first one isn't controversial as it is just about initializing the
variable that contains the values in the right place, i.e. within the
n.exec, so that this variable is local to each potential goroutine,
instead of being racily shared by all goroutines.
The second one is more worrying, i.e. having to protect the node typ
with a mutex, because a call to func (t *itype) refType actually
modifies the itype itself, which means it is not concurrent safe.
The change seems to work, and does not seem to introduce regression, but
it is still a concern as it probably is a sign that more similar
guarding has to be done in several other places.
This change allows the interpreter to execute tests and benchmarks
functions provided by packages.
The test subcommand is similar to the "go test" command and
all the relevant flags have been kept.
The ability to evaluate a directory or a package has also been added.
A new method Symbol to access exported symbol values of an interpreted
package has been added. This method is used by the test subcommand.
An EvalTest method has been added to evaluate all Go files, including "*_test.go".
The testing packages from the standard library have been added to stdlib used
symbols.
Struct fields of type interface must be converted in wrapper
values to be reachable by the runtime. Call genInterfaceWrapper
on such values.
Fixes#832.
When searching for a binary method on structures, look up on embedded
fields first, otherwise the resulting index is incorrect, as
reflect.Type.MethodByName succeeds also on container struct.
Fixes#834.
This change fixes two distinct data races:
1) some global vars of type *itype of the interp package are actually
mutated during the lifecycle of an Interpreter. Even worse: if more than
one Interpreter instance are created and used at a given time, they are
actually racing each other for these global vars.
Therefore, this change replaces these global vars with generator
functions that create the needed type on the fly.
2) the symbols given as argument of Interpreter.Use were directly copied
as reference (since they're maps) when mapped inside an Interpreter
instance. Since the usual case is to give the symbols from the stdlib
package, it means when the interpreter mutates its own symbols in
fixStdio, it would actually mutate the corresponding global vars of the
stdlib package. Again, this is at least racy as soon as several
instances of an Intepreter are concurrently running.
This change fixes the race by making sure Interpreter.Use actually
copies the symbol values instead of copying the references.
The goal is to provide greater control of input, output and error
streams of the interpreter. It is now possible to specify those
as options when creating a new interpreter. The provided values
are propagated to relevant stdlib symbols (i.e fmt.Print, etc).
Care is taken to not update the global variables os.Stdout, os.Stdin
and os.Stderr, as to not interfere with the host process.
The REPL function is now simplified. The deprecated version is removed.
The tests are updated to take advantage of the simplified access
to the interpreter output and errors.
Fixes#752.
When running TestEvalScanner with -race=true, one can observe a data race such as:
```
WARNING: DATA RACE
Read at 0x0000029f3d68 by goroutine 52:
github.com/containous/yaegi/interp.(*itype).defaultType()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1466 +0x572
...
github.com/containous/yaegi/interp.(*Interpreter).EvalWithContext.func1()
/Users/mpl/src/github.com/containous/yaegi/interp/interp.go:501 +0xf0
Previous write at 0x0000029f3d68 by goroutine 43:
github.com/containous/yaegi/interp.(*itype).refType()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1419 +0x854
github.com/containous/yaegi/interp.(*itype).TypeOf()
/Users/mpl/src/github.com/containous/yaegi/interp/type.go:1427 +0xa6
...
github.com/containous/yaegi/interp.(*Interpreter).EvalWithContext.func1()
/Users/mpl/src/github.com/containous/yaegi/interp/interp.go:501 +0xf0
```
Before this change, since closing the pipe to the REPL is done in a defer, it
means that all the i.REPL calls (and hence each goroutine for each of these
calls) are kept running and alive until the very end of the test. It should not
matter, since a new interpreter is created for each test case, and thus all the
i.REPL calls should be completely independent from each other.
And yet, by wrapping each test case in a function call, and thus making each
i.REPL call terminate as soon as the test case is over, the data race seems to
be fixed. This could suggest that the separate i.REPL calls from separate
interpreter instances are somehow sharing some memory, but I do not know how to
explain that.
The problem has yet to be fully understood, but at least this change restores
the test, without making the CI fail again.
The Go parser is able to detect some (but not all) (re-)decleration errors,
if the DeclarationErrors flag is enabled, which was not done so far.
This PR therefore enables that flag, which allows the removal of some of
the now unneeded code that was recently added to support redeclarations.
Fixes#811
In interactive mode, a line starting with the "func" keyword is usually
"wrapped", by prepending to it a "package main" statement, to make it
a valid piece of Go source code.
However, when the line is actually an anonymous function call, such as:
func() { println(3) }()
then this wrapping is not enough, as this is not valid Go in the global
context. Therefore, this kind of of expression must also be wrapped
inside a main func (as is the default case for most REPL inputs).
Since the detection and handling of such a case turned out to be quite
unelegant, this PR instead introduces a retrying phase when a parsing
error occurs for a particular class of cases. That is to say, when a
"func expression" wrapped in a main package fails to be parsed, it is
then wrapped in a main func before parsing is retried.
N.B. TestEvalScanner has been disabled for this change, because the additional test cases revealed a (most-likely already existing) data race.
Fixes#721
The input scanning is now performed in a sub goroutine and
the interrupt is listened in another goroutine, either to cancel Eval
or to cancel the current line scan.
Entering a '\n' after a 'Ctrl-C` to get the prompt is
not necessary anymore.
When determining the size of an array and a symbol is found, the symbol must be a const for the type to be valid.
While it makes sense for this check to be done in type checking, the type can be determined by GTA which would then fail. For now this check is done when getting the node type.
Fixes#825
The trick is that in reflect, the field is called Anonymous, which is
actually a different notion in the Go spec/vocable.
n.b. only works for non-recursive types for now.
Fixes#781
The recent changes that added some redeclaration checks implicitly added more
strictness related to namespaces and scopes which, among other things, broke
some uses that "accidentally" used to work.
For example, given
const script1 = `
import "fmt"
// more code
`
const script2 = `
import "fmt"
// some other code
`
If one Evals script1, then script2, with the same interpreter, without
specifying any scope, as the two fragments would be considered part of the same
(.go file) scope by default, a redeclaration error would be triggered because
import "fmt" is seen twice.
A work-around would have been to specify (a different) i.Name before each Eval
call, so that each script is considered as coming from a different .go file, and
hence are respectively in different scopes with respect to imports.
That lead us to realize we had to make specifying things such as file-scope, and
"incremental mode" (aka REPL), more obvious in the context of an Eval call.
In addition, we want to lay down the foundations for Yaegi being able to behave
more like the go tool wrt to various inputs, i.e. it should be able to take a
package directory, or an import path, as input, instead of just a .go file.
Hence the introduction of a new kind of Eval method (whose signature is not fixed yet):
func (interp *Interpreter) EvalPath(path string) (res reflect.Value, err error)
It partially solves the problem described above because:
1. the path given to EvalPath can be used as the file-scope hint mentioned
above, for now (even though the related implementation details might change).
2. Eval always runs in incremental mode, whereas EvalPath always runs in
non-incremental mode, hence clarifying the situation in that respect.
And to avoid confusion, the Name field of Interpreter is now non-exported,
since it is somewhat redundant with the path argument of EvalPath.
Note that #731 is not fully fixed (and might never be), as a requirement of the
proposed solution is to move the input bits of code into respective files
(instead of leaving them as strings).
Finally, some related bugfixes, documention changes, and some refactoring have
been included. Notably, there is no "empty scope" anymore, i.e. name defaults
to "_.go" when it is not specified.
Updates #731Fixes#778Fixes#798Fixes#789
Co-authored-by: Marc Vertes <mvertes@free.fr>
This adds type checking to SliceExpr as well as handling any required constant type conversion.
It should be noted that the check that `high` and `max` are not nil in a 3-index slice has been left out as this check is handled before type checking. Tests have been included for these cases.
In a couple of occurences, tests with enabled race detector exposed
some concurrent accesses to the cancelation callback used in select and
channel operations send and recv for EvalWithContext. This change ensure
that all accesses to this object are protected by mutex.
Fixes#815.
In the case of a pointer or alias composite literal expression, `compositeGenerator` changes the type to remove the pointer or alias. This causes a nested composite literal to have the wrong type.
Instead of changing the node type, the removal of the pointer or alias is moved to the runtime, allowing the node type to remain unchanged. This fixes potential issues in the type checking.
This adds type checking to CallExpr (excluding builtin type checking, as that is a PR in its own right) as well as handling any required constant type conversion.
This also changes constant strings and runes to be represented as `constant.Value`. Runes change `rval` type at CFG typing time to avoid having to type at AST time. There are also changes to importSpecs and `stdlib` to account for the string change. With this all `untyped` types should now be `constant.Value`s, although errors are still not returned if this is not the case to be sure we do not break things.
This also fixed a bug in `itype.methods` that would panic if the type was recursive.
Avoid goroutines leak, accumulation of defered functions and
spurious resets of signal handlers. Effectively catch interrupt
signal (Ctrl-C) to cancel current eval.
Fixes#713.
So far the REPL loop was treating any parsing error coming from
go/parser to generate the AST, as having occurred because the source
code was not yet complete (unfinished block). And it was therefore
ignoring all of them.
However, some of these errors are legitimate, and must be caught as soon
as they occur, otherwise the REPL cycle would stay in an errored state
forever (even when the block terminates), without the user getting any
feedback about it.
Therefore, this change adds an extra check when a parsing error occurs,
i.e. it verifies that it looks like an "EOF" error (unfinished block)
before it ignores it (as the user is supposed to terminate the block
eventually). Otherwise the error is treated just like a "non-parsing"
(cfg, gta, ...) error and printed out.
Fixes#637
This adds type checking to both `IndexExpr` and `CompositeLitExpr` as well as handling any required constant type conversion.
This includes a change to the type propagation to the children of a composite literal. Previously in most cases the composite literal type was propagated to its children. This does not work with type checking as the actual child type is needed.
Fix#787 changes how interfaces are set on a struct (compositeSparce). This change however makes calling the interface panic.
This PR reverts part of the change in #787 and adds a test to ensure it does not break again.
The Yaegi command line has been changed to provide subcommands.
The following sub-commands are provided:
- extract (formerly goexports)
- help
- run
- test
The previous behaviour is now implemented in run command which
is the default, so the change should be transparent.
In run command, prepare the ability to run a package or a directory
in addition to a file. Not implemented yet
The test command is not implemented yet.
The extract command is meant to generate wrappers to non stdlib
packages.
Fixes#639
The detection of special cases of interpreter functions and interfaces
is more precise. It allows at least to complete parsing of
code where type is derived from a type assertion expression.
Fixes#770.
* interp: support another vendoring case
Namely, when the vendor dir is a sibling (or an uncle) relative to the
current pkg
Fixes#758
* make linter happier
* address review comments
* fix, cleanup, add unit tests
* add dummy files to force dirs into git
The range init AST execution was skipped, and range could work
only over variables or direct function calls. By setting the
start node to the start of init and not init itself, we ensure
that the init AST is always taken into account.
Fixes#775.
* fix: constant definition loop on out of order vars
* fix: do not wire global varDecl
* fix: wire and execute global vars
* chore: add tests
* fix: refactor and lint
By returning early in case of incomplete type in CFG, we avoid
a panic, and let a chance to a new attempt after the missing
type has been parsed.
Fixes 763.
* feature: restrict symbols which can exit the interpreter process
Some symbols such as os.Exit or log.Fatal, which make the current process
to exit, are now restricted. They are replaced by a version which panics
instead of exiting, as panics are recovered by Eval.
The restricted os.FindProcess version is identical to the original
except it errors when trying to return the self process, in order to
forbid killing or signaling the interpreter process from script.
The os/exec symbols are available only through unrestricted package.
The original symbols are stored in an unrestricted package, which
requires an explicit Use, as for unsafe and syscall packages.
The Use() interpreter method has been slightly modified to allow inplace
updating of package symbols, allowing to replace some symbols but not
the entire imported package.
A command line option -unrestricted has been added to yaegi CLI to use
the unrestricted symbols.
Fixes#486.
* fix: lint
* fix: switch type from valueT in struct case
In a struct case in type assertion, if the source is a valueT, we still need to take the struct type to allow method and field resolution.
* fix: handle all ptr structs as well
In all situations where the results are set directly
to the frame, and not using a value helper, the right level of
indirections must be applied, otherwise we may end-up writing
in the wrong frame (the local one, instead of a caller or global).
Fixes#735.
* fix: copy the type in recursion
In more advanced recursive cases, setting the rtype to interface may interfear with typeing. To stop this from happening, instead of setting t.val.rtype to interface in the hope it will be set correctly later, a copy if the type is made, and the rtype of the copy is set to interface{}.
* fix: detect intermediate recursive structs
In the case of a nested recussion, each symbol can have
a different perspective on the recursion. In this case,
it is impossible to move from one struct to the next.
To keep the perspectives the same, any intermediate struct
that contains a recursion should also be set to interface{}.
so that all perspectives are the same.
* fix: handle arb recursion
* chore: refactor dref to be consistent
* fix: invalid recursive struct issue
* fix: handle checkptr issue
* fix: move unsafe into function to stop ptr check
* fix: handle deref in assign
When operations write their result to a non-local frame, the node
level field must be set accordingly, otherwise they attempt to write
in the wrong frame.
Fixes#730.
* fix: untyped constant cconverson to default type
In definition assign expression, the source type is propagated to
the assigned value. If the source is an untyped constant, the
destination type must be set to the default type of the constant
definition. A fixType function is provided to perform this.
In addition, the type conversion and check of constants is
refactored for simplifications.
Fixes#727.
* test: fix _test/const14.go
The standard library syscall package for Solaris defines unimplemented
symbols Syscall6 and RawSyscall6 which makes the build fails on
Solaris platform, now that yaegi command imports syscall symbols.
As the standard library package is locked down, this will remain
unchanged. We just skip those symbols.
Fixes#725.
* fix: make interpreter methods discoverable by runtime
When generating an interface wrapper, lookup existing wrappers by method
to get the one with the biggest set of methods implemented by interpreter.
A string method is also added to wrappers, in order to provide a string
representation of the interpreter value rather than the wrapper itself
(at least for %s and %v verbs).
This allows the runtime to pickup an interpreter method automatically
even if the conversion to the interface is not specified in the script. As
in Go spec, it is enough for the type to implement the required methods.
A current limitation is that only single wrappers can be instantiated,
not allowing to compose interfaces.
This limitation can be removed when the Go reflect issue
https://github.com/golang/go/issues/15924 is fixed.
Fixes#435.
* test: add a simpler test
* fix: detect default comm clause in select from AST
The heuristic to distinguish a default comm clause was too weak.
Make it robust by using AST.
Fixes#646.
* rename test to avoid conflict
* fix: perform send channel action for select
The CFG was wrong for select send comm clauses. If an init operation
on channel was required (like a derefence, index operation, ...) it
was skipped. The bug was invisible in case of a local var channel.
A send channel init operation consist to prepare both the data to
send (right subtree of send AST) and the channel itself (left
subtree of send AST). All channel init operation must be performed
prior to call select.
Fixes#647.
* doc: fix comment
* invert test to continue early
please note that this also changes the logic a little bit, since the
line that was:
if pn != nil {
now implictly becomes:
if an != nil && pn != nil {
(which I think is actually more correct).
* explicit chaining of init actions in select
* explicit chaining of init actions in select
Co-authored-by: mpl <mathieu.lonjaret@gmail.com>
* fix: finish support of type assertions which was incomplete
TypeAssert was optimistically returning ok without verifying that
value could be converted to the required interface (in case of
type assert of an interface type), or not checking the type in
all conditions. There is now a working implements method for itype.
Fixes#640.
* style: appease lint
* fix: remove useless code block
* doc: improve comments
* avoid test conflict
* interp: detect local redecleration
Fixes#666
* make make check happy
* fix typo in filenames, add them to the exclusion list
Co-authored-by: Marc Vertes <mvertes@free.fr>
* fix: use interface wrappers to expose interface values to runtime
If a value is assigned to, or returned as, a binary interface,
then use the interface wrapper generator to convert the value
accordingly.
Fixes#630.
* test: rename NewFoo in Foo
* test: rename NewFoo in Foo
* fix: improve branch flow to reduce indentation
* fix: correct iterator in map range on binary values
The map range iterator was not initialized correctly for values
originating from runtime.
Fixes#641.
* test: deterministic output for map28.go
* add repl input bounds check
* use bytes.Equal instead of Compare
* fix: use strings.HasPrefix to perform boundcheck on REPL input
This allows to keep most of the code unchanged and more readable.
Co-authored-by: Marc Vertes <mvertes@free.fr>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
