Self hosted parser completion #873

Hejsil · 2018-03-30T22:57:51Z

This is in no way done, but I thought, that instead of just pushing my work on the self-hosted parser to master, I would make this PR. This allows me to write a lot of tests that fail, and slowly work towards getting them all passing.

Checklist!

When all these are checked and the self-hosted parser can parse the entire std library and all tests, then it's ready for a merge.

andrewrk · 2018-03-30T23:13:18Z

Great work so far. It's been really exciting to watch your progress

andrewrk · 2018-03-30T23:16:00Z

std/zig/parser.zig

+
+    try testCanonical(
+        \\test "test array" {
+        \\    const a : [2]u8 = [2]u8{ 1, 2 };


I think canonical var decl is:
const name: type = value;
(No space between colon and var name)

bnoordhuis · 2018-03-31T04:03:45Z

std/zig/parser.zig

+    );
+
+    try testCanonical(
+        \\test "percendence" {


'precedence' typo?

Hejsil · 2018-03-31T13:41:42Z

I've now added test cases for what I hope is all of Zig's syntax. I've also added a checklist with all everything that is missing from the self-hosted parser.

andrewrk · 2018-03-31T14:32:07Z

std/zig/parser.zig

+        \\    x += 1;
+        \\    suspend |p| {
+        \\    }
+        \\    const p = async simpleAsyncFn() cache unreachable;


I hate it when the cache is unreachable :)

Lol, yes :)

Hejsil · 2018-04-03T07:59:54Z

@andrewrk The variable declaration syntax in the self-hosted parser is currently not being parsed correctly. var a: u8 = 1; is parsed intovar a: ( u8 = 1 ) ;.

In the C++ compiler, this is solved by parsing type expressions as prefix operators first and therefore requiring parentheses for infix operators to be used in type expressions. This solution cannot be used in the self-hosted parser because of the way expressions are parsed.

Do we rewrite expression parsing to make this work or disallow assignment as an expression?

Hejsil · 2018-04-03T08:04:33Z

There is also the option of checking if the type expression is an assignment and then transform var a: (u8 = 0); -> var a: u8 = 0;. The only problem with this solution is, that we are using an arena as an allocator, so we cannot free the NodeInfixOp node.

Hejsil · 2018-04-03T08:09:25Z

Also, in the C++ compiler, this is allowed:

var a: u8 = 0;
const b = a = 2;

I don't think this has any use. @typeOf(b) == void, and I don't think that is what a person intended if they wrote this piece of code.

* Slice/Array access is now not parsed in the expr contruction loop * State.ExprListItemOrEnd now takes a token id for the end token

Hejsil · 2018-04-03T12:43:19Z

This is also a problem when implementing initializers :/

fn main() void {}

Gets parsed as:

fn main() (void {})

* I also moved some tests down, as they fail in ways I can't fix yet

* Now, the arraylist from the root node is passed through the states. * This allows us to reuse the code for enums, unions and structs

andrewrk · 2018-04-03T13:42:15Z

I think assignments should not be expressions.

Hejsil · 2018-04-03T13:47:46Z

Alright. Then we just need to solve #760 :)

…atements

* We cannot render asm expressions yet

* Only allowed in while continue expr and statement expr

* Ops!

* I parse it as a type in all contexts. This is not how the C++ compiler does it, but I think typechecking should catch this

andrewrk · 2018-04-10T15:36:51Z

From the appveyor build:
lld: error: undefined symbol: memmove

Wow, I've not seen this happen yet. I guess we need to add a memmove implementation to std/special/builtin.zig.

see 0ba85ea

related: #514

andrewrk · 2018-04-11T02:47:47Z

All tests passing, merged!

Let's open new issues for whatever's left.

tiehuis · 2018-04-11T05:40:40Z

Great work, @Hejsil.

Added a lot of test cases

5118caf

andrewrk reviewed Mar 30, 2018

View reviewed changes

bnoordhuis reviewed Mar 31, 2018

View reviewed changes

std/zig/parser.zig Outdated

);

try testCanonical(

\\test "percendence" {

Copy link

Contributor

bnoordhuis Mar 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'precedence' typo?

Hejsil added 3 commits March 31, 2018 14:00

Fixed review commented code

596f4b6

Each test now have it's own test name

26e56f2

Added test cases to cover all of zigs syntax

cda3509

andrewrk reviewed Mar 31, 2018

View reviewed changes

Hejsil added 6 commits March 31, 2018 17:46

std.zig.parser now handles lib name for extern var and fn

4793c33

std.zig.parser now parses multi line strings

4d8f9e2

std.zig.parser now parses char literals

975dc5a

std.zig.parser now parses null and bool literals

aabf7cf

std.zig.parser now parses error, this and unreachable

df09c01

std.zig.parser now parses slicing and array access

b909318

Hejsil added 4 commits April 3, 2018 10:54

std.zig.parser now parses slice and array types

a2330d0

std.zig.tokenizer fixed tokens having wrong column and line

22e38ff

std.zig.parser refactored call, slice and array access to be suffix op

b424cd7

std.zig.parser Refactor:

0b9247f

* Slice/Array access is now not parsed in the expr contruction loop * State.ExprListItemOrEnd now takes a token id for the end token

Hejsil added 4 commits April 3, 2018 14:53

std.zig.parser now parses initializers... Or, it would, if it worked

5c82ed2

std.zig.parser now parses grouped expressions

9d69e94

* I also moved some tests down, as they fail in ways I can't fix yet

std.zig.parser moved container initializer tests down

40f35e9

std.zig.parser Refactored top level decl parsing

4fae452

* Now, the arraylist from the root node is passed through the states. * This allows us to reuse the code for enums, unions and structs

std.zig.ast Added ContainerDecl

d602f12

Hejsil and others added 19 commits April 9, 2018 11:17

std.zig.parser unified code for rendering and parsing semicolon in st…

e24409e

…atements

std.zig.parser now parses for loops

7dd55a8

std.zig.parser now parses if statements

c19f5a2

ast.zig.parser now parses defer statements

d04346d

std.zig.parser now parses comptime

7d32c95

std.zig.parser now parses fn types

aa55263

std.zig.parser now parses asm expressions

a09bb40

* We cannot render asm expressions yet

std.zig.parser can now render asm expressions

2c7996f

std.zig.parser changed assign expr to only be allowed in some contexts

5cd69ee

* Only allowed in while continue expr and statement expr

std.zig.parser now parses coroutine code

f85b9f2

std.zig.tokinizer: fixed failing tests

34af38e

std.zig: fixed compiler errors

1b81e40

std.zig.parser readded all tests

706e0d7

* Ops!

std.zig.parser: changed block exprs from primary expr to expr

db0812d

std.zig.parser now parses toplevel use

3b80e66

std.zig.tokinizer now treats string identifiers as identifiers

aa09e7b

std.zig.parser now parses the var type

db9a9f3

* I parse it as a type in all contexts. This is not how the C++ compiler does it, but I think typechecking should catch this

std.zig.ast: fixed none compiling code

b9cccce

std.zig.parser: removed dublicate "zig fmt: coroutines" test

c6aa637

Hejsil and others added 5 commits April 10, 2018 17:46

std.zig.parser fixed segfault when parsing cc for fn decl

0ba85ea

Merge branch 'master' into self-hosted-parser

b553b7a

fix another undefined deref

27e881c

see 0ba85ea

add memmove to builtin.o

f6c7774

related: #514

zig fmt while-else with no blocks

405a239

andrewrk merged commit 64d96ad into master Apr 11, 2018

andrewrk deleted the self-hosted-parser branch April 11, 2018 02:47

Hejsil mentioned this pull request Apr 11, 2018

self-hosted parsing std progress #909

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Self hosted parser completion #873

Self hosted parser completion #873

Hejsil commented Mar 30, 2018 •

edited

andrewrk commented Mar 30, 2018

andrewrk Mar 30, 2018

bnoordhuis Mar 31, 2018

Hejsil commented Mar 31, 2018

andrewrk Mar 31, 2018

Hejsil Mar 31, 2018

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

andrewrk commented Apr 3, 2018

Hejsil commented Apr 3, 2018

andrewrk commented Apr 10, 2018

andrewrk commented Apr 11, 2018 •

edited

tiehuis commented Apr 11, 2018

Self hosted parser completion #873

Self hosted parser completion #873

Conversation

Hejsil commented Mar 30, 2018 • edited

andrewrk commented Mar 30, 2018

andrewrk Mar 30, 2018

Choose a reason for hiding this comment

bnoordhuis Mar 31, 2018

Choose a reason for hiding this comment

Hejsil commented Mar 31, 2018

andrewrk Mar 31, 2018

Choose a reason for hiding this comment

Hejsil Mar 31, 2018

Choose a reason for hiding this comment

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

Hejsil commented Apr 3, 2018

andrewrk commented Apr 3, 2018

Hejsil commented Apr 3, 2018

andrewrk commented Apr 10, 2018

andrewrk commented Apr 11, 2018 • edited

tiehuis commented Apr 11, 2018

Hejsil commented Mar 30, 2018 •

edited

andrewrk commented Apr 11, 2018 •

edited