Status

[kddnewton]

This is a very general roadmap/status update that contains the latest information of the development of the new parser. The bullets are the nodes in syntax tree that loosely correspond to the nodes we're adding to this tree.

Tokens

• Backref - We're not matching the semantics of when these should be created or not because we need to check EXPR_FNAME.
• Backtick - We're not matching the semantics of when this is handled because CRuby will check lex state and potentially create an XStringBeg node.
• CHAR - YP_TOKEN_CHARACTER_LITERAL/StringNode
• CVar - YP_TOKEN_CLASS_VARIABLE/ClassVariableRead/ClassVariableWrite
• Comma - YP_TOKEN_COMMA
• Comment - yp_comment_t
• Const - YP_TOKEN_CONSTANT
• EmbDoc - yp_comment_t
• EmbVar - YP_TOKEN_EMBVAR
• EndContent - yp_comment_t
• FloatLiteral - YP_TOKEN_FLOAT/FloatLiteral
• GVar - YP_TOKEN_GLOBAL_VARIABLE/GlobalVariableRead/GlobalVariableWrite
• IVar - YP_TOKEN_INSTANCE_VARIABLE/InstanceVariableRead/InstanceVariableWrite
• Imaginary - YP_TOKEN_IMAGINARY_NUMBER/ImaginaryLiteral
• Int - YP_TOKEN_INTEGER/IntegerLiteral
• Ident - YP_TOKEN_IDENTIFIER
• Kw - This has been split out into individual keyword tokens for each keyword, which means: YP_TOKEN_KEYWORD___ENCODING__, YP_TOKEN_KEYWORD___LINE__, YP_TOKEN_KEYWORD___FILE__, YP_TOKEN_KEYWORD_ALIAS, YP_TOKEN_KEYWORD_AND, YP_TOKEN_KEYWORD_BEGIN, YP_TOKEN_KEYWORD_BEGIN_UPCASE, YP_TOKEN_KEYWORD_BREAK, YP_TOKEN_KEYWORD_CASE, YP_TOKEN_KEYWORD_CLASS, YP_TOKEN_KEYWORD_DEF, YP_TOKEN_KEYWORD_DEFINED, YP_TOKEN_KEYWORD_DO, YP_TOKEN_KEYWORD_ELSE, YP_TOKEN_KEYWORD_ELSIF, YP_TOKEN_KEYWORD_END, YP_TOKEN_KEYWORD_END_UPCASE, YP_TOKEN_KEYWORD_ENSURE, YP_TOKEN_KEYWORD_FALSE, YP_TOKEN_KEYWORD_FOR, YP_TOKEN_KEYWORD_IF, YP_TOKEN_KEYWORD_IN, YP_TOKEN_KEYWORD_MODULE, YP_TOKEN_KEYWORD_NEXT, YP_TOKEN_KEYWORD_NIL, YP_TOKEN_KEYWORD_NOT, YP_TOKEN_KEYWORD_OR, YP_TOKEN_KEYWORD_REDO, YP_TOKEN_KEYWORD_RESCUE, YP_TOKEN_KEYWORD_RETRY, YP_TOKEN_KEYWORD_RETURN, YP_TOKEN_KEYWORD_SELF, YP_TOKEN_KEYWORD_SUPER, YP_TOKEN_KEYWORD_THEN, YP_TOKEN_KEYWORD_TRUE, YP_TOKEN_KEYWORD_UNDEF, YP_TOKEN_KEYWORD_UNLESS, YP_TOKEN_KEYWORD_UNTIL, YP_TOKEN_KEYWORD_WHEN, YP_TOKEN_KEYWORD_WHILE, YP_TOKEN_KEYWORD_YIELD
• LBrace - YP_TOKEN_BRACE_LEFT
• LBracket - YP_TOKEN_BRACKET_LEFT
• LParen - YP_TOKEN_PARENTHESIS_LEFT
• Label - YP_TOKEN_LABEL
• LabelEnd - YP_TOKEN_LABEL_END
• Op - This has been split out into each individual operator, which means: YP_TOKEN_AMPERSAND, YP_TOKEN_AMPERSAND_AMPERSAND, YP_TOKEN_AMPERSAND_AMPERSAND_EQUAL, YP_TOKEN_AMPERSAND_EQUAL, YP_TOKEN_BANG, YP_TOKEN_BANG_AT, YP_TOKEN_BANG_EQUAL, YP_TOKEN_BANG_TILDE, YP_TOKEN_BRACKET_LEFT_RIGHT, YP_TOKEN_CARET, YP_TOKEN_CARET_EQUAL, YP_TOKEN_COLON, YP_TOKEN_COLON_COLON, YP_TOKEN_DOT, YP_TOKEN_DOT_DOT, YP_TOKEN_DOT_DOT_DOT, YP_TOKEN_EQUAL, YP_TOKEN_EQUAL_EQUAL, YP_TOKEN_EQUAL_EQUAL_EQUAL, YP_TOKEN_EQUAL_GREATER, YP_TOKEN_EQUAL_TILDE, YP_TOKEN_GREATER, YP_TOKEN_GREATER_EQUAL, YP_TOKEN_GREATER_GREATER, YP_TOKEN_GREATER_GREATER_EQUAL, YP_TOKEN_KEYWORD_AND, YP_TOKEN_KEYWORD_NOT, YP_TOKEN_KEYWORD_OR, YP_TOKEN_LESS, YP_TOKEN_LESS_EQUAL, YP_TOKEN_LESS_EQUAL_GREATER, YP_TOKEN_LESS_LESS, YP_TOKEN_LESS_LESS_EQUAL, YP_TOKEN_MINUS, YP_TOKEN_MINUS_AT, YP_TOKEN_MINUS_EQUAL, YP_TOKEN_MINUS_GREATER, YP_TOKEN_PERCENT, YP_TOKEN_PERCENT_EQUAL, YP_TOKEN_PIPE, YP_TOKEN_PIPE_EQUAL, YP_TOKEN_PIPE_PIPE, YP_TOKEN_PIPE_PIPE_EQUAL, YP_TOKEN_PLUS, YP_TOKEN_PLUS_AT, YP_TOKEN_PLUS_EQUAL, YP_TOKEN_QUESTION_MARK, YP_TOKEN_SLASH, YP_TOKEN_SLASH_EQUAL, YP_TOKEN_STAR, YP_TOKEN_STAR_EQUAL, YP_TOKEN_STAR_STAR, YP_TOKEN_STAR_STAR_EQUAL, YP_TOKEN_TILDE, YP_TOKEN_TILDE_AT
• Period - YP_TOKEN_DOT
• QSymbolsBeg - YP_TOKEN_PERCENT_LOWER_I
• QWordsBeg - YP_TOKEN_PERCENT_LOWER_W
• RBrace - YP_TOKEN_BRACE_RIGHT
• RBracket - YP_TOKEN_BRACKET_RIGHT
• RParen - YP_TOKEN_PARENTHESIS_RIGHT
• RationalLiteral - YP_TOKEN_RATIONAL_NUMBER/RationalLiteral
• RegexpBeg - YP_TOKEN_REGEXP_BEGIN
• RegexpEnd - YP_TOKEN_REGEXP_END
• SymBeg - YP_TOKEN_SYMBOL_BEGIN - This is definitely not being handled correctly as it needs to differentiate itself from a ternary operator through a bunch of state that we don't currently track.
• SymbolsBeg - YP_TOKEN_PERCENT_UPPER_I
• TLamBeg - YP_TOKEN_LAMBDA_BEGIN - We're not actually handling this correctly yet though, only in the cases where there are no parameters.
• TLambda - YP_TOKEN_MINUS_GREATER
• TStringBeg - YP_TOKEN_STRING_BEGIN
• TStringContent - YP_TOKEN_STRING_CONTENT
• TStringEnd - YP_TOKEN_STRING_END
• WordsBeg - YP_TOKEN_PERCENT_UPPER_W

Lists

• ArrayLiteral - ArrayNode
• QSymbols - ArrayNode
• QWords - ArrayNode
• Symbols - ArrayNode
• Word - StringNode/InterpolatedStringNode
• Words - ArrayNode

Blocks

• BEGINBlock - PreExecutionNode
• BlockArg - BlockParameterNode
• ClassDeclaration - ClassNode
• Def - DefNode
• DefEndless
• Defs
• ENDBlock - PostExecutionNode
• KwRestParam - KeywordRestParameterNode
• ModuleDeclaration - ModuleNode
• Params - ParametersNode
• RestParam - RestParameterNode
• SClass - SClassNode

Control flow

• Begin
• BodyStmt
• Break - BreakNode
• Case
• Else - ElseNode
• Elsif - IfNode
• Ensure - EnsureNode
• For - ForNode
• If/IfMod - IfNode
• IfOp - Ternary
• Next - NextNode
• Redo - RedoNode
• Rescue/RescueEx/RescueMod - RescueModifierNode/RescueNode
• Retry - RetryNode
• Return/Return0 - ReturnNode
• Super/ZSuper - SuperNode/ForwardingSuperNode
• Unless/UnlessMod - UnlessNode
• Until/UntilMod - UntilNode
• When
• While/WhileMod - WhileNode
• Yield/Yield0 - YieldNode

Expressions

• Alias - AliasNode
• Assign - CallNode, ClassVariableWrite, ConstantPathWriteNode, GlobalVariableWrite, InstanceVariableWrite, LocalVariableWrite
• Assoc - AssocNode
• AssocSplat - AssocSplatNode
• Binary - CallNode, AndNode, and OrNode
• ConstPathField/ConstPathRef - ConstantPathNode and ConstantPathWriteNode
• ConstRef - ConstantRead
• Defined - DefinedNode
• Dot2/Dot3 - RangeNode
• Field - CallNode
• HashLiteral - HashNode
• Lambda
• LambdaVar
• MAssign
• MLHS - MultiTargetNode
• MLHSParen
• MRHS
• Not - CallNode
• OpAssign - OperatorAssignmentNode, OperatorAndAssignmentNode, and OperatorOrAssignmentNode
• Paren - ParenthesesNode
• Program - Program
• Statements - Statements
• TopConstField/TopConstRef
• VarAlias - AliasNode
• VarField - LocalVariableWrite
• VarRef - TrueNode, FalseNode, NilNode, and LocalVariableRead.
• VoidStmt - Skipped
• Unary - UnaryNode
• Undef - UndefNode

Strings

• DynaSymbol - SymbolNode and InterpolatedSymbolNode
• EmbExprBeg/EmbExprEnd - These are StringInterpolated nodes now.
• Heredoc
• HeredocBeg
• HeredocEnd
• RegexpContent/RegexpLiteral - RegularExpressionNode/InterpolatedRegularExpressionNode
• StringConcat
• StringContent - This is effectively now wrapped up into StringNode.
• StringDVar - GlobalVariableRead/ClassVariableRead/InstanceVariableRead
• StringEmbExpr - StringInterpolatedNode
• StringLiteral - StringNode and InterpolatedStringNode
• SymbolContent/SymbolLiteral - SymbolNode
• XString/XStringLiteral - XStringNode/InterpolatedXStringNode

Method calls

• ARef/ARefField
• ArgBlock
• ArgParen
• ArgStar
• Args
• ArgsForward
• BareAssocHash
• BlockVar
• BraceBlock
• Call - CallNode
• Command
• CommandCall
• DoBlock
• ExcessedComma
• FCall - CallNode
• MethodAddBlock
• VCall - CallNode

Pattern matching

• AryPtn
• FndPtn
• HshPtn
• In
• PinnedBegin
• PinnedVarRef
• RAssign

[iliabylich]

A few questions:

Backref - We're not matching the semantics of when these should be created or not because we need to check EXPR_FNAME.

Are you going to emulate those states in YARP? I'd really like to turn lexer into (almost) stateless module if possible.

Comment - We're creating the tokens but doing nothing to ignore them in the parser yet. We'll need to handle these by creating a separate list that we can optionally attach to the AST/serialized list later.

WDYT about emitting tokens separately? They do not participate in AST construction at all. In lib-ruby-parser I moved them to a separate field on parser result.

LabelEnd - We haven't handled this yet.

What is this token responsible for?

Params

Please correct if I'm wrong but arguments is a part of the block/method declaration and parameters is what is passed to a method call, that's what I think I've in all parsers, but of course I can't find it explicitly stated anywhere. Is this node about method/block definition?

DefEndless

I'd recommend to merge it with Def/Defs

IfOp

What is this node?

Rescue/RescueEx/RescueMod

Can it be a single rescue node?

Yield/Yield0

I understand the difference between super/zsuper, but are these nodes different?

Assign

What is this node? Is for a setter call?

ConstPathField/ConstPathRef

LambdaVar

What are these nodes?

MLHS/MLHSParen

Can we merge these nodes into a single MLHS with optional begin/end locations?

Paren

Is it a node type? Is it for empty parentheses? Can it be a Statements node?

VarAlias

Is it for gvar/nthref alias? If so, can be it merged with Alias node?

VoidStmt

Or is this node for empty parentheses? If so same question as above: Can it be a Statements node?

EmbExprBeg

parser gem emits is as Begin (Statements in our case), can we do the same thing? Or is it a token type?

ArgParen

BlockVar

What is this node? Or is a token too?

MethodAddBlock

Are we going to introduce these nodes at all?

Sorry if the list wasn't ready. Feel free to ignore this comment if so, I'll update it later.

[kddnewton]

You can find the definitions of most of the nodes here: https://ruby-syntax-tree.github.io/syntax_tree/. For the most part they mirror the names that come from ripper. If I've listed the nodes together on the same line, I want to merge them into individual nodes. I'm planning on collapsing a bunch more nodes as well.

Comments/EmbDoc should go into a separate list that can be optionally attached to the AST.

[kddnewton]

@iliabylich yeah I'd like it to be stateless as well, but unfortunately there's a couple of really complicated semantics we're going to have to mirror to get 100% compatibility. Check this out: https://github.com/tric/trick2022/blob/master/04-tompng/remarks.markdown.

[eregon]

Please correct if I'm wrong but arguments is a part of the block/method declaration and parameters is what is passed to a method call, that's what I think I've in all parsers, but of course I can't find it explicitly stated anywhere. Is this node about method/block definition?

It's the opposite actually. A method/block definition has parameters. A call passes arguments to a method/block.
In Ruby docs the term arguments are often used for both, but when the term parameters is used it's always for the definition side, and notably the term was used about keyword-related changes in Ruby 3.0.
See https://en.wikipedia.org/wiki/Parameter_(computer_programming)#:~:text=Parameters%20appear%20in%20procedure%20definitions,an%20argument%20is%20an%20instance.

[kddnewton]

@eregon, @iliabylich, @enebo, @chrisseaton I've tagged v0.1.0 with a status report of everything I could think of. I've also updated this document with the current state of the project.

Obviously tons of stuff still to do, but I'm happy with the progress so far. If you have any glaring issues with the direction we're going in, let me know. Otherwise we'll keep trucking next week.

[kddnewton]

@eregon, @iliabylich, @enebo, @chrisseaton I've tagged v0.2.0 with a status report if you're interested.