🚀 Socket Launch Week Day 5:Introducing Repository Access Permissions and Custom Roles.Learn more →
Sign In

ruby_parser

Package Overview
Dependencies
Maintainers
1
Versions
80
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

ruby_parser - rubygems Package Compare versions

Comparing version
3.12.0
 to
3.13.0
+18
debugging.md
# Quick Notes to Help with Debugging
## Comparing against ruby / ripper:
```
% rake cmp3 F=file.rb
```
This compiles the parser & lexer and then parses file.rb using both
ruby, ripper, and ruby_parser in debug modes. The output is munged to
be as uniform as possible and diffable. I'm using emacs'
`ediff-files3` to compare these files (via `rake cmp3`) all at once,
but regular `diff -u tmp/{ruby,rp}` will suffice for most tasks.
From there? Good luck. I'm currently trying to backtrack from rule
reductions to state change differences. I'd like to figure out a way
to go from this sort of diff to a reasonable test that checks state
changes but I don't have that set up at this point.
lib/ruby26_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby26_parser.y

Sorry, the diff of this file is not supported yet

+216
tools/munge.rb
#!/usr/bin/ruby -ws
$v ||= false
stack = []
last_token = nil
reduce_line = nil
def munge s
renames = [
"'='", "tEQL",
"'!'", "tBANG",
"'%'", "tPERCENT",
"'&'", "tAMPER2",
"'('", "tLPAREN2",
"')'", "tRPAREN",
"'*'", "tSTAR2",
"'+'", "tPLUS",
"','", "tCOMMA",
"'-'", "tMINUS",
"'.'", "tDOT",
"'/'", "tDIVIDE",
"';'", "tSEMI",
"':'", "tCOLON",
"'<'", "tLT",
"'>'", "tGT",
"'?'", "tEH",
"'['", "tLBRACK",
"'\\n'", "tNL",
"']'", "tRBRACK",
"'^'", "tCARET",
"'`'", "tBACK_REF2",
"'{'", "tLCURLY",
"'|'", "tPIPE",
"'}'", "tRCURLY",
"'~'", "tTILDE",
'"["', "tLBRACK",
# 2.0 changes?
'"<=>"', "tCMP",
'"=="', "tEQ",
'"==="', "tEQQ",
'"!~"', "tNMATCH",
'"=~"', "tMATCH",
'">="', "tGEQ",
'"<="', "tLEQ",
'"!="', "tNEQ",
'"<<"', "tLSHFT",
'">>"', "tRSHFT",
'"*"', "tSTAR",
'".."', "tDOT2",
'"&"', "tAMPER",
'"&&"', "tANDOP",
'"&."', "tLONELY",
'"||"', "tOROP",
'"..."', "tDOT3",
'"**"', "tPOW",
'"unary+"', "tUPLUS",
'"unary-"', "tUMINUS",
'"[]"', "tAREF",
'"[]="', "tASET",
'"::"', "tCOLON2",
'"{ arg"', "tLBRACE_ARG",
'"( arg"', "tLPAREN_ARG",
'"("', "tLPAREN",
'rparen', "tRPAREN",
'"{"', "tLBRACE",
'"=>"', "tASSOC",
'"->"', "tLAMBDA",
'":: at EXPR_BEG"', "tCOLON3",
'"**arg"', "tDSTAR",
'","', "tCOMMA",
# other
'tLBRACK2', "tLBRACK", # HACK
"' '", "tSPACE", # needs to be later to avoid bad hits
"/* empty */", "none",
/^\s*$/, "none",
"keyword_BEGIN", "klBEGIN",
"keyword_END", "klEND",
/keyword_(\w+)/, proc { "k#{$1.upcase}" },
/\bk_([a-z_]+)/, proc { "k#{$1.upcase}" },
/modifier_(\w+)/, proc { "k#{$1.upcase}_MOD" },
"kVARIABLE", "keyword_variable", # ugh
# 2.6 collapses klBEGIN to kBEGIN
"klBEGIN", "kBEGIN",
"klEND", "kEND",
/keyword_(\w+)/, proc { "k#{$1.upcase}" },
/\bk_([^_][a-z_]+)/, proc { "k#{$1.upcase}" },
/modifier_(\w+)/, proc { "k#{$1.upcase}_MOD" },
"kVARIABLE", "keyword_variable", # ugh: this is a rule name
# UGH
"k_LINE__", "k__LINE__",
"k_FILE__", "k__FILE__",
"k_ENCODING__", "k__ENCODING__",
'"defined?"', "kDEFINED",
'"do (for condition)"', "kDO_COND",
'"do (for lambda)"', "kDO_LAMBDA",
'"do (for block)"', "kDO_BLOCK",
/\"(\w+) \(modifier\)\"/, proc { |x| "k#{$1.upcase}_MOD" },
/\"(\w+)\"/, proc { |x| "k#{$1.upcase}" },
/@(\d+)(\s+|$)/, "",
/\$?@(\d+) */, "", # TODO: remove?
]
renames.each_slice(2) do |(a, b)|
if Proc === b then
s.gsub!(a, &b)
else
s.gsub!(a, b)
end
end
s.strip.squeeze " "
end
ARGF.each_line do |line|
case line
when /^(Stack now|Entering state|Shifting|Cleanup|Starting)/ then
# do nothing
when /^vtable_/ then
# do nothing
when /Gem::MissingSpecError/ then
# do nothing -- ruby 2.5 is being bitchy?
when /^Reading a token: Next token is token (.*?) \(\)/ then
token = munge $1
next if last_token == token
puts "next token is %p (%p)" % [token, last_token]
last_token = token
when /^Reading a token: / then
next # skip
when /^read\s+:(\w+)/ then # read :tNL(tNL) nil
token = munge $1
next if last_token == token
puts "next token is %p (%p)" % [token, last_token]
last_token = token
when /^Next token is token (\S+)/ then
token = munge $1
next if last_token == token
puts "next token is %p (%p)" % [token, last_token]
last_token = token
when /^read\s+false/ then # read false($end) "$end"
puts "next token is EOF"
when /^Now at end of input./ then
# do nothing
when /^.:scan=>\["([^"]+)"/ then
puts "scan = %p" % [$1]
when /^Reducing stack by rule (\d+) \(line (\d+)\):/ then
reduce_line = $2.to_i
when /^ \$\d+ = (?:token|nterm) (.+) \(.*\)/ then
item = $1
stack << munge(item)
when /^-> \$\$ = (?:token|nterm) (.+) \(.*\)/ then
stack << "<none>" if stack.empty?
item = munge $1
x = stack.map { |s| s.strip }.join " "
if x != item then # prevent kdef -> kdef
if $v && reduce_line then
puts "reduce #{x} --> #{item} at #{reduce_line}".squeeze " "
else
puts "reduce #{x} --> #{item}".squeeze " "
end
puts
end
reduce_line = nil
stack.clear
when /^reduce/ then # ruby_parser side
puts munge line.chomp
puts
when /^(\w+_stack)\.(\w+)/ then
# TODO: make pretty, but still informative w/ line numbers etc
puts line.gsub("true", "1").gsub("false", "0")
# puts "#{$1}(#{$2})"
when /^(\w+_stack(\(\w+\))?: \S+)/ then
# _data = $v ? line.chomp : $1
# puts line
# TODO: make pretty, but still informative w/ line numbers etc
puts line.gsub("true", "1").gsub("false", "0")
when /^lex_state: :?([\w|]+) -> :?([\w|]+)(?: (?:at|from) (.*))?/ then
if $3 && $v then
puts "lex_state: #{$1.upcase} -> #{$2.upcase} at #{$3}"
else
puts "lex_state: #{$1.upcase} -> #{$2.upcase}"
end
when /debug|FUCK/ then
puts line.chomp
when /^(#.*parse error|on )/ then
puts line.chomp
when /^(goto|shift| +\[|$)/ then # racc
# do nothing
# when /^Reading a token: Now at end of input./ then
# # puts "EOF"
# when /^Reading a token: Next token is token (.+)/ then
# puts "READ: #{$1.inspect}"
when /^accept/ then
puts "DONE"
else
puts "unparsed: #{line.chomp}"
end
end
+23
tools/ripper.rb
#!/usr/bin/env ruby -ws
$d ||= false
$p ||= false
require "ripper/sexp"
require "pp" if $p
ARGV.each do |path|
src = File.read path
rip = Ripper::SexpBuilderPP.new src
rip.yydebug = $d
sexp = rip.parse
puts "accept" unless rip.error?
if $p then
pp sexp
else
p sexp
end
end
+29
-2
compare/normalize.rb

@@ -1,2 +0,2 @@

#!/usr/bin/ruby -w
#!/usr/bin/env ruby -w

@@ -56,2 +56,3 @@ good = false

'"&&"', "tANDOP",
'"&."', "tLONELY",
'"||"', "tOROP",

@@ -85,2 +86,3 @@

/^\s*$/, "none",
"keyword_BEGIN", "klBEGIN",

@@ -93,3 +95,28 @@ "keyword_END", "klEND",

/@(\d+)\s+/, "",
# 2.6 collapses klBEGIN to kBEGIN
"klBEGIN", "kBEGIN",
"klEND", "kEND",
/keyword_(\w+)/, proc { "k#{$1.upcase}" },
/\bk_([^_][a-z_]+)/, proc { "k#{$1.upcase}" },
/modifier_(\w+)/, proc { "k#{$1.upcase}_MOD" },
"kVARIABLE", "keyword_variable", # ugh: this is a rule name
# UGH
"k_LINE__", "k__LINE__",
"k_FILE__", "k__FILE__",
"k_ENCODING__", "k__ENCODING__",
'"defined?"', "kDEFINED",
'"do (for condition)"', "kDO_COND",
'"do (for lambda)"', "kDO_LAMBDA",
'"do (for block)"', "kDO_BLOCK",
/\"(\w+) \(modifier\)\"/, proc { |x| "k#{$1.upcase}_MOD" },
/\"(\w+)\"/, proc { |x| "k#{$1.upcase}" },
/@(\d+)(\s+|$)/, "",
]

@@ -96,0 +123,0 @@ 

+47
-0
History.rdoc

@@ -0,1 +1,48 @@

=== 3.13.0 / 2019-03-12
* 3 major enhancements:
* Removed 1.8 and 1.9 support. Moved to ruby_parser-legacy gem.
* Added tentative 2.6 support.
* Updated ruby_parser.yy to ruby 2.4 architecture.
* 22 minor enhancements:
* Added debug3 and cmp3 rake tasks that use the above tools.
* Added tLONELY to compare/normalize.rb
* Added tools/munge.rb, tools/ripper.rb, and debugging.md
* 2.6: Added trailing .. and ... support.
* Extended StackState to log more state changes, making debugging easier.
* Extended StackState#store to take an initial value.
* Improved logging / debugging in StackState.
* Improved normalization and parser compare tasks.
* Improved tools/munge.rb output.
* In f_arglist, track in_kwarg if no parens.
* In process_newline_or_comment, handle NL if in_kwarg there vs normal.
* Refactored normalized values to WORDLIST.
* Refactored parser: push up to relop and rel_expr.
* Removed Keyword.keyword18.
* Removed RubyLexer version attr_accessors.
* Removed long dead deprecations internal to RubyParserStuff.
* Removed version arg to RubyLexer#initialize (default nil + no-op).
* Renamed Keyword.keyword19 to Keyword.keyword.
* Renamed process_bracing to process_brace_close
* Renamed process_curly_brace to process_brace_open
* Report first parse error message if all parser versions fail, not last.
* Updated parser to track against 2.6.
* 11 bug fixes:
* Fix some shift/reduce errors.
* Fixed BEGIN blocks having different arg value that END blocks. (mvz)
* Fixed all reported unused non-terminals/rules.
* Fixed bug 272.
* Fixed bug in interpolated symbol lists. (strviola)
* Fixed bug where block shadow arguments were not registered as lvars. (akimd)
* Fixed bug where kwsplat args weren't treated as lvars. (mvz)
* Fixed lex_state and other internals in many cases.
* Fixed shebang in compare/normalize.rb to use env. Avoids rubygems freaking.
* Fixed some more internal state bugs.
* Fixed tRCURLY -> tSTRING_DEND for interpolation brace closes.
=== 3.12.0 / 2018-12-04

@@ -2,0 +49,0 @@ 

+0
-7
lib/rp_extensions.rb

@@ -13,9 +13,2 @@ # :stopdoc:

end
# I hate ruby 1.9 string changes
class Fixnum
def ord
self
end
end unless "a"[0] == "a"
# :startdoc:

@@ -22,0 +15,0 @@ 

+116
-118
lib/ruby_lexer.rb

@@ -37,2 +37,6 @@ # frozen_string_literal: true

EXPR_BEG_ANY = [:expr_beg, :expr_mid, :expr_class ]
EXPR_ARG_ANY = [:expr_arg, :expr_cmdarg, ]
EXPR_END_ANY = [:expr_end, :expr_endarg, :expr_endfn]
ESCAPES = {

@@ -81,3 +85,3 @@ "a" => "\007",

attr_accessor :command_start
attr_accessor :command_state
attr_accessor :cmd_state # temporary--ivar to avoid passing everywhere
attr_accessor :last_state

@@ -114,16 +118,9 @@ attr_accessor :cond

##
# What version of ruby to parse. 18 and 19 are the only valid values
# currently supported.
attr_accessor :version
attr_writer :comments
def initialize v = 18
self.version = v
def initialize _ = nil
@lex_state = :expr_none
self.cond = RubyParserStuff::StackState.new(:cond, $DEBUG)
self.cmdarg = RubyParserStuff::StackState.new(:cmdarg, $DEBUG)
self.cond = RubyParserStuff::StackState.new(:cond, $DEBUG)

@@ -349,13 +346,20 @@ reset

def is_arg?
in_lex_state? :expr_arg, :expr_cmdarg
in_lex_state?(*EXPR_ARG_ANY)
end
def is_beg?
in_lex_state? :expr_beg, :expr_value, :expr_mid, :expr_class, :expr_labelarg
# TODO: in_lex_state?(*EXPR_BEG_ANY) || lex_state == [:expr_arg, :expr_labeled]
in_lex_state?(*EXPR_BEG_ANY, :expr_value, :expr_labeled)
end
def is_end?
in_lex_state? :expr_end, :expr_endarg, :expr_endfn
in_lex_state?(*EXPR_END_ANY)
end
def lvar_defined? id
# TODO: (dyna_in_block? && dvar_defined?(id)) || local_id?(id)
self.parser.env[id.to_sym] == :lvar
end
def ruby22_label?

@@ -366,3 +370,3 @@ ruby22plus? and is_label_possible?

def is_label_possible?
(in_lex_state?(:expr_beg, :expr_endfn) && !command_state) || is_arg?
(in_lex_state?(:expr_beg, :expr_endfn) && !cmd_state) || is_arg?
end

@@ -378,2 +382,6 @@

def lambda_beginning?
lpar_beg && lpar_beg == paren_nest
end
def matched

@@ -420,3 +428,4 @@ ss.matched

def process_bracing text
def process_brace_close text
# matching compare/parse23.y:8561
cond.lexpop

@@ -428,9 +437,5 @@ cmdarg.lexpop

self.brace_nest -= 1
self.lex_state = :expr_endarg
self.lex_state = :expr_endarg # TODO: :expr_end ? Look at 2.6
# TODO
# if (c == '}') {
# if (!brace_nest--) c = tSTRING_DEND;
# }
return :tSTRING_DEND, matched if brace_nest < 0
return :tRCURLY, matched

@@ -474,7 +479,9 @@ when "]" then

def process_curly_brace text
def process_brace_open text
# matching compare/parse23.y:8694
self.brace_nest += 1
if lpar_beg && lpar_beg == paren_nest then
if lambda_beginning? then
self.lpar_beg = nil
self.paren_nest -= 1
self.paren_nest -= 1 # close arg list when lambda opens body

@@ -484,11 +491,15 @@ return expr_result(:tLAMBEG, "{")

token = if is_arg? || in_lex_state?(:expr_end, :expr_endfn) then
:tLCURLY # block (primary)
elsif in_lex_state?(:expr_endarg) then
:tLBRACE_ARG # block (expr)
else
:tLBRACE # hash
end
token = case lex_state
when :expr_labeled then
:tLBRACE # hash
when *EXPR_ARG_ANY, :expr_end, :expr_endfn then
:tLCURLY # block (primary)
when :expr_endarg
:tLBRACE_ARG # block (expr)
else
:tLBRACE # hash
end
self.command_start = true unless token == :tLBRACE
# TODO: self.lex_state |= :expr_label if token != :tLBRACE_ARG
self.command_start = true if token != :tLBRACE

@@ -532,3 +543,3 @@ return expr_result(token, "{")

!is_end? &&
(!is_arg? || space_seen)) then
(!is_arg? || space_seen)) then # TODO: || in_state(:expr_labeled)
tok = self.heredoc_identifier

@@ -538,3 +549,10 @@ return tok if tok

return result(:arg_state, :tLSHFT, "\<\<")
if in_arg_state? then
self.lex_state = :expr_arg
else
self.command_start = true if lex_state == :expr_class
self.lex_state = :expr_beg
end
return result(lex_state, :tLSHFT, "\<\<")
end

@@ -549,2 +567,3 @@

# TODO: handle magic comments
while scan(/\s*\#.*(\n+|\z)/) do

@@ -564,5 +583,18 @@ hit = true

return if in_lex_state?(:expr_beg, :expr_value, :expr_class,
:expr_fname, :expr_dot)
# TODO: remove :expr_value -- audit all uses of it
c = in_lex_state?(:expr_beg, :expr_value, :expr_class,
:expr_fname, :expr_dot) && !in_lex_state?(:expr_labeled)
# TODO: figure out what token_seen is for
# TODO: if c || self.lex_state == [:expr_beg, :expr_labeled] then
if c || self.lex_state == :expr_labeled then
# ignore if !fallthrough?
if !c && parser.in_kwarg then
# normal newline
return result(:expr_beg, :tNL, nil)
else
return # skip
end
end
if scan(/([\ \t\r\f\v]*)(\.|&)/) then

@@ -586,7 +618,3 @@ self.space_seen = true unless ss[1].empty?

def process_paren text
token = if ruby18 then
process_paren18
else
process_paren19
end
token = process_paren19

@@ -599,21 +627,2 @@ self.paren_nest += 1

def process_paren18
self.command_start = true
token = :tLPAREN2
if in_lex_state? :expr_beg, :expr_mid then
token = :tLPAREN
elsif space_seen then
if in_lex_state? :expr_cmdarg then
token = :tLPAREN_ARG
elsif in_lex_state? :expr_arg then
warning "don't put space before argument parentheses"
end
else
# not a ternary -- do nothing?
end
token
end
def process_paren19

@@ -673,4 +682,3 @@ if is_beg? then

if is_end? then
state = ruby18 ? :expr_beg : :expr_value # HACK?
return result(state, :tEH, "?")
return result(:expr_value, :tEH, "?")
end

@@ -697,4 +705,3 @@

# ternary
state = ruby18 ? :expr_beg : :expr_value # HACK?
return result(state, :tEH, "?")
return result(:expr_value, :tEH, "?")
elsif check(/\w(?=\w)/) then # ternary, also

@@ -710,7 +717,3 @@ return result(:expr_beg, :tEH, "?")

if version == 18 then
return result(:expr_end, :tINTEGER, c[0].ord & 0xff)
else
return result(:expr_end, :tSTRING, c)
end
return result(:expr_end, :tSTRING, c)
end

@@ -764,3 +767,5 @@

return expr_result(token, "[")
# TODO: this is done by expr_result except "|EXPR_LABEL")
# SET_LEX_STATE(EXPR_BEG|EXPR_LABEL);
expr_result token, "["
end

@@ -781,5 +786,2 @@

rb_compile_error "symbol cannot contain '\\0'" if
ruby18 && symbol =~ /\0/
return result(:expr_end, :tSYMBOL, symbol)

@@ -808,7 +810,10 @@ end

result(:expr_labelarg, :tLABEL, [symbol, self.lineno])
result(:expr_labeled, :tLABEL, [symbol, self.lineno]) # TODO: expr_arg|expr_labeled
end
def process_token text
# matching: parse_ident in compare/parse23.y:7989
# TODO: make this always return [token, lineno]
self.last_state = lex_state
token = self.token = text

@@ -832,14 +837,12 @@ token << matched if scan(/[\!\?](?!=)/)

if !ruby18 and is_label_possible? and is_label_suffix? then
if is_label_possible? and is_label_suffix? then
scan(/:/)
return result(:expr_labelarg, :tLABEL, [token, self.lineno])
# TODO: :expr_arg|:expr_labeled
return result :expr_labeled, :tLABEL, [token, self.lineno]
end
# TODO: mb == ENC_CODERANGE_7BIT && !in_lex_state?(:expr_dot)
unless in_lex_state? :expr_dot then
# See if it is a reserved word.
keyword = if ruby18 then # REFACTOR need 18/19 lexer subclasses
RubyParserStuff::Keyword.keyword18 token
else
RubyParserStuff::Keyword.keyword19 token
end
keyword = RubyParserStuff::Keyword.keyword token

@@ -849,8 +852,6 @@ return process_token_keyword keyword if keyword

# TODO:
# if (mb == ENC_CODERANGE_7BIT && lex_state != EXPR_DOT) {
# matching: compare/parse23.y:8079
state = if is_beg? or is_arg? or in_lex_state? :expr_dot then
command_state ? :expr_cmdarg : :expr_arg
elsif not ruby18 and in_lex_state? :expr_fname then
cmd_state ? :expr_cmdarg : :expr_arg
elsif in_lex_state? :expr_fname then
:expr_endfn

@@ -862,4 +863,5 @@ else

if not [:expr_dot, :expr_fname].include? last_state and
self.parser.env[token.to_sym] == :lvar then
state = :expr_end
(tok_id == :tIDENTIFIER) and # not :expr_fname, not attrasgn
lvar_defined?(token) then
state = :expr_end # TODO: EXPR_END|EXPR_LABEL
end

@@ -873,34 +875,34 @@

def process_token_keyword keyword
state = keyword.state
# matching MIDDLE of parse_ident in compare/parse23.y:8046
state = lex_state
self.lex_state = keyword.state
value = [token, self.lineno]
self.command_start = true if state == :expr_beg and lex_state != :expr_fname
return result(lex_state, keyword.id0, value) if state == :expr_fname
self.command_start = true if lex_state == :expr_beg
case
when lex_state == :expr_fname then
result(state, keyword.id0, keyword.name)
when keyword.id0 == :kDO then
case
when lpar_beg && lpar_beg == paren_nest then
self.lpar_beg = nil
when lambda_beginning? then
self.lpar_beg = nil # lambda_beginning? == FALSE in the body of "-> do ... end"
self.paren_nest -= 1
expr_result(:kDO_LAMBDA, value)
result(lex_state, :kDO_LAMBDA, value)
when cond.is_in_state then
result(state, :kDO_COND, value)
when cmdarg.is_in_state && lex_state != :expr_cmdarg then
result(state, :kDO_BLOCK, value)
when in_lex_state?(:expr_beg, :expr_endarg) then
result(state, :kDO_BLOCK, value)
when lex_state == :expr_end # eg: a -> do end do end
result(state, :kDO_BLOCK, value)
result(lex_state, :kDO_COND, value)
when cmdarg.is_in_state && state != :expr_cmdarg then
result(lex_state, :kDO_BLOCK, value)
when [:expr_beg, :expr_endarg].include?(state) then
result(lex_state, :kDO_BLOCK, value)
else
result(state, :kDO, value)
result(lex_state, :kDO, value)
end
when in_lex_state?(:expr_beg, :expr_value, :expr_labelarg) then
result(state, keyword.id0, value)
when [:expr_beg, :expr_labeled].include?(state) then
result(lex_state, keyword.id0, value)
when keyword.id0 != keyword.id1 then
result(:expr_beg, keyword.id1, value)
result(:expr_beg, keyword.id1, value) # TODO: :expr_beg|:expr_label
else
result(state, keyword.id1, value)
result(lex_state, keyword.id1, value)
end

@@ -1011,4 +1013,4 @@ end

self.cond.reset
self.cmdarg.reset
self.cond.reset
end

@@ -1022,6 +1024,2 @@

def ruby18
RubyParser::V18 === parser
end
def scan re

@@ -1164,7 +1162,3 @@ ss.scan re

re = if qwords then
if HAS_ENC then
/[^#{t}#{x}\#\0\\\s]+|./ # |. to pick up whatever
else
/[^#{t}#{x}\#\0\\\s\v]+|./ # argh. 1.8's \s doesn't pick up \v
end
/[^#{t}#{x}\#\0\\\s]+|./ # |. to pick up whatever
else

@@ -1233,2 +1227,3 @@ /[^#{t}#{x}\#\0\\]+|./

def process_string # TODO: rewrite / remove
# matches top of parser_yylex in compare/parse23.y:8113
token = if lex_strterm[0] == :heredoc then

@@ -1242,2 +1237,3 @@ self.heredoc lex_strterm

# matches parser_string_term
if ruby22plus? && token_type == :tSTRING_END && ["'", '"'].include?(c) then

@@ -1254,3 +1250,4 @@ if (([:expr_beg, :expr_endfn].include?(lex_state) &&

self.lex_strterm = nil
self.lex_state = (token_type == :tLABEL_END) ? :expr_labelarg : :expr_end
# TODO: :expr_beg|:expr_label
self.lex_state = (token_type == :tLABEL_END) ? :expr_label : :expr_end
end

@@ -1361,2 +1358,3 @@

when scan(/#[{]/) then
self.command_start = true
return :tSTRING_DBEG, nil

@@ -1363,0 +1361,0 @@ when scan(/#/) then

+8
-8
lib/ruby_lexer.rex.rb

@@ -73,5 +73,5 @@ # encoding: UTF-8

return process_string if lex_strterm
self.command_state = self.command_start
self.cmd_state = self.command_start
self.command_start = false
self.space_seen = false
self.space_seen = false # TODO: rename token_seen?
self.last_state = lex_state

@@ -91,3 +91,3 @@

when text = ss.scan(/[\]\)\}]/) then
process_bracing text
process_brace_close text
when ss.match?(/\!/) then

@@ -107,3 +107,3 @@ case

when ss.skip(/\./) then
action { result :expr_dot, :tDOT, "." }
action { self.lex_state = :expr_beg; result :expr_dot, :tDOT, "." }
end # group /\./

@@ -189,3 +189,3 @@ when text = ss.scan(/\(/) then

when text = ss.scan(/\{/) then
process_curly_brace text
process_brace_open text
when ss.match?(/\*/) then

@@ -209,3 +209,3 @@ case

when ss.skip(/\<\<\=/) then
action { result :arg_state, :tOP_ASGN, "<<" }
action { result :expr_beg, :tOP_ASGN, "<<" }
when text = ss.scan(/\<\</) then

@@ -221,3 +221,3 @@ process_lchevron text

when ss.skip(/\>\>=/) then
action { result :arg_state, :tOP_ASGN, ">>" }
action { result :expr_beg, :tOP_ASGN, ">>" }
when ss.skip(/\>\>/) then

@@ -233,3 +233,3 @@ action { result :arg_state, :tRSHFT, ">>" }

when expr_dot? && (ss.skip(/\`/)) then
action { result((command_state ? :expr_cmdarg : :expr_arg), :tBACK_REF2, "`") }
action { result((cmd_state ? :expr_cmdarg : :expr_arg), :tBACK_REF2, "`") }
when ss.skip(/\`/) then

@@ -236,0 +236,0 @@ action { string STR_XQUOTE, '`'; result(nil, :tXSTRING_BEG, "`") }

+88
-106
lib/ruby_parser_extras.rb

@@ -10,5 +10,6 @@ # encoding: ASCII-8BIT

module RubyParserStuff
VERSION = "3.12.0"
VERSION = "3.13.0"
attr_accessor :lexer, :in_def, :in_single, :file
attr_accessor :in_kwarg
attr_reader :env, :comments

@@ -94,19 +95,2 @@

def block_var18 ary, splat, block
ary ||= s(:array)
if splat then
splat = splat[1] unless Symbol === splat
ary << "*#{splat}".to_sym
end
ary << "&#{block[1]}".to_sym if block
if ary.length > 2 or ary.splat then # HACK
s(:masgn, *ary.sexp_body)
else
ary.last
end
end
def array_to_hash array

@@ -157,4 +141,6 @@ case array.sexp_type

when :shadow then
name = arg.last
self.env[name] = :lvar
if Sexp === result.last and result.last.sexp_type == :shadow then
result.last << arg.last
result.last << name
else

@@ -294,5 +280,2 @@ result << arg

# TODO: remove in 4.0 or 2018-01, whichever is first
deprecate :get_match_node, :new_match
def gettable(id)

@@ -341,6 +324,8 @@ lineno = id.lineno if id.respond_to? :lineno

v = self.class.name[/1[89]|2[01]/]
v = self.class.name[/2\d/]
raise "Bad Class name #{self.class}" unless v
self.lexer = RubyLexer.new v && v.to_i
self.lexer.parser = self
self.in_kwarg = false

@@ -439,5 +424,2 @@ @env = RubyParserStuff::Environment.new

# TODO: remove in 4.0 or 2018-01, whichever is first
deprecate :logop, :logical_op
def new_aref val

@@ -481,2 +463,6 @@ val[2] ||= s(:arglist)

def new_brace_body args, body, lineno
new_iter(nil, args, body).line(lineno)
end
def argl x

@@ -590,3 +576,3 @@ x = s(:arglist, x) if x and x.sexp_type == :array

def new_defn val
(_, line), name, _, args, body, * = val
(_, line), (name, _), _, args, body, * = val
body ||= s(:nil)

@@ -612,3 +598,3 @@

def new_defs val
recv, name, args, body = val[1], val[4], val[6], val[7]
recv, (name, _line), args, body = val[1], val[4], val[6], val[7]
body ||= s(:nil)

@@ -631,2 +617,6 @@

def new_do_body args, body, lineno
new_iter(nil, args, body).line(lineno)
end
def new_for expr, var, body

@@ -756,5 +746,2 @@ result = s(:for, expr, var).line(var.line)

o += v
# encoding options are ignored on 1.9+
k = c if c =~ /[esu]/ if RUBY_VERSION < "1.9"
end

@@ -797,2 +784,6 @@

def new_rescue body, resbody
s(:rescue, body, resbody)
end
def new_resbody cond, body

@@ -828,3 +819,3 @@ if body && body.sexp_type == :block then

str = val[0]
str.force_encoding("ASCII-8BIT") unless str.valid_encoding? unless RUBY_VERSION < "1.9"
str.force_encoding("ASCII-8BIT") unless str.valid_encoding?
result = s(:str, str)

@@ -837,3 +828,3 @@ self.lexer.fixup_lineno str.count("\n")

str = val[1]
str.force_encoding("ASCII-8BIT") unless str.valid_encoding? unless RUBY_VERSION < "1.9"
str.force_encoding("ASCII-8BIT") unless str.valid_encoding?
result = s(:str, str)

@@ -892,3 +883,2 @@ self.lexer.fixup_lineno

else
debug20 24
sym = s(:dsym, "", sym || s(:str, ""))

@@ -1004,5 +994,2 @@ end

# TODO: remove in 4.0 or 2018-01, whichever is first
deprecate :node_assign, :new_assign
##

@@ -1164,3 +1151,3 @@ # Returns a UTF-8 encoded string after processing BOMs and magic

def value_expr oldnode # HACK
def value_expr oldnode # HACK: much more to do
node = remove_begin oldnode

@@ -1213,53 +1200,58 @@ node.line = oldnode.line if oldnode

#
# :expr_beg = ignore newline, +/- is a sign.
# :expr_end = newline significant, +/- is a operator.
# :expr_arg = newline significant, +/- is a operator.
# :expr_cmdarg = newline significant, +/- is a operator.
# :expr_endarg = newline significant, +/- is a operator.
# :expr_mid = newline significant, +/- is a operator.
# :expr_fname = ignore newline, no reserved words.
# :expr_dot = right after . or ::, no reserved words.
# :expr_class = immediate after class, no here document.
# :expr_beg = ignore newline, +/- is a sign.
# :expr_end = newline significant, +/- is an operator.
# :expr_endarg = ditto, and unbound braces.
# :expr_endfn = ditto, and unbound braces.
# :expr_arg = newline significant, +/- is an operator.
# :expr_cmdarg = ditto
# :expr_mid = ditto
# :expr_fname = ignore newline, no reserved words.
# :expr_dot = right after . or ::, no reserved words.
# :expr_class = immediate after class, no here document.
# :expr_label = flag bit, label is allowed.
# :expr_labeled = flag bit, just after a label.
# :expr_fitem = symbol literal as FNAME.
# :expr_value = :expr_beg -- work to remove. Need multi-state support.
wordlist = [
["alias", [:kALIAS, :kALIAS ], :expr_fname ],
["and", [:kAND, :kAND ], :expr_beg ],
["begin", [:kBEGIN, :kBEGIN ], :expr_beg ],
["break", [:kBREAK, :kBREAK ], :expr_mid ],
["case", [:kCASE, :kCASE ], :expr_beg ],
["class", [:kCLASS, :kCLASS ], :expr_class ],
["def", [:kDEF, :kDEF ], :expr_fname ],
["defined?", [:kDEFINED, :kDEFINED ], :expr_arg ],
["do", [:kDO, :kDO ], :expr_beg ],
["else", [:kELSE, :kELSE ], :expr_beg ],
["elsif", [:kELSIF, :kELSIF ], :expr_beg ],
["end", [:kEND, :kEND ], :expr_end ],
["else", [:kELSE, :kELSE ], :expr_beg ],
["case", [:kCASE, :kCASE ], :expr_beg ],
["ensure", [:kENSURE, :kENSURE ], :expr_beg ],
["false", [:kFALSE, :kFALSE ], :expr_end ],
["for", [:kFOR, :kFOR ], :expr_beg ],
["if", [:kIF, :kIF_MOD ], :expr_beg ],
["in", [:kIN, :kIN ], :expr_beg ],
["module", [:kMODULE, :kMODULE ], :expr_beg ],
["elsif", [:kELSIF, :kELSIF ], :expr_beg ],
["def", [:kDEF, :kDEF ], :expr_fname ],
["next", [:kNEXT, :kNEXT ], :expr_mid ],
["nil", [:kNIL, :kNIL ], :expr_end ],
["not", [:kNOT, :kNOT ], :expr_arg ],
["or", [:kOR, :kOR ], :expr_beg ],
["redo", [:kREDO, :kREDO ], :expr_end ],
["rescue", [:kRESCUE, :kRESCUE_MOD ], :expr_mid ],
["not", [:kNOT, :kNOT ], :expr_beg ],
["then", [:kTHEN, :kTHEN ], :expr_beg ],
["yield", [:kYIELD, :kYIELD ], :expr_arg ],
["for", [:kFOR, :kFOR ], :expr_beg ],
["self", [:kSELF, :kSELF ], :expr_end ],
["false", [:kFALSE, :kFALSE ], :expr_end ],
["retry", [:kRETRY, :kRETRY ], :expr_end ],
["return", [:kRETURN, :kRETURN ], :expr_mid ],
["self", [:kSELF, :kSELF ], :expr_end ],
["super", [:kSUPER, :kSUPER ], :expr_arg ],
["then", [:kTHEN, :kTHEN ], :expr_beg ],
["true", [:kTRUE, :kTRUE ], :expr_end ],
["if", [:kIF, :kIF_MOD ], :expr_beg ],
["defined?", [:kDEFINED, :kDEFINED ], :expr_arg ],
["super", [:kSUPER, :kSUPER ], :expr_arg ],
["undef", [:kUNDEF, :kUNDEF ], :expr_fname ],
["break", [:kBREAK, :kBREAK ], :expr_mid ],
["in", [:kIN, :kIN ], :expr_beg ],
["do", [:kDO, :kDO ], :expr_beg ],
["nil", [:kNIL, :kNIL ], :expr_end ],
["unless", [:kUNLESS, :kUNLESS_MOD ], :expr_beg ],
["until", [:kUNTIL, :kUNTIL_MOD ], :expr_beg ],
["unless", [:kUNLESS, :kUNLESS_MOD ], :expr_beg ],
["or", [:kOR, :kOR ], :expr_beg ],
["next", [:kNEXT, :kNEXT ], :expr_mid ],
["when", [:kWHEN, :kWHEN ], :expr_beg ],
["redo", [:kREDO, :kREDO ], :expr_end ],
["and", [:kAND, :kAND ], :expr_beg ],
["begin", [:kBEGIN, :kBEGIN ], :expr_beg ],
["while", [:kWHILE, :kWHILE_MOD ], :expr_beg ],
["yield", [:kYIELD, :kYIELD ], :expr_arg ],
["BEGIN", [:klBEGIN, :klBEGIN ], :expr_end ],
["END", [:klEND, :klEND ], :expr_end ],
["__FILE__", [:k__FILE__, :k__FILE__ ], :expr_end ],
["__LINE__", [:k__LINE__, :k__LINE__ ], :expr_end ],
["class", [:kCLASS, :kCLASS ], :expr_class ],
["__FILE__", [:k__FILE__, :k__FILE__ ], :expr_end ],
["END", [:klEND, :klEND ], :expr_end ],
["BEGIN", [:klBEGIN, :klBEGIN ], :expr_end ],
["while", [:kWHILE, :kWHILE_MOD ], :expr_beg ],
["alias", [:kALIAS, :kALIAS ], :expr_fname ],
["__ENCODING__", [:k__ENCODING__, :k__ENCODING__], :expr_end],

@@ -1270,23 +1262,7 @@ ].map { |args| KWtable.new(*args) }

WORDLIST18 = Hash[*wordlist.map { |o| [o.name, o] }.flatten]
WORDLIST19 = Hash[*wordlist.map { |o| [o.name, o] }.flatten]
WORDLIST = Hash[*wordlist.map { |o| [o.name, o] }.flatten]
WORDLIST18.delete "__ENCODING__"
%w[and case elsif for if in module or unless until when while].each do |k|
WORDLIST19[k] = WORDLIST19[k].dup
WORDLIST19[k].state = :expr_value
def self.keyword str
WORDLIST[str]
end
%w[not].each do |k|
WORDLIST19[k] = WORDLIST19[k].dup
WORDLIST19[k].state = :expr_arg
end
def self.keyword18 str # REFACTOR
WORDLIST18[str]
end
def self.keyword19 str
WORDLIST19[str]
end
end

@@ -1352,3 +1328,3 @@

@stack = [false]
warn "#{name}_stack(set): 0" if debug
log :reset if debug
end

@@ -1361,2 +1337,3 @@

def is_in_state
log :is_in_state if debug
@stack.last

@@ -1366,3 +1343,2 @@ end

def lexpop
warn "#{name}_stack.lexpop" if debug
raise if @stack.size == 0

@@ -1372,8 +1348,16 @@ a = @stack.pop

@stack.push(a || b)
log :lexpop if debug
end
def log action
c = caller[1]
c = caller[2] if c =~ /expr_result/
warn "%s_stack.%s: %p at %s" % [name, action, @stack, c.clean_caller]
nil
end
def pop
r = @stack.pop
warn "#{name}_stack.pop" if debug
@stack.push false if @stack.size == 0
@stack.push false if @stack.empty?
log :pop if debug
r

@@ -1384,12 +1368,9 @@ end

@stack.push val
return unless debug
c = caller.first
c = caller[1] if c =~ /expr_result/
warn "#{name}_stack(push): #{val} at line #{c.clean_caller}"
nil
log :push if debug
end
def store
def store base = false
result = @stack.dup
@stack.replace [false]
@stack.replace [base]
log :store if debug
result

@@ -1400,4 +1381,5 @@ end

@stack.replace oldstate
log :restore if debug
end
end
end
+5
-7
lib/ruby_parser.rb

@@ -37,3 +37,3 @@ require "ruby_parser_extras"

rescue Racc::ParseError, RubyParser::SyntaxError => exc
e = exc
e ||= exc
end

@@ -70,7 +70,5 @@ end

# Unfortunately a problem with racc is that it won't let me namespace
# properly, so instead of RubyParser::V18, I still have to generate
# the old Ruby23Parser and shove it in as V23.
# properly, so instead of RubyParser::V25, I still have to generate
# the old Ruby25Parser and shove it in as V25.
require "ruby18_parser"
require "ruby19_parser"
require "ruby20_parser"

@@ -82,2 +80,3 @@ require "ruby21_parser"

require "ruby25_parser"
require "ruby26_parser"

@@ -87,2 +86,3 @@ class RubyParser # HACK

class V26 < ::Ruby26Parser; end
class V25 < ::Ruby25Parser; end

@@ -94,4 +94,2 @@ class V24 < ::Ruby24Parser; end

class V20 < ::Ruby20Parser; end
class V19 < ::Ruby19Parser; end
class V18 < ::Ruby18Parser; end
end
+5
-4
Manifest.txt

@@ -9,9 +9,6 @@ .autotest

compare/normalize.rb
debugging.md
lib/.document
lib/rp_extensions.rb
lib/rp_stringscanner.rb
lib/ruby18_parser.rb
lib/ruby18_parser.y
lib/ruby19_parser.rb
lib/ruby19_parser.y
lib/ruby20_parser.rb

@@ -29,2 +26,4 @@ lib/ruby20_parser.y

lib/ruby25_parser.y
lib/ruby26_parser.rb
lib/ruby26_parser.y
lib/ruby_lexer.rb

@@ -39,1 +38,3 @@ lib/ruby_lexer.rex

test/test_ruby_parser_extras.rb
tools/munge.rb
tools/ripper.rb
+41
-28
Rakefile
# -*- ruby -*-
$:.unshift "../../hoe/dev/lib"
require "rubygems"

@@ -13,2 +11,3 @@ require "hoe"

Hoe.add_include_dirs File.expand_path "~/Links/SP/lib" # HACK
Hoe.add_include_dirs "../../sexp_processor/dev/lib"

@@ -18,5 +17,3 @@ Hoe.add_include_dirs "../../minitest/dev/lib"

V1 = %w[18 19]
V2 = %w[20 21 22 23 24 25]
V1_2 = V1 + V2
V2 = %w[20 21 22 23 24 25 26]

@@ -33,3 +30,3 @@ Hoe.spec "ruby_parser" do

if plugin? :perforce then # generated files
V1_2.each do |n|
V2.each do |n|
self.perforce_ignore << "lib/ruby#{n}_parser.rb"

@@ -57,5 +54,3 @@ end

end
end
V1_2.each do |n|
file "lib/ruby#{n}_parser.rb" => "lib/ruby#{n}_parser.y"

@@ -119,2 +114,4 @@ end

rp_out = "lib/ruby#{v}_parser.output"
rp_y = "lib/ruby#{v}_parser.y"
rp_y_rb = "lib/ruby#{v}_parser.rb"

@@ -126,4 +123,5 @@ c_diff = "compare/#{diff}"

c_tarball = "compare/#{tarball}"
normalize = "compare/normalize.rb"
file tarball do
file c_tarball do
in_compare do

@@ -139,5 +137,5 @@ dl version

if File.exist? "tool/id2token.rb" then
sh "ruby tool/id2token.rb --path-separator=.:./ id.h parse.y > ../#{parse_y}"
sh "ruby tool/id2token.rb --path-separator=.:./ id.h parse.y | expand > ../#{parse_y}"
else
cp "parse.y", "../#{parse_y}"
sh "expand parse.y > ../#{parse_y}"
end

@@ -149,3 +147,3 @@ end

file c_mri_txt => c_parse_y do
file c_mri_txt => [c_parse_y, normalize] do
in_compare do

@@ -158,5 +156,5 @@ sh "bison -r all #{parse_y}"

file rp_out => :parser
file rp_out => rp_y_rb
file c_rp_txt => rp_out do
file c_rp_txt => [rp_out, normalize] do
in_compare do

@@ -172,5 +170,5 @@ sh "./normalize.rb ../#{rp_out} > #{rp_txt}"

task c_diff => [c_mri_txt, c_rp_txt] do
file c_diff => [c_mri_txt, c_rp_txt] do
in_compare do
system "diff -du #{mri_txt} #{rp_txt} > #{diff}"
sh "diff -du #{mri_txt} #{rp_txt} > #{diff}; true"
end

@@ -187,20 +185,22 @@ end

task :clean do
rm_f Dir[c_parse_y, c_mri_txt, c_rp_txt]
rm_f Dir[c_mri_txt, c_rp_txt]
end
task :realclean do
rm_f Dir[tarball]
rm_f Dir[c_parse_y, tarball]
end
end
ruby_parse "1.8.7-p374"
ruby_parse "1.9.3-p551"
# ruby_parse "1.8.7-p374"
# ruby_parse "1.9.3-p551"
ruby_parse "2.0.0-p648"
ruby_parse "2.1.9"
ruby_parse "2.2.6"
ruby_parse "2.3.3"
# TODO ruby_parse "2.4.0"
ruby_parse "2.2.9"
ruby_parse "2.3.8"
ruby_parse "2.4.5"
ruby_parse "2.5.3"
ruby_parse "2.6.1"
task :debug => :isolate do
ENV["V"] ||= V1_2.last
ENV["V"] ||= V2.last
Rake.application[:parser].invoke # this way we can have DEBUG set

@@ -240,9 +240,22 @@ Rake.application[:lexer].invoke # this way we can have DEBUG set

task :debug_ruby do
file = ENV["F"] || ENV["FILE"]
sh "/Users/ryan/Desktop/DVDs/debugparser/miniruby -cwy #{file} 2>&1 | ./yuck.rb"
task :debug3 do
file = ENV["F"]
verbose = ENV["V"] ? "-v" : ""
munge = "./tools/munge.rb #{verbose}"
abort "Need a file to parse, via: F=path.rb" unless file
ENV.delete "V"
sh "ruby -y #{file} 2>&1 | #{munge} > tmp/ruby"
sh "./tools/ripper.rb -d #{file} | #{munge} > tmp/rip"
sh "rake debug F=#{file} DEBUG=1 V=25 2>&1 | #{munge} > tmp/rp"
end
task :cmp3 do
sh %(emacsclient --eval '(ediff-files3 "tmp/ruby" "tmp/rip" "tmp/rp")')
end
task :extract => :isolate do
ENV["V"] ||= V1_2.last
ENV["V"] ||= V2.last
Rake.application[:parser].invoke # this way we can have DEBUG set

@@ -249,0 +262,0 @@ 

lib/ruby18_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby18_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby19_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby19_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby_lexer.rex

Sorry, the diff of this file is not supported yet

lib/ruby_parser.yy

Sorry, the diff of this file is not supported yet

lib/ruby20_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby20_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby21_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby21_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby22_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby22_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby23_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby23_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby24_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby24_parser.y

Sorry, the diff of this file is not supported yet

lib/ruby25_parser.rb

Sorry, the diff of this file is too big to display

lib/ruby25_parser.y

Sorry, the diff of this file is not supported yet

test/test_ruby_lexer.rb

Sorry, the diff of this file is too big to display

test/test_ruby_parser.rb

Sorry, the diff of this file is too big to display