Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lark-parser: 2017-12-18 -> 0.6.5 #49865

Merged
merged 1 commit into from Nov 7, 2018
Merged

lark-parser: 2017-12-18 -> 0.6.5 #49865

merged 1 commit into from Nov 7, 2018

Conversation

flokli
Copy link
Contributor

@flokli flokli commented Nov 7, 2018

This bumps lark-parser to a released version that doesn't loop forever
and consumes all memory on certain grammars.

erezsh/lark redirects now to lark-parser/lark, so also update homepage.

Motivation for this change

The current version in nixpkgs is outdated and burns CPU in some cases.
The released version fixes that, shortlog from the old revision to that release also is full of improvements:

AlexV (2):
      toxing for easy multiversion tests
      cleaning up tox.ini

Bruno Pedrosa (1):
      New example showing how to use lark+qscintilla (Issue #150)

Chris McKinney (1):
      Fixed TypeError on pretty-printing tuples in tree

Daniel Drake (1):
      docs: clarify end_column value

DrSlump (1):
      example driven parser errors

Ehud Tamir (1):
      Merge CYK parser.

Eliot Alter (2):
      Fix a small typo
      Fix a small typo

Erez Shinan (197):
      Cleaned up the LALR(1) parser
      Merge branch 'cleaner_lalr'
      Example JSON parser is now LALR
      Version 0.5.0 is out with lots of improvements
      BUGFIX: 'python setup.py test' now works.
      Version bump
      BUGFIX: Create unless returned duplicate tokens (Issue #56)
      Refactoring for LALR, added the ParseTable class
      Big Refactor: Grammars now build in half the time. Code shorter & cleaner.
      Post-refactor cleanup
      Improved grammar validation and refactored the lexers
      Minor refactoring for the standalone tool (in progress)
      Updated README
      Update README.md
      More refactoring, untangling grammar compilation and parse-tree creation
      More refactoring towards standalone
      Stand-alone tool working for LALR+traditional lexer (first commit)
      Corrections to the standalone tool
      Standalone tools now supports postlex
      Standalone: Significantly reduced generated code size
      Updated README to mention standalone
      Version bump
      Added standalone example
      Minor fixes in lexer
      BUGFIX: Internally repetitive rules are now handled silently (Issue #60)
      Better error message for bad regexps (Issue #62)
      Validate against zero-width terminals in XEarley (Issue #63)
      BUGFIX: Repeating subrules are now allowed (Issue #61)
      Small fix in Python grammars (misread tuples, issue #68)
      Merge pull request #69 from NighttimeDriver50000/master
      Merge branch 'master' of https://github.com/ehudt/lark into ehudt-master
      All relevant tests passing. Also indentation and other refactoring.
      Idiomatic function names and a few other style fixes
      Small fix
      Merge pull request #65 from ehudt/master
      A few more adaptations
      Merge branch 'cyk'
      Version bump
      Updated README
      Fixed propagate positions. Added lexer_callbacks option.
      Added more information in UnexpectedInput exception (Issue #78)
      Simplified grammar analysis. Improved performance of FIRST/FOLLOW calculations
      Better error message for reduce collisions in lalr
      Fixed reconstruct (Issue #72)
      Style fixes
      Refactoring and bugfixes in reconstruct.py
      FEATURE: Added support for ranged-repeat for rules and terminals (Issues #75, #19)
      BUGFIX: SIGNED_FLOAT matched an int... (Issue #84)
      BUGFIX: Undefined symbols (Issue #87)
      Version bump
      Merge pull request #89 from psboyce/patch-1
      BUGFIX: Token and Derivation get compared in _compare_drv (Issue #91)
      Refactored parse_tree_builder, much small and cleaner now
      Merge pull request #93 from micaiahparker/master
      Update README.md
      BUGFIX: Non-linearity in tree construction, causing performance issues for large inputs (Issue #94)
      Added 'considered_rules' to exceptions, to help users debug
      Version bump
      Merge pull request #102 from night199uk/functools
      Standalone generator working again. Updated examples
      Added tests for standalone generator
      Merge pull request #88 from cclauss/patch-2
      Updated setup.py. How long has it been saying 'Alpha'? Hmm..
      Merge pull request #104 from night199uk/token_pickle
      Added error message for the alias syntax in terminals (Issue #97)
      Fixed bug in Earley: A tree builder optimization clashed with explicit ambiguity
      Fixed bug in Earley prioritization
      Version bump
      Significantly better memory performance (Thanks @drslump!)
      Significant reduction in memory consumption (Saving only parse-table instead of analysis instance)
      Added SlottedTree
      A minor style fix (a possible memory usage improvement)
      Improvements based on the PR of @drslump (https://github.com/erezsh/lark/pull/125)
      Initial: Added transformers.py, and Meta to tree
      Some more normalizing
      Snap more things into place
      More steps towards a good solution
      Merge pull request #128 from psboyce/patch-2
      Corrections to PR and added get_context
      Merge branch 'pr_accept' (Containing #129 with corrections)
      Added example for error reporting with LALR
      Improved Lark's error reporting for grammar syntax errors (Based on PR #129)
      Update README.md
      Merge pull request #133 from Gentle/interpreter_defaults
      BUGIX in lexer: Embedding strings overwrote priority (Issue #121)
      Interface improvements for the Lark instance
      .lark (preparing)
      Added examples/lark.g - Reference implementation of the Lark grammar (inspired by issue #116)
      Update README.md
      Symbols instead of strings - initial
      Earley working too
      All tests passing
      Breaking backwards compatibility:
      Small refactoring step
      Refactoring to introduce Symbol instances before creating anons
      Fixed token visibility rules (Issue #109)
      Anon terminals no longer need to start with _
      Simplify PrepareAnonTerminals
      Merge branch 'symbols' into 0.6.0
      Columns now start at 1
      Merge branch '0.6.0' into new_transformers
      Missed those at merge
      Refactored inline_args with smart_decorator
      Refactored transformers, better code
      args decorators actually work now
      More work
      Updated README - Build now refering to lark-parser
      Merge pull request #144 from asmodehn/toxing
      New transformers near completion
      Fixes
      Improvements
      A requirements file for the examples is an overkill
      Merge branch 'brupelo-master'
      Added %declare
      Fixed Python grammars, and a bug in newline detection
      Added cheatsheet PDF
      Merge branch 'file-ext-change' of https://github.com/RobRoseKnows/lark into RobRoseKnows-file-ext-change
      .gitignore should never have been added
      Merge branch 'RobRoseKnows-file-ext-change' into 0.6.0
      Added %declare to examples/lark.lark
      Update README.md
      Update README.md
      visitor_args -> v_args
      Fixed nearley
      Merge branch 'new_transformers' into 0.6.0
      Added some docstrings, removed is_terminal from common
      Moved and restructured exceptions
      Updated performance comparison to include ANTLR
      Merge pull request #168 from lark-parser/0.6.0
      Updated version
      Dynamic lexer is now returns the maximum match only. Complete lexing beahvior moved to "dynamic_complete"
      Improved documentation and code style (minor)
      Fixed reconstruct
      Fixed examples for error reporting
      Fix for Python 3.7
      Added option to provide a custom lexer (with example)
      Added custom_lexer to examples/README
      Lark grammars are now utf8 by default (Issue #179)
      Fixed issue where Lark would throw RecursionError for huge grammars (Issue #181)
      Improved error messages (as pointed out in issue #181)
      Version bump
      Added support for v_args(tree=True)
      Bug in PropagatePositions when supplying a reducing transformer
      Version bump
      Merge pull request #180 from julienmalard/master
      Refactoring and fixes for merge 2fd0087
      Merge pull request #196 from julienmalard/master
      Fixed reconstructor. All tests passing
      Merge branch 'fix_recons'
      BUGFIX: Repeated use of optional rules tripped up the simplifier, manifesting when aliases were used (Issue #197)
      BUGFIX: Bad naming scheme in tree-builder callbacks caused conflicts (Issue #198)
      Fixed missing import (Issue #202)
      Fix error
      Cleanup after pylint
      Version bump
      Added MkDocs documentation
      Added list of pages to yml (docs)
      Improved docs
      Fix for issue #188
      BUGFIX - Fixed 2 issues with line counting
      Bug in pretty printing of Earley derivations (issue #210)
      BUGFIX affecting standalone generator (Issue #212)
      Version bump
      Fixed bugs in the standalone generator (Issue #212)
      Fixed the propagate_positions implementation, and added start_pos/end_pos attributes to Tree.Meta
      Added gitter link (Issue #206)
      Merge pull request #225 from ciphergoth/patch-1
      BUGFIX: Standalone parser didn't retain filter_out values for terminals (Issue #222)
      Misleading error message at unexpected EOF (Issue #220)
      BUGFIX: Automatic terminal names didn't respect existing terminals(Issue #224)
      Added visitors & transformers to standalone (Issue #223)
      Added the 'tartiflette' project to the readme
      Slightly better error message (UnexpectedToken)
      Standalone parser now uses contextual lexer instead of traditional one (Issue #212)
      EOF token now gets proper line/column number (Issue #228)
      Tiny refactor
      Refactoring around terminals / tokens
      Small refactoring in standalone
      Point to readthedocs instead of wiki
      Update README.md
      Merge pull request #232 from evandrocoan/patch-1
      Merge pull request #234 from ealter/patch-1
      Merge pull request #235 from ealter/patch-2
      Refactored lexer + tiny bugfix
      Bugfix for propagate_positions
      Added documentation for lexer_callbacks in recipes
      Merge pull request #239 from evandrocoan/add_gitignore
      More to .gitignore
      Experimental support for importing rules
      Tiny improvement to the LALR parser
      Tiny fix for docs
      Update README.md
      Merge pull request #241 from dsd/master
      Merge pull request #243 from evandrocoan/pydot_tree_orientation
      Fixed a deep bug in grammar analysis involving empty rules (Issue #250)
      Updated mention of GPL for standalone tool (Issue #247)
      Fixed v_args handling of staticmethod and classmethod (Issue #246, #249)

Julien Malard (4):
      From and relative type imports seem to work.
      Reimplemented relative and multiple imports.
      Fixed test.
      Tests for reconstructors

Micaiah Parker (1):
      added UnexpectedToken to __init__ for help unit testing

Parker (2):
      Use loops for flow control instead of catching exceptions
      Fix order of members when pickling Token

Paul Crowley (1):
      Fix typo in sample code: gramamr -> grammar

Ramon Klass (2):
      Implemented a new visitor class (Interpreter) that works top-down (PR #130)
      interpreter: default behavior changed to return the values instead of discarding them, added test showcasing the behavior

Rob Rose (5):
      Changed Lark standalone file extension
      Changing changes from .lrk to .lark
      Merge branch 'master' of https://github.com/lark-parser/lark into file-ext-change
      Resolved upstream changes to use new file-extension
      Added .gitignore for pyenv

cclauss (1):
      Python 3.2 and 3.3 are End of Life

ehudt (3):
      Merge branch 'master' into master
      Merge pull request #1 from erezsh/cyk_after_fixes
      Update LICENSE

evandrocoan (4):
      Fix #231
      Added missing binary files to .gitignore
      Allow to pass to lark/tree.py pydot__tree_to_png the pydot tree
      Fixed SyntaxError: positional argument follows keyword argument on

night199uk (2):
      Allow functools partials to be used with the InlineTransformer
      Ensure Tokens can be pickled correctly
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@flokli
lark-parser: 2017-12-18 -> 0.6.5
d4a4de5 
This bumps lark-parser to a released version that doesn't loop forever
and consumes all memory on certain grammars.

erezsh/lark redirects now to lark-parser/lark, so also update homepage.
@flokli flokli requested a review from FRidh as a code owner November 7, 2018 10:02
@FRidh
Copy link
Member

FRidh commented Nov 7, 2018

@GrahamcOfBorg build python2.pkgs.lark-parser python3.pkgs.lark-parser

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python2.pkgs.lark-parser, python3.pkgs.lark-parser

Partial log (click to expand)

Successfully installed lark-parser-0.6.4
/build/source
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/1s0nrd1sz18x0ayipzj07nlfibrv4h0v-python3.6-lark-parser-0.6.5
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/1s0nrd1sz18x0ayipzj07nlfibrv4h0v-python3.6-lark-parser-0.6.5/lib
patching script interpreter paths in /nix/store/1s0nrd1sz18x0ayipzj07nlfibrv4h0v-python3.6-lark-parser-0.6.5
checking for references to /build in /nix/store/1s0nrd1sz18x0ayipzj07nlfibrv4h0v-python3.6-lark-parser-0.6.5...
/nix/store/cc0lhqh47wd5m9pwndqqrv77hm9yd33p-python2.7-lark-parser-0.6.5
/nix/store/1s0nrd1sz18x0ayipzj07nlfibrv4h0v-python3.6-lark-parser-0.6.5

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python2.pkgs.lark-parser, python3.pkgs.lark-parser

Partial log (click to expand)

checking for references to /build in /nix/store/a0ja516az2m634h3619i8pz7gsf2qbya-python2.7-lark-parser-0.6.5...
/build/source
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/8pfiwsz8fjy07qlr8nnb0gb0wyha1c4g-python3.6-lark-parser-0.6.5
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/8pfiwsz8fjy07qlr8nnb0gb0wyha1c4g-python3.6-lark-parser-0.6.5/lib
patching script interpreter paths in /nix/store/8pfiwsz8fjy07qlr8nnb0gb0wyha1c4g-python3.6-lark-parser-0.6.5
checking for references to /build in /nix/store/8pfiwsz8fjy07qlr8nnb0gb0wyha1c4g-python3.6-lark-parser-0.6.5...
/nix/store/a0ja516az2m634h3619i8pz7gsf2qbya-python2.7-lark-parser-0.6.5
/nix/store/8pfiwsz8fjy07qlr8nnb0gb0wyha1c4g-python3.6-lark-parser-0.6.5

@FRidh
Copy link
Member

FRidh commented Nov 7, 2018

Interesting to see another Nixpkgs user use this package :-)

@FRidh FRidh merged commit becf887 into NixOS:master Nov 7, 2018
@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: python2.pkgs.lark-parser, python3.pkgs.lark-parser

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/l6b3qjw4982gjspcinzp5z7aa92znb0v-python3.6-lark-parser-0.6.5/lib
patching script interpreter paths in /nix/store/l6b3qjw4982gjspcinzp5z7aa92znb0v-python3.6-lark-parser-0.6.5
Successfully installed lark-parser-0.6.4
/private/tmp/nix-build-python2.7-lark-parser-0.6.5.drv-0/source
post-installation fixup
strip is /nix/store/g5r4apl0za012ffs6ladinwa5w0m1l3k-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/rqc5aiii15i3a8qvxx2ns561gcz4jxv8-python2.7-lark-parser-0.6.5/lib
patching script interpreter paths in /nix/store/rqc5aiii15i3a8qvxx2ns561gcz4jxv8-python2.7-lark-parser-0.6.5
/nix/store/rqc5aiii15i3a8qvxx2ns561gcz4jxv8-python2.7-lark-parser-0.6.5
/nix/store/l6b3qjw4982gjspcinzp5z7aa92znb0v-python3.6-lark-parser-0.6.5

@flokli flokli deleted the lark-parser-0.6.5 branch November 7, 2018 10:13
@flokli
Copy link
Contributor Author

flokli commented Nov 7, 2018

@FRidh yes, it helped parsing some very painful text formats. Would you mind backporting to 18.09 too?

@FRidh
Copy link
Member

FRidh commented Nov 7, 2018

Backported as 1349894.

@flokli
Copy link
Contributor Author

flokli commented Nov 7, 2018

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@flokli @FRidh @GrahamcOfBorg