Add some basic docs on compiler parts

opal · Oct 27, 2013 · 1c052ec · 1c052ec
1 parent 492db0c
commit 1c052ec
Show file tree

Hide file tree

Showing 2 changed files with 46 additions and 0 deletions.
diff --git a/doc/compiler.md b/doc/compiler.md
@@ -0,0 +1,42 @@
+# Opal Compiler
+
+Opal is a source to source compiler. It accepts ruby code as a string and
+generates javascript code which can be run in any environment. Generated
+code relies on the opal runtime which provides the class system and some
+other runtime helpers.
+
+## Compiler stages
+
+The compiler can be broken down into 3 separate stages:
+
+* lexing
+* parsing
+* code generation
+
+### Lexer
+
+The [opal lexer](../lib/opal/parser/lexer.rb) is implemented in pure ruby using
+the `StringScanner` class from the opal stdlib. The source code is scanned
+and tokens are then provided to the parser. This process simply converts
+the ruby code given as a string, into a list of tokens representing the
+parts of the ruby code.
+
+### Parser
+
+The [opal parser](../lib/opal/parser/grammar.y) is implemented using a standard
+bison like syntax, but relies on `racc`, a ruby implementation of yacc/bison
+which is again available in the standard library. The parser takes these tokens
+generated by the lexer and builds a syntax tree representing the ruby code.
+This syntax tree is represented by [sexps](../lib/opal/parser/sexp.rb). As
+ruby is such a complex and dynamic language, there is a lot of interaction
+between the parser and the lexer, namely through a preserved `lex_state`.
+
+### Code generation
+
+The [opal compiler](../lib/opal/compiler.rb) takes these sexps from the parser
+and generates ruby code from them. Each type of sexp has [its own node type](../lib/opal/nodes/base.rb)
+used to generate javascript. Each node creates an array of one or more
+[fragments](../lib/opal/fragment.rb) which are the concatendated together to
+form the final javascript. Fragments are used as they contain the generated
+code as well as a reference back to the original sexp which is useful for
+generating source maps afterwards.
diff --git a/doc/home.md b/doc/home.md
@@ -11,3 +11,7 @@
 [Static Application](static_applications.md) Just build your app and dependencies to a build.js file
 
 [Using Sprockets](using_sprockets.md) Use rack/sprockets to auto-recompile an opal application
+
+## Compiler
+
+[Opal Compiler](compiler.md) An overview of how the compiler works