Arc Forumnew | comments | leaders | submitlogin
1 point by akkartik 2300 days ago | link | parent

"One of the beauties of lisp is that the prefix notation is simple, making the grammar almost trivial to parse. Bullet's reader has to work a little harder, needing one token of look-ahead to be able to handle its infix operators.."

I chose to make the tokenizer oblivious to infix syntax. The only delimiters it recognizes are quotes, unquote (comma), parens and whitespace. The infix operators[1] are implemented in a later phase; it can transform each token without needing any context around it.

Hmm, when I look at it now I realize that I could do this earlier in the pipeline. I could do this transformation before I construct the cons cells for expressions, and it'd probably be more efficient.

I'd love to chat more about the implementation details since it seems we've dealt with a lot of the same problems. Talking to you about it has already given me two ideas for improvements, thanks.

[1] Arc calls them ssyntax; I think the name is intended to line up with s-expressions.

4 points by seertaak 2299 days ago | link

The pipeline in bullet is:

1. line processor: text lines are surrounded by brackets (textually!) where appropriated (i.e. 2-indents subject to nesting rules). Infix operators aren't touched. Result is stored in an array of CodeFragment, which is basically text + source code location.

2. reader: the forms are "read" and turned into B-expressions. You could call this the parsing stage, but lisp is a bit strange in some sense: there are two levels of parsing: the reader is the first, and the second is in the...

3. interpreter/compiler: every lisp has a starting phase which takes the "low-level" S-expressions comprising the code and parses it into the true semantic expression of the language: e.g. IfExpr, LambdaExpr, FnApplyExpr, etc.

Indeed, I think this is what makes lisp unique: all other languages have parsers which take you directly to step 3. The AST for semantic analysis purposes trees is basically just the parse tree. In lisp, you have to do work to get the AST -- but that's also why you get flexibility in macro writing.

I'd be happy to exchange implementation details. As I mention in the article, my code resides at

My email is

martin at percossi dot com

feel free to email me if you have questions. It's nice to know other people are working on/thinking about the same problems! :)