Arc Forumnew | comments | leaders | submitlogin
Jaclyn: The Jarc Compiler (fandle.com)
4 points by jazzdev 5162 days ago | 22 comments


2 points by jazzdev 5162 days ago | link

This Jarc 12 release (with the compiler), also includes arc 3.1 semantics. And I believe I have fixed all the bugs that rocketnia has mentioned in this forum.

Thanks to Ken Shirriff for this blog post on What's new in arc3 and to rocketnia for all his detailed Jarc comments in this forum.

-----

2 points by rocketnia 5161 days ago | link

This is pretty awesome. So... I'm not trying to rain on your parade, but I noticed two things that break Lathe right away:

- When you call (split "hello" 1), it doesn't call the Arc strings.arc split function; instead it tries to call the Java String.split() method, which doesn't accept a numeric parameter. It doesn't help to say (let split ...) or (apply split "heyo" 1 nil) either. Seems like after http://www.arclanguage.org/item?id=11730 you changed your mind or something. :)

- There's a subtle difference in the implementation of setforms that messes me up. It isn't as hackable:

  (Arc 3.1)
  
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> (mac foo (x) 'bar)
  #(tagged mac #<procedure: foo>)
  arc> (setforms 'foo.0)
  ((gs1722 bar) gs1722 (fn (gs1723) (assign bar gs1723)))
  arc> (let s setforms (= setforms [do (prn "trace: " _) s._]))
  #<procedure: setforms>
  arc> (setforms 'foo.0)
  trace: foo.0
  trace: (foo 0)
  ((gs1724 bar) gs1724 (fn (gs1725) (assign bar gs1725)))
  arc>
  
  
  (Jarc)
  
  Jarc> (mac foo (x) 'bar)
  #3(tagged mac #<procedure>)
  Jarc> (setforms 'foo.0)
  ((gs20 bar) gs20 (fn (gs21) (assign bar gs21)))
  Jarc> (let s setforms (= setforms [do (prn "trace: " _) s._]))
  #<procedure>
  Jarc> (setforms 'foo.0)
  trace: foo.0
  ((gs22 bar) gs22 (fn (gs23) (assign bar gs23)))
Notice how the Arc 3.1 implementation is self-recursive and that if it's overwritten, the old version ends up calling the new version instead of itself. Jarc's worked this way for a while too, but now either you've changed your definition of 'setforms or else you've made one too many assumptions in your occasional tail-call optimization.

Neither of these issues is a big deal for me with Lathe; I can just define another function called "xsplit" or something, and I can just ssexpand the parameter of my setforms extension myself rather than relying on the recursion to do it. But until I do those things and get farther, these potential incompatibilities may be worth your attention.

I'm really optimistic about Jarc 12+. Thanks a lot, again, for all you've done. ^_^

-----

1 point by rocketnia 5160 days ago | link

I've got Lathe's buggy-jarc branch working with Jarc 12 now. Woo!

I ended up reasoning that Lathe would be better off not needing to redefine basic Arc functionality at all, so I fiddled around and got it working that way. The biggest changes are a) that the codebase now uses '=fn and '=mc in place of 'def and 'mac, and b) that namespace applications like my.foo now expand to (global 'gs1234-foo) rather than just gs1234-foo, which lets (= my.foo 2) work without the 'setforms extension I was using.

These design changes aren't in the Lathe master branch yet, but they should get there eventually; whether or not hackability is important to Jarc, I feel it's important for Lathe not to depend on it, so that it's as compatible as possible with other codebases that do depend on redefinition (or indeed, Arc implementations which don't support it).

Anyway, back to Jarc. There were only a couple more hurdles I came across. The first is that cut's optional parameter isn't really optional in Jarc; it was easy enough to just call 'cut with three parameters all the time. The other issue was more insidious: Quasiquotation is still messed up.

- Giving `,2 should result in 2 rather than (unquote 2).

- Giving `(1 `(2 ,(3 ,4))) should result in (1 (quasiquote (2 (unquote (3 4))))) rather than (1 (quasiquote (2 (unquote (3 (unquote 4)))))). This isn't even consistent with Jarc 11. :)

- The following should result in (1) rather than (2):

  (mac foo () 1)
  (def bar () `(,(foo)))  ; remember ,@ and ssyntax too
  (mac foo () 2)
  (bar)
- The following should result in (2) rather than (1):

  (def foo () `(1))
  (= ((foo) 0) 2)
  (foo)
I thought about implementing quasiquote myself as an Arc macro to give you a reference implementation, but as I started in on that I noticed just how inconsistent Arc's own quasiquotation is.

In particular, in Arc 3.1, (iso ``',,@'(x) '`',x) works, but (iso ``',@,@'(x) '`',@x) gives an error. Also, (iso ``(unquote x y) ``,x) is true, which entails that not every expression which can be quote-mutated (the way (def foo () `(1)) is mutated above) is copied verbatim from the original quasiquote form.

Given these inelegances, it's quite a bit more difficult to emulate Arc than I expected. That said, had I understood these details going in, I probably would have gotten a lot farther by now, so I hope by mentioning them I can help you out too.

-----

1 point by jazzdev 5155 days ago | link

I've just released Jarc 13 which addresses most of these issues.

- quasiquote

I've learned that quasiquote is quite a deep rabbit hole! Jarc 13 handles quasiquote much better. I wrote a Java-based qq-expand that is simpler than qq.arc. It doesn't handle dotted lists and it doesn't do any optimizing, but it does handled nested quasiquotes properly. And qq.arc is now included in Jarc 13. So if you want it, do (use 'qq).

In Jarc 13, quasiquote is now a macro instead of a special form. This makes it much easier for the interpreter and compiler to share the code for handling it. Since neither has to do any special handling for it. I was able to rip out some code from Closure and from compile.arc. Always like it when I can do that!

- split, cut and compiled functions with optional params

Fixed. Jaclyn now handles dotted destructed args too.

- strings

Jarc 13 includes strings.arc, code.arc and pprint.arc when it starts.

- eval

Unrelated to this thread, but when I "fixed" eval to ignore the lexical environment it broke the pr-tag macro in jtml.arc. Since pr-tag is a macro it didn't need to call eval anyway, so I just rewrote it.

- Changing the definition of a function after it's defined

This scenario is still problematic:

  (def foo () `(1))
  (= ((foo) 0) 2)
  (foo)
While I agree with the hackability of Arc, I think it's perhaps a problem that you can change the definition of a function after it's defined. You are essentially reaching into foo and changing the cons inside it.

The behavior of this is dependent on how you optimize qq-expand.

Arc 3.1 yeilds (2) for the above code. Jarc 13 yields (1) for the above code. Jarc 13 with (use 'qq) yields (2). Jarc 13 with (use 'qq) (toggle-optimize) yields (1)

As CatDancer pointed out in http://arclanguage.org/item?id=9962 there are multiple ways to expand quasiquote that all result in the same expression when eval'd.

Unless there's a precise definition for how `(1) should be expanded, I don't see how we can define what the output of the above code should be.

-----

1 point by rocketnia 5155 days ago | link

Cool, I'll check it out really soon.

While I agree with the hackability of Arc, I think it's perhaps a problem that you can change the definition of a function after it's defined. You are essentially reaching into foo and changing the cons inside it.

Yeah, I don't like this feature either. It makes me a bit paranoid about using quasiquote at all sometimes, since I might be unintentionally opening up parts of my macro code to being modified. I don't let that affect my coding style--I just don't usually mutate macro-expansions and stuff--but it's sort of a pet peeve.

I'm sure there's a pg post somewhere around here about it being a bit of effort to get things to work this way, but I can't find it.

Unless there's a precise definition for how `(1) should be expanded...

I think the obvious one is '(1), which is to say, replacing the 'quasiquote symbol with 'quote. If 'quasiquote is taken out of the core, then 'quote is really the only fundamental form left that you can "reach into" this way. Furthermore, Jarc 12 already worked for '(1):

  (def foo () '(1))
  (= ((foo) 0) 2)
  (foo)            ; results in 2
If (use 'qq) gives you something that yields (2) and it's a macro, then it probably expands `(1) into '(1), and so the work is probably done for you.

This probably also means that your optimizer stops (quote (1)) from returning a reference to a single mutable (1) and makes it cons up a whole new (1) instead. If that's the case, then it isn't even much of an optimization. ^_^ I wonder if the compiled JVM bytecode can contain a sort of quote-closure field for each 'quote form, so that the compiled Java object/class can be initialized with a direct reference to a subtree of foo's syntax.

On the other hand, if this is something you don't want to fix, I'll agree with you there. :-p

-----

1 point by rocketnia 5154 days ago | link

Okay, I've brought Lathe's buggy-jarc branch up-to-date with Jarc 13. The code I had for Jarc 12 already worked, so I took the opportunity to remove all the unnecessary workarounds that were still in there. It sure is an awesome relief having the diff with the master branch get so small, but there are still two bugs in my way right now:

- Jarc's metafns aren't metafns. This has been a problem for quite a while, but usually I phrase it in terms of (my:foo ...) not working, since that's the way my namespace system encourages macro calls to look.

Each of the following should result in t rather than trying to apply 1 as a function:

  (iso ((compose do) '(1 2 3)) '(1 2 3))
  (iso ((andf do) '(1 2 3)) '(1 2 3))
For compatibility, this should work even if someone rebinds 'compose, 'andf, or some other metafn name to a completely different value. Metafn forms are identified purely by name in official Arc.

- Jarc's a&b ssyntax no longer works. The result of (ssexpand 'a&b) is (andf a b) as it should be, and (andf a b) does work correctly in most cases (the above bug being an exception), but for some reason using a&b in the code itself causes an error. It tries to look up the symbol a&b.

-----

1 point by fallintothis 5154 days ago | link

I'm sure there's a pg post somewhere around here about it being a bit of effort to get things to work this way, but I can't find it.

http://arclanguage.org/item?id=10248

-----

1 point by jazzdev 5154 days ago | link

It makes me a bit paranoid about using quasiquote

This isn't particular to quasiquote. You can reach into any function that returns a literal cons or string.

If this is something you don't want to fix, I'll agree with you there

Yeah, I'm disinclined to try to emulate Arc precisely in this regard because it appears to be an arbitrary artifact of how Arc does quasiquote expansion, and not a defined feature.

-----

2 points by fallintothis 5154 days ago | link

Yeah, I'm disinclined to try to emulate Arc precisely in this regard because it appears to be an arbitrary artifact of how Arc does quasiquote expansion, and not a defined feature.

I think the only thing going on here is quotation. Generally, quoted things work like pointers (see http://arclanguage.org/item?id=10248). Because you can still do

  arc> (def foo () '(1)) ; note: just quote, no quasiquote
  #<procedure: foo>
  arc> (= ((foo) 0) 2)
  2
  arc> (foo)
  (2)
but

  arc> (def foo () (cons 1 nil)) ; note: cons creates new data each call
  *** redefining foo
  #<procedure: foo>
  arc> (= ((foo) 0) 2)
  2
  arc> (foo)
  (1)
Then, the reason the quasiquoted thing gets weird results with the qq.arc I ported is that optimizing `(1) manages to switch between always consing new data with (list '1) and trying to reduce runtime consing with '(1).

-----

1 point by jazzdev 5153 days ago | link

I'm not (terribly) surprised that the ability to modify literals was intentional. Thanks for that link. I'm still wondering if `(1) expanding to '(1) is intentional or not.

Clearly '(foo) is a literal and `(foo ,bar) is not a literal, right? I guess it's not a huge stretch to say that `(foo) is a literal. Unfortunately, qq-expand doesn't always expand `(...) with no commas into '(...)

  Jarc> (qq-expand '(foo))
  (quote (foo))  ; literal
  Jarc> (qq-expand '(foo bar))
  (quote (foo bar)) ; literal
  Jarc> (qq-expand '(foo nil))
  (list (quote foo) nil) ; not a literal
But Arc does expand `(foo nil) into a literal

  arc> (def foo(s) (let a `(3 nil) (= (car a) (cons s (car a)))))
  #<procedure: foo>
  arc> (foo 2)
  (2 . 3)
  arc> (foo 1)
  (1 2 . 3)

-----

2 points by aw 5152 days ago | link

I'm still wondering if `(1) expanding to '(1) is intentional or not.

A quasiquote expander can expand `(1) into '(1) or (list 1), or even (join '(1) ' nil), which is what Bawden's simple, correct, but inefficient quasiquote expander does.

The expander may choose '(1) as being more efficient than (list 1), but isn't required to do so to be a quasiquote expander.

Since it's an optimization, I wouldn't write code that relies on a particular instance of `(1) evaluating to the same list every time, that's just an accidental result of the optimization. Instead, use a plain quote for that.

-----

1 point by fallintothis 5153 days ago | link

That's a bug. It didn't think that nil was a constant. Apparently I didn't know about literal 327 days ago. Thanks for finding it. :)

http://bitbucket.org/fallintothis/qq/changeset/6c0dab5f091e

(Really, all of that code needs a good rewrite. It was straight-ported from some Common Lisp, and suffers greatly for it. Bleh.)

-----

3 points by fallintothis 5152 days ago | link

There. Just updated qq.arc, removing almost 100 lines of cruft in the process. It still passes all the old tests, and (with any luck) is much more pleasant to work with.

http://bitbucket.org/fallintothis/qq/src/tip/qq.arc

-----

2 points by aw 5152 days ago | link

Yay!

-----

1 point by rocketnia 5154 days ago | link

This isn't particular to quasiquote. You can reach into any function that returns a literal cons or string.

I know, hence why I talked about 'quote. On principle, I'm equally paranoid about 'quote, 'quasiquote, and literal strings, but I've never mutated strings in my own projects, and I almost exclusively quasiquote my "escaping" cons cells rather than quoting them, so I just glossed over those facets of my paranoia. :-p

it appears to be an arbitrary artifact of how Arc does quasiquote expansion, and not a defined feature.

Like I said, I think there's a relevant pg comment around here somewhere. I've searched some more, but I still can't find it. Maybe it was just my imagination. ^_^;

Oh, wait, fallintothis's post links to it.

Arrgh, it would help if Google weren't so inconsistent. It's a result for ("behavior can be quite useful in some contexts"), but it doesn't show up in the results for ("behavior can be quite useful in some"), and ("can be quite useful in some contexts") has no results at all. Of course, observing this property and posting about it will no doubt change the search results. XD

Ah, here we go, http://af.searchyc.com/ is much nicer. ^_^

-----

1 point by aw 5159 days ago | link

I haven't looked at your quasiquote examples in particular, but note that standard Arc uses MzScheme's quasiquote expander, which is buggy for nested quasiquote expansions.

Fortunately it's easy to use a working quasiquote expander: it's a one line change to the Arc compiler so that if a "quasiquote" macro has been defined, it gets used instead of the builtin MzScheme version. Then we can load a quasiquote implementation written in Arc, such as fallintothis' port of the Common Lisp quasiquote expander.

http://arclanguage.org/item?id=9912

-----

1 point by jazzdev 5157 days ago | link

The problem with cut is the reason why 'split' wasn't working also--it calls cut without the optional parameter. Fixed now. I'm going to try fallintothis' quasiquote in Jarc. I'll need the Java-based one for bootstrapping, I think, so will need to replace this one in. Shouuld be okay. Will need some testing.

Thanks all!

-----

1 point by jazzdev 5157 days ago | link

You're not creating any rain. You're just reporting the weather, and I appreciate the information.

Yeah, even non-tail recursive functions get optimized by the compiler to avoid any run-time lookup of syms in the global environment. I'm not sure that's the right trade-off, but it seems okay for a compiler. Although hackability is important too. I'll have to think about that and research what other Lisp compilers do.

The split problems seems to be a compiler bug. I'll have to investigate that more.

And Jarc should include strings.arc, there's a conflict with java.lang.String.trim() but that should be okay.

-----

1 point by akkartik 5159 days ago | link

rocketnia: what is this lathe you keep talking about? :) I've been unable to find a link. Is it publicly available?

-----

2 points by rocketnia 5159 days ago | link

I talk about Lathe at http://arclanguage.org/item?id=11610, which links to a blog post, which links to the GitHub repo thaddeus already linked to. ^_^

The blog post is still a pretty comprehensive look at Lathe. A bit has happened since then, but the most useful thing is Jarc compatibility, and even that isn't on the master branch yet; Lathe's approach to namespaces still doesn't harmonize with Jarc as well as I'd like.

-----

1 point by akkartik 5159 days ago | link

Ah that article. Yes, I just didn't associate it with lathe.

Google couldn't find it when I searched either (lathe is in one of the comments). Arc needs a search box. /me ducks :)

-----

3 points by thaddeus 5159 days ago | link

http://github.com/rocketnia/lathe

-----