It's been 15 days since I've pushed any changes to arc2js, so I'll describe some of the major changes.
For starters, I basically threw out the old compiler (which was single-pass) and started over by writing a multi-pass compiler. Before you start cringing and throwing tomatoes at me, I only rewrote the compiler, which means that most of my old work is still there, just with some minor tweaks.
This new compiler is much better than the old one. It's roughly comparable in number of lines, but it was much easier to implement and is much easier to reason about and understand, and it supports some really nifty features that the old compiler basically would have barfed on.
Probably the biggest change is optimizations. I've put quite a bit of work into optimizing common Arc idioms into super-efficient JS code. I figure if I can optimize basic things like "let" and "do" well enough, it'll be fast, even if I ignore optimizations in less-commonly-used things.
My old compiler optimized "let" blocks, but only in very specific situations: it had to be the only expression in the function's body, and it had to be in statement position. My new compiler optimizes "let"s almost everywhere, even in expression position and at the global scope.
Here's an example:
(fn ()
(foo (let a 10 a)))
Let's compile it:
function () {
var a = 10;
return foo(a)
};
Woah, neat, it compiled into a "var" statement. But what about this...?
(fn ()
(if a (let a 10 a)))
It will compile into this:
function () {
var b;
return a && (b = 10, b)
};
Huh? What's going on here? There are multiple phases in my compiler. One is the "shorten" phase. That phase's job is to take local variable names and replace them with gensyms. Later on, those gensyms can be replaced with an identifier like "a" or "b", etc.
This, combined with static free variable analysis enables me to safely optimize things like "let" blocks into "var" statements, because of variable renaming.
Another phase is the "optimize" phase, which does what it says it does. But the way it does it is pretty cool. Most of the phases deal with plain old Arc lists, they don't deal with JS code at all. What basically happens is, when the optimize phase sees something like this:
(foo (let a 10 a))
It will rewrite it to this:
(let a 10
(foo a))
And now because the "let" is on the outside, it's in statement position, which means my compiler can rewrite it to a "var" statement. That's fine and dandy if the "let" is the first argument, but what about this?
(if a (let a 10 a))
It'll be rewritten to this:
(let b nil
(if a (do (= b 10) b)))
We still move the "let" to the outside, but now we initialize it to nil and do the assignment inline. This works no matter how nested the "let"s are. Let's consider a more complex example. This time, we're gonna generate some DOM nodes using the dom.arc library:
First, let's turn off optimizations and see what the above compiles into:
(function (a) {
a.setAttribute("foo", "bar");
a.appendChild((function (a) {
a.setAttribute("bar", "qux");
a.appendChild((function (a) {
a.setAttribute("qux", "corge");
a.appendChild((function (a) {
a.setAttribute("corge", "foobar");
return a
})(document.createElement("div")));
return a
})(document.createElement("div")));
return a
})(document.createElement("div")));
a.appendChild((function (a) {
a.setAttribute("yes", "no");
return a
})(document.createElement("div")));
return a
})(document.createElement("div"));
Okay, so we can clearly see the nested structure caused by the "let" blocks (which is pretty cool), but we can do better. Let's turn on optimizations:
(function (a) {
a.setAttribute("foo", "bar");
var b = document.createElement("div"),
c,
d;
b.setAttribute("bar", "qux");
c = document.createElement("div");
c.setAttribute("qux", "corge");
d = document.createElement("div");
d.setAttribute("corge", "foobar");
c.appendChild(d);
b.appendChild(c);
a.appendChild(b);
var e = document.createElement("div");
e.setAttribute("yes", "no");
a.appendChild(e);
return a
})(document.createElement("div"));
Woah! Much better. Notice how it took our nested (and inefficient) code and optimized it into flat super-efficient code using "var" statements. This lets you use Arc idioms, and know that it'll be expanded into fast JS code.
Now, I said my compiler optimizes "let"s at the global scope. Let me give you an example of what I'm talking about. First, the Arc code:
(do (let a 5 a)
(let a 10 a)
(let a 15 a))
Let's compile it without optimizations:
(function (a) {
return a
})(5);
(function (a) {
return a
})(10);
(function (a) {
return a
})(15);
Okay, so it creates one function per "let" block, perfectly normal. Let's turn on optimizations now:
(function (a, b, c) {
a;
b = 10;
b;
c = 15;
return c
})(5);
Huh! Would you look at that. Now it only created a single function, with the 2nd and 3rd "let" blocks being ordinary variables inside of it. What's especially cool about this is that it's transforming Arc code into Arc code, like a macro.
Thus, theoretically, an Arc compiler like ar could make these same optimizations, flattening out nested "let" blocks. In practice, though, Racket probably does optimizations of it's own, so that probably wouldn't help much. But it's definitely very useful for an Arc to JS compiler like arc2js.
As a final example, let's consider the "each" macro. In JS, the idiom for looping is the "for" loop:
for (var i = 0; i < foo.length; i++) {
... foo[i] ...
}
You end up using the humble "for" loop a lot. In Arc, however, you can just do this:
(each x foo
... x ...)
What does the above compile into...?
(function (a) {
var b = 0,
c;
while (b < a) {
c = foo[b];
...
c;
...
++b
}
})(foo.length);
Well, dang. Look at all that boilerplate! But you don't need to worry about that, you can just use "each" and rest assured it will result in super-fast code.
At this point I consider arc2js suitable for production use (i.e. I would use it for my own code).
Oh yeah, by the way, I've split arc2js into two parts: arc2js.arc and compiler.arc
Most of compiler.arc contains stuff that isn't specific to JS, and then arc2js.arc adds in the JS-specific stuff. Thus, it should hopefully be pretty easy to take compiler.arc and then write, say, arc2python, without needing to change too much.
So, all those optimizations I talked about are in compiler.arc, which means they would work with a hypothetical arc2python as well.
By the way, I just realized that if sref were a macro, you could indeed use this syntax. The problem is that : is already used for composition, and sref is a function, so it would try to compose two numbers (which would of course fail spectacularly if you tried to call it).
On the other hand, if sref were a macro, then I could have it check for (compose ...) and then you could use : syntax for list slices/steps. Alas, that would be a pretty big change, so I think that would be better suited for Arubic, rather than base ar. This did give me the idea to make sref into a macro, so thanks for that.
If not ':' then '..' is also possibility, like (a 5..9),
but to be honest i like your version, it's more Lispy. But maybe syntax '..' could improve readability? Compare:
(a b c)
(a b..c)
You write (about ar) "Much of Arc remains unimplemented", so is it better to use mzsheme version for now?
Nope, no can do. "." is used for ssyntax as well. To be specific, foo.bar is equivalent to (foo bar) Thus, setf would need to be a macro, or we'd need to give up on ssyntax. I like ssyntax, so I'd go the macro route, personally.
---
"You write (about ar) "Much of Arc remains unimplemented", so is it better to use mzsheme version for now?"
Oh, no! I've been using ar for quite some time now, it works beautifully. I think awwx is being silly when he says that "much of Arc" is unimplemented. It's mostly implemented, and the stuff that isn't implemented, I don't use, so I don't notice... I'm at the point where the idea of using Arc 3.1 is absurd: ar is definitely the way to go.
My libraries are designed specifically to work only with ar, though some may work on Arc 3.1 with some tweaks... My apologies for the incredible number of files in the ar branch, there's not much I can do about that...
---
By the way, you shouldn't use my old branch. It's there for historical reasons, and so that I can pull in changes piece-meal into current ar. Using the latest version of ar (I give the link above, in this post) is the way to go. Sorry for the confusion.
"Nope, no can do. "." is used for ssyntax as well."
Even though . is used for ssyntax, .. isn't, at least in Arc 3.1. While (ssyntax 'a..b) returns t, ssexpanding anything with .. in it throws an error.
You've talked about classifying ssyntax as prefix, postfix, and infix before, so I suspect you'd want a..b to expand through (a .b) to (a (get b)). As for me, I think it's nicer to treat multiple-character sequences like .. as their own operators.
"(though Racket is better on docs, but maybe I'm wrong)"
The docs for Arc range from nonexistent to outdated to poor. Racket's docs aren't that much better, but I agree that Arc's documentation is horrible.
However I haven't actually needed Arc documentation, because I can just open up arc.arc and read the definitions themself. My hardest thing with Arc is writing my own custom function/macro, only to realize later that it already exists. An example is `or=` which I didn't know about until rocketnia pointed it out to me. Ditto for `awhen`
---
As for your suggestion for slice syntax, I already did that. You can find it here:
It won't work on the latest version of ar. I have a fixed version on my harddrive, but haven't pushed it yet...
With that library, you can now do this:
(foo 0 1) ; same as foo[0:1] in Python
(foo 0 5 2) ; same as foo[0:5:2] in Python
Basically, it takes (foo ...) and converts it into (cut foo ...) so (foo 0 1) is the same as (cut foo 0 1).
It also supports negative indices, just like Python. I tried to keep it very close to Python's semantics, as I think this is one of the (few) awesome things about Python. In fact, the only difference is the following:
(foo 5 0) -> error: start index must be smaller than the end index
foo[5:0] -> []
Great that it's already there. Is there a way to do foo[5:]?
And about docs - you are right, the best is to read the source-code. But the problem (as you pointed out) is when you have something you want to solve, it's hard to search for it in source-code. If you want to append strings and you don't know that + does it, source code will not give you an answer (unless you read the whole thing carefully). In Racket, you can search using human language in Racket Reference or Racket Guide.
Anyway, is there a centralized page, where you can get most links to Arc resources?
It's not as short as the Python version, but it works well enough. Basically, you can pass nil to mean the "default value".
That means that (foo nil nil) is the same as foo[:] in Python (this creates a copy of the list). The defaults are (foo 0 (len foo) 1)
---
"If you want to append strings and you don't know that + does it, source code will not give you an answer (unless you read the whole thing carefully)."
Right. I was working on solving that as well, by grouping arc.arc into different "sections". So you'd have a section for "assignment", a section for "scripts", a section for "file system", etc. I believe this approach is actually better than English prose documentation. Unfortunately, I have not done this for the latest version of ar... it's only on my old branch[1], which is sorely outdated.
---
"Anyway, is there a centralized page, where you can get most links to Arc resources?"
If you're looking for libraries, you can try Anarki[2], or rocketnia's Lathe[3], or my lib branch[4].
My branch currently has a library that implements slicing (already shown), a lib that implements a very simple and lightweight object system, a lib that implements a hacky (but working!) namespace/module system that uses inheritance, and a compiler that can take an Arc expression and convert it into a JavaScript string, that can then be executed in your favorite JS engine.
I should warn you that my arc2js library is severely outdated. I've made many many changes and improvements, and plan to push them to my branch (eventually). In addition, the import.arc library is very very hacky (in particular, the way it deals with macros), so I personally wouldn't use it just yet. It's mostly a proof-of-concept at this point.
I'm not shader, but I'm guessing that it's because... you can't.... bind values to strings...?
I mean, yeah, you could have a table where the keys are strings, and treat that as the metadata, but uuugh it's clunky and I've found myself disliking the whole "global table to hold data" thing more and more, recently.
Then again, I don't know how their compiler works, so I might be completely off-base here...
I think this should be configurable in ar, with the default being to follow Arc/3.1, but still allow the programmer to make nil different from 'nil.
Hey awwx, how hard would it be to make this change in ar right now? From what I can see, "nil" is just a global variable bound to the symbol 'nil. So, what if I did this...?
(ail-code (racket-set! nil (uniq)))
It seems to work okay in the couple tests I did, but I wonder if it would break anything...
Yep, that's the vision :) Macros stay, anything else can go.
(I hope to also lose the parens around the recursive call to and.)
I'm not sure it'll be a good idea to rely this much on indentation. It starts looking like concatenative languages with their reliance on 'words'. But hey, it's up to the programmer to use tools tastefully, and to figure out what good taste is.
"(I hope to also lose the parens around the recursive call to and.)"
You mean remove the parens inside quasiquote? Good luck with that.
---
"But hey, it's up to the programmer to use tools tastefully, and to figure out what good taste is."
Exactly. I'd rather have implicit parentheses, because then I can choose whether to include parens or not... if I understand your system correctly, you can still wrap everything in parens, just like in Arc. So it's up to the programmer to decide whether to use parens or not. I think that's a very good thing.
"if I understand your system correctly, you can still wrap everything in parens, just like in Arc."
Yes, that's the hypothesis of paren-insertion. So far I've seen two troubling cases, where you have to worry about paren-insertion even if you are explicitly adding parens everywhere:
a) A long data list that wraps to multiple lines:
(a b c d ...
e f g h ..)
My solution is to distinguish this from code indentation by using a single space.
(a b c d ...
e f g h ...)
Most of the time you can use any number of spaces to indicate indent, just like in python. But I'm hoping nobody wants an indent of 1.
b) The second case is arc's multi-branch if statements:
(if
cond1 then1
cond2 then2
else)
or pg+rtm's approach:
(if cond1
then1
cond2
then2
else)
I'm not sure what to do about that. It's why the body of and has two nested ifs where one would seem to do -- I couldn't figure out how to indent a single if :/
Hmm, now that I see your cases and mine I realize in all these cases you can always back off to inserting parens everywhere, and things will just work. Lines where the first token is a '(' never insert a fresh '(' before.
[1] Sorry I edited grandparent to add the pg+rtm case you were adding at the same time in your response.
"Hmm, now that I see your cases and mine I realize in all these cases you can always back off to inserting parens everywhere, and things will just work. Lines where the first token is a '(' never insert a fresh '(' before."
Right. At least, that's how I would expect it to behave. Worst-case scenario, you just put in parens just like Arc. But even then, you can at least leave off the initial parens:
if (foo)
(bar)
(qux)
There is one thing that concerns me, though... it's very common to want to return a value from an "if", like so:
if (condition? x)
+ x 10
x
Whoops, the last "x" is wrapped in parens... but you don't want to call x, you want to return it directly. So you have to wrap the "if" in parens:
A single word on a line is never wrapped in parens, though it may close parens inserted in a previous line.
I keep thinking there's an example similar to this one of yours that won't parse correctly unless you explicitly wrap the if in parens:
if (foo)
(bar)
(qux)
But I can't think of a counter-example :)
Even if the algo is unambiguous this approach will fail if people can't easily wrap their heads around what it will do. But that's a matter of taste, I suppose.
Ah, right, I thought you meant getting rid of `( ... ) Yeah, that seems okay. But in that case, you'll probably never be able to completely fix the issue where you have a long list of data, as described in the other post.
You could probably come up with a compromise that works the majority of the time, though. For instance, you could disable indentation inside of '( ... ) but still allow for indentation with `( ... )
Yeah that's a good point. If this approach is too aggressive I can back off and disable paren-insertion inside parens. I already do that to make things like this work:
Yeah if I could get rid of that one we'd be able to add macros to C or python :) But imagine a language that can look like python most of the time. But if you want macros, then the parens return.
"...they might as well not have eval in the first place."
Although I agree with you in principle, eval is needed in at least one place: load. So I don't think we can necessarily get rid of it completely, unless we used a different method of loading/evaling files...
eval is also used in a few other places in ar, but not very many. So rather than getting rid of it, I think a better solution is to say, "eval is generally a bad idea, so use something else unless you absolutely positively must use eval" which is exactly what we do.
By the way, I'll note that when I tried to use eval to expand a macro at runtime, it ended up being horrendously slow. I changed it to use a function (which did the same thing, but with thunks) and suddenly my code ran at least 100 times faster than it did with eval.
Ah, you're right. You may not be able to do away with eval.
And do you know what you just helped me realize? If we've already identified ourselves as a practical lisp, why should we try to get rid of features like eval in the first place?
A theoretical lisp tries to get rid of unnecessary features to approach being as minimal as possible. A practical lisp has already decided that it would rather be useful than theoretically minimal.
So a practical lisp probably shouldn't worry about getting rid of things unless they are actively causing problems. If it wants to be as useful as possible, it might as well leave in unnecessary features. Because they're probably still useful some of the time to some people, even if they're not totally necessary.
By the way, regarding the title of this post... I've already been working on arc2js[1], and evanmurphy also has LavaScript[2]. So yes, we are working on a "comparable" thing for Arc. I'm in the process right now of rewriting large chunks of my compiler, and hopefully I'll be able to make a release within the next couple weeks.
I've been adding in some stuff like static analysis, so it can know which variables are free, and use that information to do things like shorten variable names. I'll note that the version of arc2js that's on git is far outdated compared to the version on my harddrive.
In practice, in both my Arc and JS code, I use eval only when I absolutely have to... which is almost never. It's usually when I'm doing something incredibly hacky, like trying to eval macros at runtime...
Anyways, the traditional wisdom in JS is to avoid eval as much as possible. 99.99% of the time, eval is bad. It slows down your code, makes things like static analysis harder (or impossible), and it's usually completely unnecessary.
So, I don't see any practical problems with removing eval, but I do see some philosophical problems. eval has been a part of Lisp for so long, that it almost feels like a Lisp without eval isn't really a Lisp...
In any case, from a practical perspective, a lack of eval isn't a big deal at all. But I can see why some people might want eval (for non-practical reasons).
