"I just realized: it's possible to add in hyper-static scope to Arc while retaining full backwards compatibility . (Crazy, no?)"
I think you forgot the main reason why backwards-compatibility isn't very feasible: Macros.
; ===== util.file ====================================================
(fn (func . seqs)
; Lets you write (map x seq (+ 1 x)) in place of
; (fun-map (fn (x) (+ 1 x)) seq).
(mc (x seq . body)
; ===== this-is-fuuun.file ===========================================
(var fun-map (list " | O | X"
"X | X | O"
" | X | "))
(pr (map line fun-map (+ line "\n")))
Arc macros behave as though macroexpansion were simply about constructing some lists of symbols. But we really want each macro-inserted symbol to be looked up in that macro's lexical environment.
In Penknife, I handled macros by taking advantage of an existing assumption I was making about modules: Assume there are no side effects during the loading of the program, so that we can record the macroexpansion results to a file as a precompiled program without corrupting the program's behavior. Then any code that had a macro in scope at compile time will have a doppelganger of that macro in scope at run time. Whenever we encounter a variable during program execution, we can resolve it by looking up a macro value, accessing its lexical environment, and repeating until the original variable binding is in scope.
Penknife didn't really embrace the hyper-static global environment, but it would have been built upon the same sort of basis: Each file would have started in its own fresh environment, and some commands (namely, imports) would have worked by replacing the current environment.
"The definition of "=" is the same: if the variable exists, mutate it, otherwise create a new variable."
The behavior I'd use is that any compile-time variable access (even under a lambda) creates a new, uninitialized variable binding if a binding doesn't already exist.
; Create bindings for 'even and 'odd, then set the value of 'even.
(= even (fn (x) (case x 0 t (~odd:- x 1))))
; Set the value of 'odd.
(= odd (fn (x) (case x 1 t (~even:- x 1))))
If you wait to create the variable bindings until assignment time, then even's reference to "odd" is initially unbound, and you have to somehow associate it with the binding of 'odd created in the second line.
"I think you forgot the main reason why backwards-compatibility isn't very feasible: Macros."
I didn't forget: Nulan completely solved the macro hygiene problem after all. But that's a more extensive change so I figured I'd save it for after the basic hyper-static scope system is in place.
In fact, assuming Arcueid does implement my proposal, I would actually go in and make a new version of arc.arc that uses "var" and has hygienic macros by default. Then you could simply load up the new arc.arc to get all the shininess. But loading the old arc.arc would have full compat with existing Arc 3.1 programs.
"I think you've been resolving this by writing macros so that the procedures are inserted directly into the macro result, rather than referred to indirectly by symbols. Right?"
Nope. Macro hygiene in Nulan just uses the already existing box implementation. It's really easy, really simple, and really fast. Seriously, boxes are awesome. No need to complicate things.
The way to solve it in Arc: just provide a function called "get-variable-box" which is only available at compile-time and it returns the box for the variable.
Then you change quasiquote so it uses "get-variable-box" rather than inserting the symbol directly. Bam, hygienic macros with no additional runtime cost, and extremely small compile-time cost. And they look and feel just like Arc macros, so you don't lose any power or convenience. No clunky Scheme macros, huzzah!
Once I understood that the fundamental problem was with dynamic scope, and the best way to solve it is with boxes (or similar), everything became super easy and awesome.
"The behavior I'd use is that any compile-time variable access (even under a lambda) creates a new, uninitialized variable binding if a binding doesn't already exist."
Yeah I'd do that too, if I wanted to graft dynamic variables onto a hyper-static system. But since Arc uses dynamic variables, I proposed to graft hyper-static onto it instead.
"But that's a more extensive change so I figured I'd save it for after the basic hyper-static scope system is in place."
The middle ground doesn't seem worthwhile to me. When programmers work with with Arc-style unhygienic macros, at each use site, the variables in scope must (mostly) match the variables the macro author expected. So I think people who like using macros will be happiest if they systematically keep the variable names consistent across all the code in their program (even others' code), at which point namespace mechanisms just get in the way.
"Nope. Macro hygiene in Nulan just uses the already existing box implementation . It's really easy, really simple, and really fast. Seriously, boxes are awesome. No need to complicate things."
I think you caught me on a technicality. :) I see "procedures are inserted directly into the macro result" as a general approach. Mutable boxes make it possible for this approach to achieve late binding. Elsewhere in this discussion you tilt the technicality closer to my phrasing, since you recommend to let users build boxes out of getter and setter procedures.
Anyway, I'm a fan of that approach when it works, but it doesn't work so well when compilation is involved: The macroexpanded code contains unserializable values--namely, the procedures or boxes we're talking about. This is a lesson I learned with Penknife, where I at first had macros insert boxes, and then had to reengineer this so macros inserted step-by-step treasure maps for how to find a variable from the run time environment.
"Yeah I'd do that too, if I wanted to graft dynamic variables onto a hyper-static system. But since Arc uses dynamic variables, I proposed to graft hyper-static onto it instead."
How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix.
"The middle ground doesn't seem worthwhile to me."
Retaining Arc compatibility in general doesn't seem worthwhile to me, but a lot of people want it, so I gave a system that retains Arc compatibility while tacking on some new shininess. Nulan doesn't have to worry about Arc compatibility, so it has pure hyper-static scope and hygienic macros by default.
"How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix."
Easy: I have a macro called "defs" that handles mutual recursion:
(if (is x 0)
(odd (- x 1)))
(if (is x 0)
(even (- x 1))))
The above macroexpands into this:
(= even (fn (x)
(if (is x 0)
(odd (- x 1)))))
(= odd (fn (x)
(if (is x 0)
(even (- x 1)))))
Basically, it first creates new boxes, and then it assigns the functions to the boxes. This is one of a few reasons why I prefer mutable boxes over immutable boxes. Though, you could probably have "defs" expand to a Y-combinator instead, if you really wanted immutability...
"I think you caught me on a technicality. :)"
Maybe it was just a simple miscommunication. What you were talking about sounded exactly like the technique of splicing function values using macros:
What I'm talking about happens entirely at compile-time using boxes. The effect is very similar, but the implementation is very different.
"Anyway, I'm a fan of that approach when it works, but it doesn't work so well when compilation is involved: The macroexpanded code contains unserializable values--namely, the procedures or boxes we're talking about"
Also, what's the point in serializing boxes since functions still can't be serialized? If you found a way to serialize functions, then it'd be much more useful to be able to serialize boxes.
"Easy: I have a macro called "defs" that handles mutual recursion"
While I appreciate 'defs, it's a non-answer. The even/odd example I posted and the evenp/oddp example dido posted are idiomatic Arc code. While you and I don't care much about Arc compatibility, it's something dido wants for Arcueid, so these examples should work without modification.
I'm about to disagree with myself, but first I want to reiterate and clarify what I was saying at "caught me on a technicality":
For this discussion I don't see a much of a reason to distinguish between macros which insert mutable boxes and macros which insert functions. Either system can pretty much support the other as a special case: We can translate spliced boxes into spliced getter/setter functions, and we can translate spliced functions into spliced functions-in-the-box. Because of that equivalence, these systems share the disadvantage of being challenging to serialize.
If dido considers compilation to be important (do you, dido?), then this hygiene approach might be unsuitable, and thus the use of first-class namespaces might be unsuitable. (As I explained at "match the variables the macro author expected," first-class namespaces make hygiene more important.)
"What I'm talking about happens entirely at compile-time using boxes."
Ah. I think you have a point!
I seem to remember understanding this before, when you and I talked about Nulan compilation in depth, but I guess I had to retrace the steps just now.
Anyhow, get-variable-box is fantastic IMO, but first-class namespaces still might not be ideal for Arcueid due to Arc's unhygienic macros.
dido, are you comfortable with breaking existing Arc macro idioms in favor of hygiene?
I have a convoluted but surprisingly comprehensive idea of how to integrate get-variable-box into a system that's compatible with unhygienic Arc macros, but I've put it in a separate simultaneous post: http://arclanguage.org/item?id=17464
Actually, it's two separate posts, because it's otherwise too long for the forum. If this becomes a tl;dr scenario, I won't be surprised. ^_^
"While I appreciate 'defs, it's a non-answer. The even/odd example I posted and the evenp/oddp example dido posted are idiomatic Arc code. While you and I don't care much about Arc compatibility, it's something dido wants for Arcueid, so these examples should work without modification."
For this example, let's suppose there was a file "foo.arc" that contained idiomatic Arc code that implements evenp/oddp. This code works in Arc 3.1. It will work in my system as well, because undefined symbols automatically create new boxes. Basically, it'll work, but name collisions are possible, just like in Arc 3.1.
If you then write a new file "bar.arc" that uses hyper-static idioms (var, defs, etc.), it can import "foo.arc" and everything will work fine. "foo.arc" will clobber any existing evenp/oddp definitions, but "bar.arc" will not clobber "foo.arc". And of course "bar.arc" can use "w/include" and "w/exclude" to prevent "foo.arc" from clobbering things.
If you wanted to make it so that "foo.arc" behaves correctly without needing to use "w/include" and "w/exclude", you would indeed need to rewrite it to use "defs". But it's still usable even without a rewrite. So it's a perfectly graceful degradation.
My system is designed so that it can correctly use all existing Arc 3.1 code, while new code is written with the hyper-static idioms. Then, slowly, old code can be migrated to use hyper-static scope, until eventually you could make Arc purely hyper-static.
There's three issues I see with my proposal:
1) If you're writing Arc code in a hyper-static fashion, you really want "arc.arc" to be changed to be hyper-static. But old Arc code will need the non-hyper-static "arc.arc". I think the simplest solution to this is to have two versions of "arc.arc", one that uses hyper-static scope, and one that doesn't. Then you would need to make sure to load the non-hyper-static version before loading Arc 3.1 code. This could be automated a tiny bit by using a macro, something like "w/arc3".
2) "load" occurs at run-time, which is why my definition of "w/include" needed to use "eval". Nulan doesn't have this problem because file importing occurs at compile-time. Perhaps the best way to solve this is to keep "load" as-is, and add in a new "import" macro that does all its work at compile-time.
3) If you think (eventually) making Arc purely hyper-static is a bad thing, you won't like my proposal.
"Am I getting this right? This sounds very workable. :) And whaddayaknow, Nulan works. ^_-"
Yes, that's more or less correct. The one detail that's different is... Nulan doesn't have a "get-variable-box" function. The reason is because "quote" internally uses (the equivalent of) "get-variable-box". So in Nulan, rather than using "get-variable-box", you'd just use "quote". And if you want to break hygiene, you'd explicitly use the "sym" function.
I mostly followed along, but I don't understand "It will work in my system as well, because undefined symbols automatically create new boxes." You were talking about having them create new boxes at assignment time, and I was recommending compiling-a-reference-time instead so that we don't get an unbound variable error in the first definition.
How it works is, anytime the compiler sees an undefined symbol, it creates a new box for it like as if it had been created with "var".
Another way to think about it is... the compiler would replace this:
(= foo (fn () ... bar ...))
(= bar (fn () ... foo ...))
(= foo (fn () ... bar ...))
(= bar (fn () ... foo ...))
What happened is, when it encountered the undefined variable "bar", it created a new box for it. Then it encountered the undefined variable "foo", so it created a new box for it. Then it did the assignments like normal.
Given how you said "compiling-a-reference-time", I think we're talking about the same thing. Why did you mention assignment time?
Ah, sorry, huge miscommunication and misunderstanding on my part. I've actually been agreeing with you all along.
A large part of the problem is that I've been thinking about my proposal as two separate parts: one part deals with backwards compat with Arc, and the other part describes a hyper-static system for Arc.
When I was talking about "defs", I was talking about the hyper-static part. But you were talking about the backwards compat part. Hilarity (?) ensues.