I think the fundamental reason for this is that macros don't evaluate their arguments, but functions do. Your example macro is written like as if it were a function, but it's not a function. If you instead wrote it as a macro, like so:

  (mac foo (a b) `(cons ',a ',b))

---

"This apply works for some macros (like Pauan's def example), just not all of them."

It still works if you call it like this: (apply foo '(1 '(2 3)))

As I mentioned in my e-mail, I agree that is an inconsistency when trying to make apply work with fexprs, which is part of why I prefer Kernel's approach of requiring you to wrap the fexpr before calling apply on it... but that doesn't mean that a version of apply that works on fexprs is necessarily broken, just a bit hacky and inconsistent.

I wouldn't call apply broken for the reason you demonstrate above (with the foo macro), because foo really should be written as a function, or its definition should be changed to make use of the non-evaluating nature of its operands.

---

It seems to me that when you talk about apply "working correctly" you're expecting it to somehow guess that the foo macro is behaving like a function, so applying it should treat it like a function, even though it's a macro. But precisely because macros/fexprs might or might not evaluate their arguments, apply cannot guess that.

apply can know whether its argument is a function or fexpr, but not whether the fexpr will actually evaluate its operands or not... so it seems your concept of apply "working correctly" would require apply to essentially read your mind, or do some sort of crazy static analysis and make guesses.

Let me put it like this. When calling apply on something, let's pretend that apply maps quote over the argument list, so that these two are equivalent:

  (foo 1 '(2 3))
  (apply foo '(1 (2 3)))

  (foo 1 (2 3))
  (apply foo '(1 (2 3)))

Really, if you want to solve this, I would suggest having two operators, one for the function-like behavior, and one for the fexpr-like behavior. I'll call these two "apply" and "operate", respectively. So, assuming foo is a function, then the following are equivalent:

  (foo 1 '(2 3))
  (apply foo '(1 (2 3)))
  (operate foo '(1 '(2 3)))

  (apply (wrap foo) '(1 '(2 3)))

  (eval `(,foo 1 '(2 3)))

---

"I'd love comments on this way of decomposing things."

"Do you have a better example of a macro that breaks with apply, a macro that's actually written as a macro, rather than as an almost-function?"

How about and in the previous thread? I tried to use it like this:

  (def all-trues(xs)
    (apply and xs))

  (all-true (list '(1 2))) ; => error

But that's completely incorrect because you always always have to know whether it's a function or a macro, even in a language like Arc or Scheme, because they are fundamentally different things.

Even fexprs aren't the same as functions because they don't evaluate their arguments, and argument evaluation is important, so it's very important that you understand whether something is a function or fexpr, and if it's a fexpr, which arguments are evaluated, and how.

That's a large part of the reason Kernel prefixes fexprs with $ by convention: it really helps you to notice right away the difference between the two, and it's an important difference.

---

"How about and in the previous thread?"

Yeah, I think that qualifies as a good example. It breaks for the exact same reason that your foo macro breaks: it evaluates its arguments, so you have to double-quote them:

  (all-true '('(1 2)))

"It's pretty clear that 'apply for macros' is an ill-posed, arguably meaningless problem, and I thank you for helping me understand that."

But the whole point is that the list is just a list of elements! Imagine it's coming from a whole complex sequence of functions. Don't think of it like this:

  (all-true '((1 2))

  (all-true l) ; with l bound to ((1 2))

"that's completely incorrect because you always always have to know whether it's a function or a macro, even in a language like Arc or Scheme"

What I called the 'point of lisp' was my sense of the ideal. The promise of a high-level language is that you can create new names and use them like primitives that the language came with.

  (let x 3
    bar.x) ; Does the language come with let? Who cares?

Sure, I already considered that. That is, after all, the most common use for apply. And I agree that's ugly. You'd have to say something like (all-true (wrap-quote l)) or somesuch. Ugly.

---

"Isn't that simply hideous?"

Sure is!

---

"The promise of a high-level language is that you can create new names and use them like primitives that the language came with."

And fexprs/macros let you do that. That wasn't my point. What you said is that the point of macros is being able to call things without caring whether the thing is a macro/function/whatever. That view is incorrect because the meaning of the expression differs depending on whether it's a function or macro.

For instance, in the "let" expression you just gave, it doesn't evaluate the variable "x", but you only know that because "let" is a macro. If it were a function it would throw a "x is undefined" error. So you have to know whether something is a function or macro in order to use it correctly.

The same is true of apply: it has two different (and inconsistent) behaviors depending on whether its argument is a fexpr or a function. And that's a problem because it's confusing and makes it really easy to get it wrong. Hence why Kernel makes you explicitly use wrap to specify your intent.

---

"You've explained the technical reasons why 'apply for macros' is hard, and that makes sense. But the solution is to not use apply with macros, not tie ourselves up in knots with quote."

Sure, and even with that approach, if your language has wrap/unwrap, then you can still use apply on macros, you just need to wrap them first:

  (apply (wrap foo) ...)

  (eval `(,foo ...))

No, I thought about that when I built let. And then I forgot about it. Now when I see that expression I think, "oh, I'm creating a binding." The point of having power is to not have to derive things from first principles every time we see them.

"And fexprs/macros let you do that. That wasn't my point. What you said is that you want to be able to call things without caring whether the thing is a macro/function/whatever."

Yes, but you still need to carry around that knowledge in your subconscious, and retrieve that knowledge every time you use "let". That was my point: that the distinction between them is important.

---

"The point of having power is to not have to derive things from first principles every time we see them."

Yes... and I'm agreeing with you.

You said that the point of macros is not having to care whether something is a function/macro or not. But on a very fundamental level you do have to care, just like how you have to care about whether a variable is a string or a number.

I was only disagreeing with that statement, that you don't need to care about the differences between fexprs/functions. All the problems with apply come about because of misunderstandings of the differences between the two:

  ;; functions
  (foo '1 '(2 3))
  (apply foo '(1 (2 3)))

  ;; fexprs
  (foo 1 (2 3))
  (apply foo '(1 (2 3)))

  wart> = l '((1 2))
  wart> (and @l) ; error
  wart> (wrap.and @l) ; error
  wart> (eval `(,wrap.and ,@l)) ; error
  wart> (eval `(,wrap.and ,@(map (fn(_) (cons quote _)) l))) ; error

  (def operate (f args)
    (eval `(,f ,@(map (fn (_) (list quote _)) args))))

I'm not saying that's necessarily a good idea, but if it comes up often enough, might as well write a helper function for it.

---

And when I said "bring out eval" I meant like this:

  (eval `(,and ,@l))

Ah, this works :)

  (eval `(,and ,@(map (fn(_) (cons quote _)) l)))

  def all-true(xs)
    (and @mapquote.xs)

  > > Isn't that simply hideous?
  >
  > Sure is!

Can you implement vau like this?

  ; (vau (a b c) env ...body...)
  ; ==>
  ; (fn '(a b c) (let env caller-scope ...body...))
  ;
  (def vau '(parms env body)
    (eval (list fn (list quote parms)
            (list* let env 'caller-scope
              body))
          caller-scope))

What is 'caller-scope bound to at the top level?

---

"It seems all its drawbacks wrt vau involve hygiene, which I want to explore ignoring."

What drawbacks are those?

---

"Phew, that was a two-week-long tangent that involved me ripping wart down to its foundations before building it up again, and revisiting all my assumptions about how things work."

Or perhaps you're talking specifically about functions mutating their environment... well you can do that in Kernel too:

  (wrap ($vau ... env ...))

  ($vau ... env1
    ($vau ... env2
      ...))

  (fn '...
    (let env1 caller-scope
      (fn '...
        (let env2 caller-scope
          ...)))

I think so, yes!

I got to properly understand vau, to realize that its environment arg is never explicitly passed in.

  wart> caller-scope
  007scope.cc:122 No binding for caller-scope

If I just saw 'caller-scope in pseudocode somewhere, I'd implement it as an anaphoric variable of 'fn (and I'd implement a non-anaphoric variant of 'fn to base it on, so we'd basically be at vau again :-p ). So at the global scope, my implementation would just treat 'current-scope as a global variable.

  (env= caller-scope foo 5)

  (let global caller-scope
    ...
    (env= global foo 5)
    ...
    (eval ... global))

  ($let ((global (get-current-environment)))
    ...
    ($set! global foo 5)
    ...
    (eval ... global))

On the other hand, it's quite easy to define wart's magical fn in terms of vau... with no leaky abstractions! The only catch is that caller-scope will always refer to the global scope inside vau forms... that could be changed too, by extending/overwriting vau, but I didn't do that:

  (= get-current-environment (wrap (vau () e e)))

  ;; don't define this if you don't want caller-scope at the global level
  (= caller-scope (get-current-environment))
  
  ;; simple fn, without magical quoted args or caller-scope
  (= fn (vau (parms . body) env
          (wrap (eval `(,vau ,parms nil ,@body)))))

  ;; omitting other stuff like afn, no, caris, list, etc.
  ;; but they can all be defined in terms of simple fn
  ... 
          
  ;; makes these variables hygienic
  (with (with  with
         eval  eval
         let   let)
         
    ;; this recreates the part of wart that is currently handled in C++
    (= parse-fn (fn (parms env)
                  (if (caris x 'quote)
                      (list (cdr parms) nil)
                      ((afn (x vars parms)
                         (if (no x)
                               (list (rev parms)
                                     (rev vars))
                             (caris (car x) 'quote)
                               (self (cdr x)
                                     vars
                                     (cons (cdar x) parms))
                             (let c (car x)
                               (self (cdr x)
                                     `((,eval ,c ,env) ,c ,@vars)
                                     (cons c parms)))))
                       parms nil nil))))
    
    ;; overwrite with the complex fn which has quoted args and caller-scope
    (= fn (vau (parms . body) env
            (w/uniq new-env
              (let (parms vars) (parse-fn parms new-env)
                (eval `(,vau ,parms ,new-env
                         (,with ,vars
                           (,let caller-scope ,new-env
                             ,@body)))
                      env))))))

  (fn (a 'b . c)
    ...)

  (vau (a b . c) g1
    (with (a (eval a g1)
           c (eval c g1))
      (let caller-scope g1
        ...)))

One more caveat: I'm not sure how you implemented caller-scope. If it's hardcoded into every function such that functions can't have an argument called "caller-scope" then the above will work the same way. But if it's a global implicit parameter like in Nu and ar, then you'll need to search the arg list for an argument called "caller-scope", which is doable but would require some more code, especially for handling argument destructuring.

---

Anyways, I'm not saying you should necessarily do it this way in wart, but... I think it's cleaner, conceptually, for functions to be defined in terms of vau, rather than vau being defined in terms of functions.

Hmm... yeah, it's a pretty leaky abstraction. It's one of these features with ambient authority:

  (if designed like a global variable assigned at the beginning of each
  call)
  
  Right now, get the most recently started call's caller's environment.
  
  
  (if designed like a continuation mark)
  
  Right now, get the deepest-on-the-stack call's caller's environment.
  
  
  (if designed like a local variable bound by each (fn ...))
  
  Given a non-global lexical environment, its local variables must have
  been initially bound by some call, so get the lexical environment
  that was used to evaluate that call.

...Actually, the first two of these designs are broken! Say I define a simple fexpr like this one:

  (def idfn '(result)
    (eval result caller-scope))

  ((fn (a) ((fn () (idfn caller-scope)))) 1)  ; eval'd in global scope
  ((fn () (idfn caller-scope)))  ; eval'd in local scope of (fn (a) ...)
  (idfn caller-scope)            ; eval'd in local scope of (fn () ...)
  (eval result caller-scope)     ; eval'd in local scope of idfn
  caller-scope                   ; eval'd in local scope of (fn () ...)

Does Wart have this awkward behavior, akkartik?

---/change of topic---

To get back to the point, the local-variable-style version of the feature (the only version that works?) doesn't strike me as completely undesirable. It lets the programmer evaluate expressions in arbitrary ancestor stack frames, and that isn't a shocking feature to put in a debugger.

Wart returns the scope with a set to 1. Is that the one you mean? It's different from the value of:

  ((fn (a) caller-scope) 1)

http://github.com/akkartik/wart/blob/98685057cd/008eval.cc#L...

But I don't think it will become Kernel. Kernel seems really elegant as long as you give up quote and quasiquote. I suspect if you bolt them onto Kernel it'll have slid from its sweet-spot peak of elegance. A language that encourages quoting and macros will diverge significantly from Kernel.

http://github.com/akkartik/wart/blob/98685057cd/008eval.cc#L...

It tries to bind caller-scope after the params, so this fails:

  wart> def foo(caller-scope) 34
  wart> foo 3
  007scope.cc:77 Can't rebind within a lexical scope
  dying

In wart scopes are just tables. It's not a good idea to modify them, but nobody will stop ya :)

"Kernel's "get-current-environment" is not the same as the environment argument passed to fexprs."

Can you elaborate? In

  ($vau foo env
    ..)

  ($let ((env1 (get-current-environment)))
    ...
    ($vau foo env2
      ...
      ($let ((env3 (get-current-environment)))
        ...))

env2 is the dynamic scope where the $vau is called, it does not include bindings for foo, env1, env2, or env3.

env3 is the environment of the $vau itself. This environment inherits from env1, and thus is lexical just as env1 is. And so, it contains all the bindings of env1, in addition to bindings for foo, env1, and env2, but not env3.

So, by using this combination of lexical and dynamic scope, you can express a huge variety of different things in Kernel.

---

Let me explain via a more elaborate example:

  ($let ((global (get-current-environment)))
    (display (eval 'x global))
    
    ($set! global $foo
      ($vau (x) env
        (display (eval 'x env))
        (display (eval 'env env))
        
        ($let ((inner (get-current-environment)))
          (display (eval 'x inner))
          (display (eval 'env inner))
          (display (eval 'global inner))))))
          
  ($let ((x 10))
    ($foo 20))

Then, we create a $vau form and assign it to the variable $foo in the global environment. This is the same as saying ($define! foo ...) at the top level, but is necessary when inside a $let.

Now, at the bottom, we use $let to bind the variable 'x to 10 and then call $foo with the single argument 20. First, $foo tries to evaluate the variable 'x in the dynamic environment "env". This should display 10, because it's using the binding in the $let where $foo is called.

We then evaluate the variable 'env in the $vau's dynamic environment. But the variable 'env does not exist in the dynamic environment, so it throws an error.

Then, we use a $let inside the $vau to grab a hold of the $vau's inner environment and bind it to the variable 'inner. We then evaluate the variable 'x in the $vau's inner environment. This should display 20, because it's using the binding inside the $vau, which is bound to the $vau's argument 'x which was 20 when calling the $vau.

We then evaluate the variable 'env in the $vau's inner environment, which evaluates to the dynamic environment, because inside the $vau, the variable 'env is bound to the dynamic environment.

Lastly, we evaluate the variable 'global in the $vau's inner environment. This evaluates to the global environment because the $vau's inner environment inherits from the top-level $let where the variable 'global is bound.

--

Here is a picture showing the environments in the above code:

http://img42.imageshack.us/img42/1870/vauenvironments.png

The global environment is pink, the first blue environment is the $let's environment that inherits from the global environment, plus a binding for the variable 'global. The green is the $vau's environment which inherits from the $let's environment, plus a binding for the variables 'x and 'env. And the yellow is the $let's environment that inherits from the $vau's environment.

In Kernel, the dynamic environment is exactly analogous to dynamic scope. The difference is that lexical scope is the default, and you have to explicitly ask to evaluate things with dynamic scope, rather than earlier Lisps which used dynamic scope by default and didn't even have lexical scope.

Special variables in Common Lisp are similar to dynamic scope, except you're flagging only certain variables to be dynamic, whereas the dynamic environment in Kernel lets you evaluate any variable with dynamic scope.

---

"If we only create bindings using let and combiner parameters, the dynamic environment is just the caller's static environment, is that right?"

Yes. In fact, in Kernel, $let is defined just like it is in Arc, using $lambda. And $lambda uses $vau, and $vau is the only way to create new environments in Kernel, as far as I know. So, environments are all about $vau.

---

  ($define! $foo
    ($vau (x) env
      (list x env)))
      
  ($define! $bar
    ($vau (x) env
      ($foo x)))
      
  ($bar 10)

  ($define! $bar
    ($vau (x) env
      ($foo x env)))

  ($let ((x 10))
    ($bar x))

  ((wrap ($vau (x) env
           ($bar x env)))
   10)

---

"I thought about whether I could build lazy eval directly using $vau"

Ah! I considered this approach when I was building dynamic variables in wart, but it seemed less confusing to attach the scope to the binding. Wart is unlike common lisp in this respect. SBCL:

  * (defvar foo 34)
  * (defun bar() (foo))
  * (bar)
  34
  * (let ((foo 33)) (bar))
  33 ; foo is still dynamic. WEIRD.

  wart> = foo 34
  wart> def bar() foo.
  wart> bar.
  34
  wart> let foo 33 bar.
  34 
  wart> making foo 33 bar.
  33 ; dynamic scope

  ($vau () env
    (eval 'x env))

---

If you're talking about something similar to Racket's parameterize, you should be able to build that using dynamic-wind. Then something like this...

  ($parameterize ((foo 10))
    ...)

  ($let ((env   (get-current-environment))
         (orig  foo))
    (dynamic-wind ($lambda ()
                    ($set! env foo 10))
                  ($lambda ()
                    ...)
                  ($lambda ()
                    ($set! env foo orig))))

In fact, I'm gonna write a $parameterize form right now:

  ($define! $parameterize-redirect
    ($vau (target tree . body) env
      ($let ((inner (get-current-environment))
             (env   (eval target env)))
        (dynamic-wind
          ($lambda ()
            ($set! inner orig
              (map ($lambda ((var expr))
                     (list var (eval var env)))
                   tree))
            (for-each ($lambda ((var expr))
                        (eval (list $define! var expr) env))
                      tree))
          ($lambda ()
            (eval (cons $sequence body) env))
          ($lambda ()
            (for-each ($lambda ((var expr))
                        (eval (list $define! var expr) env))
                      orig))))))
                      
  ($define! $parameterize
    ($vau (tree . body) env
      (eval (list* $parameterize-redirect env tree body) env)))

  ($let ((x 5))
    ... x is 5 ...

    ($parameterize ((x 10))
      ... x is 10 in here ...
      )

    ... x is 5 again ...
    )

What about this?

  (def get-current-environment ()
    caller-scope)

  get-current-environment.
  thunk.caller-scope.

  (get-current-environment)
  ((fn () caller-scope))     ; Wart
  (($vau () e e))            ; Kernel