foo <- (fn (n) (eval `(+ ,n 1) caller_scope))

  foo <- (fn '(n) (eval ((fn () `(+ ,n 1))) caller_scope))

---

"I'm still not convinced I fully understand why eval and fn each add an indirection to caller_scope ."

(For the moment, I'm going to overlook eval here.)

That's just how you chose to build it, right? Isn't the whole point of caller_scope that it's specific to the current call?

I think the main issue you're encountering is that you always use the same name, "caller_scope", resulting in shadowing. Kernel uses an explicit parameter, as in ($vau () env ...), but Wart doesn't. This isn't necessarily a problem, because you can do something like this to work around the shadowing when necessary:

  (fn (x)
    (let my_scope caller_scope
      (fn (y)
        ...)))

I think this code will evaluate the ... as though it's just inside the (fn (x) ...), so that it can see the correct binding of caller_scope:

  (fn (x)
    (let fnx_scope ((fn () caller_scope))
      (fn (y)
        (eval '... fnx_scope))))

If you tweak the language a bit, you could make the implicit caller_scope parameter get shadowed by the explicit parameters. That is, if a function is of the form (fn (caller_scope) ...), you could let it ignore its implicit caller_scope binding altogether. (Currently Wart treats this just like any other duplicate argument name, and it throws an error.) In that case, you could use this technique:

  (fn (x)
    (let my_scope caller_scope
      (fn (y)
        (let caller_scope my_scope
          ...))))

---

"I'm still not convinced I fully understand why eval and fn each add an indirection to caller_scope ."

(Now to tackle eval.)

Are you sure 'eval adds a layer of indirection? I think the only indirection you're encountering is caused by 'fn. I just tried it out in Wart to make sure:

  ((fn () ((fn () (caller_scope = caller_scope)))))
  => ''1
  ((fn () ((fn () (caller_scope = (eval 'caller_scope caller_scope))))))
  => nil
  ((fn () ((fn () (caller_scope = ((fn () (eval 'caller_scope caller_scope))))))))
  => ''1

Within this second layer, we get a true value for (caller_scope = caller_scope). This is just a sanity check.

We false value for (caller_scope = (eval 'caller_scope caller_scope)), presumably because we're comparing the two different environments we've established.

Finally, it seems we can successfully use ((fn () (eval '<foo> caller_scope)))) in place of <foo>, because although it introduces a new environment, it evals in the old one.

Here's another way we can verify that eval doesn't introduce its own binding for caller_scope:

  ((fn () (eval 'caller_scope caller_scope)))
  020lookup.cc:28 no binding for caller_scope
  => nil
  ((fn () (eval '2 caller_scope)))
  => 2

---

"But I needed it to expand to this:"

  let $old push
    mac (push x s)
      `(if (isa stack ,s)
         (push ,x (rep ,s))
         (,$old ,x ,s))

  mac (push x s) :case `(isa stack ,s)
    `(push ,x (rep ,s))
  
  -->
  
  let $old push
    mac (push x s)
      `(if ,`(isa stack ,s)
         ,`(push ,x (rep ,s))
         (,$old ,x ,s))

In between the , and the ` you have "room to operate on the names before starting on the backquote":

  mac (foo n) :case (let a .. `(isa foo-type ,a))
    with (a ..
          b ..)
      `(bar ,a ,b)
  
  -->
  
  let $old foo
    mac (foo n)
      `(if ,(let a .. `(isa foo-type ,n))
         ,(with (a ..
                 b ..)
            `(bar ,a ,b))
         (,$old ,n))

---

"But even if there isn't, I'm starting to question if lisp is indeed the hundred-year programming model. I'm starting to feel at the end of my tether, at the limits of what macroexpansion can cleanly support."