"All of this suggests to me that all talk of lisp 'permitting macros'
because it is 'homoiconic' is a canard. You can build parse trees and
macros for any syntax. It's just that the people who want such syntax
haven't cottoned on to the power of macros. Or they haven't realized
the value of optional syntax, or of redundant syntax where multiple
tokens mean exactly the same thing."
I don't think the problem is that macros are impossible in a syntax-rich language... the problem is that as the syntax moves further and further away from the AST, it becomes harder to understand macros, because you have to do all these mental syntax twists in your head. And so the harder it is to understand/read/write macros, the less likely you are to do so, and the more likely you are to conclude that macros are an unnecessary waste of time.
If the above is true, then any attempt to make a Lisp-like language with macros and lots of syntax is doomed to failure, not because it's impossible, but because the more syntax you add, the less useful macros become: normal people don't understand why macros are so great because they're used to syntax, and the Lisp crowd doesn't like syntax because it makes reasoning about macros harder.
This also suggests that if you want a language with more normalish syntax but with decent support for vaus/macros, then it should have a very simple and consistent syntax to minimize the mental overhead. Magpie seems like it fits the bill, essentially having a simpler and more consistent Pythonish/Rubyish syntax.
---
Nulan is based on the premise that syntax should be designed not to make Lisp more similar to normal languages, but instead to aid the programmer. So although Nulan does have syntax, I've tried to give it very clear 1-to-1 translations to S-expressions. I believe this makes vaus/macros a bit harder to reason about (due to the syntax) but in turn you gain some brevity and readability. My hope is that this will be an overall net gain.
---
Interestingly enough, Ruby has first-class environments and three kinds of `eval` that support strings and blocks as the first argument... and with string interpolation, you can write code that looks a lot like a macro...
Essentially, Ruby has all the things it needs for first-class vau, and it's actually idiomatic to use `eval` in certain situations... for instance, here's some Ruby code I wrote a while back:
def self.make_toggle_setter *args
args.each do |arg|
class_eval "def #{arg} x
toggle_#{arg} if x != is_#{arg}?
end"
end
end
Looks a lot like a vau/macro, doesn't it? Except it uses string interpolation rather than building up a list. So Ruby has pretty much all the benefits of (unhygienic) macros despite not having a homoiconic syntax.
This suggests that all you really need for vau is some way to build up data (string, list, AST, whatever) which is then evaluated in a context. And as long as the data looks similar enough to the language itself, it'll be okay. Which is why interpolated strings work out well in Ruby's case: it looks like you're writing Ruby code.
The question then isn't whether the language is homoiconic or not... the question is whether it's easy to write data that looks like the language. In Ruby, you use a string to treat the language as data. Magpie uses quotations. Lisp uses lists.
"The point then isn't whether the language is homoiconic or not... the question is whether it's easy to write data that looks like the language. In Ruby, you use a string to treat the language as data. In Magpie it's quotations. In Lisp it's lists."
By the way, in Lisp, a macro receives a list and returns a list. As shown above, Ruby can get the same (overall) effect using runtime eval + string interpolation.
But there is something that you can do with lists that is much harder with strings/ASTs, and that's to inspect or manipulate them. I've written macros in Arc that will essentially code-walk their body and do different things depending on what they find. This is fairly trivial with lists, because Lisp usually has plenty of functions to inspect/manipulate them.
The same is not true of blocks/strings in Ruby. In Ruby, you can inspect the arity of a block, and you can evaluate the block in a context, but that's about all you can do. Because of the power of first-class lambdas, you can still do a lot of nifty things with Ruby's eval, but it's not quite the same as manipulating lists in Lisp. And you can use regexps to inspect/manipulate strings, but that's obviously a big pain.
So, for simple cases, Ruby's interpolated strings work out just as well as unhygienic macros in Arc, but if you want to do anything more complicated that actually involves inspecting/manipulating the string/block, it'll be a lot harder if not impossible (excluding writing your own compiler/interpreter, of course :P)
---
By the way... because of their runtime nature, it's also impossible (or at least ridiculously difficult) to write a "deep" code-walker in a language that uses vau. I see two kinds of code-walkers: shallow and deep.
An example of a "shallow" code walker would be the `loop` macro in Common Lisp: it walks the macro's body, but it doesn't need to macro-expand anything, it just uses the immediate forms. Vaus can do those kinds of code-walks easily.
A "deep" code-walker would be something that needs to macro-expand the body and then walk the macro-expanded form. An example would be Arc's `setforms` (which is used by `=`, `zap`, etc). Vaus can't do that.
This suggests that for code-walkers, macros are actually more powerful than vau. But "deep" code-walkers are usually not created even in Lisps that have macros, so I think it's fine for vaus to give them up.
"..for simple cases, Ruby's interpolated strings work out just as well as unhygienic macros in Arc, but if you want to do anything more complicated that actually involves inspecting/manipulating the string/block, it'll be a lot harder if not impossible."
Yeah. Another way of saying that is that packing the parsing of the AST into a macro-like feature is less powerful than exposing it to the rest of the language. This is why I don't consider string-interpolation-and-eval to be macro support.
Here are a few languages that have tried to build lisp macro support (none successfully, IMO):
Lispers will hold these up as examples that you need s-expressions to do macros 'right'. But I'm starting to question this. Is smalltalk 'homoiconic'? I think you could argue yes. And factor certainly supports a very powerful notion of quotations.
"This is why I don't consider string-interpolation-and-eval to be macro support."
Well it's not a macro anyways because `eval` runs at runtime. But it is roughly analogous to vaus in Kernel/Nulan, which in turn are a superset of (runtime) macros. Except that, in Ruby, it's a very hacky vau which is limited in power. :P
---
"Lispers will hold these up as examples that you need s-expressions to do macros 'right'."
I think it's just as I said. The more syntax you add (and especially the more complex the syntax is), the harder it is to deal with macros. That's all. S-expressions happen to be the simplest, most consistent, and readable syntax we've found, which makes it ideal for macros.
But Nulan is an example where it is possible to add syntax to a Lisp, just so long as the syntax translates easily into S-expressions underneath. Arc is another milder example of the same idea.
---
"My final evidence that the word 'homoiconic' is so fuzzy as to be useless"
I don't see how it's useless... as those pages suggest, "homoiconity" describes not a binary 1/0 but a continuum of languages. That does make it a fuzzy analog word, but a lot of words we use are like that: we humans think in fuzzy analog ways. That doesn't make it useless.
---
"But I'm starting to question this. Is smalltalk 'homoiconic'? I think you could argue yes. And factor certainly supports a very powerful notion of quotations."
I think you're worrying too much about "macros". What I care about isn't macros, but the ability to create new constructs that look and behave just like existing constructs.
If the language doesn't have any special forms (everything is a function), then the language only needs functions and not macros/vaus.
But Lisps usually do have special forms (lambda, if, etc.) and the primary way to define new special forms is with macros. Kernel/Nulan/wart achieve the same thing with vau.
Ruby's eval makes it possible to define certain constructs which would otherwise be impossible without eval. But you still can't define new things that look and act like "if", "def", etc. because the only way you can turn off evaluation in Ruby is with a block.
---
So, Ruby's string interpolation + eval is more powerful than languages like Python which don't have it, but it's still less powerful than macros/vaus in Lisp-like languages. And in languages with laziness like Haskell, you might not even need macros/vaus at all!
If Ruby had some way to plug into the parser, then it could add new syntax. But Ruby's syntax is quite complicated, so dealing with things at the parser level would probably be very hard. In any case, it's moot because Ruby doesn't let you do that, as far as I know.
Magpie does have the ability to plug new things into the parser, therefore Magpie is just as powerful as macros in Lisp. And Magpie's syntax seems simple enough to me that I think it can actually work out okay. Whether it's as easy as S-expressions or not is another story... I think the primary benefit of S-expressions is that they make macros easy, not that they make it possible.
---
I would say that Magpie is homoiconic, due to it representing its source code as a user-manipulable object, and it also has quotations which make it easy to create said objects.
That's why the word "homoiconity" is a fuzzy continuum: different languages have different abilities, and they achieve their abilities in different ways. And even two languages with very different abilities or ways of achieving those abilities can still be equivalent in power.
What I care about is the power to do things, so I'm trying not to get too hung up on one particular technique like macros. As long as it works well enough, it's fine by me, whether it's strings + eval, Magpie's quotations, Haskell's laziness, macros, vau, etc.
---
By the way, I'm not trying to argue that Ruby is homoiconic: I don't think it is. I was merely pointing out that even a non-homoiconic language with lots of complex syntax like Ruby can still get a lot of the same power that (unhygienic Arc-like) macros have. Just not all of it.
The same has been said of macros: they give you a lot of the same power that vaus have, just not all of it. Then again, there are some things that macros can do that vaus can't... so it seems that they are more like an evolutionary split in the tree: vaus don't replace macros, instead they grow alongside them in a different branch.
"Here are a few languages that have tried to build lisp macro support (none successfully, IMO):"
I dunno, Nemerle and Perl seem okay to me. Can't really speak about Dylan or Template Haskell, except to say that it seems to me the only reason Haskell needs macros is due to its restrictive type system.
