I noticed that there seems to be some confusion and resistance about my idea. The resistance is probably due to confusion, so I'm going to attempt to conclusively prove that message passing is significantly superior to the current way of creating new data types. A bold claim, can I back it up? Read on.
I will write this post as a series of examples. These examples will show things that should be trivial, but in fact end up being far more complicated in wart/Anarki/pgArc.
For the rest of this post, when I refer to "wart" I am referring to wart/Anarki, because Anarki does things in basically the same way. For every example, pgArc will be at least as verbose as wart, and in some cases it is impossible to make the example work in pgArc.
This is not a post designed to say, "wart sucks", instead it's a post designed to say, "look! message passing is awesome!"
First, let us define a new table type, which we shall call a "clone". A clone behaves completely identically to a table in every way, with only one exception: if it can't find a key, it will check it's parent. On the left side, I will show the way to accomplish this in wart. On the right side, I will show the same way using message passing.
(def clone(x) ; (def clone(p)
(annotate 'heir (obj val (table) parent x))) ; (let t (table)
; (annotate 'table (fn (m k v)
(defcall heir h ; (case m
[or rep.h!val._ ; 'keys (join (keys p) (keys t))
rep.h!parent._]) ; 'get (or (t k v) (p k v))
; 'set (sref t k v))))))
(defset heir(h key value) ;
(= rep.h!val.key value)) ;
;
(def iso(x y) :case (isa x 'heir) ;
...) ;
;
(def fill-table(tb data) :case (isa x 'heir) ;
...) ;
;
(def keys(x) :case (isa x 'heir) ;
(join (keys rep.x!val) ;
(keys rep.x!parent))) ;
;
(def vals(x) :case (isa x 'heir) ;
(join (vals rep.x!val) ;
(vals rep.x!parent))) ;
I gave up on iso and fill-table because they would have ended up being much too verbose and complicated. I expect if I had tried to support them, wart would have become twice as verbose as it already is. All of that just to make a measely table type? Wart doesn't even work according to the spec... because it needs to create a new 'heir type. Why is this a problem? Hold that thought, let me show you another example.
In this case, we're going to use AVL trees (http://arclanguage.org/item?id=14181) to represent a table. This is nice because AVL trees have good worse-case performance:
I'm not going to actually create avl-tables, so you'll have to imagine the implementation. Once again, in wart, we ended up needing to create a new type 'avl-table, and then re-extend all the table built-ins, and call defcall and defset on the new type... Not only is this significantly more verbose, but it's also inefficient and inextensible.
To answer why it's inextensible, I will ask a question: what does it mean when something has a type of 'table? You cannot say, "it's implemented as a hash" because Arc specifically leaves the implementation undefined. A table in Arc could be a hash table, or an alist, or AVL trees, or anything, really. So what does "table" mean? It is an interface. When Arc code sees a variable my-table that has a type of 'table, it expects all the following to work on it:
(my-table 'foo) -> get the key 'foo
(= mytable.'foo 5) -> set the key 'foo
(keys my-table) -> get the keys
(vals my-table) -> get the values
The above is the interface for tables. If your data type can do all of the above, then you can legitimately call it a table. If it can't, then it's not a table. In other words, types represent an interface. They specify behavior. When something is of type 'table, you expect it to behave like a table. You don't care whether the table is created with AVL trees or conses, you only care that it supports the table interface.
So the problem with wart (and Anarki, and pgArc) is that it requires you to create a new type every time you want to create data. But we don't care about the implementation, we only care about the interface. Consider the function `each`. It has different behavior depending on whether the variable is a cons or a string or a table. Let's represent that as follows:
The thing is... in wart, 'heir and 'avl-table support the table interface. But they don't have a type of 'table, so now `each` needs to be re-written like so:
(def each (x)
(let t (type x)
(if (in t 'table 'avl-tree 'heir) ...
(case t
'cons ...
'string ...)))
This is ridiculous! `each` doesn't care about how the tables are implemented, it only cares that they behave like tables. Can it be made better? Yes, by creating a global function `table?`:
But that doesn't solve the problem of needing to extend all the relevant built-in types... it applies a bandaid to the problem, but doesn't solve it. Meanwhile, message passing doesn't even have that problem, since your table variants can have a type of 'table. No need for a `table?` function.
Let's look at one final example. Let's create an alist type... specifically, something that combines the behavior of 'cons and 'table. Here's what I mean by that:
How would you go about doing this in wart...? Impossible or at the very least super incredibly difficult. How would you do this with message passing?
(def alist args
(let l (pair args)
(annotate 'cons (fn (m k v)
(case m
'get (assoc l k)
'set ...
(attr l k v))))))
Perhaps you are scared because I used OOP words like "methods" and "prototypes". For that, I apologize. Instead, I will focus on a simpler, yet more powerful idea: messages. Alan Kay (the creator of Smalltalk) said that objects aren't important: messages are [1]. I think I understand now what he meant by that.
Hopefully we all here realize that modern OOP is a mistake. They claim that it leads to more extensible code, yet I have not seen evidence for that. What I have seen is that it tends to result in a lot of unnecessary verbosity and fiddling with type hierarchies. There is a simple, fundamental, core reason for this: they try to use objects for everything.
Functions are like verbs. Most programs do something, so writing in a functional style is very good for writing most programs. In the OOP world, however, they try to write everything using nouns... which of course is ridiculous.
I claim, that functional programming (though it has great merits), is not a silver bullet that is perfect in every situation. When representing action, functional style is great. But when representing data, it falls apart. I claim that this results in the same verbosity and rigidness that OOP creates.
Don't believe me? Look at the examples above. Now look again. Now say with a straight face, "functional style can create data types just as easily as message passing"
Message passing is a natural, easy, short, and extensible way to represent data: things like conses, tables, input streams. Functional style is a natural, easy, short, and modular way to represent programs: doing something to data. By combining these two styles, Arc can become exceedingly short, and hackable.
"OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP. There are possibly other systems in which this is possible, but I'm not aware of them."
Note: I've probably written less than 10 lines of Smalltalk, but that doesn't change the fact that I can clearly see how message passing reduces verbosity and increases extensibility.
Nobody here is scared of anything, least of all 'OOP words'. You may find it more helpful to phrase us as 'unconvinced that methods are needed in a lisp'. A previous iteration of def :case was called defgeneric, named after the mechanism in CLOS, so there's plenty of history of people trying out OOP ideas in lisp, or in this forum.
Yes, except rocketnia doesn't like coerce either. I do like coerce, and I think it's a good idea for `each` to coerce to 'cons (or a similar iterable thing). I think coerce + extend + message passing would make for fantastically short and extensible code.
---
"Nobody here is scared of anything, least of all 'OOP words'. You may find it more helpful to phrase us as 'unconvinced that methods are needed in a lisp'."
I have already produced plenty of evidence that extend is over twice as verbose as message passing, more difficult to use, and doesn't play well with modules. When it comes to describing data types, I have not seen a single advantage that extend has over message passing. What more do you want? Should message passing do tap dancing too?
(def tap-dancer ()
(annotate 'dancer
(fn (m)
(case m
'tap-dance (prn "tap dancing now")
(err:string "did not understand the message " m)))))
(= dancer (tap-dancer))
(dancer 'tap-dance) -> "tap dancing now"
(dancer 'break-dance) -> error: did not understand the message break-dance
"Yes, except rocketnia doesn't like coerce either."
For the record, that's true, but I do follow a coercion pattern sometimes. See the end of http://arclanguage.org/item?id=14220: "To digress a bit, coercion is a positively useful design pattern when...."
As it happens, 'each is one of those times I'd use a coercion pattern. I'd probably define these utilities, with these dependencies:
each --> some --> seqify --> entries --> keys --> unwrap-my-table
To introduce the type 'my-table and give it support for 'each, I only need to extend 'keys so that it tries 'unwrap-my-table.[1] If I do that, 'seqify will already be able to coerce my type to a lazy list.
[1] If the system doesn't support failcall, I may also have to extend 'supports-keys so that it tries 'isa-my-table. The diagram would be about twice as big if I included these predicates.
I am not convinced your tiny example proves anything. We aren't playing code golf here to make things shorter by any means. Expressive constructs make for shorter programs, and I still believe methods are no more expressive than anything else. I think you're moving parens around.
One way to add evidence is to actually have an implementation of an arc-like language with methods (like I have done with wart). And I don't mean half-baked pseudocode that I have to spend time massaging. (http://arclanguage.org/item?id=14288) Who knows if I would massage it like you would have? You're leaving too much unspecified to persuade me.
"What more do you want? Should message passing do tap dancing too?"
I'm not going to respond if you talk like that.
Keep running with your ideas instead of waiting for us to catch up. If they're good you'll eventually bring us around. Expecting validation from others before moving onward is a recipe for misery. So we're fools. Ignore us.
Look at the first example there. That is a perfectly reasonable and trivial thing to do (creating a table that can inherit from another table). Message passing is less than half the size of wart... And that was without implementing iso and fill-table, which would make wart even more verbose...
That is not "moving parens around". This is a clear example where message passing is less than half as verbose as extend. Message passing should always be at least as good as extend, and in some situations will be significantly better. If you would rather use extend and write programs that are over twice as long, then go ahead... but I thought one of the main points of Arc was brevity.
---
"I'm not going to respond if you talk like that."
It was a joke.
---
"One way to add evidence is to actually have an implementation of an arc-like language with methods (like I have done with wart)."
As said, I plan to implement this in py-arc.
---
"Keep running with your ideas instead of waiting for us to catch up. If they're good you'll eventually bring us around. Expecting validation from others before moving onward is a recipe for misery. So we're fools. Ignore us."
Yes, you're right. I simply don't understand why, when I present clear evidence that extend is either equal to, or worse than message passing in every way pertaining to data types... it isn't accepted. What is it that is preventing acception? Is the evidence not clear? Are my words confusing?
If you would rather use extend and write programs that are over twice as long, then go ahead
How would your approach clone from strings rather than from tables? Those are the sorts of questions I am talking about. Are methods useful if they can only clone tables?
The main point of arc is brevity across a range of programs.
"As said, I plan to implement this in py-arc."
I enjoy our discussions, but I don't understand how I am expected to be persuaded in the absence of running code. I'm happy to agree to disagree until you have something I can take the trouble to install, run, and make your complaints in http://arclanguage.org/item?id=14313 about :)
Strings are annotated conses, so you can use message passing for them too. Message passing works for all data types, even int and num. What does "cloning a string" even mean, by the way?
Let me look at an example that is a bit more string-oriented: we want to make a string that can do substring searches with function notation. In other words, we want this to work:
(def pos-string (x)
(annotate 'string
(fn (m k . rest)
(case m
'get (case (type 'k)
'string (posmatch k x)
(apply x m k rest))
(apply attr x m k rest)))))
How would you do this with extend? Can it be done without extending every single built-in that uses strings?
Here's a hopefully clearer and less buggy version of your example:
(def pos-string (orig-string)
(annotate 'string
(fn (message key . rest)
(if (and (is message 'get) (isa key 'string))
; This message represents a function call with a string as the
; first argument. Let's use 'posmatch for that.
(posmatch key orig-string)
; We don't handle this message. Let's use 'attr to propagate
; raw message to the original.
(apply attr orig-string message key rest)))))
---
"Can it be done without extending every single built-in that uses strings?"
There aren't that many axiomatic things strings do in Arc. In my mind, a custom string type would only have to support two operations: being called as a function; and being coerced to a lazy list, sent through a function, and coerced back. If a string-like thing supports those two behaviors, the core utilities should be able to handle it.
Arc's strings also support 'sref, but I pretend they don't. :-p
The "being coerced to a lazy list, sent through a function, and coerced back" utility may seem overcomplicated. It's the solution I prefer to enable this behavior:
arc> (map inc "abcde")
"bcdef"
This is implemented in pg-Arc using a 'coerce back to the input type, but if a pos-string has a type of 'string, then coercing the output to its original type will probably just make a core string, rather than a pos-string.
"less buggy"? You're using k when it should be key. :P It is shorter, though, so thanks for that.
---
"There aren't that many axiomatic things strings do in Arc"
Yeah, I know. Tables seem to be the big pain point. Possibly conses too, if somebody wanted to extend those... tables seem like they'd be the most common thing to extend, though, and also the trickiest.
---
"[...] being coerced to a lazy list, sent through a function, and coerced back [...]"
Oh! Oh! I know! Let's use message passing! :P We can give strings an 'iter message! Just like Python! Whooo! I'm reminded of the Jamie Zawinski quote, "Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems."
Actually, I think it'd just be easier to define a `coerce` rule for 'iter, no need for message passing in that situation. Now, as for the actual creation of iterators, that could be done with message passing. :P
Then again... you're right that if pos-string uses the same type 'string as built-in strings, then `coerce` won't be able to know that it should use pos-string... unless you added in a message to the iterator, of course. But then that won't play as well with coerce...
I guess that is one example where the current way of coerce + extend + new-type-every-time is actually better.
"Tables seem to be the big pain point. Possibly conses too[...]"
I could be wrong, but I think it's enough for a table to extend 'keys, function call behavior, and 'sref. <Insert usual disclaimer that there'd need to be predicates like 'supports-keys in the absence of failcall.>
For Penknife's purposes, the only reason I care about cons cells is that they're eager implementations of nonempty lazy lists. They should support function call behavior, 'sref, and the one and only lazy list operation:
(iffirst first rest seq
<...thing to do with first and rest if the seq isn't empty...>
<...thing to do if the seq is empty...>)
If cons cells are mutable and allowed to be improper, then they need 'scdr too, and it might be more elegant to add 'scar, 'car, and 'cdr.
"The main point of arc is brevity across a range of programs."
Message passing is general... it can apply to all the built-in types, and can also apply to all user-created types. It does increase brevity across a range of programs. You could even write all Arc code in that style, but I don't recommend that. I'm restricting my discussion of message passing to data types because message passing is clearly superior with regard to data types. I make no claims about the superiority of message passing with regard to things other than data types.
I've been using specific examples because as aw mentioned (http://arclanguage.org/item?id=14264) it's important to explain the problem that it solves, and also demonstrate how it solves the problem... but the concept is general.
In any case, since you're demanding runnable code, it's up to me to put my money where my mouth is.
I will also note that all it would take to convince me that I'm wrong is to demonstrate an alternate system that is just as short and extensible as message passing. I have not seen such a system.
rocketnia seems to dislike types, and I agree: a proliferation of types is troublesome. But we only have two routes: we can accept that types exist in Arc, or we can make Arc typeless, like GEDANKEN [1].
Simply saying "don't use isa" doesn't solve the problem... because you still need to extend everything. I have clearly demonstrated that with the current type system, creating new types, or variants of existing types, is far too difficult. But with message passing, it becomes trivial. Do you have a better suggestion?
A few random observations, in case they are helpful to you:
- I often don't get anyone agreeing with me in the forums (http://arclanguage.org/item?id=14263 notwithstanding :)... I notice that people often will spend their time instead to share their own ideas, concerns, and criticisms... or say nothing if they don't have anything additional to contribute. (Some of my best hacks have gotten little response in the forum when I first post about them). Thus I suspect you shouldn't necessarily take lack of positive agreement as evidence of disagreement.
- One pattern I've noticed on the forum is that someone will drop by and say "Arc would be a better solution for my problem if it had (immutable lists | software transactional memory | message passing | documentation | unit tests | libraries | ...)". It's more rare for this to be followed up by an actual runnable implementation of the feature.
- When someone does provide an implementation, they sometimes appear to be disappointed when their solution isn't adopted by other people. I suspect maybe there may be some confusion about the nature of problems and solutions: even if say X is an excellent and succinct solution to problem A, someone like myself for example may not take the trouble of adapting it if I happen not to have problem A. And... even if I do have problem A, X may or may not be a useful contribution to a solution for my problems A+B. When I'm trying to solve A+B, I may remember X and then try it out and see if it is useful for A+B, but it can be hard to know in advance whether I'll personally find X useful until I'm seriously looking at the details of A+B.
- I enjoy sharing solutions I've implemented to problems I've had, and I'm pleased when I learn that someone else has found them useful as well. But it can take a year or more after I first publish a hack for me to learn that someone is using it... which is fine for me since I'm aware that there can be a long feedback cycle, but I notice other people become discouraged when nothing seems to be happening in a shorter time frame.
To maximize the likelihood that other people will be able to make use of your solution, you may find this template helpful:
- explain the problem the solution solves. (People often omit this step: they immediately start off with describing the solution. But while it may be easy to see that a solution is elegant, it can be hard to see what it's a solution for).
- describe the solution. (Most people do this part).
- demonstrate how the implementation solves the problem. (This part is also often omitted, but a few concrete examples can do wonders to help people make the connection between the solution and a problem they have).
The forum itself tends to be a poor place to do this because of its modest formatting capabilities and the inability to edit posts later means that descriptions become disjointed, spread over multiple comments and posts.
I agree with most of what you're saying, but I will add a few things:
There are, generally speaking, two types of people who visit this forum. The first, type-A let's call them, are hardcore hackers, like yourself(ves), who are interested in the language design and the underbelly mechanics of language assembly. Then there are people, like myself, whom are only interested in using the language to build applications.
When I read: > "Arc would be a better solution for my problem if it had...", "It's more rare for this to be followed up by an actual runnable implementation of the feature." I will suggest the two types of people represent the divide within the pattern you're recognizing and I will going to even further to suggest this is not a bad thing. I would like to encourage more type-B user engagement - even, if only, so that I am not all alone :)
While I don't have the time or inclination to understand the full content/depth within these type-A posts, I do try to learn what I can even if only at a surface level. I really do enjoy reading these posts and do appreciate them so I hope they don't go to a back channel as akkartik has mentioned a possibility for.
Furthermore, I would like to highlight some of the points aw's message that I feel are really important (and with my interpretation):
* Inline comments are not the best way to present your ideas. I've struggled assembling all the bits and pieces and knowledge requirements from prior posts/threads. The order alone messes everything up and I simply do not have the time to do the paper chase.
* Writing an external blog post will likely force you to present a fuller picture, with its surrounding context, to an audience that can't always be on top of things.
All that said, please don't give up on the ideas, we need more of them - not less.
Oh, and by the way, there's an infinite amount of room available in the wiki for other things such as proposals, ideas, arguments, implementation approaches, and so on. So if you're looking for a place where you can write things out and edit them later, the wiki is great place to do that. (You can of course also write your own blog post if you want, and we can link to it from the wiki as well).
I took a break to write a blogging platform from scratch that includes my own custom markdown.
I even posted my first arc topic here: http://blackstag.com/blog.posting?id=2. I then quickly discovered the auto spam bots are moving from the arc forum to my blog site every night. So I took another break, lol, just to put some spam management code in. I'm glad had read enough of pg's code to implement some good techniques similar to news.arc.
Now I'm ready to do more blogging - I hope. I've accumulated a list of 9 or 10 posts and I plan to get some arc stuff in there as well.
I notice other people become discouraged when nothing seems to be happening in a shorter time frame.
I went through the five stages of grief after submitting http://arclanguage.org/item?id=12657. It took me literally weeks to get right, and it sat at I think 1 point for nearly a week (with just evanrmurphy's positive comment[1]).
But eventually I realized much of what you said :) I have more realistic expectations now. Points are nothing, and responses come in time.
[1] While evan was active you could tell when he'd seen something because it would go to 2 points :) (without suggesting that he upvoted everything)
Evan would often articulate agreement or admiration even if he had nothing to add, and it was useful. But I'm not sure we should all do it.
Perhaps all this is a sign that we need a back channel.
Yeah it seems people here don't express agreement unless they can add something useful.
Some of my best hacks have gotten little response in the forum when I first post about them
Because some of your best hacks left me speechless :)
Things I see here often take a while to digest, days or weeks. It's taken me over a year to fully appreciate extend, a road that took me through various dead-ends.
It was just when I got up this morning (thinking about http://arclanguage.org/item?id=14253) that I realized what you were suggesting wasn't just a syntactic thing of putting objects before methods, but doing away with types altogether. You're saying the language shouldn't allow us to say isa x 'table, only responds_to x tableness (http://arclanguage.org/item?id=14258).
So yeah sometimes it takes people a while to digest the implications of a solution (http://arclanguage.org/item?id=14264), but sometimes they just haven't understood what you're saying :)
What I'm suggesting is to get rid of the current view of types. The `type` and `annotate` and `isa` functions would still be used, just for a different purpose: defining interfaces.
So in that sense, yes, you're right. But it's not to the extreme of rocketnia; I'm merely suggesting to change how they're used.
Hmm, I wonder if you're getting Pauan and me mixed up. >.> I'm against 'isa and 'type almost altogether, but I think Pauan considers 'type to be part of the high-level interface of a value:
I consider it to be a part of an interface. If we're using duck typing (which I recommend), what happens when two functions define different behavior, but with the same names? All sorts of subtle bugs ensue. But by leveraging the type mechanism, we can group these differing behaviors into interfaces, avoiding the problem completely.
"For the rest of this post, when I refer to "wart" I am referring to wart/Anarki, because Anarki does things in basically the same way."
I'm flattered :) but I think people are about as reluctant to try out my wart as they are to try out your methods.
I was tempted to more aggressively push wart in this forum, but I'm glad I was ruled by my doubts, because wart turns out to have more warts than arc :) Any new idea takes a while to be digested, to try out in various situations.
These are great, valid reasons. I wasn't complaining, was I? How do they not apply to your suggestions? I'm still not sure what you would have us do. Say, "yes you're right"? Go off and rebuild arc like you suggest? Why would that then not suffer the problem of 'learning how it differs from old arc'?
Think of a situation where Arc currently throws an error, like when you try to use `apply` on a macro. Current code cannot use apply on a macro, because of the error.
Now let's suppose we fix that, so apply works on macros... nobody needs to "learn" anything, other than "hey, apply works on macros now!" because the changes are completely backwards compatible.
Arc doesn't expose the internal details of it's data types, what I'm proposing is a way to expose them. Existing Arc programs can't be relying on the internal details because Arc programs can't access the internal details.
Thus, message passing is backwards compatible. The only time people would need to learn message passing is if they actually plan on creating a new data type.
So how does implementing message passing differ from wart? It doesn't change the way existing Arc programs are run. Can I take any Arc program, plop it in, and have it work the same in wart? Message passing is backwards compatible with existing Arc programs.
In other words... you could take an existing Arc program, plop it into the new version of Arc (that has message passing), and it should work just fine. Only the low-level details have changed, but the high-level ones remain the same.
Yeah, I know, but I was referring more to, "what do I need to learn to get stuff to work in wart". I admit that my terminology was less precise than it probably should have been.
