I will write this post as a series of examples. These examples will show things that should be trivial, but in fact end up being far more complicated in wart/Anarki/pgArc.

For the rest of this post, when I refer to "wart" I am referring to wart/Anarki, because Anarki does things in basically the same way. For every example, pgArc will be at least as verbose as wart, and in some cases it is impossible to make the example work in pgArc.

This is not a post designed to say, "wart sucks", instead it's a post designed to say, "look! message passing is awesome!"

First, let us define a new table type, which we shall call a "clone". A clone behaves completely identically to a table in every way, with only one exception: if it can't find a key, it will check it's parent. On the left side, I will show the way to accomplish this in wart. On the right side, I will show the same way using message passing.

  (def clone(x)                                   ;  (def clone(p)
    (annotate 'heir (obj val (table) parent x)))  ;    (let t (table)
                                                  ;      (annotate 'table (fn (m k v) 
  (defcall heir h                                 ;                         (case m
    [or rep.h!val._                               ;                           'keys (join (keys p) (keys t))
        rep.h!parent._])                          ;                           'get (or (t k v) (p k v))
                                                  ;                           'set (sref t k v))))))
  (defset heir(h key value)                       ;
    (= rep.h!val.key value))                      ;
                                                  ;
  (def iso(x y) :case (isa x 'heir)               ;
    ...)                                          ;
                                                  ; 
  (def fill-table(tb data) :case (isa x 'heir)    ;
    ...)                                          ;
                                                  ;
  (def keys(x) :case (isa x 'heir)                ;
    (join (keys rep.x!val)                        ;
          (keys rep.x!parent)))                   ;
                                                  ; 
  (def vals(x) :case (isa x 'heir)                ;
    (join (vals rep.x!val)                        ;
          (vals rep.x!parent)))                   ;
          

In this case, we're going to use AVL trees (http://arclanguage.org/item?id=14181) to represent a table. This is nice because AVL trees have good worse-case performance:

  (def avl-table(x)                                  ;  (def avl-table()
    (annotate 'avl-table ...))                       ;    (annotate 'table (fn (m k v)
                                                     ;                       (case m
  (defcall avl-table h                               ;                         'keys ...
    ...)                                             ;                         'get ...
                                                     ;                         'set ...))))
  (defset avl-table(h key value)                     ;
    ...)                                             ;
                                                     ;  
  (def iso(x y) :case (isa x 'avl-table)             ;                           
    ...)                                             ;
                                                     ; 
  (def fill-table(tb data) :case (isa x 'avl-table)  ;
    ...)                                             ;
                                                     ;
  (def keys(x) :case (isa x 'avl-table)              ;
    ...)                                             ;
                                                     ; 
  (def vals(x) :case (isa x 'avl-table)              ;
    ...)                                             ;
    

To answer why it's inextensible, I will ask a question: what does it mean when something has a type of 'table? You cannot say, "it's implemented as a hash" because Arc specifically leaves the implementation undefined. A table in Arc could be a hash table, or an alist, or AVL trees, or anything, really. So what does "table" mean? It is an interface. When Arc code sees a variable my-table that has a type of 'table, it expects all the following to work on it:

  (my-table 'foo)    -> get the key 'foo
  (= mytable.'foo 5) -> set the key 'foo
  (keys my-table)    -> get the keys
  (vals my-table)    -> get the values
  

So the problem with wart (and Anarki, and pgArc) is that it requires you to create a new type every time you want to create data. But we don't care about the implementation, we only care about the interface. Consider the function `each`. It has different behavior depending on whether the variable is a cons or a string or a table. Let's represent that as follows:

  (def each (x)
    (case (type x)
      'cons ...
      'table ...
      'string ...))
      

  (def each (x)
    (let t (type x)
      (if (in t 'table 'avl-tree 'heir) ...
          (case t
            'cons ...
            'string ...)))
          

  (def table? (x)
    (in (type x) 'table 'avl-tree 'heir))

  (def each (x)
    (if (table? x) ...
        (case (type x)
          'cons ...
          'string ...)))

Let's look at one final example. Let's create an alist type... specifically, something that combines the behavior of 'cons and 'table. Here's what I mean by that:

  (= my-alist (alist x 1 y 2))
  
  (my-alist 'x)     -> 1
  (= my-alist.'x 5) -> 5
  (car my-alist)    -> (x 5)
  (type my-alist)   -> cons
  

  (def alist args
    (let l (pair args)
      (annotate 'cons (fn (m k v)
                        (case m
                          'get (assoc l k)
                          'set ...
                               (attr l k v))))))

Hopefully we all here realize that modern OOP is a mistake. They claim that it leads to more extensible code, yet I have not seen evidence for that. What I have seen is that it tends to result in a lot of unnecessary verbosity and fiddling with type hierarchies. There is a simple, fundamental, core reason for this: they try to use objects for everything.

Functions are like verbs. Most programs do something, so writing in a functional style is very good for writing most programs. In the OOP world, however, they try to write everything using nouns... which of course is ridiculous.

I claim, that functional programming (though it has great merits), is not a silver bullet that is perfect in every situation. When representing action, functional style is great. But when representing data, it falls apart. I claim that this results in the same verbosity and rigidness that OOP creates.

Don't believe me? Look at the examples above. Now look again. Now say with a straight face, "functional style can create data types just as easily as message passing"

Message passing is a natural, easy, short, and extensible way to represent data: things like conses, tables, input streams. Functional style is a natural, easy, short, and modular way to represent programs: doing something to data. By combining these two styles, Arc can become exceedingly short, and hackable.

* [1]: http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-... http://userpage.fu-berlin.de/~ram/pub/pub_jf47ht81Ht/doc_kay...

"OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP. There are possibly other systems in which this is possible, but I'm not aware of them."

In wart, each doesn't check the type of the sequence.

https://github.com/akkartik/wart/blob/81a6dc27f7ebe58bd89fc0...

https://github.com/akkartik/wart/blob/81a6dc27f7ebe58bd89fc0...

This works because coerce is extensible. All I had to do was define how to coerce tables into lists: https://github.com/akkartik/wart/blob/81a6dc27f7ebe58bd89fc0... and arc's transparent destructuring takes care of the rest.

---

Nobody here is scared of anything, least of all 'OOP words'. You may find it more helpful to phrase us as 'unconvinced that methods are needed in a lisp'. A previous iteration of def :case was called defgeneric, named after the mechanism in CLOS, so there's plenty of history of people trying out OOP ideas in lisp, or in this forum.

---

---

"Nobody here is scared of anything, least of all 'OOP words'. You may find it more helpful to phrase us as 'unconvinced that methods are needed in a lisp'."

I have already produced plenty of evidence that extend is over twice as verbose as message passing, more difficult to use, and doesn't play well with modules. When it comes to describing data types, I have not seen a single advantage that extend has over message passing. What more do you want? Should message passing do tap dancing too?

  (def tap-dancer ()
    (annotate 'dancer
      (fn (m)
        (case m
          'tap-dance (prn "tap dancing now")
                     (err:string "did not understand the message " m)))))

  (= dancer (tap-dancer))
  (dancer 'tap-dance)   -> "tap dancing now"
  (dancer 'break-dance) -> error: did not understand the message break-dance

For the record, that's true, but I do follow a coercion pattern sometimes. See the end of http://arclanguage.org/item?id=14220: "To digress a bit, coercion is a positively useful design pattern when...."

As it happens, 'each is one of those times I'd use a coercion pattern. I'd probably define these utilities, with these dependencies:

  each  -->  some  -->  seqify  -->  entries  -->  keys  -->  unwrap-my-table

One way to add evidence is to actually have an implementation of an arc-like language with methods (like I have done with wart). And I don't mean half-baked pseudocode that I have to spend time massaging. (http://arclanguage.org/item?id=14288) Who knows if I would massage it like you would have? You're leaving too much unspecified to persuade me.

"What more do you want? Should message passing do tap dancing too?"

I'm not going to respond if you talk like that.

http://arclanguage.org/item?id=14261

Look at the first example there. That is a perfectly reasonable and trivial thing to do (creating a table that can inherit from another table). Message passing is less than half the size of wart... And that was without implementing iso and fill-table, which would make wart even more verbose...

That is not "moving parens around". This is a clear example where message passing is less than half as verbose as extend. Message passing should always be at least as good as extend, and in some situations will be significantly better. If you would rather use extend and write programs that are over twice as long, then go ahead... but I thought one of the main points of Arc was brevity.

---

"I'm not going to respond if you talk like that."

It was a joke.

---

"One way to add evidence is to actually have an implementation of an arc-like language with methods (like I have done with wart)."

As said, I plan to implement this in py-arc.

---

"Keep running with your ideas instead of waiting for us to catch up. If they're good you'll eventually bring us around. Expecting validation from others before moving onward is a recipe for misery. So we're fools. Ignore us."

If you would rather use extend and write programs that are over twice as long, then go ahead

How would your approach clone from strings rather than from tables? Those are the sorts of questions I am talking about. Are methods useful if they can only clone tables?

The main point of arc is brevity across a range of programs.

"As said, I plan to implement this in py-arc."

Let me look at an example that is a bit more string-oriented: we want to make a string that can do substring searches with function notation. In other words, we want this to work:

  (= my-string (pos-string "foobar"))
  (my-string "foo") -> 0
  (my-string "bar") -> 3
  (my-string 0)     -> #\f
  (my-string 3)     -> #\b

  (def pos-string (x)
    (annotate 'string
      (fn (m k . rest)
        (case m
          'get (case (type 'k)
                 'string (posmatch k x)
                         (apply x m k rest))
               (apply attr x m k rest)))))

  (def pos-string (orig-string)
    (annotate 'string
      (fn (message key . rest)
        (if (and (is message 'get) (isa key 'string))
          
          ; This message represents a function call with a string as the
          ; first argument. Let's use 'posmatch for that.
          (posmatch key orig-string)
          
          ; We don't handle this message. Let's use 'attr to propagate
          ; raw message to the original.
          (apply attr orig-string message key rest)))))

"Can it be done without extending every single built-in that uses strings?"

There aren't that many axiomatic things strings do in Arc. In my mind, a custom string type would only have to support two operations: being called as a function; and being coerced to a lazy list, sent through a function, and coerced back. If a string-like thing supports those two behaviors, the core utilities should be able to handle it.

Arc's strings also support 'sref, but I pretend they don't. :-p

The "being coerced to a lazy list, sent through a function, and coerced back" utility may seem overcomplicated. It's the solution I prefer to enable this behavior:

  arc> (map inc "abcde")
  "bcdef"

---

"There aren't that many axiomatic things strings do in Arc"

Yeah, I know. Tables seem to be the big pain point. Possibly conses too, if somebody wanted to extend those... tables seem like they'd be the most common thing to extend, though, and also the trickiest.

---

"[...] being coerced to a lazy list, sent through a function, and coerced back [...]"

Oh! Oh! I know! Let's use message passing! :P We can give strings an 'iter message! Just like Python! Whooo! I'm reminded of the Jamie Zawinski quote, "Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems."

Actually, I think it'd just be easier to define a `coerce` rule for 'iter, no need for message passing in that situation. Now, as for the actual creation of iterators, that could be done with message passing. :P

Then again... you're right that if pos-string uses the same type 'string as built-in strings, then `coerce` won't be able to know that it should use pos-string... unless you added in a message to the iterator, of course. But then that won't play as well with coerce...

Fixed! Now it oughta be less buggy. ^^

---

"Tables seem to be the big pain point. Possibly conses too[...]"

I could be wrong, but I think it's enough for a table to extend 'keys, function call behavior, and 'sref. <Insert usual disclaimer that there'd need to be predicates like 'supports-keys in the absence of failcall.>

For Penknife's purposes, the only reason I care about cons cells is that they're eager implementations of nonempty lazy lists. They should support function call behavior, 'sref, and the one and only lazy list operation:

  (iffirst first rest seq
    <...thing to do with first and rest if the seq isn't empty...>
    <...thing to do if the seq is empty...>)

Message passing is general... it can apply to all the built-in types, and can also apply to all user-created types. It does increase brevity across a range of programs. You could even write all Arc code in that style, but I don't recommend that. I'm restricting my discussion of message passing to data types because message passing is clearly superior with regard to data types. I make no claims about the superiority of message passing with regard to things other than data types.

I've been using specific examples because as aw mentioned (http://arclanguage.org/item?id=14264) it's important to explain the problem that it solves, and also demonstrate how it solves the problem... but the concept is general.

rocketnia seems to dislike types, and I agree: a proliferation of types is troublesome. But we only have two routes: we can accept that types exist in Arc, or we can make Arc typeless, like GEDANKEN [1].

Simply saying "don't use isa" doesn't solve the problem... because you still need to extend everything. I have clearly demonstrated that with the current type system, creating new types, or variants of existing types, is far too difficult. But with message passing, it becomes trivial. Do you have a better suggestion?

* [1]: http://portal.acm.org/citation.cfm?id=362364

- I often don't get anyone agreeing with me in the forums (http://arclanguage.org/item?id=14263 notwithstanding :)... I notice that people often will spend their time instead to share their own ideas, concerns, and criticisms... or say nothing if they don't have anything additional to contribute. (Some of my best hacks have gotten little response in the forum when I first post about them). Thus I suspect you shouldn't necessarily take lack of positive agreement as evidence of disagreement.

- One pattern I've noticed on the forum is that someone will drop by and say "Arc would be a better solution for my problem if it had (immutable lists | software transactional memory | message passing | documentation | unit tests | libraries | ...)". It's more rare for this to be followed up by an actual runnable implementation of the feature.

- When someone does provide an implementation, they sometimes appear to be disappointed when their solution isn't adopted by other people. I suspect maybe there may be some confusion about the nature of problems and solutions: even if say X is an excellent and succinct solution to problem A, someone like myself for example may not take the trouble of adapting it if I happen not to have problem A. And... even if I do have problem A, X may or may not be a useful contribution to a solution for my problems A+B. When I'm trying to solve A+B, I may remember X and then try it out and see if it is useful for A+B, but it can be hard to know in advance whether I'll personally find X useful until I'm seriously looking at the details of A+B.

- I enjoy sharing solutions I've implemented to problems I've had, and I'm pleased when I learn that someone else has found them useful as well. But it can take a year or more after I first publish a hack for me to learn that someone is using it... which is fine for me since I'm aware that there can be a long feedback cycle, but I notice other people become discouraged when nothing seems to be happening in a shorter time frame.

To maximize the likelihood that other people will be able to make use of your solution, you may find this template helpful:

- explain the problem the solution solves. (People often omit this step: they immediately start off with describing the solution. But while it may be easy to see that a solution is elegant, it can be hard to see what it's a solution for).

- describe the solution. (Most people do this part).

- demonstrate how the implementation solves the problem. (This part is also often omitted, but a few concrete examples can do wonders to help people make the connection between the solution and a problem they have).

There are, generally speaking, two types of people who visit this forum. The first, type-A let's call them, are hardcore hackers, like yourself(ves), who are interested in the language design and the underbelly mechanics of language assembly. Then there are people, like myself, whom are only interested in using the language to build applications.

When I read: > "Arc would be a better solution for my problem if it had...", "It's more rare for this to be followed up by an actual runnable implementation of the feature." I will suggest the two types of people represent the divide within the pattern you're recognizing and I will going to even further to suggest this is not a bad thing. I would like to encourage more type-B user engagement - even, if only, so that I am not all alone :)

While I don't have the time or inclination to understand the full content/depth within these type-A posts, I do try to learn what I can even if only at a surface level. I really do enjoy reading these posts and do appreciate them so I hope they don't go to a back channel as akkartik has mentioned a possibility for.

Furthermore, I would like to highlight some of the points aw's message that I feel are really important (and with my interpretation):

* Inline comments are not the best way to present your ideas. I've struggled assembling all the bits and pieces and knowledge requirements from prior posts/threads. The order alone messes everything up and I simply do not have the time to do the paper chase.

* Writing an external blog post will likely force you to present a fuller picture, with its surrounding context, to an audience that can't always be on top of things.

I added an "Arc Wish List" page to the wiki: https://sites.google.com/site/arclanguagewiki/wish-list It's currently empty :-) Perhaps some type-B users would like to fill in some things they'd like implemented? ^_^

I took a break to write a blogging platform from scratch that includes my own custom markdown.

I even posted my first arc topic here: http://blackstag.com/blog.posting?id=2. I then quickly discovered the auto spam bots are moving from the arc forum to my blog site every night. So I took another break, lol, just to put some spam management code in. I'm glad had read enough of pg's code to implement some good techniques similar to news.arc.

Now I'm ready to do more blogging - I hope. I've accumulated a list of 9 or 10 posts and I plan to get some arc stuff in there as well.

I went through the five stages of grief after submitting http://arclanguage.org/item?id=12657. It took me literally weeks to get right, and it sat at I think 1 point for nearly a week (with just evanrmurphy's positive comment[1]).

But eventually I realized much of what you said :) I have more realistic expectations now. Points are nothing, and responses come in time.

[1] While evan was active you could tell when he'd seen something because it would go to 2 points :) (without suggesting that he upvoted everything)

Evan would often articulate agreement or admiration even if he had nothing to add, and it was useful. But I'm not sure we should all do it.

Some of my best hacks have gotten little response in the forum when I first post about them

Because some of your best hacks left me speechless :)

What I'm suggesting is to get rid of the current view of types. The `type` and `annotate` and `isa` functions would still be used, just for a different purpose: defining interfaces.

  (type my-alist)   -> cons

I'm flattered :) but I think people are about as reluctant to try out my wart as they are to try out your methods.

I was tempted to more aggressively push wart in this forum, but I'm glad I was ruled by my doubts, because wart turns out to have more warts than arc :) Any new idea takes a while to be digested, to try out in various situations.

1) downloading SBCL and wart, and fiddling with it to get it working

2) learning how wart differs from Arc, especially with regards to core functions

Now let's suppose we fix that, so apply works on macros... nobody needs to "learn" anything, other than "hey, apply works on macros now!" because the changes are completely backwards compatible.

Arc doesn't expose the internal details of it's data types, what I'm proposing is a way to expose them. Existing Arc programs can't be relying on the internal details because Arc programs can't access the internal details.

Thus, message passing is backwards compatible. The only time people would need to learn message passing is if they actually plan on creating a new data type.

So how does implementing message passing differ from wart? It doesn't change the way existing Arc programs are run. Can I take any Arc program, plop it in, and have it work the same in wart? Message passing is backwards compatible with existing Arc programs.

They still need to 'learn how it differs from arc' (your words). Unless you convince PG to include methods :)

"Message passing is backwards compatible with existing Arc programs."

First, what is our goal? I want to allow users to create new data types that behave like existing Arc datatypes... I want them to fit in seamlessly. What am I talking about?

Suppose you wish to create something that behaves almost identically to a table, but does something special when you call = on it:

  (= my-table.'foo 5) ; do something special

---

But first, what is the problem? In order to explain this, I must first explain the concept of private/public data. Public data is exposed everywhere. An example would be a global variable or a global function.

Private data is data that is hidden from the outside world completely. In Arc, we can use closures to create private data:

  (let priv 5
    (def get-priv () (+ 2 priv)))

  priv       -> error
  (get-priv) -> 7

Okay, so what? We use closures all the time, this isn't exactly new... hold that thought for a moment. What is an input port, in Arc? They are obviously implemented in Racket, and they contain private data, just like the variable priv.

This is important. Arc code cannot access the private data of the stdin port. Only the Racket functions readc, peekc, readb, etc. can access stdin's private data. This means two things:

1) The above-mentioned functions (readc, peekc, etc.) must be implemented in Racket, because only they can access stdin's private data.

2) This makes it much harder and more verbose to create "fake" input ports: a data type that is written in Arc, but designed to behave like an input port.

Let's consider an example. We wish to create two input ports, foo and bar. Here is the obvious way:

  (let stream '(#\f #\o #\o)
    (def foo ())
    
    (extend peekc (x) (is x foo)
      (car stream))
      
    (extend readc (x) (is x foo)
      (pop stream)))
      
      
  (let stream '(#\b #\a #\r)
    (def bar ())
    
    (extend peekc (x) (is x bar)
      (car stream))
      
    (extend readc (x) (is x bar)
      (pop stream)))

  (peekc foo) -> #\f
  (readc foo) -> #\f
  (readc foo) -> #\o
  (readc foo) -> #\o
  (readc foo) -> nil

  (peekc bar) -> #\b
  (readc bar) -> #\b
  (readc bar) -> #\a
  (readc bar) -> #\r
  (readc bar) -> nil

Now, that was just streams. Tables are a whole different ballgame. In fact, in pgArc it's impossible to make a fake table (something that looks and behaves like a table, but isn't). This is because you can't properly extend eval and apply in Arc. But even if you could, it would be just as clunky as the stream example (or worse).

Why is this? It is for the exact same reason: tables contain private data that only Racket knows about. Arc cannot access that private data, nor can it create it. But before we solve that problem, let's take a little detour...

---

There are fundamentally two ways of calling a function. Let's consider a simple example:

  (+ 1 5)  -> 6
  (1 '+ 5) -> 6

  (+ 1 5) -> (1 '+ 5)
  (- 1 5) -> (1 '- 5)
  (* 1 5) -> (1 '* 5)
  ...

  (car my-list)       -> (my-list 'car)
  (len my-list)       -> (my-list 'len)
  (my-table 'foo)     -> (my-table 'get 'foo)
  (= my-table.'foo 5) -> (my-table 'set 'foo 5)

But... wait... functions can accept arguments. Consider the two streams again. foo is a self-contained stream, that has access to private data. We want a way to access that private data, which is why we use the functions readc and peekc. The problem is, that the behavior of the private data varies depending on how foo is called.

If foo is called by readc, we want it to consume a character... but if foo is called by peekc, we don't want it to consume a character. But foo doesn't know what function called it! It doesn't know whether it was called by readc or peekc, so how does it know what to do? Simple, we give it a message, telling it what to do:

  (foo 'peek)
  (foo 'read)

  (let stream '(#\f #\o #\o)
    (= foo (annotate 'input (fn (x)
                              (case x
                                'peek (car stream)
                                'read (pop stream))))))
                                
  (let stream '(#\b #\a #\r)
    (= bar (annotate 'input (fn (x)
                              (case x
                                'peek (car stream)
                                'read (pop stream))))))

Okay, that's great for streams, but what about tables? They work too! My idea is that when `apply` sees something that is annotated with type 'table, it will do a behind-the-scenes conversion:

  ; before            -> after
  (my-table 'foo)     -> (my-table 'get 'foo)
  (= my-table.'foo 5) -> (my-table 'set 'foo 5)
  (keys my-table)     -> (my-table 'keys)

  (annotate 'table (fn (x)
                     (case x
                       'keys ...
                       'get ...
                       'set ...)))

  (def attr (x . args)
    (apply (annotate 'fn x) args))

  (def cons (x y)
    (annotate 'cons
      (fn (m)
        (case m
          'len (+ (len y) 1)
          'car x
          'cdr y))))

  (def car (x)
    (attr x 'car))

  (def cdr (x)
    (attr x 'cdr))

  (def table ()
    (let alist '((nil nil))
      (annotate 'table
        (fn (m k v)
          (case m
            'keys (map [car _] alist)
            'get (alist k v)
            'set (sref alist k v))))))

  (def keys (x)
    (attr (coerce x 'table) 'keys))

  (def len (x)
    (case type.x
      'cons  (attr x 'len)
      'table (len (keys x))))

  (def vals (x)
    (map [x _] (keys x)))

It's untested, and py-arc doesn't actually support my idea (yet), so I might not be testing it anytime soon. As such, there's probably bugs in it. But it should give you a taste of the power of this idea: the ability to define core datatypes in Arc itself, using only functions (no piggybacking!), but do so in a way that it's easy for user code to extend.

As a side note, what's with the attr function? Well, you see, when Arc sees (my-table 'foo) it expands it to (my-table 'get 'foo), so calling (my-table 'keys) doesn't work. But by annotating the table with type 'fn, you can get at the actual "hidden" attributes, like 'keys, 'get, and 'set. This means that (my-table 'keys) will never conflict with (my-tables 'get 'keys).

Functional programming tends to emphasize making new operations on existing values, while object-oriented programming tends to emphasize making new kinds of values with support for existing operations. It's less common to have a language support both, and it naturally gets less and less common the more one demands other properties of the solution. Languages which already prioritize certain problems have more difficulty tackling the expression problem.

Well, when Philip Wadler coined the term "expression problem," he had certain other problems in mind that made it difficult: He was going for a way to statically check that all the necessary extensions existed, and he wanted to do it in a language with separate compilation of modules. These are hardly problems Arc's designed to solve, being such a dynamic language, but according to some people they're essential to what makes the expression problem a problem. http://lambda-the-ultimate.org/node/4136#comment-63009

As long as we don't make those demands in Arc, it's a non-problem indeed, and one easy solution is 'extend:

  ; Define a new type A with support for existing ops B and C.
  (= A-type (uniq))
  (def isa-A (x) (isa x A-type))
  (extend B (self . args) isa-A.self ...)
  (extend C (self . args) isa-A.self ...)
  
  ; Define a new op A with support for existing types B and C.
  (def A (self . args) (err "Can't A these args."))
  (extend A (self . args) isa-B.self ...)
  (extend A (self . args) isa-C.self ...)

This 'extend kind of approach is widely known as "predicate dispatch," unsurprisingly. Unfortunately, Because the predicates can have Turing-complete dependence on the run time arguments, predicate dispatch about as difficult to statically reason about as it gets. This in turn makes it hard to make a module system that automatically determines a precedence order among extensions made in different libraries. http://lambda-the-ultimate.org/node/3700#comment-52728

Those are downsides that hardly have alernatives in a language that's mostly dynamic anyway, like our Arc-like languages. In fact, once I get back around to Blade, my extensible static language idea, I expect to still like predicate dispatch: Blade will probably have no (built-in) static type system, and I already design libraries so they extend only cases they introduce. If and when precedence is an issue, I can probably handle it very nicely using my self-organizing rule precedence system from Lathe.

Ohhh, predicate dispatch. Predicate dispatch, la la la~

ahem

The prototype system doesn't seem (to me) to actually solve the expression problem, instead erring on the side of being too object-oriented: It focuses on making new kinds of values with support for existing operations, but doesn't make it easy to make a new operation (message) that acts on existing kinds of values. (I don't think this is anything I haven't said before, but maybe it's cast in more common terms now. ^_^ )

No. I am talking only about the issue of creating new values. Message passing is not designed to solve the problem of creating new operations. That job is already fulfilled by `extend`. Message passing solves the issue of creating new data types. Nothing more, nothing less.

---

I am well aware that extend can solve it... Message passing solves it while being significantly shorter and easier to use. Also, extending does not work well with modules...

If one module tries to create or modify an existing data type, and it uses `extend`, only that module will see the extensions. Thus you are forced to either jump through even more hoops, or extend __built-ins* which is bad because then safe modules can't properly create new data types...

Extend is at least twice as verbose [1] as message passing, it is more difficult to use, and it causes problems when you try to introduce a module system. Message passing has none of those issues...

---

I'm not talking about prototypes [2], I'm talking about message passing, which is different. It is possible to create prototypes using message passing, but message passing is on a lower level, in the same way that it's possible to create iterators (high level) using continuations (low level).

Message passing simply means that you (the caller) tell the function what you want it to do. This message would probably be a symbol like 'get or 'keys or whatever, but it doesn't have to be a symbol [3].

The only important thing to understand about message passing is that it inverts control: rather than a function returning a value, and then you operating on that value, you send an operator (the message) to the function, and then the function uses that operator to do something.

This is good for describing data types, which are self-contained. It is not so good for describing programs. That's why I'm proposing a mixture of message passing (at the lowest level) and functional (at higher levels).

---

"It focuses on making new kinds of values with support for existing operations, but doesn't make it easy to make a new operation (message) that acts on existing kinds of values."

You can use extend to create new operators, just as we do right now. But extend does not work well for creating new data types that use the existing operators. A combination of extend + message passing is ideal.

Extend is a beautiful thing, but when all you have is a hammer, everything looks like a nail. That doesn't mean a hammer is always the appropriate tool for the job. Use message passing when it makes sense, and use extend when it makes sense.

---

Also, it seems you still don't think that message passing is more extensible... what is extensibility? It can't be "the capability to do something" because any Turing-complete language can do that. It's how easy it is to do things, and how easy it is to make changes to your program.

If you try to make a new data type with extend, you end up jumping all over the place, extending several built-ins, and now good luck making it play nicely with modules and existing code.

With message passing, you just change one place and everything just works perfectly. Thus, they can both do the same thing, but message passing allows for changes and additions to data types in a way that requires you (the programmer) to do less work. Thus, I would call that more extensible.

---

* [1] http://arclanguage.org/item?id=14261

* [2] The original post in this thread does talk about prototypes, but that's because I was still figuring stuff out, and didn't realize that message passing was better. It is possible to create prototypes using message passing, so I'm going to ditch the prototype idea and focus on message passing. If we ever need/want prototypes, we can easily add them using message passing.

---

"No. I am talking only about the issue of creating new values."

Okay then, we're getting somewhere. ^_^ Nevertheless, back when you said "The naive approach of creating a new data type and extending sref is not extensible or scalable" (emphasis yours), I don't think you actually suggested anything more extensible. Maybe we just have a difference in meaning here, or maybe the "scalable" part was crucial to what you meant, or maybe you've changed your mind now. Is it any of those things?

---

"Message passing solves the issue of creating new data types. Nothing more, nothing less."

Then your approach doesn't solve a problem I find important. In fact, in the examples you've given, it would make it a harder problem to solve, thanks to encapsulation. So not only would I not find the approach useful myself, but I would find libraries that did follow that approach less helpful.

Thus I continue to advocate 'extend, which solves the issue of creating new data types and solves even more problems I have (or expect to have).

Note that there are other problems message-passing might solve, notably parallelism. I'm not saying you can magically get parallelism in a few steps from where you're at, lol, but it's probably a natural way to begin approaching the expression problem in a system that already has a parallelism focus.

---

"If one module tries to create or modify an existing data type, and it uses `extend`, only that module will see the extensions."

Not necessarily. If modules are implemented the way I like (which is biased since I like 'extend), an imported variable could have the same mutable binding as the binding it has in its home module, in which case an 'extend extension would be visible in every part of the program that uses that module.

You could argue that it's not always a good idea for extensions to be visible everywhere, and I agree. It might not be a problem in my own code, since I almost always write libraries so that they only extend cases they introduce (so that no two libraries' cases overlap), but it is a problem in general, and I've been wrestling with recently. I believe there is actually a solution to this problem that lets me keep using predicate dispatch, but your style of message-passing doesn't inspire me.

---

"Extend is at least twice as verbose [(example)] as message passing"

Your example isn't fair to 'extend. You assume 'iso, 'fill-table, and 'vals would be derived functions in PyArc, but you don't give Wart the same advantage. You use the same (m k v) argument list for the 'keys, 'get, and 'set functions, when really different methods will tend to have different argument lists (at least in my mind).

Meanwhile, you overlook that the 'extend approach is extensible enough to support a message-passing system like yours inside it (which is what should really matter in the core, IMO). Plenty of times I design types that support only a fixed set of behaviors and fully encapsulate those behaviors; for me, they tend to be shaped as lists of functions (virtual method tables?), but from time to time I use a dispatch-on-a-symbol approach like yours. The way I make those types work is straightforward: I use 'extend and delegate to the functionality held within the value. There's a bit of boilerplate involved in this right now--not as much as I think you see, but some. (Once we've established a type, we can use it over and over.)

Since you also say "extend does not work well for creating new data types that use the existing operators" (emphasis yours), here's an illustration, extrapolating from my earlier example:

  ; Define a new type A with support for existing ops B and C and with
  ; nothing else special about it.
  
  (= A-type (uniq))
  
  (def isa-A (x)
    (isa x A-type))
  
  (def make-A (B-behavior C-behavior)
    (annotate A-type (list B-behavior C-behavior)))
  
  (extend B (self . args) isa-A.self
    (apply rep.self.0 self args))
  
  (extend C (self . args) isa-A.self
    (apply rep.self.1 self args))
  
  ; Later on:
  ;
  ; (make-A (fn (this <arguments to B>) ...)
  ;         (fn (this <arguments to C>) ...))

However, I currently don't have enough motivation for encapsulation to consider it a positive aspect of my designs; I try to see types like these as placeholders for more open-faced types, and I make them only in the most simplistic, irreducible cases where the boilerplate hardly matters. It'll be a while before I have a meaningful opinion on which way to go from here (and I welcome you to surpass me in the meantime).

---

"I'm not talking about prototypes, I'm talking about message-passing, which is different."

Oops, sorry about that. I was just using "prototypes" as a name for your general approach, and I forgot you drew a distinction. You've illustrated message-passing without prototypes now (the 'tap-dancing example), and I think that helps.

---

"Also, it seems you still don't think that message-passing is more extensible... what is extensibility?"

---

"Maybe we just have a difference in meaning here, or maybe the "scalable" part was crucial to what you meant, or maybe you've changed your mind now. Is it any of those things?"

I think we just have a different view of extensibility.

---

"In fact, in the examples you've given, it would make it a harder problem to solve, thanks to encapsulation."

Um... extending is just as encapsulated as message passing is... I'm not sure what you were trying to say.

---

"Thus I continue to advocate 'extend, which solves the issue of creating new data types and solves even more problems I have (or expect to have)."

Please give a single example where extending is better than message passing, with regards to creating or modifying data types.

I'm not arguing that we get rid of extend. Extend is great, it gets to stay. I'm arguing that implementing the low-level data types using message passing would be less verbose and easier than using extend.

Even if we adopted message passing, extend would still have a large role to play. In fact, I envision extend being used to add/modify/remove messages:

  (def bad-function (m)
    (case m
      'get ...          ; bad-function has a bug in it!
      'set ...))
      
  (extend bad-function (m) (is m 'get) ; fix bad-function, hurray!
    ...)

  (extend table-proto* (m) (is m 'my-message)
    ...)

---

"I believe there is actually a solution to this problem that lets me keep using predicate dispatch, but your style of message-passing doesn't inspire me."

I'd be interested in hearing that.

---

"Your example isn't fair to 'extend. You assume 'iso, 'fill-table, and 'vals would be derived functions in PyArc, [...]"

I'll note that with message passing, it can use a type of 'table, and thus work seamlessly with code that uses isa, whereas in wart you need to create a new type, and then extend code (like iso and filltable) to recognize your new type.

But... let's assume that there was a built-in called `table?` that tested for "tableness", and that the other built-ins used that rather than isa:

  (def clone(x)                                   ;  (def clone(p)
    (annotate 'heir (obj val (table) parent x)))  ;    (let t (table)
                                                  ;      (annotate 'table (fn (m k v)
  (defcall heir h                                 ;                         (case m
    [or rep.h!val._                               ;                           'keys (join (keys p) (keys t))
        rep.h!parent._])                          ;                           'get (or (t k v) (p k v))
                                                  ;                           'set (sref t k v))))))
  (defset heir(h key value)                       ;
    (= rep.h!val.key value))                      ;
                                                  ;
  (def table?(x) :case (isa x 'heir) t)           ;
                                                  ;
  (def keys(x) :case (isa x 'heir)                ;
    (join (keys rep.x!val)                        ;
          (keys rep.x!parent)))                   ;

I'll also note that I dislike how this requires you to create a new type every time you want to extend something... thus even if we moved to a `table?` approach, we'd still be keeping isa and type around, solely to support extensions... So if they're in the language, why not co-opt them for a more useful purpose (like message passing)? Or figure out a better way to create data types that doesn't even use isa?

---

"You use the same (m k v) argument list for the 'keys, 'get, and 'set functions, when really different methods will tend to have different argument lists (at least in my mind)."

You're right: that's done for simplicity, but that is easily solved. For instance, consider a macro like this:

  (object table
    keys () ...
    get (k v) ...
    set (k v) ...)

  (object cons
    call args ...
    car () ...
    cdr () ...)

---

"Meanwhile, you overlook that the 'extend approach is extensible enough to support a message-passing system like yours inside it (which is what should really matter in the core, IMO)."

Uh huh. And how are you going to get that to play nicely with the built in table type? Or the built in cons type? It's much easier to design things so the built-in types already use message passing.

---

"[...] here's an illustration, extrapolating from my earlier example: [...]"

Now do that with tables. Or lists. Or numbers. Harder, ain't it?

---

"However, I currently don't have enough motivation for encapsulation to consider it a positive aspect of my designs;"

Once again... how is message passing "more encapsulated"?

---

"and at the beginning of your post you claimed that your message-passing only tries to tackle one half of that problem: extending the system with new kinds of value, not extending it with new operations."

Yes, because extend is already pretty darn good at covering the other half. So by combining the two, you can get the best of both worlds.

---

My point is not that "extend can't do that" because extend is definitely powerful enough to do it. OOP is also powerful enough to describe any program. That doesn't mean every program should be written in OOP style.

If we define a whole bundle of behaviors in one closure, or in a nearby set of closures, then the value can naturally encapsulate variables in the lexical scope. Essentially, the value itself is a closure but with a different interface.

If instead we extend the behaviors at the top level, outside the construction context, then we have to design the value's representation in such a way that those extensions can inspect it. This encourages a lesser degree of encapsulation, which in turn gives future programmers more freedom in using the code. On the other hand, sometimes there are just no other operations worth adding, and this just makes the code more scattered and harder to reason about.

Oh, I overlooked your post at http://arclanguage.org/item?id=14307. The encapsulation aspect is exactly the part that allows for certain optimizations.

---

"Please give a single example where extending is better than message passing, with regards to creating or modifying data types."

Well, I gave an example before the part you replied to, and I gave a more in-depth version in the same post as you replied to. I dunno, I think they're "better," but keep in mind that we're solving different problems: in my case, the expression problem, and in your case, convenient construction of custom data types. Based on the motivations you've expressed, I believe my problem has a strictly broader scope than yours, but by the same token, I do believe a narrower scope can allow for a more elegant special-case solution.

---

"I'm arguing that implementing the low-level data types using message passing would be less verbose and easier than using extend."

That actually sounds like a pretty nifty way to go, not because of verbosity and ease of implementation ('cause I have a pretty high tolerance of complexity in the core) but because it may eliminate some primitive types.

There'd be trouble eliminating all of them though: If you pass 'car a message, then when that message is parsed from the argument list, there may be an infinite loop.

---

"Voila, now all tables support your new message."

I think I specifically said it would be hard to add behaviors low on the prototype hierarchy. :) Specifically, it would be hard to add behaviors to things whose behaviors are determined by their constructors, rather than their prototypes; I'll get to this topic at ( * ).

---

"I'd be interested in hearing [how you plan to reconcile predicate dispatch and modules]."

I think I'd rather start a new thread for that. I'll try to get around to it soon.

---

"I'll note that with message passing, it can use a type of 'table, and thus work seamlessly with code that uses isa, whereas in wart you need to create a new type, and then extend code (like iso and filltable) to recognize your new type."

I haven't heard you say what happens when something needs to act seamlessly both as a 'table and as an 'input (my example at http://arclanguage.org/item?id=14258). You did have an example of alists being tables and cons cells, but I don't see how it would work, since you said (type my-alist) was 'cons. (http://arclanguage.org/item?id=14261)

---

"But with message passing, you get correct behavior with modules out of the box..."

( * ) Actually, I don't see the connection between message-passing and modules at all. That's probably your point, but if I did end up wanting to add a new operation on heirs (a desire you don't claim to be prepared for), the most straightforward way would be to redefine 'clone completely, in a way that intentionally defies module separation so that all the program's heirs use the new definition.

---

"[...]even if we moved to a `table?` approach, we'd still be keeping isa and type around, solely to support extensions[...]"

In http://arclanguage.org/item?id=14265 I mentioned that we could get rid of 'type and 'rep completely by using a (deftype wrap-foo unwrap-foo isa-foo) form, but that I'd prefer to let people have access to 'type and 'rep too, if only for emergencies.

---

"For instance, consider a macro like this"

How about failcall and a macro like this:

  ; This makes a new type, extends 'keys, 'ref, and 'sref with rules
  ; that check that their first argument is that type and delegate to
  ; its inner behaviors, and defines a function 'fn-make-foo and a macro
  ; 'make-foo for constructing values of the new type.
  ;
  (def-extension-type fn-make-foo make-foo
    keys ref sref)
  
  (make-foo
    ()                         ...
    (key (o default))          ...
    (new-val key (o default))  ...)

Note the fact that you have to say 'def-extension-type this way, but that each construction of the type is slightly shorter since it doesn't have to reiterate which "methods" it implements.

Anyway, I expect one could actually duplicate this exact syntax in a message-passing system that has failcall. So you probably still have a potential brevity advantage, thanks to the fact that your problem has a narrower scope than mine, but it's pretty close.

---

"Uh huh . And how are you going to get that to play nicely with the built in table type?"

You're right, if message-passing is merely a library, there's a good chance it won't be seamless with existing things in the language.

---

"Now do that with tables. Or lists. Or numbers. Harder, ain't it?"

I address tables and lists here: http://arclanguage.org/item?id=14337

Numbers, on the other hand, are more interesting. ^_^ It's super-tempting to make them efficient with the help of algorithms that manipulate the binary representation of the number. IMO, that means numbers should ideally be represented as binary strings.

Yeah, that's what message passing does... gives a way to get at the internal details of the function. The alternative is to use a closure + extend. Both systems use closures to hide internal data, they just have different interfaces for "getting at" the internal data.

---

"Based on the motivations you've expressed, I believe my problem has a strictly broader scope than yours, but by the same token, I do believe a narrower scope can allow for a more elegant special-case solution."

Yeah, you're right. A good solution to the expression problem would be nice, but in the meantime, how about sprinkling a little sugar on the current difficult areas?

---

"There'd be trouble eliminating all of them though: If you pass 'car a message, then when that message is parsed from the argument list, there may be an infinite loop."

How so? If cons is defined as follows:

  (def cons (x y)
    (object cons
      car () x
      cdr () y))

  (def car (x)
    (attr (coerce x 'cons) 'car))

  (def cdr (x)
    (attr (coerce x 'cons) 'cdr))

---

"I think I specifically said it would be hard to add behaviors low on the prototype hierarchy. :) Specifically, it would be hard to add behaviors to things whose behaviors are determined by their constructors, rather than their prototypes;"

But isn't a prototype a constructor? The whole point of prototypes are that they remove the distinction between class/instance, so in a sense, every prototype is both an instance and a class at the same time.

---

"I haven't heard you say what happens when something needs to act seamlessly both as a 'table and as an 'input"

I was actually mulling that over last night... I see two (compatible) ways to do that: duck typing and allowing values to have multiple types.

For instance, let's say you wanted to make a table that also happens to be an iterator... tables have a 'keys, 'call, and 'set message. Iterators have a 'next message. So you would just combine them together:

  (object table
    keys ()    ...
    call (k d) ...
    set  (k v) ...
    next ()    ...)

  (object (iterator table)
    keys ()    ...
    call (k d) ...
    set  (k v) ...
    next ()    ...)

But by using types-as-interfaces, this lets you give context to the messages. So an iterator that has a 'next message is different from something else that has a 'next message. If you really don't care about the type, you can just use `attr`.

---

"Actually, I don't see the connection between message-passing and modules at all. That's probably your point, but if I did end up wanting to add a new operation on heirs (a desire you don't claim to be prepared for), the most straightforward way would be to redefine 'clone completely, in a way that intentionally defies module separation so that all the program's heirs use the new definition."

The point is that you want one module to be able to create a new data type, and then let other modules use it. Because isa uses a symbol, and symbols are interned across modules, (isa foo 'table) would work just fine. This is not true with things like `table?` unless you jump through some hoops or have every module extend __built-ins*

As for adding a new operation on 'heir... it's possible with message passing, but I don't think it would be particularly superior to extend. In fact, it may just be easier to define that new behavior using extend, since message passing is more concerned with data and interfaces than behavior.

---

"Anyway, I expect one could actually duplicate this exact syntax in a message-passing system that has failcall. So you probably still have a potential brevity advantage, thanks to the fact that your problem has a narrower scope than mine, but it's pretty close."

By the time someone adds all the messages needed to extract the private data out of the type (which they can't do without modifying the library), it'll

---

"A good solution to the expression problem would be nice, but in the meantime, how about sprinkling a little sugar on the current difficult areas?"

Not to belittle, but it doesn't strike me as difficult. I'd rather sprinkle sugar on 'extend if I need to, using something like my 'def-extension-type example, but I haven't needed to yet.

---

"How so? If cons is defined as follows[...]"

If 'car is implemented as a function and you call it and 'car tries to get its argument using 'car, you're in trouble. There's a similar loop if you're sending the message 'car rather than calling a function 'car, but removing the built-in implementation of 'car for built-in cons cells is the problem here, not message-passing.

---

"But isn't a prototype a constructor?"

In your example at http://arclanguage.org/item?id=14331, the (annotate 'table (fn ...)) expression is an example of constructing an instance, and you specify instance-specific behavior in that expression, and they get to refer to some encapsulated data. It's hard, if not impossible, for someone using 'clone as a library to add behaviors to that instance which have access to the same data, whether we used prototypes or not.

---

"So then comes in the other idea... allowing values to have multiple types at once."

Would 'type return a list of types? I see no problem with the way this is going, but it feels more complicated now.

---

"This is nice because it allows for data to use the same message name, without causing conflicts."

Actually, I bet you'd get a conflict when a value needed to implement two interfaces that had completely different meanings for 'next. Whether you consider that important is up to you; it's a common gotcha in OO languages with inheritance. I consider it to be a namespacing issue, and that encourages me to use the global namespace somehow. I'd put 'next in the global namespace as an extensible function, but it could easily be a global variable holding a gensym message name instead.

---

"The point is that you want one module to be able to create a new data type, and then let other modules use it."

Yep, I understand that's what you want, and I understand that's all you expect your technique to tackle.

To phrase the other half of the expression problem that in those terms: I want one module to be able to create a new data type, and then let other modules and application code interpret that value according to interfaces of their own devising.

"Because isa uses a symbol, and symbols are interned across modules, (isa foo 'table) would work just fine. This is not true with things like `table?` unless you jump through some hoops[...]"

If two modules use different notions of table, then a single value like 'table can't uniquely identify them both. There's a name clash.

If they use the same notion, then they should both depend on whatever module introduces that concept anyway (probably the core), and that module can provide them access (and extension access) to functions like 'table? and 'keys.

---

"Or to put it another way, rulebooks could be implemented as a library on top of message passing."

Well, the kind of message-passing system you're talking about makes encapsulation very easy. The kind of 'extend system I'm talking about only thrives on unencapsulated types. They have an adversarious relationship right now.

That said, if you make your message receivers less encapsulated, rulebooks might indeed work as a library. ^_^

Failcall not so much, though. I managed to get failcall mostly working as an Arc library (in Lathe's arc/failcall.arc), but any preexisting code that uses 'apply is a potential abstraction leak. Failcall pretty much needs to be in the core to be seamless.

---

"When you extend a function, it has some overhead, yes?"

It's not using the function car to get the message... it's using the symbol 'car. An infinite loop should only happen in the following situation (or similar):

  (= foo (object cons
           car () (car foo)))

  (= foo (cons 1 2))

  (= bar (object cons
           car () (car foo)))

  (car bar) -> 1

  (def attr (x . args)
    (apply (coerce x 'fn) args))

  (attr foo 'car)
  (attr foo 'cdr)

"In your example at http://arclanguage.org/item?id=14331, the (annotate 'table (fn ...)) expression is an example of constructing an instance"

Hm... solvable, but a lot trickier. That's a good point.

---

"Would 'type return a list of types? I see no problem with the way this is going, but it feels more complicated now."

That would be the simple way to do it, yeah, which is what I was planning on doing with py-arc. There are other ways, but I doubt they'd be any simpler. Then isa would be defined like so:

  (def isa (x t)
    (in (type x) t))

"Actually, I bet you'd get a conflict when a value needed to implement two interfaces that had completely different meanings for 'next."

That's where coerce comes into play. Let's say we had two interfaces, foo and bar. They both have the 'something message, but the two interfaces give it different meanings. You could then use coercion to choose which meaning you want:

  (attr (coerce x 'foo) 'something) ; use foo's meaning
  (attr (coerce x 'bar) 'something) ; use bar's meaning

---

"If two modules use different notions of table, then a single value like 'table can't uniquely identify them both. There's a name clash."

Right, that's why you would use a different type... each type would map to a single interface. Two interfaces == two types.

---

"Well, the kind of message-passing system you're talking about makes encapsulation very easy. The kind of 'extend system I'm talking about only thrives on unencapsulated types. If you make your message receivers less encapsulated, rulebooks might indeed work as a library. ^_^"

I'm really not understanding you when you say "encapsulation", which seems to be very different from the way I think of "encapsulation". Please explain how extend is less encapsulated, and how message passing is too encapsulated.

---

"Actually, it can, but I expect almost all programs to have a small constant number of rules in each rulebook, and I expect (and encourage) constant-time failure conditions so one rule doesn't encumber the others. So I expect most rulebooks to have constant-time overhead, and that's the same complexity I expect of message-passing."

If you wish to create a new type that behaves like tables... you need to extend at least keys, sref, and ref, correct? Now if you wish to have 5 different table types, then that means that those functions need to be extended 5 times. Message passing should always be O(1) but extend is O(n) where n is the number of times the function has been extended.

Did you stop reading halfway through what you quoted? "There's a similar loop if you're sending the message 'car rather than calling a function 'car," and here's the loop:

1. The cons pair (car . nil) receives the message 'car with no arguments.

2. The message-passing function implementing the cons pair (car . nil) receives the argument list '(car).

3. The message-passing function implementing foo needs to determine the first element of its argument list to see what message it should handle. It sends that argument list the message 'car with no arguments.

4. Go to 1.

---

That's where coerce comes into play.

I don't see it. I don't think something really counts as two types unless you can use that value as either type right away. The way you're using 'coerce, every method pass will need to have an accompanying 'coerce just to be sure, and a few method-passing functions will need to check the current type as well as the method name. At that point you might as well just combine the type name and the method name into one symbol and go back to square one.

---

"Right, that's why you would use a different type..."

Two modules can be developed by two people who don't realize they're using the "same" type. You might as well tackle the issue of a module system by saying "that's why you would use a different global variable."

---

"I'm really not understanding you when you say "encapsulation", which seems to be very different from the way I think of "encapsulation". Please explain how extend is less encapsulated, and how message passing is too encapsulated."

This again.... Okay, here's an explanation by example. ^_^

  (def counter ()
    (let count 0
      (fn ()
        ++.count)))

  arc> (= foo (coun­ter))
  #<procedure: counter>
  arc> type.foo
  fn
  arc> rep.foo
  #<procedure: counter>
  arc> (is foo rep.f­oo)
  t

This value is encapsulated. Specifically, we can think of its implementation as encapsulating the mutable variable 'count by storing it in a lexical closure hidden within the 'fn value. This hypothetical implementation doesn't matter to what we can do with the value, but it does indicate how to reimplement it with less encapsulation:

  (def counter ()
    (annotate 'counter list.0))
  
  (defcall counter (self)
    (++ rep.self.0))

  arc> (= foo (coun­ter))
  #(tagged counter (0 . nil))
  arc> type.foo
  counter
  arc> rep.foo
  (0)

  (extend peek (str) (isa str 'counter)
    (inc rep.str.0))

So, back to message-passing. Message-passing constructions like (annotate 'table (fn ...)) capture their lexical surroundings just like regular closures do, making it natural to use them for implementations similar to the first version of 'counter.

---

"Now if you wish to have 5 different table types, then that means that those functions need to be extended 5 times."

If you implement those five types using (annotate 'table (fn ...)), then you have to implement three methods per type, for the same overall result. I believe this is what akkartik meant by "moving parens around."

Nope! Just not understanding why it's a problem, since I don't see how that would cause an infinite loop.

---

"1. The cons pair (car . nil) receives the message 'car with no arguments.

2. The message-passing function implementing the cons pair (car . nil) receives the argument list '(car).

3. The message-passing function implementing foo needs to determine the first element of its argument list to see what message it should handle. It sends that argument list the message 'car with no arguments."

You mean like this?

  ; argument list '(car)

  (fn (m)
    (case m
      'car 'car
      'cdr nil))


  ; cons pair (car . nil)

  (fn args
    (case (attr args 'car)
      'car car
      'cdr nil))

"At that point you might as well just combine the type name and the method name into one symbol and go back to square one."

Yeah, I admit it's not that great at handling the situation where a value wants to implement two different interfaces, but the interfaces have the same message name.

---

"This again.... Okay, here's an explanation by example. ^_^"

Alright, I'm starting to get it. Rather than representing types as a function that maps between keys/behaviors, you're representing it as a simple flat list. But that's quite possible to do with message passing as well:

  (def counter ()
    (let methods (obj call  (fn () (++ methods.'value))
                      value 0)
                      
      (annotate 'counter methods)))


  (= foo (coun­ter))
  (type foo) -> counter
  (rep foo)  -> #hash((call ...) (value 0))

It also serves as self-documenting code. It also allows the built-in functions like `keys` to dispatch on the 'keys method... which doesn't work so well with a list. It also means we can easily write a macro to do the wrapping for us:

  (def counter ()
    (object counter
      call () (++ self.'value)
      value 0))

  (fn (m)
    (case m
      'keys ...
      'set  ...))

I don't know what you're getting at, but the cons cell (car . nil) and the argument list '(car) are supposed to be 'iso in my example; otherwise, why would I say it's a loop? I'm just referring to them in different ways to emphasize their different roles in the story.

I'm sure you're confused about something else in my example, but I don't know what it is. Here's a quick take at summarizing the issue anyway: Inspecting a list by sending it messages is incompatible with decoding a message by inspecting an argument list.

---

"The only downside is that now they're carrying around an internal table... but how is that worse than carrying around an internal list?"

Here's an old post of mine where I go over the pros and cons: http://arclanguage.org/item?id=12076. (Since then, I've learned to stop worrying about 'rep and 'annotate and love predicate dispatch. I put inheritance-aware multimethods in Lathe after that too, but I found they were less useful than I expected for what I was doing.)

With a predicate dispatch approach, it hardly matters whether the internal representation is a table or a list or even a function which dispatches to a complete getter/setter protocol, since I usually only interact with it in a single clump of extensions. Nevertheless, I do find this feature of list-based types pretty helpful:

  (let (foo bar baz) rep.x
    ...)

"So... would you consider the above (using a table to implement message passing) to be "more encapsulated" than your idea?"

Actually, I consider that example to be less encapsulated. Future code can come in and replace the 'call method and add new fields. That would be just fine with me.

Okay... let me see if I understand you. What you're saying is, that if a function is given an argument list (like with rest args), then it can't extract the data out of that list, by sending messages? Why not? A cons is represented as the following form:

  (obj type 'cons
       rep ...
       car ...
       cdr ...)

  (attr foo 'car)
  (attr foo 'cdr)
  ...

  (= args '(car))

  (= args (obj type 'cons
               car 'car
               cdr nil))
               
  (= args.'rep args)

  (fn args ...)

"If 'car is implemented as a function and you call it and 'car tries to get its argument using 'car, you're in trouble."

Which would be represented as follows...

  (def car args
    (car args)
    ...)

  (def car args
    (attr args 'car)
    ...)

---

"Nevertheless, I do find this feature of list-based types pretty helpful:"

Hm... that does seem pretty useful/neat/nifty! Not sure if it offsets the advantages of a table, though.

---

"Here's an old post of mine where I go over the pros and cons: http://arclanguage.org/item?id=12076. "

Neat! Great summary. For the record, I'm leaning toward the second form, with type being an attribute on the table. Then `type` can just be a convenience function (or defined for backwards compatibility).

"no way for different types' x fields to be accessed using the same code without doing something like standardizing the field order"

Yeah, that's kinda the killer (for me) for list-based types. Using lists seems so... short and light-weight, though.

Also, I found this amusing: "Hmm, that gives me an idea. Maybe what I miss most of all is the ability to tag a new datatype so that an existing utility can understand it"

Which is precisely what message passing does. But as you say, you seem to have moved onto other... different directions.

---

"...Do note that tables would have to be built into the core that way though. ;) It's to avoid loops again: Getting a table key by sending a message is incompatible with sending a message by getting a table key."

No worse than having fns in the core. :P Especially if I figure out a way to define fn using message passing. Then... tables would be primitives... and fn would be layered on top...?

---

Because that just sends arguments to another function, and that function tries to extract them, so it sends arguments to another function, and so on. At no point does any function succeed in determining what its first argument is.

"Why `attr`?"

FWIW, I believe I understand what 'attr is for. ^_^ It passes direct messages to a value without passing a "call" message. It just doesn't solve the paradox.

"But that is solved by using attr"

But how does the 'attr function extract its first argument?

---

"Not sure if it offsets the advantages of a table, though."

Me neither. ^_^ To even the field a bit:

Clojure has its own kinda wordy form of table destructuring. An Arc macro can do better than that when it's only trying to do non-recursive destructuring.

  (w/keys a b c rep.x  ; gets the 'a, 'b, and 'c keys
    ...)

Personally, I'd probably go for 'def-extension-type and blissfully ignore the internal representation altogether (except when I'm reading things at the REPL, hm...).

---

"['the ability to tag a new datatype so that an existing utility can understand it'] is precisely what message passing does."

It's what 'extend does too. The tagging is just done in a meta way. The way I see things now, the meaning of a value is only the op-understands-value relationships it supports (duck typing?), so extending an op is a valid way to change (tag) the meaning of a value.

---

"Especially if I figure out a way to define fn using message passing. Then... tables would be primitives... and fn would be layered on top...?"

Hm... well, I still don't get what you're saying, but I'll try implementing it and see if it's a problem or not.

---

"But how does the 'attr function extract its first argument?"

In a worst case scenario, it can be a built-in and use Python magic to avoid the paradox. Having one built-in (attr) is still waaaay better than having a ton of other built-ins (car, cons, cdr, maptable, etc.)

---

"Clojure has its own kinda wordy form of table destructuring. An Arc macro can do better than that when it's only trying to do non-recursive destructuring."

JavaScript also has a form of object destructuring [1]:

  {a, b} = {a: "foo", b: "bar"}

  // or was it like this...?

  [a, b] = {a: "foo", b: "bar"}

---

"Personally, I'd probably go for 'def-extension-type and blissfully ignore the internal representation altogether (except when I'm reading things at the REPL, hm...)."

That's all well and good for high-level coding, but somebody's going to have to determine what the low-level data type will be. I'm going with a table, in py-arc, thanks to our discussions.

---

"It's what 'extend does too. The tagging is just done in a meta way. The way I see things now, the meaning of a value is only the op-understands-value relationships it supports (duck typing?), so extending an op is a valid way to change (tag) the meaning of a value."

Ah, but the key difference is: where is the operator stored? Right now we're only using global functions as operators, but we could also store them as functions in a table. This is better in some situations... and extend is better in other situations.

So yeah, I'm leaning toward duck typing as well, and actually getting rid of the `type` function completely. My idea is that the "type" of something is based solely on it's prototype: `table` would be both a prototype and a constructor of tables (at the same time). You can then create new prototypes that inherit from `table`, and voila, they're tables.

This is sorta-kinda like how JavaScript does it, except JavaScript actually screwed up prototypes, so we can do a lot better in Arc. For starters, in JavaScript you can access the prototype, but you can't change it. That's definitely gonna change: in Arc, you should be able to dynamically change the prototype.

Something's a table, but want it to be a cons? Just change it's 'prototype attribute and bam, it now inherits from cons. Anyways, then the `isa` function would test for inheritance, rather than type. So rather than saying (isa foo 'table) you'd say (isa foo table), which would mean, "does foo inherit from table?" [2].

As far as I can see, the only downside is that isa would now be O(n) rather than O(1), but the prototype chain should be small enough that it won't make a big difference in ordinary programs.

---

"I think you gotta encode behavior in some type or other. I see two ways to avoid functions: You could have tables hold s-expressions and interpret them, or you could have tables encode the behavior themselves. The latter would make tables objects, and they'd be indistinguishable (on a high level) from message-passing closures."

I'm kinda leaning toward the second approach. Even in a worst case scenario, having two built-in primitives that can't be created in Arc (tables and fns) is still better than the current situation.

---

* [1]: https://developer.mozilla.org/en/New_in_JavaScript_1.7#Destr...

Okay, I just realized something... what if we changed destructuring so it worked on tables too? Then this would work:

  (let '(a b c) (obj a 1 b 2 c 3)
    ...)

  (def foo ('(a b c))
    (prn a b c))

  (foo (obj c 1 b 2)) -> nil 2 1

  (foo :c 1 :b 2) -> nil 2 1
  (foo {c 1 b 2}) -> nil 2 1

  (foo '(1 2 3)) -> 1 2 3

  (def annotate (type rep)
    (fn (m . args)
      (case m
        'type type
        'rep  rep
              (apply attr rep m args))))
    
  (def type (x)
    (attr x 'type))
    
  (def rep (x)
    (attr x 'rep))

...but if creating a single table with extend is twice as verbose as message passing, then creating 5 tables with extend will still be twice as verbose as message passing. That seems to be more than just "moving parens around", though perhaps you could argue that the 2x difference doesn't matter.

"Message passing should always be O(1) but extend is O(n) where n is the number of times the function has been extended."

Whoa. Should methods/messages be first-class like functions? Can I pass them around, store them in symbols, generate objects with gensym'd methods? How does 'call by name' interact with all that?

Sure, why not? I actually envision them being implemented as functions, but other ways are possible. In JavaScript, methods are ordinary functions.

As for messages being first-class... they sorta already are, in the sense that the messages are symbols. I don't see much reason to give messages a special type; they can just be symbols.

---

"How does 'call by name' interact with all that?"

Hm... not sure.

---

"Why expend so much collective attention so inefficiently when you can just build it and see what happens?"

:) I think you're underestimating the difficulty of this. You need isa to somehow simulate a set of messages to an object before deciding that yes, it fits that interface. I have no idea how to do that without causing undesirable side effects. Performance is another concern.

---

You were probably more intelligible to everyone else; I seem to have been uncommonly dense :) What you're describing is indeed message passing. I think I was misled by your code snippets.

We're probably going to rediscover why a s-expression syntax for smalltalk is a bad idea, but it should be a fun process.

So... what... just use duck typing for everything? :P Duck typing is easier, but has a greater possibility for strange bugs and conflicts. But Arc is a LFSP, so that might be a good fit for Arc.

---

Neat! In the link you mentioned, I found this: http://okmij.org/ftp/Scheme/oop-in-fp.txt

Which, in fact, describes something almost completely identical to my message passing idea, except that it's more verbose in Scheme than in Arc. It even demonstrates prototypical inheritance.

So rather than trying to completely solve the expression problem by creating a unified system, I'm instead proposing to split it up into two parts: message passing for data, and extend for programs. This is not as elegant as a unified model, but it sure as hell is easier to do, and I don't even know if a good unified model is possible.

---

Also, see this link:

http://www.cs.utexas.edu/users/wcook/papers/OOPvsADT/CookOOP...

I believe that is relevant to the discussion. What I am proposing is message passing (PDA), what you are proposing is extend (ADT). What I'm saying is that both are in conflict with each other, and that sometimes one is better than the other (depending on the situation)...

So rather than using extend for everything (your proposal), how about using extend for some things, and message passing for other things? That is what I'm proposing: using whatever model leads to shorter, more extensible code. I'm focusing on data types because data types are the only area (that I've seen so far) that message passing is significantly better than extend.

---

A quote that I found in the above pdf:

"In 1975, John Reynolds published a paper called “User-defined data types and procedural data structures as complementary approaches to data abstraction” [39] in which he compares procedural data abstraction to user-defined data types. He argued that they are complementary, in that they each has strengths and weaknesses, and the strengths of one are generally the weaknesses of the other. In particular, he found that PDAs offer extensibility and interoperability but obstruct some optimizations. ADTs, on the other hand, facilitate certain kinds of optimizations, but are difficult to extend or get to interoperate"

"doesn't make it easy to make a new operation (message) that acts on existing kinds of values."

I can't imagine how anything could. Isn't this equivalent to "doesn't do your programming for you"?

"Functional programming tends to emphasize making new operations on existing values"

You're saying lisps cover this side, right? So it makes sense to focus on just the other half.

I think Arc already supports both sides, but it almost seems an accidental property of the implementation. Most lisps don't support extensibility, they lock down the primitives. I've complained about this before: http://news.ycombinator.com/item?id=2329680, http://arclanguage.org/item?id=12803.

I speculate that in the dawn of time a lisp introduced equal because they needed to compare lists and couldn't change eq. Or they assumed eq was set in stone since it was in the original McCarthy paper (http://paulgraham.com/rootsoflisp.html). I think of this as the original sin (http://www.arclanguage.org/item?id=13690).

I demonstrated how 'extend could, and although there'd often be more code to write given languages or core library designs where the extension problem is hard, I wouldn't phrase it in as magical a way as doing your programming for you.

I'm not suggesting that a new operation will work on existing values without you specifying the behavior for those values; I'm suggesting that you're given the ability to specify behavior for existing values in the first place.

---

"You're saying lisps cover this side, right? So it makes sense to focus on just the other half."

It's not something that can be solved one piece at a time. Pauan's prototype system makes it hard for a programmer to tell whether a given value is really a core 'table or just inherits from 'table, and that makes makes it hard to specify new behaviors that only apply to cases low on the inheritance hierarchy.

Using the prototype system, I'm sure we can make a new kind of value with support for certain existing operations, but then it'll be hard to make a new operation with support for that kind of value.

---

"I think Arc already supports both sides, but it almost seems an accidental property of the implementation. Most lisps don't support extensibility, they lock down the primitives."

Right, that's a big reason I still use Arc, rather than some other language that's solved its expression problem. I see that as being kind of an emergent feature of having this particular language be so unrestrictive of the programmer. Intentional or not, it's a good language to explore the kind of extensibility I'm interested in. ^^

---

"I speculate that in the dawn of time a lisp introduced equal because they needed to compare lists and couldn't change eq."

Whatever the case, I use weak hash tables, I'd expect a well-designed weak hash table to do an 'eq comparison (the alternative being for unreachable keys to stick around in case an equivalent one is constructed), and at that point I could implement 'eq like so:

  (def eq (a b)
    (.b:weak-table a t))

Even if we accept prototypes (which aren't necessary for message passing), that's not true. Data types could have a 'type message that would return it's low-level implementation, in the rare event that you actually needed it. Or they could have a 'proto message that would return what their prototype is.

Also, perhaps you are unaware of how prototypes work in JavaScript... it is quite easy to tell (once you know how) whether something is an Array, or simply inherits from Array [1] (it is pretty much never necessary to actually do this, though...) It is also possible to get completely fine-grained control, so you can extend all arrays, or a specific subset of arrays, or an individual array.

But that's beside the point, because I'm not talking about prototypes (http://arclanguage.org/item?id=14305).

---

* [1] Object.prototype.toString.call(foo) will return "[object Array]" if and only if foo is an actual array. If it's a "subclass" of Array, then it will be "[object Object]" Clunky? Yes. But that has nothing to do with prototypes; it's clunky because prototypes in JavaScript suck. Prototypes can be done in a much more elegant way.

Pretty much. That's what I'm doing: focusing on the other side. Which, in the case of data types at least, makes programs significantly shorter and easier to write.

I haven't gotten all the way through your post, but I'm pretty sure it's impossible to define callable types in pg-Arc without dropping to Racket, and even then it might be impossible to do it cleanly unless you've loaded ac.scm with Racket's 'compile-enforce-module-constants parameter set to #f. Because of this, I recommend Anarki and Rainbow when it's necessary to have callable types, since they have 'defcall.

But hey, here's an unclean way. ^_^

The only things callable in pg-Arc are functions, tables, strings, and lists, and the only one of those that can have significantly custom behavior is the function. (This is set in stone thanks to 'ar-apply being on the Racket side, where the Racket compiler determines it's a constant.) We'll need to represent custom tables directly as Racket functions that provide the getter behavior.

This doesn't give us an easy way to get the internal representation of the table, or even to distinguish our tables from other functions. Fortunately, we can tack extra metadata onto an encapsulated value like a function by stuffing it into a weak hash table. We just need to make sure our tables aren't 'eq? to any other functions, which in our case should be true thanks to the fact that they capture the local variable(s) holding the storage for this particular table.

(I think pg-Arc does allocate a new function every time a (fn ...) form executes, meaning we don't necessarily need to make sure the function captures something, but Rainbow will compile certain (fn ...) forms into constants, and for all I know Racket may change its behavior in a future version.)

  (mac $ (x) `(cdr `(nil . ,,x))  ; bug exploit to drop to Racket
  
  (= isa-my-table ($.make-weak-hasheq))
  (= rep-my-table ($.make-weak-hasheq))
  
  (def my-table ()
    (let internal-representation (...)
      (let result (fn (key (o default))
                    ...behavior for getting...)
        (= isa-my-table.result t)
        (= rep-my-table.result internal-representation)
        result)))
  
  (extend sref (self val (o default) . rest)
            (and isa-my-table.self no.rest)
    ...behavior for setting...)

We could overwrite 'type, but that doesn't change the type seen internally by 'ac-macro?, 'ar-tag, and 'ar-coerce in ac.scm. They all use 'ar-type directly. Fortunately, nothing uses 'ar-tag and 'ar-coerce but Arc's 'annotate, '+, and 'coerce, all of which we can overwrite on the Arc side. That leaves 'ac-macro? as the one holdout... and that one checks specifically for the type 'mac, so there's nothing lost if it sees 'fn rather than our table type.

Consider this function:

  (def foo (x y)
    (case x
      'add (+ 1 y)
      'sub (- 1 y)))

  (foo 'add 5) -> 6
  (foo 'sub 5) -> 4

  (add foo 5)  -> 6
  (foo 'add 5) -> 6

A table could have methods as well:

  (= foo (obj
           add (fn (x) (+ 1 x))
           sub (fn (x) (- 1 x))))
           
  (foo.'add 5) -> 6
  (foo.'sub 5) -> 4

  var foo = {
      add: function (x) { return x + 1 },
      sub: function (x) { return x - 1 }
  };

  foo.add(5) -> 6
  foo.sub(5) -> 4

http://en.wikipedia.org/wiki/Guarded_Command_Language#Guarde... http://en.wikibooks.org/wiki/Haskell/Control_structures#Guar...

Guards work particularly well with case expressions.

The traditional answer is "in the global scope" or maybe in a let block or something. That's all well and good, in most circumstances. In the case of a data-type, though, you want the possibility to create custom data types, like a custom table, or a custom stream.

The problem is... in this case, because things like peekc and readc are global functions, you end up needing to do some hacky stuff, extending them in a very verbose way to add in your custom stream type.

So instead, let's think of a different place to define the functions. In this case, the functionality is local to the stream: each individual stream decides how it is read. Thus, rather than defining a global function that operates on streams... we define a 'peek and 'char function on the stream itself. That, is the fundamental concept of methods.

And pointing out that a language with these 'places to put functions' doesn't need functions anymore. Your 'methods' are just a form of dispatch, and you want programmable dispatch policies. In the simplest case, no dispatch policy. In a language with your methods you need no concept called functions.

And yes, I'm aware that the term "method" is overloaded, but I still feel it's appropriate. Consider earlier where I showed functions that are placed into a table. I don't think that could reasonably be described as guards, yet they can be reasonably described as methods.

Also, you're right that methods (in OOP languages) can simulate functions, and vice versa, but I feel like functions are "lighter weight". For starters, a method must always be attached to something, it can't be global. I like functions, so I'd like to keep 'em, I just feel like methods are more appropriate in certain (probably rare?) circumstances.

By the way, I'm curious, how would you go about creating a custom table type in wart? In my hypothetical ideal Arc, it would be like this:

  (annotate 'table (def my-table (x n v)
                     (case x
                       'get ...
                       'set ...)))

As I explained in that post, it's impossible to do this seamlessly in Arc:

  (= my-proto (clone nil x "foo"))

  (my-proto 'x)           -> "foo"
  (= (my-proto 'x) "bar") -> "bar"
  (keys my-proto)         -> (x)
  (vals my-proto)         -> ("bar")

  $ git clone git@github.com:akkartik/wart.git
  $ cd wart
  $ git checkout a5d25c805a97347d1207 #Current HEAD for those trying this in future

  $ cat > 099heir.wart # Just kidding; copy and paste the following lines
                       # until the '^D' into 099heir.wart

  ; A new type: heir (because it inherits from a parent)
  (def clone-prototype(x)
    (annotate 'heir (obj val (table) parent x)))

  ; Extend ssyntax
  (defcall heir h
    [or rep.h!val._
        rep.h!parent._])

  ; Extend assignment
  (defset heir(h key value)
    (= rep.h!val.key value))

  ; Extend keys
  (def keys(x) :case (isa x 'heir)
    (join (keys rep.x!val)
          (keys rep.x!parent)))
  ^D

  # Now start your engines
  $ wart

  wart> ; And try the following expressions at the prompt
  (= a (table))
  (= b (clone-prototype a))

  ; inheriting from parent works
  (= a!x 23)
  (prn b!x) ; => 23

  ; assignment works
  (= b!y 34)
  b!y ; => 34
  a!y ; => nil

  ; keys works
  keys.a ; => (x)
  keys.b ; => (x y)

  ; vals is left as an exercise for the reader :)

  wart> (quit) ; or hit ctrl-d a few times

It's nice that wart makes it so easy to extend stuff (with :case and defcall), but that doesn't help with the situation of having to extend multiple built-in functions (like keys, vals... anything that relies on tables, even user functions!) just to define one new type.

I can't think of a situation where I'll need to override user functions once I've overridden the basic operations. Can you? I think you just need to override these ops: https://github.com/akkartik/wart/blob/a5d25c805a97347d1207e7...

As for extending user functions... consider this: what about code that expects something of type 'table? Your prototype is of type 'heir... so that won't work. My system does work in that situation. That can be mitigated by having functions coerce to 'table, and then define a defcoerce for heir, but why jump through so many hoops when you can get all that for free?

I blame that code for its lack of extensibility. It should just go ahead and use the value as a table, trusting the extensions to be there (and using failcall to recover, of course ^_^ ).

If it must do a boolean check to determine the value's features (perhaps because failcall isn't available, or because it needs to fail fast), it shouldn't do a direct type check. Instead, it should call another global function dedicated to the inquiry of tableness (which is probably too specific an inquiry anyway, most of the time). That way, people can extend the tableness function too.

  (def table? (x) (isa x 'table))

You're trying to solve the symptom... I'm trying to solve the problem. What I'm saying is that re-extending all the relevant built-ins is ridiculously inefficient and verbose, and that there's a far better way. Message passing not only drastically reduces verbosity, but it also solves the problem you mentioned.

I suggest you go and read http://arclanguage.org/item?id=14237 which already explains this in more depth.

I think the current way that people think about types is fundamentally wrong, and I'll write a post about that in a moment.

Yes. In a world with failcall, it can try treating the value as a table in one rule, as a cons as another rule, and as a string in a third rule. In a world without failcall, it can do the same thing, but with the help of extensible global functions 'atable, 'acons, and 'astring. That's merely a specific example of what I just said.

---

"How is using (table? foo) better than (isa foo 'table)? Both are the same as far as extensibility goes."

Sure, there's not a whole lot of difference between extending 'table? and extending 'isa. However, in my mind 'isa has a clear role as a way to check the most specific possible feature of a value: the kind of internal data it has.

One could go all the way down, making a (deftype wrap-foo unwrap-foo isa-foo) form that eliminates the need for 'annotate, 'rep, 'isa, and 'type altogether. In this example, 'isa-foo would be a function that indicates support for 'unwrap-foo, and 'wrap-foo would be a simple data constructor that makes an encapsulated value with nothing but innate support for 'unwrap-foo. (In a system with failcall, 'isa-foo is pretty much redundant.)

This is basically how the Kernel draft deals with custom types (except without an eye toward extensibility), but I haven't been inspired to go quite as far as this in practice. I think programmers should be free, if discouraged, to break encapsulation of values they don't know the nature of, like Arc programmers can when they use 'type and 'rep.

"And in fact, you can do that right now, in Arc[...]"

Cool, huh? :-p

Obviously pg-Arc hasn't been built from the ground up this way, but Arc Forumgoers have had an awful lot of extensibility ideas since Arc 3.1 was released, and pursuing any one of them in Anarki would change the core so much as to turn it into a separate language. That's why I don't call Penknife a version of Arc.

---

"And you're also missing the whole point of message passing... message passing enables duck typing, which means it's possible to write a function that doesn't care what it's type is, as long as it supports the required message."

That's the whole point of extending global functions too. I don't care what something's type is, as long as all the utilities I use have been extended to deal with it.

As I mentioned above with 'deftype, the 'extend approach can go all the way down to the level of making the 'type function itself merely a reflection tool. Obviously pg-Arc isn't built this way, but we've had an awful lot of extensibility ideas here since Arc 3.1 was released, and pursuing any one of them in Anarki would practically turn it into a separate language. (That's why I don't call Penknife a version of Arc.)

---

"I suggest you go and read http://arclanguage.org/item?id=14237 which already explains this in more depth."

I've already read that, thanks. I just wasn't convinced, and I found more interesting arguments to make over here. ^^

"Edit: here's the post http://arclanguage.org/item?id=14261 "

How? I've provided concrete examples where my solution is significantly shorter at creating new data types than the current solution, while still retaining `isa`, while avoiding problems. If you have a solution that can do all that, then please present it.

---

"In this example, 'isa-foo would be a function that indicates support for 'unwrap-foo, and 'wrap-foo would be a simple data constructor that makes an encapsulated value with nothing but innate support for 'unwrap-foo. (In a system with failcall, 'isa-foo is pretty much redundant.)"

Why? My system already does things similar to that, but in a very simple way, that's compatible with pgArc.

---

"However, in my mind 'isa has a clear role as a way to check the most specific possible feature of a value: the kind of internal data it has."

This is precisely what I meant when I said that I think the current view of types is fundamentally wrong. A table is not a hash. It's an interface that could potentially be implemented as a hash... or something else. Arc leaves it unspecified, which I consider to be a Good Thing.

---

"That's the whole point of extending global functions too. I don't care what something's type is, as long as all the utilities I use have been extended to deal with it."

Note that they're using the message-passing idea: after an iterator has been created, you can call it with 'status?, 'dead?, 'alive?, or 'kill! to get/set it's private data.

What I'm suggesting is to treat all compound data types like that. You can then create function wrappers around them, which is in fact what they did too: note the definition of `iterator-empty?`

Haha... that's what I've been explaining all along! That's the whole point of message passing: defining a low-level framework that makes it easy to both create new types, and also extend existing types.

I still don't see what this has to do with methods, though. (I still can't follow that humongous comment[1]). Both C++ and Java have methods. No language but Go tests for an interface by seeing the functions it supports.

And this isn't the same as duck typing at all. Arc does duck typing just fine. In arc or python there's no way to convey tableness or any other ness. You just use the operations you care about, and hopefully any new types remembered to implement them.

Duck typing has no contract, Java has too rigid and inextensible a contract. Go is just right. And you're hosed in any of them if you write code of the form if (isa x table) ..

[1] I think I've missed half your comments in recent weeks. Either because they're so long and hard to follow, or because there's just too many of them and I forgot to go to the second page of new comments. The latter is a weakness of this forum structure. For the former, well, I think you spend too much time describing methods. Most of us have seen them in other languages. What you suggest is subtly different, and the subtleties are lost in the verbiage.

Which is why runnable examples are useful. You're right that wart differs from arc in subtle ways. That's why I have unit tests, to enumerate all the subtleties. I think you're calling it out (fairly) for lots of gritty details, and then doing the same thing even more egregiously.

So... Go's interfaces actually look an awful lot like message passing. Here's a comparison:

  type reader interface {  // Go
      Read(b []byte) (ret int, err os.Error)
      String() string
  }


  (annotate 'reader        ; py-arc
    (fn (m b)
      (case m
        'read   ...
        'string ...)))


  (object reader           ; py-arc, with sugar
    read   (b) ...
    string ()  ...)

  (attr foo 'read)                   ; duck typing
  
  (if (isa foo 'reader)              ; interface checking
    (attr foo 'read))
     
  (attr (coerce foo 'reader) 'read)  ; coerce to reader

Think of it like continuations and iterators. Continuations are lower-level than iterators, and it's possible to create iterators by using continuations. Message passing is the low-level idea, which can be used to create higher-level things, like prototypes, objects, or interfaces.

---

"Duck typing has no contract, Java has too rigid and inextensible a contract. Go is just right. And you're hosed in any of them if you write code of the form if (isa x table) .."

But what if `isa` tested for an interface? In other words, (isa x 'table) would be like saying, "does x support the table interface?"

---

"I think I've missed half your comments in recent weeks. [...] For the former, well, I think you spend too much time describing methods."

Yeah, I've probably been a bit too excitable about this... ^^;

---

"I think you're calling it out (fairly) for lots of gritty details, and then doing the same thing even more egregiously."

I'm not saying it's a bad thing that wart differs from pgArc, I was just explaining why I haven't immediately jumped in and started using wart. I think it's good to try out different approaches. Nor am I expecting people to immediately jump in and start using message passing... especially since message passing probably isn't useful for everybody.

I was presenting my ideas in the hope of getting useful feedback... so that my ideas can be improved, or rejected. I know that you guys are smart, and by presenting my ideas here, you all have already helped me improve them. I was just looking for some feedback (which I'm finally getting, which is good).

Agreed. I'm giving you feedback on what would help me understand the idea. Sometimes if an idea has too many subtle nooks and crannies it helps me to see running code. That doesn't compel you to provide it, of course.

"I was just looking for some feedback."

You're not just looking for feedback, you're aggressively/excitably demanding it :) But you'll be more likely to get it from me (as opposed to super boring meta-feedback, and agonized grunts about your writing quality) if you provide runnable code. Pseudocode is no different than english prose.

"Nor am I expecting people to immediately jump in and start using message passing.."

I'm not sure what you're expecting when you say things like "what more do I have to do?". What more do you have to do for what?

(Btw, the answer to the question 'what more do I have to do' is.. give me runnable code :) I usually start out talking about an idea; if it is neither shot down nor embraced I go build it. If it's still neither shot down nor embraced I try to build stuff atop it. Keep doing that until you run into limitations or (slim chance) you build something so impressive people have to pay attention. Platforms need killer apps.)

---

"Go's interfaces actually look an awful lot like message passing."

Argh, here we go again..

  (annotate 'reader        ; py-arc
    (fn (m b)
      (case m
        'read   ...
        'string ...)))

"what if `isa` tested for an interface?"

This seems to be the absolute crux of your idea. This needs changes to ac.scm. But it has absolutely nothing to do with message passing. Messages are not types, and your suggestion is to augment the type system with a keyword called interface or object that isa can accept. If you made that change, arc would support 'message passing' according to you. Am I right?

I'll summarize my position again. I buy that interfaces are foundational, but lots of things are foundational. Unification like in prolog, pattern matching like in haskell, lazy evaluation like in haskell, messages like in smalltalk, function calls like in lambda calculus. That something is foundational doesn't necessarily make it a good fit for a specific language. To show that something foundational is a good fit in arc and also backwards-compatible, provide runnable examples that look good/short and behave intuitively. Show us that existing mechanisms in arc can coexist with the new feature, that it occupies a niche that they don't already provide. No hand-wavy ellipses.

---

"I'm not saying it's a bad thing that wart differs from pgArc, I was just explaining why I haven't immediately jumped in and started using wart."

Having built-in types like table/cons/etc. written in the same way, rather than as opaque blobs in Racket.

---

"I'm not sure what you're expecting when you say things like "what more do I have to do?". What more do you have to do for what?"

"What more do I have to do to demonstrate that message passing has advantages over extend?"

I asked that because it seemed that people weren't convinced that message passing was actually better than extend. This was unfortunate because:

1) It appeared to ignore the evidence that I was presenting

2) It seemed to basically dismiss message passing, but without explaining why. How am I supposed to know how message passing is flawed, unless people explain?

If the dismissal was caused by not understanding my idea, then it's my fault for not explaining well enough.

---

"[...] This needs changes to ac.scm."

Yup!

---

"But it has absolutely nothing to do with message passing. Messages are not types, and your suggestion is to augment the type system with a keyword called interface or object that isa can accept."

You're right: message passing is the low-level idea. Interfaces are built on top of message passing, but aren't necessary to support message passing.

---

"If you made that change, arc would support 'message passing' according to you. Am I right?"

No, in order for Arc to support message passing (at least in my mind), it would be necessary for the built-in types to also use message passing. As you pointed out, user-created Arc code can already use message passing, but that's not useful if you want to create something that behaves like a built-in type.

---

"[...] provide runnable examples that look good/short and behave intuitively."

"in order for Arc to support message passing, it would be necessary for the built-in types to also use message passing"

Are you planning to replace (f x) everywhere with (x f)? That hardly seems backwards-compatible. (Forgive me if this is a stupid question. I have zero 'expertise' since I haven't read 75% of what you have written in this thread.)

If you provide a way to handle (f x) as an 'f message to x, then you shouldn't need to implement primitives.

"It seemed to basically dismiss message passing, but without explaining why. How am I supposed to know how message passing is flawed, unless people explain?"

You have to try it :) It's pretty clear that nobody here has tried quite what you're proposing, so what you thought of as dismissal was just people thinking (like me for the past few weeks) that they had nothing to add since they haven't tried it, and wanting to reserve judgement until they had an opportunity to play with an implementation. But maybe that's just me :)

Nooope. Let's assume my-table is a table. On the left side is the current pgArc interpretation. On the right side is message passing:

  (my-table 'something)       -> (my-table 'get 'something)
  (= (my-table 'something) 5) -> (my-table 'set 'something 5)
  (keys my-table)             -> (my-table 'keys)

---

"Or are you seeing dismissal in statements rather than silence?"

That's a bit of a tautology. ^_^ If I'm not mistaken, you took the term "message-passing" from its object-oriented "foo.bar( baz )" context to begin with.

---

"Yeah, I've probably been a bit too excitable about this... ^^;"

I see, yeah you're right. The alternative to duck typing would be subtypes. Ugh.

No need to extend Arc's built-ins to convince it that, "yes I really really really want this to be a table."

You're saying more than that, you're saying how. That logic is common to both approaches. Your approach is superior anytime user code has language like if (isa x 'table) ... Otherwise the two are equivalent in power.

  (def my-table ()
    ...)
  
  (def my-table (m) :case (is m 'keys)
    ...)
  
  (def my-table (m k) :case (is m 'get)
    ...)
    
  (def my-table (m k v) :case (is m 'set)
    ...)

  (= my-table (fn (m k v)
                (case m
                  'keys ...
                  'get ...
                  'set ...)))

My solution is general... it works on all compound data types: tables, input, lists, etc.

  ; Arc

  (case x
    "foo" ...
    "bar" ...)


  // JavaScript

  switch (x) {
  case "foo":
      ...
      break;
  case "bar":
      ...
      break;
  }

  if (x === "foo") {
      ...
  } else if (x === "bar") {
      ...
  }

Arc and Javascript case functions are not expected to do the same thing. JS does not allow expressions for input arguments while Arc can. And that break; statement is a feature that some would say Arc lacks. ie, what if you want your case statement to fall through? - now Javascript is golden and one could say Arc is lacking.

I also don't believe developers, generally speaking, prefer if/else chains over switch/case statements as they are different tools intended for different purposes.

They both are not perfect, both are missing features that the other could benefit from.

  switch (true) {
  case 5 > 1:
      console.log("5 > 1");
      break;
  }

On the contrary, switch should not have a break statement! It should have a fallthru; or continue; statement. Why make the common case difficult? This should be uncontroversial... it's been well established that switch's design is poor, and they should have made it explicit fallthru (like with a continue statement) rather than implicit.

As for Arc "lacking" fallthru... can't you implement that with continuations? That would be the equivalent of explicit fallthru.

---

I think that's because the above semantics correspond more directly to assembly language, which I imagine was done because "switch" was defined back when that was either important or just not seen to be bad. (Perhaps I'm just making that up, though.) Here's how a switch statement would look in x64 assembly:

  switch:
        ;put desired thing in rax, let's say
  case_1:
        cmp rax, val_1
        jne case_2      ;jump if not equal
        <case 1 code>
  case_2:
        cmp rax, val_2
        jne case_3
        <case 2 code>
  case_3:
        cmp rax, val_2
        jne done
        <case 3 code>
  done:
        <whatever>

  switch:
        ;put desired thing in rax, let's say
  case_1:
        cmp rax, val_1
        jne case_2      ;jump if not equal
        <case 1 code>
        jmp done        ;jump.
  case_2:
        cmp rax, val_2
        jne case_3
        <case 2 code>
        jmp done
  case_3:
        cmp rax, val_2
        jne done
        <case 3 code>
  done:
        <whatever>

> JavaScript allows arbitrary expressions in switch statements

That's right... looks like it was the test data I was thinking of, which isn't entirely relevant, given you can accomplish the same thing in the end.

> Why make the common case difficult?

  switch (true) {
  case true:
      console.log(1);
  case true:
      console.log(2);
  case true:
      console.log(3);
      break;
  case true:
      console.log(4);
  }

  switch (true) {
  case true:
      console.log(1);
      continue;
  case true:
      console.log(2);
      continue;
  case true:
      console.log(3);
  case true:
      console.log(4);
  }

In fact, in my years and years of programming in JavaScript, I have used if/else blocks many many times (some of which could have been switch statements), and only wanted fallthru a handful of times. I think allowing fallthru is fine, but it should be made explicit, rather than implicit.

The only possible argument I can think of in favor of implicit fallthru is using a switch inside a `for` loop:

  for (var i = 0; i < array.length; i += 1) {
      switch (array[i]) {
      case "foo":
          continue;
      }
  }

---

Vaporware announcement: I've been trying today to make streams and generators duals. So (read stream) and (yield value) would be ways to bounce control between an input stream and its user.

I started playing with this after rocketnia's observation about accum (http://arclanguage.org/item?id=14198), which name had been bothering me ever since I started using waterhouse's accumulate on a regular basis. "Ooh, so I can replace (accum acc ... (acc x)) with just (as list ... (yield x))? Sign me up."

If someone replaces rb-core-call with something other than a core function or rulebook, they'll get an infinite loop, and it'll be their own fault.

In my Penknife draft, every function can either return a value, raise an exception, or fail with a failure message. Failure means the function doesn't meaningfully apply in this case, and it lets the caller try something else if they they happened to use failcall to provide an alternative. If they didn't use failcall, it just raises an exception.

A Penknife rulebook can be called as a function. It's represented by a sequence of functions that are called one at a time until one returns a value or raises an error. If they all fail instead, then the whole rulebook fails.

A rulebook is like a series of 'extend cases, but being its own type, it's less encapsulated. There could theoretically be rulebook utilities that let you count the number of extensions, etc. While an implementation of 'extend could do this too, it's a little unheard of.