The "add hacks not options" idea is intriguing, and I adore the simplicity it both requires and begets. I'm not sure how to control damage from conflicting patches though. And I'd much rather share hacks via libraries - it's so easy to just redefine any core arc function, and your "extend" function makes that even easier and safer. I suppose there are some kinds of hacks that are harder to share this way though - especially if you're hacking ac.scm.
Are all those functions defined in arc-tokeniser just to keep them out of the top level namespace?
Yes. I've noticed that most arc code is not like this, so maybe arc-tokeniser is really bad style. What's the correct way to deal with namespace clashes? I'd like to avoid having a whole bunch of functions called arc-tokeniser-<something>.
Although upon reflection, it's true that popping these kinds of functions up into the toplevel makes them more readily hackable (by simple substitution, or 'extend-ing). I wonder if there's a way to do that without spawning a crowd of verbosely-named little functions?
I don't have answers to any of these questions - my brain is still wired mostly in java, and in the process of re-wiring I can't write well in any language ...
I'm not sure how to control damage from conflicting patches though.
We publish a commit which is a merge of the two patches. You can see an example in my arc2.testify-table0+testify-iso0 commit, which is a merge of my arc2.testify-table0 and my arc2.testify-iso0 patches. Here's the original testify from arc2:
(def testify (x)
(if (isa x 'fn) x [is _ x]))
My arc2.testify-table0 patch makes testify treat tables like it does functions:
(def testify (x)
(if (in (type x) 'fn 'table) x [is _ x]))
My arc2.testify-iso0 patch has testify use "iso" instead of "is":
(def testify (x)
(if (isa x 'fn) x [iso _ x]))
And the merge of the two:
(def testify (x)
- (if (isa x 'fn) x [iso _ x]))
- (if (in (type x) 'fn 'table) x [is _ x]))
++ (if (in (type x) 'fn 'table) x [iso _ x]))
The first "-" line shows the arc2.testify-table0 patch, the second "-" line shows the arc2.testify-iso0 patch, and the "++" shows how I merged the two patches. (You can get this output by using the -c option to git-log: "git log -p -c")
and your "extend" function makes that even easier and safer
Right, I think that functions like "extend" would arise from seeing patterns in code and abstracting them out, making the code more succinct. It was more my thought process I was commenting on, I had thought "patches are hard to deal with so I'll make functions like extend". Now I'm thinking, "what if patches were easy?".
What's the correct way to deal with namespace clashes?
I've been wondering about that. One possibility I've wondered about is to have an abbreviation macro:
So the actual name of the function would be arc-tokeniser-make-token, but anyone can refer to it by make-token by using the abbreviation macro if that would be convenient for them.
This is even more speculative, but I've also wondered if maybe it needn't be your job as a library author to worry about namespace clashes. What if the user of the library said "oh, look, make-token is clashing", and could easily load it with particular symbols renamed...
I think the first step is not to worry about namespace clashes. Instead think, "oh, namespace clashes are easy to deal with, so if they happen no problem". Otherwise you end up doing work (and maybe making the code more complicated or harder to extend) to avoid a namespace clash that may never happen.
Where w/locals expands into a big (withs ...) form with key-value pairs taken from the list.
Use an alternative def to put function definitions in such a table instead of the global namespace:
(def-in tokeniser-helpers make-token (kind tok start length) ...)
This way, arc-tokeniser would be less horribly big, easier to hack, and non-namespace-polluting. As a kind of plugin system, it doesn't seem terribly obtrusive does it? The disadvantage is that you need to search further for the definitions of your helper functions.
Is this something like how your prefix-abbrev macro would work?
I think it's not just a question of worrying about clashes that may never happen - it also feels inelegant, dirty even, to have globally-accessible functions that are relevant only in a very specific context. Otherwise I would completely agree - it would be a kind of premature optimisation to worry about them.
it also feels inelegant, dirty even, to have globally-accessible functions that are relevant only in a very specific context
Yes, but how do you know that your Arc parser functions are only going to be relevant in the code you've written? Perhaps someday I'll be writing my own parser, or something completely different, and I'll find it useful to use one of your functions in a way that you didn't think of!
I suggest trying out writing your code is in the simplest possible way. For example, in your original:
(def arc-tokeniser (char-stream)
(withs (make-token (fn (kind tok start length)
(list kind tok start (+ start length)))
"make-token" does not use "char-stream", so we can make this simpler:
(def make-token (kind tok start length)
(list kind tok start (+ start length))
Now I can look at "make-token" in isolation. I can easily understand it. I know that all that other stuff in arc-tokeniser isn't affecting it in some way. And, if I'm writing my own parser and I want to use "make-token", I can do so easily.
And sure, down the road there may be some other library that also defines "make-token". At that point, it will be easy to make a change so that they work together. Perhaps by renaming one or the other, or by doing something more complicated. The advantage of waiting is that then we'll know which functions actually conflict, instead of going to a lot of work now to avoid any possibility of future conflict, the majority of which may never happen.
Now of course I'm not saying to pull every single function out of arc-tokenizer. You've some functions that depend on char-stream and token and states and so on. So those it makes perfect sense to leave inside arc-tokenizer. My claim is to today write the simplest possible parser.arc library, explicitly not worrying about future namespace clashes. That it is better to deal with them in the future, when they actually happen.
Version n of anything usually consists of incremental improvements to version n-1. Including bug-fixes as well as new features. To deliver a new version written from scratch would be to deliver a new set of bugs written from scratch.
timing tests ... ok, endmatch totally wins. Especially for a mismatch in which case endmatch returns early, but match-end shows no gain. These tests were run on anarki arc:
average time im ms over 100000 runs of [_ ".jpg" "hello.jpg"]
endmatch 0.029
match-end 0.064
average time im ms over 100000 runs of [_ ".jpg" "hello.png"]
endmatch 0.016
match-end 0.062
So it seems performance is a likely explanation for the way endmatch is implemented.
(interestingly, for comparison, on rainbow the time for endmatch is roughly similar (0.028,0.019) but for match-end is disastrous (0.11,0.11). I'll need to figure out why.)
You're right about my match-end not working on arc2 - I mostly use anarki as it has a bunch of convenient changes including negative offsets for cut.
Though keep in mind that effort making a particular function run faster is only useful if a significant percentage of a program's execution time is spent in that function. For example, even if your match-end runs four times slower than endmatch in rainbow, if a program spends only 0.01% of its time matching the end of strings, then making match-end run faster won't help the program run any faster.
Certainly as an optimization arc2's implementation of endmatch would appear to be premature one, considering that endmatch isn't even used in the arc2 and HN source code, much less identified (as far as I know :) as a hot point slowing down the execution of someone's program. And while I do personally find it to be a fun piece of code (demonstrating how loop unrolling can be done using macros), your implementation is a clear winner in clarity and succinctness. :-)
If you are using anarki and still getting this error, then it's a mystery. It might be necessary to expand "~/tmp/arc" into a full path - I know rainbow doesn't expand "~", and I don't know if the shell does. And "~/tmp/arc" should contain arc.arc and the usual arc files from the distribution.
There is a continuation library for java at http://rifers.org/wiki/display/RIFE/Web+continuations - but it's wrapped in a web server. Apparently it does all the stack copying for you. I looked at it briefly and ran away fast. But it might be the right approach if you're compiling to bytecode, and bytecode compilation is probably the only way to make arc go fast on java.
Rainbow's continuation-passing style means arc code is not using the native java stack, which probably slows it down immensely.
I doubt a stack copy is very feasible. For one thing, it would need a potentially different implementation from JVM to JVM (and from context to context, since a JVM might optimize something in several different stack-involving ways depending on which is best at the time). Also, based on what I see at http://www.cs.princeton.edu/sip/pub/oakland98.pdf and http://www.ccs.neu.edu/scheme/pubs/cf-toplas04.pdf, it looks like swapping out the stack would blatantly undermine the security model of JVMs that rely on stack inspection, meaning that lots of JVMs are probably actively motivated to keep native code from getting at the wrong part of the stack. Then again, I haven't actually tried it, so maybe I'm just being pessimistic.
In any case, stack copy isn't RIFE's approach. RIFE's approach is to use a custom ClassLoader on selected classes (classes implementing a particular interface) to transform the Java bytecode into explicit continuation-passing style at runtime. Ultimately, it does nothing the Java programmer couldn't have already done by writing those particular classes in continuation-passing style to begin with. If I'm already committed to having a continuation-passing-style Arc interpreter in my Java project, I can just do my to-be-CPS-transformed code directly in Arc and be done with it.
On the opposite side of the spectrum, JauVM (http://jauvm.blogspot.com/2005/07/so-what-does-it-do.html) has an approach I think is about as comprehensive as it gets. The idea is that it's a wrapper of an existing JVM, delegating just about everything except exactly what it needs to do in order to provide first-class continuations. Unfortunately, I think JauVM does this by directly interpreting every instruction, so it's probably horribly slow.
I think the best approach for Arc-in-Java is ultimately a compromise between RIFE and JauVM: A ClassLoader that's used for everything and CPS-transforms it unless it's a specially-marked class whose programmer is prepared to manually pass continuations.
Congratulations! Is this going to become a full arc interpreter? The stated goals include compatibility with arc, but jarc is missing tail-call optimisation and first-class continuations. Implementing these in java turned my hair gray overnight and required a complete rethink of rainbow.
Competition for rainbow ... re-open project ... I will just have to stop sleeping or something ...
Competition can be good, but I'm open to cooperation. You can send email to my jazzdev gmail account. I've avoided looking at rainbow until I got Jarc released. Now I'm curious to see what you've done.
Rainbow is 2x faster than Jarc, and first class continuations are impressive!
I used ccc in an arc parser I wrote (it's at lib/parser.arc in anarki) ... I confess however that the choice of using ccc was driven more by the desire to experience this exotic construct than by the problem itself (as well as needing something to test rainbow's ccc with). The whole thing could be rewritten more simply.
That's a good example because in this case ccc in only used within the library. Thus an implementation option is to support ccc within a library while giving up something else while running inside the library (speed, or interoperability with Java classes, etc.) that makes to feasible or easy to support ccc inside the library. (After all worst case is that you write an Arc interpreter in which it is easy to support full ccc but it runs 20 times slower). Outside you don't support full continuations, but you have your regular Java Arc implementation (which runs really fast etc.)
ccc is also used by the 'point macro in arc.arc - but it's only used as an escape continuation, which is only jumping back up the stack, not copying it, so can be implemented very easily in java without introducing performance issues.
'point is also the only place I could find ccc used in the original arc distribution.
