One of the changes is that mzscheme's FFI import a few names starting with an underscore, thus clashing with Arc's namespace. I modified ac.scm so that Arc's names start with a double underscore.
The other problem is that the mzscheme type #<cpointer> is not recognized by Arc : I had to change ac-type so as to add it.
I love Pyrex too. However, that means making (and maintaining) another compiler that should be compatible with Arc and still allow the extension features. That's a problem with Pypy, actually. You can't use any Python code in it, and it is quite restrictive in some cases.
However, you're quite right as I took the names cdef, cint, etc. from Pyrex.
Well, the client-server stuff can easily be done through sockets and a dedicated protocol on the native-process side. There are many such protocols, but this is not what I am after. I need ease of use (and, for other cases, speed).
Very standard, I know, but (as for now) I just want to use POSIX functions for a system utility. Picking in a .so file is just what I need. I don't want to start a server process, secure it (we're talking about functions manipulating my system and available to others through the net) and then communicate with it.
As for JVM and .Net, I totally agree with you. When I'll have something working, doing (for example) a JNI wrapper should be easy.
Personally, I use '() when I mean "the empty list" and nil when I mean either "the false boolean" or "no value yet". It is just a matter of readability, since they're both as long to type ;)
Simply () works too, and is more than 33% shorter ;)
The following shows four ways to express the empty list, and proves that they mean the same in Arc. What's a little peculiar, however, is that one gets printed differently;
I totally agree with you. I do a lot of system scripts (log analysis, job management on a cluster, etc.) and I usually use Perl or Python for these. I'd love switching to Arc for that, but as for now speed is really an issue. Especially startup time : a script is something you possibly run very frequently (sometimes many thousands instances at a time), and even two small seconds on startup is a killer. That's why I spent a few time trying to fix this in arc-exe, btw.
I will soon have to try classification algorithms (you know, bayesian nets, neural nets, these kinds of things). It will be prototypes, so full speed is not required yet. Well even there, something a little faster would be good.
Well, pg promised a profiler and proposed (in a recent poll) to work on an FFI a day or another. I guess that,with these two, we will have what we need in the future...
The FFI is certainly important and I wouldn't use Arc if it didn't have a well-defined standard FFI in its eventual form (to my mind that was one of the biggest mistakes of CL).
I don't think that a FFI is a good substitute for a fast Arc implementation though. It's true that I can re-code any time-critical parts of my application in C, but why should I have to? What you're saying amounts to an unpalatable compromise, and to my mind it's not a compromise that a hundred year language can afford to make.
It will be good to have a profiler for Arc.
I don't agree with your argument that a profiler and FFI will gives us what we need for the future. Python has both a profiler and an FFI. I've used both, in combination, for numerical simulation. It's a highly inelegant and ultimately unsatisfying solution. The python part of the app was never fast enough, and whilst profiling helped, it also lead to some very ugly code. The C-coded parts were... well... C code! I think I would be preaching to the converted if I started in on C's deficiencies in this forum :-)
CL for all its faults shows us that a lisp can be fast. I can't see any reason why Arc implementations won't be fast eventually too. If someone more knowledgeable can see one I'd be very interested to learn about it, but this thread doesn't seem to have come up with anything concrete.
[A word of explanation: the Python/C combination wasn't my decision, and in hindsight I wouldn't recommend it to anyone starting a new project involving numerical simulation.]
You make a good point there. However, look at the very-efficient CL code : it looks like C code. Written in CL, sure, but it often deals with calculations on vectors of fixnums where the type of everything was predefined and type checking reduced to nothing. No dynamic dispatch of generic functions on heterogenous lists here.
But an FFI does not necessarily mean you have to write C code everytime you need an efficient calculation. Have a look at the following code I've just tested :
(def fib (n)
(if (<= n 2)
n
(+ (fib (- n 1)) (fib (- n 2)))))
(time:fib 30)
-> time: 47498 msec.
<< Hidden code here >>
(def ffib (n)
(if (<= n 2)
n
(+ (ffib (- n 1)) (ffib (- n 2)))))
(time:ffib 30)
-> time: 4635 msec.
More than 10 times faster, and they are both pure Arc code. What's the magic part ? Well, it just imports new declarations of +, - and <= I previously wrote in C (with the FFI I worked on recently). These definitions only deal with fixnums, that's why they are fast. Here is the full code :
(w/inline "
char inf (long a, long b){
return a <= b;
}
long minus (long a, long b){
return a - b;
}
long plus (long a, long b){
return a + b;
}"
(cdef _<= "inf" cbyte (clong clong))
(cdef - "minus" clong (clong clong))
(cdef + "plus" clong (clong clong))
(def <= (a b) (is 1 (_<= a b))))
Well, that declaration could be encapsulated into something like :
(declare-numeric
(def ffib ...
And you would end up with something as fast as CL (and looking like optimized CL code). Well, not really, but this is still alpha version.
Actually, if you are using Anarki, a lot of the slowness comes from infix.arc redefining math operators. Just removing infix.arc from libs.arc will increase the speed of math operations by about an order of magnitude. In fact, removing infix.arc actually produced more of a speedup for me than using your ffi.
With infix.arc:
arc> (def fib (n) (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2)))))
#<procedure: fib>
arc> (time (fib 30))
time: 34672 msec.
1346269
Without infix.arc:
arc> (def fib (n) (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2)))))
#<procedure: fib>
arc> (time (fib 30))
time: 4219 msec.
1346269
Yeah, sorry about that. I wasn't thinking about performance when I originally put infix.arc up on Anarki, and since I was doing a lot of testing at the time I found it convenient to add it to libs.arc. Perhaps it would be best to leave it out by default though.
Oh that's right... On my machine, the FFI version is still a little faster (about 20% faster), but not that much. It now deals more efficiently with boolean values however, that might explain it... Anyway, I don't know if it's worth using FFI this way now... Not until we can compile Arc code to efficient C code at least :)
Thanks for the counter example, you've made a very elegant point. It's quite encouraging in fact. It seems to point to the strength of Arc. I think this would require a lot more effort to achieve in Python.
Of course in this case one would hope that the fundamental arithmetic and relational operators would already be implemented efficiently in an Arc native binary compiler, but the principle you've illustrated still holds regardless.
As for speed, Stalin Scheme is amazing for example. Not very complete and well documented for example. However, CL is full of "efficiency hacks". Scheme is full of purity, not always easy to implement efficiently. Optimizing CL is easier than Scheme, IMHO.
Why do you think CL is easier to optimize that Scheme? This isn't a hostile question; I'm just curious. Intuitively, I feel like "purer" languages should be easier to optimize.
First, CL has in the standard many possibilities for making declarations reagrding optimization : for the functions you want, you can compile code, declare types (e. g. this var only holds fixnums), declare you want to optimize the speed and ignore type safety, etc. This way, you end up writing code the way you would write it in C. There is no such thing in the Scheme standard. Individual implementations could, of course, but as far as I know no one does.
Abother example is call/cc. This is a very interesting beast, only existing in Scheme. But it is hard to implement efficiently.
The last example I can think of is 'nil. Using nil as false and the empty list is very interesting in this regard : you can implement nil as the NULL pointer, which is also 0, the false boolean. Less manipulations to do on the bare metal. Distinguishing between #f and '(), on the contrary, implies making more tests at the lower levels.
re: call/cc - I think a bit of the lambda the ultimate series of papers eventually boils down to the realization that a machine language jump-to-subroutine is equivalent to a call/cc, and the target of the call/cc just has to access the return address on the stack as a function address.
I don't get it. There's the whole stack copying for call/cc, so call/cc is much more expensive.
(I read the "Lambda the ultimate GOTO" paper you referenced earlier; it's about goto vs structured programming, not call/cc. As an aside, it's very interesting to reflect on just how controversial structured programming was.)
Implementing call/cc efficiently has been well-reasearched in the Scheme community. For a very well-written account of a non-stack-copying implementation see
Then continue at ReadScheme at "Compiler Technology/Implementation Techniques and Optimization"
to see further developments (Look especially for Clinger's papers).
In a function call, the state (the current computation being done) is saved anyway, and therefore "copied" if that is your preferred term. So ideally, call/cc should have the same overhead as an ordinary function call; the only difference is that in call/cc the continuation state is the value given to the function, while in a function call it's just one of the values given to the function.
Note however that much of the theoretical analyses of call/cc make a basic assumption of a "spaghetti stack", which would mean that partially unwound stacks would be saved implicitly as long as any continuation exists which refers to that stack, and all stacks themselves are subject to garbage collection. Most machines don't actually have a spaghetti stack and can't make a spaghetti stack anyway ^^. That said a spaghetti stack could be implemented as a simple list, with push == cons and pop = cdr.
Alternatively store the local variables on a garbage-collected heap, and include a reference to the local variables with the continuation/return address (you'll probably need to save the pointer-to-local-variables anyway, since the target function is likely to use that pointer for its own locals). Again, no additional overhead over plain function calls, except perhaps to restructure the return address and the pointer-to-local-variables.
Don't know about MIT Scheme, but if I were to implement call/cc on stock hardware and compiling down to machine language that's what I'd do ^^
I'm still totally not understanding your claim that call/cc should have the same overhead as an ordinary function call.
I read the Clinger "Implementation Strategies for Continuations" paper and they found call/cc about 10 times slower than function calls on the tak/ctak tests. I tried those tests on PLT Scheme and the overhead I saw is even worse: .7 seconds with function calls vs 51.8 seconds with continuations on (tak 24 16 8).
Clinger says about the stack strategy: "When a continuation is captured, however, a copy of the entire stack is made and stored in the heap. ... Variations on the stack strategy are used by most implementations of Scheme and Smalltalk-80."
Components of function call: (1) put arguments somewhere (2) put return address somewhere (3) jump to function.
Components of call/cc: (1) put return address as argument somewhere (2) put return address somewhere (3) jump to function.
That said, continuations generally assume that "put X somewhere" means somewhere != stack. Yes, even return addresses are not intended to be on the stack when using continuations. The main problem is when compiling down to C, which always assumes that somewhere == stack. If you're compiling down to machine language where you have access to everything, then you can just ignore the stack, but not in C, which inherently expects the stack.
Ah, I get it. I suppose typing is one of the big issues affecting speed. If the language standard insists on dynamic typing, there might be no way to get certain kinds of optimizations.
And yeah, call/cc is probably always going to be a bear. But man is it cool. :)
I suppose this goes against what a few of us (including me) were saying above -- that the language and the speed are really separate issues. Or maybe it's more coherent to say that language standards (as opposed to "languages" generally understood) can have a profound effect on speed. If they don't give implementors a lot of choice, they can box people into certain corners.
It's interesting that Scheme actually mandates optimization in at least one case (tail recursion). I don't know how many language standards make those kinds of demands, but I suspect there aren't many.
Tail recursion is interesting as it is not especially an optimization for speed but as a way to make programmers rely primarily on functional programming : if you don't have it, functional programming is rapidly a dead-end as you can make the stack explode really fast. As a bonus, it is faster :)
it looks like a bug in news.arc. In a first time, you can just comment (or remove) the string "news.arc" in the file libs.arc. You won't be able to use the news application however, but to run the tutorial you don't need it.
Exactly. Sorry, I never realized that adding news to the libs
would break the tutorial. In future, if anyone notices a bug this major, please email me about it or at least post something here.
ccc is one of the strangest things in Scheme (and Arc now). Hard to get everything behind it, but it can be used to implement return statements, try-catch à la C++/Java, coroutines à la Lua, generators à la Python and of course continuation-based web apps à la Arc...
A very strange beast, and as far as I know the thing Scheme has that CL hasn't (and cannot trivially implement).
Actually, the form of closure used by Arc is highly similar to a continuation passing style, so although it doesn't use 'ccc, it does use continuations.
cl-cont is a library that implements closures and runs on many CL implementations. So it's not a matter of CL can't do it, CL just has to use a library to do it.
I think there are conceptual problems mixing unwind-protect-like constructs and call/cc.
The cl-cont lib appears to only support a subset of CL (in particular unwind-protect is not supported).
So cl-cont doesn't demonstrate that CL can support it, only that a restricted subset of CL can.
I don't know if it is possible to sensibly accommodate unwind-protect and call/cc in the same language, it seems smarter people than I have avoided doing so.
Scheme's dynamic-wind serves the same purpose as unwind-protect (except with entry conditions as well, since a continuation could suddenly return into your protected code), and plays nicely with continuations. I'm not sure about the details, though.
Thanks for that, pretty interesting. I guess that makes writing entry and exit conditions a little more interesting, since they could end up being invoked multiple times.
