In less than a month marcvs has setup an AWS server, installed racket and arc, tweaked the arc codebase and the HN app, configured email, and translated all the copy in the HN codebase to spanish. All this without having much programming experience at all. Kudos!
Hmm, do you care about threading and consistency at all? If not, you could probably do everything with just arc macros over the existing flat file approach..
I think that some form of scalability would be valuable, but that could easily be achieved with some sort of single threaded worker for each db 'server', and then have multiple instances running to provide the scalability. In order to make the single threaded semantics work well even in a multi-threaded application, I already have a short library for erlang-style pattern matched message passing.
Given the data volumes I've been planning on working with, I mostly want to use the permanent storage for history and fault tolerance, as opposed to live access. That could probably be handled in-memory for the most part. So maybe some form of flat file system would work without causing too many problems.
I originally started using git to effectively achieve that design without having to manage the trees, history, and diff calculation myself, but I've discovered that storing thousands of tiny objects in the git index may not be very efficient. I still think something similar is a good idea, but I would want to separate 'local' version states for each object from the 'global' version, so that it doesn't take forever to save the state of a single object. Maybe storing each object in a git 'branch' with the guid of the object as the branch name would work, since only one object would be in each index. The overhead for saving each object would be slightly higher, but it should be constant, rather than linear with the total number of objects.
Any obvious flaws with that idea that I'm missing? Have any better ideas or foundations to build off of?
That code is pretty rudimentary, but allows low level access to git commands from arc, plus storage and retrieval of arc objects. After my previous comment though, I'll probably change it so that each object gets a separate branch, with 'meta branches' listing which branches to load if necessary.
Pauan has a Pratt parser baked into his language (https://github.com/Pauan/nulan) while my attitude for wart has been "if you want to change the syntax, hack the parser". Not to be flippant; my goal explicitly is to make the code dead simple for anybody* to hack on. Not there yet, but I'd love for you to take a stab at your regex idea with it :) Perhaps we could pair on it sometime?
* who knows C. Other terms and conditions may apply.
I would like to point out that although it's a Pratt parser, it's been specifically modified to work better with Lisps, meaning it operates on lists of symbols rather than on a single expression. I have not seen another parser like it.
This makes it much more powerful while also being much easier to use. Using the Nulan parser, adding in new syntax is as easy as writing a macro!
For instance, in Nulan, the "->" syntax is used for functions:
foo 1 2 3 -> a b c
a + b + c
The above is equivalent to this Arc code:
(foo 1 2 3 (fn (a b c)
(+ a b c)))
And here's how you can implement the "->" syntax in Nulan:
$syntax-rule "->" [
priority 10
order "right"
parse -> l s {@args body}
',@l (s args body)
]
As you can see, it's very short, though it might seem cryptic if you don't understand Nulan. Translating it into Arc, it might look like this:
(syntax-rule "->"
priority 10
order "right"
parse (fn (l s r)
(with (args (cut r 0 -1)
body (last r))
`(,@l (,s ,args ,body)))))
The way that it works is, the parser starts with a flat list of symbols. It then traverses the list looking for symbols that have a "parse" function.
It then calls the "parse" function with three arguments: everything to the left of the symbol, the symbol, and everything to the right of the symbol. It then continues parsing with the list that the function returns.
So basically the parser is all about manipulating a list of symbols, which is uh... pretty much exactly what macros do.
Going back to the first example, the "parse" function for the "->" syntax would be called with these three arguments:
1 (foo 1 2 3)
2 ->
3 (a b c (a + b + c))
It then destructures the 3rd argument so that everything but the last element is in the variable "args", and the last element is in the variable "body":
args (a b c)
body (a + b + c)
It then generates the list using "quote", which is then returned.
Basically, it transforms this:
foo 1 2 3 -> a b c (a + b + c)
Into this:
foo 1 2 3 (-> (a b c) (a + b + c))
As another example, this implements Arc's ":" ssyntax, but at the parser level:
$syntax-rule ":" [
priority 100
order "right"
delimiter %t
parse -> l _ r
',@l r
]
So now this code here:
foo:bar:qux 1 2 3
Will get transformed into this code here:
foo (bar (qux 1 2 3))
I've never seen another syntax system that's as easy and as powerful as this.
Oh yeah, and there's also two convenience macros:
$syntax-unary "foo" 20
$syntax-infix "bar" 10
The above defines "foo" to be a unary operator with priority 20, and "bar" to be an infix operator with priority 10.
Nulan's parser is powerful enough that almost all of Nulan's syntax can be written in Nulan itself. The only thing that can't be is significant whitespace.
Even the string syntax (using "), whitespace ( ), and the various braces ({[]}) can be changed from within Nulan.
I'm actually fairly convinced that my original idea as stated above wouldn't work, because without specific syntactical support already built in for whatever new reader you're trying to add, the original reader would have no good way of knowing when your macro ended, i.e.:
(re /\)*/)
How does the original reader know to pass all of "/\)*/" in to the 're macro? Maybe there's some clever way to tell the difference between "escaped" special characters and normal ones, but it would limit the possibilities on the temp reader.
Maybe one option would be to have more flexibility when defining macros in the first place by taking advantage of the fact that macros can be defined and evaluated at the reader stage, so they can specify their own read semantics for their evaluation if they choose. I.e. make it so that macro definitions can specify a more generic level of control than just unevaluated, pre-parsed s-exps. That would make them more of scoped 'reading macros' that shadow the existing reader, rather than 'reader macros' that just hook into it.
Your train of thought is very similar to a factor in several of my designs over time: Jisp[1], Blade[2], Penknife[3], Chops[4], and now the syntax I'm planning to use with Era[5].
I've always used use bracket nesting to determine where the syntax's body begins and ends. This way, most code requires no escape sequences, and the few times escape sequences are necessary, at least they stand a chance of being consistent when cutting-and-pasting across staging levels.
(re /\)*/) ; Broken.
(re /(?#()\)*/) ; Fixed with a regex comment. (Does this work?)
(re /\>*/) ; Fixed by redesigning the escape sequence.
Reader macros are a different approach, where the entire rest of the file/stream has unknown syntax until the reader reaches that point. That's simple in its own way, but I prefer not to make my code that lopsided in its semantics I guess. :)
(EDIT: Half an hour after I posted this, I realized I had a big mess of previous drafts tacked onto the end. I've deleted those now.)
---
[1] Jisp was one of the first toy languages I made, and it was before I programmed in Arc (or any other lisp). When it encounters (foo a b c), it resolves "foo" to an operator and sends "a b c" to that operator.
[2] Blade didn't get far off the ground, but it was meant to have a similar parser, with the explicit goal of making it easy to combine several languages into a single compiled program, with an extra emphasis on having no accidental code-order-dependent semantics at the top level. I switched to square brackets-- [foo a b c] --since these didn't require the shift key.
[3] Penknife was meant to be a REPL companion to Blade, and I developed the syntax into a more complicated, Arc-like combination of infix and prefix syntaxes (http://www.arclanguage.org/item?id=13071). Penknife was complete enough that I used it as a static site generator for a while. However, at this point I realized all the custom syntax processing I was doing was really killing the compile-time performance.
[4] Chops is a JavaScript library that achieves Blade-like parsing without any goals for infix treatment or general-purpose programming. I use it as a markup language and a JavaScript preprocessor now that my static site generator runs in the browser.
$rc.rcPage( "/", $cg.parseIn( [str RocketN[i I]A.com] ),
"19-Nov-2012", $cg.parseIn( [str 2005[en]2010, 2012] ),
{ "title": "RocketNIA.com, Virtual Index of Ross Angle",
"breadcrumbs": $cg.parseIn(
[str RocketN[i I]A.com: Virtual Index of Ross Angle] ) },
$cg.parse( [str
((This is the open source version of my site.
[out http://www.rocketnia.com/ The online version] has a bit more
content.))
...
] ) )
[5] Era is a module system I'm making, and I intend to compile to those modules from a lisp-like language. I've switched back to parentheses-- (foo a b c) --because smartphone keyboards tend to omit square brackets. The code (foo a b c) parses as a four-element list of symbols in the hope of more efficient processing, but the code foo( a b c) parses as a single symbol named "foo( a b c)".
The bouncing between parens and square brackets is interesting ^_^ I weakly feel it's not worth optimizing for what's easy to type because things can change (or be changed, with keybindings, etc.) so easily. Better to optimize for how things look. But even there, parens vs brackets is in the eye of the beholder.
Whoops, I can't believe I didn't use the phrase "balanced brackets" instead. ^_^
The following two pieces of text may be similar, but I'd give them significantly different behavior as code:
(foo a b (bar c d) (baz e) f)
(foo a b bar c d) (baz e f)
My systems don't provide any way to pass the string "a b bar c d) (baz e f" to an operator.
---
"staging levels"
Staged programming is where a program generates some code to run later on, perhaps as a second program in some sense--especially if that boundary is enforced by a need to serialize, transmit, or sandbox the second program rather than executing it here and now. Staged programming has some implications for syntax, since it's valuable to be able to see the code we're generating.
Most languages use " to denote the beginning and end of a string, so they can't also use " to represent the character " inside the string. This can makes it frustrating to nest code within code. I'll use a JavaScript example.
While all these stages are JavaScript code, they all effectively use different syntax. It's not easy to copy and paste code from one stage to another.
Suppose we identify the end of the string by looking for a matching bracket, possibly with other pairs of matched brackets in between. I'll use ~< and >~ as example string brackets.
This fixes the issue. The same code is used at every level.
In JavaScript, technically we can implement delimiters like these if we're open-minded about what a delimiter looks like. We just need a function str() that turns a first-class string into a string literal.
Now the code is consistent! Consistently infuriating. :-p
In Arc, we delimit code using balanced ( ). The code isn't a string this time, but the use of balanced delimiters has the same advantage.
> (eval '(eval '(eval '(eval '(+ 1 2)))))
3
This advantage is crucial in Arc, because any code that uses macros already runs across this issue. A macro call takes an s-expression, which contains a macro call, which takes an s-expression....
Since we're now talking about macros that take strings as input, let's see what happens if Arc syntax is based on strings instead of lists.
> (let total 0 (each x (list 1 2 3) (++ total x)) total)
6
> "let total 0 \"each x \\\"list 1 2 3\\\" \\\"++ total x\\\"\" total"
6
If we use balanced ( ) to delimit strings, we're back where we started, at least as long as we don't look behind the curtain.
> (let total 0 (each x (list 1 2 3) (++ total x)) total)
6
If you want working code for a language like this, look no further than Penknife. :)
---
"It also wasn't clear what the escape sequence is in the third example."
Are you talking about this one?
(re /\>*/)
The original code would be broken in my approach because it uses ) in the middle of the regex, so the macro's input would stop at "/\". This fix addresses the issue by using a hypothetical escape sequence \> to match a right parenthesis, rather than using the standard escape sequence \).
If you're talking about my Penknife code sample, the "qq." part is quasiquote, and the \, part is unquote. Quasiquotation is relatively complicated here due to the fact that it generates soup, which is like a string with little pieces floating in it. :-p Penknife has no s-expressions, so it was either this foundational kludge or the hard-to-read use of manual AST constructors.
It's hard to count these examples with a whole number. XD Let me know if you were talking about my Blade code sample (the third code block in the post) or my Jisp code sample (third if you count the two example regex fixes separately).
It might be worth pointing out that the LTS appearing in my examples is more pronounced than it needs to be. The usual escape sequence \\ for backslashes creates ridiculous results like \\\\\\\". If we use \- to escape \ we get the more reasonable \--" instead, and then we can see the nonuniform nesting problem without that other distraction:
I ran into a similar issue with regex syntax when attempting to incorporate it into Arcueid's reader. There seems to be no easy way to parse a regular expression using Perl-like /.../ syntax, not if you also want symbols that use /'s for other things, e.g. the division function. Arcueid thus uses for now, r/.../ for regular expressions, and that syntax could be more easily distinguished from other legitimate uses of symbols with a minimum of fuss.
Wart's tokenizer already knows about backslashes inside strings, so "\"" becomes one token. It seems plausible to try tokenizing in a backslash-aware way everywhere and not just inside strings. Other than that you would have to treat slashes as a delimiter like double-quotes.
It might be an ugly design, but I think it would work, and it would be worth trying out.
Interesting, although I don't know if it can be done with efficiency in the worst case. Backtracking regexes can take in their worst-case exponential time for matching, as illustrated in the regex in my original post.
I've thought about this as well for a while. Arc's approach seems to be flat files so far, which means any move to multiple servers is terra incognita on some level[1]. I mean to build a nosql system at some point, but I want to do something besides yet another[2] project. I want to understand[3] why we must couple technology stack choices with where we want to be in CAP space[4]. Why can't we have a single project that lets us tweak one knob for RAM vs persistent store, strong vs eventual consistency, and so on? Laying out the design space in one place may give us some chance at least accumulating lessons as we reinvent the wheel.
It might be simplest to just install postfix (I've tried it several times, and the three-screen configuration wizard Just Works) and then follow the steps in http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix... to have postfix route through SES.
I'll update anarki later today with a sample for sending email through postfix.
Great question; I'm looking forward to other people's answers.
"When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need."
Building a programming language is a pretty natural itch. For a certain kind of programmer, at least. I'll give you some reasoning and rationalizations for wart, without any certainty that I can separate the two.
I've been vaguely aware of lisp almost since I was exposed to computers, back in 1996. But in India algol-style languages reigned. Basic, fortran, cobol, pascal, C/C++. A little behind the times. I took a lisp course when I moved to the US back in '99 and I could program little things[1] in common lisp after that. But I'd still stare at things like the recursive definition for reverse and wonder precisely what it was that made it so different from what I'd write in C. I periodically go back to that question, but more on that further down.
After programming mostly in C for 10 years, burning out, recovering, the usual coming-of-age stuff, I found myself in the bay area doing rails and the startup scene for a couple of years ('07-'10). I'd been interested in social software and recommendation systems since Reddit got started. I even signed an NDA with the Reddit guys to get at their code and try to build a recommendation system for them. But the lawyers took too damn long and I'd lost interest by the time they sent it to me (sometime '07). But around '08 I started thinking about recommendation systems again. Arc was out by this point[2], and it acted as a great gateway drug. Here, finally, was a lisp that helped filter out all the cruft and let me focus on how lisp was different from C, and that was ergonomic enough that I had to work to stop programming in it. At some point I looked at the logs generated by news.arc, and noticed that they were just s-exprs that could be read back in if I so desired. Hmm, crazy idea: a recommendation system that can see the recent actions in a session to inform recommendation decisions. The easiest way to do that would be to stick with lisp.
So I left rails behind and started a fresh (and my final, so far) startup in arc, to build a feedreader that didn't require understanding RSS. I never did do much with the idea of adapting to the current session, because I could never get to a point where I knew enough about a user to make any intelligent decisions. Providing a bootstrap experience that kept them coming back long enough turned out to be an eternally unsolved problem. I found out the importance of UX and how much I sucked at it. A year later the project was dead and I was working at Google. But the interest in lisp stuck, and I continued doing stuff with it, noodling on a series of little questions about its design, why it did things this way and not that. Sometimes I learned the hard way something people knew back in the '60s. Occasionally I discovered something new[3].
But I was still dissatisfied. One of the things that had bothered me all through this period was webservers and memory leaks. No matter what platform I used -- rails, python, racket, common lisp -- it seemed any non-trivial website would periodically run out of memory and die. Everywhere I looked, people dealt with this in a brutal way: by periodically restarting their servers and clearing the slate. Ugly, and it seemed to point at an endemic problem: languages spent tons of effort tweaking their garbage collectors, but programmers would mess up in some subtle way and accidentally prevent stale objects from being collected. Often this was because of interactions between multiple projects. Often there were no tools for the low-level debugging required. Or if there was, you needed to learn too much about the high-level internals of the stack you were relying on.
There were other dissatisfactions. Common lisp was a lisp-2, and its APIs were ancient and baroque[4]. I could sense that I could get used to them, but the mind rebelled at polluting my brain with all those warts. Racket was more modern and a lisp-1, but it gave up a lot of the dynamism of lisp with its phase-separation rules, hygienic macros and module system. Both had oodles of documentation, but common lisp's docs were poor and examples often wouldn't run. It had a 'standard' constraining creativity and unionizing bad APIs with lots of detailed rules, but the standard hadn't been updated in years so you were often doing stuff out of its ambit anyway (I still don't understand its condition system). It was the worst of both worlds. Racket has a much nicer documentation system, and yet it was too overwhelming for reasons I still struggle to pinpoint[5]. At some point, it seemed, the documentation was complex enough to ask, why can't I just deal with the codebase directly rather than some interface to it? How can I fork racket and throw out hygienic macros until I gain the wisdom of their need?
So eventually, after leaving arc, after trying to build arc out of common lisp macros[6], after some time agonizing over just switching to Factor[7], I bit the bullet, threw out all the dependencies, and returned to my roots in C. My goals were to create a dynamic lisp-1 with non-hygienic macros where I could define code in any order[8], and to do so with as little code as possible, in such a way that others could come after me and query the design space I had explored, asking what-if questions: what if I want to turn off this feature? Why do we implement this like so? The goal was never to build a 'real' language the way the world thinks about it. I wanted to see how far I could push a design without any black-box interfaces, where programmers had to understand the entire stack, and therefore where the entire stack had to be simple enough to fit in a single head[9], one that encouraged the combination of high- and low-level fluency necessary to track down problems like memory leaks, one where the documentation was intertwined inextricably with the implementation, so it couldn't go out of date and it could be queried interactively[10]. One that didn't spend thousands of lines of code trying to 'optimize' arbitrary code, but provided programmers the tools to measure their applications and attack bottlenecks for themselves without compromising readability, the way 'macros' provide a la carte mechanisms for reducing our own boilerplate. That's The Lisp Way.
What I have so far is a far cry from this goal. Sometimes I worry that I'm just playing in the shallows where it's fun[11] and avoiding the scary abyss. Programming languages are fun, but if I can stay disciplined wart will be more than a language. And nobody will have to ever rediscover why something is built just like so.
[5] Many of my initial annoyances were bugs in arc at the time (http://arclanguage.org/item?id=13616), or a result of the legacy mzscheme language (since fixed by Arc/Nu). Would I have embarked on this if the timing had been just a little different?
[10] I don't want to require new programmers to understand every last line of code before they can use a program. I want the program to reward curiosity, so that newcomers are able to drill down into the right bits when they run into problems, to answer questions for themselves without needing to find an expert to talk to (http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...), and to iteratively try tweaking the code with instant feedback about regressions they introduce.
One more footnote. This article catalyzed the focus on memory leaks for me when I read it back in 2011: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-.... It turned out that the memory leak was caused in a library which defined a specific method (del) that (was documented to) disabled garbage collection in some cases. You look here and the blame goes there, you look there and the blame goes here. Something is very wrong with software; who's in charge here?
1. Arc 3.1 uses a kludgy uniq that doesn't actually guarantee unique symbols. I hooked it up with racket's infrastructure at some point, which happens to permit an optional arg.
$ racket
> (gensym)
'g25
> (gensym 'tmp)
'tmp26
2. Also, anarki generalizes arc 3.1 by permitting any number of args to [], including zero:
anarki> ([prn "abc" _1 _2] 34 35)
"abc3435"
This is indeed incompatible with arc 3.1 in one way:
arc3.1> ([prn "abc"] 34) # Must provide one arg, but ok to not use it.
"abc"
anarki> ([prn "abc"] 34) # Must provide exactly as many args as expected.
Error: .. arity mismatch ..
Well, how did the change not break setforms in arc.arc? Was the original definition for make-br-fn silently overwritten somewhere? Last I checked there is still a (let argsyms (map [uniq](cdr expr) ...) inside of the definition of setforms in arc.arc even in the latest versions of Anarki. This will not work with the change to bracket functions that has been made.
Frankly, I think that bracket functions should always have at least one argument permitted.
"Well, how did the change not break setforms in arc.arc? Was the original definition for make-br-fn silently overwritten somewhere?"
Yes. The [...] syntax is redefined in make-br-fn.arc after arc.arc is loaded.
I described my opinion of this at https://github.com/nex3/arc/commit/957501d81bafecab070268418.... It did break my code, but I don't mind breaking changes these days (maybe because I'm hardly a user!), and I think a:b syntax is already more useful than either version of [...].
"Frankly, I think that bracket functions should always have at least one argument permitted."
Groovy does this. The Closure syntax { ... } is shorthand for { it = null -> ... }, which is a one-argument Closure whose argument "it" defaults to null. When I moved from Groovy to Arc, I missed this.
What do people think of this? Basically I want all list operations to treat atoms as degenerate dotted lists. Does this have any adverse implications for other aggregates (tables)?
Update: As I expected, taking out the round-trip to table in the first version above saves us some consing. All the changes to get to the second version don't seem to result in any extra consing over the first.
"What do people think of this? Basically I want all list operations to treat atoms as degenerate dotted lists. Does this have any adverse implications for other aggregates (tables)?"
I don't see a problem, but I may not be the right person to ask. If it came down to it, I'd be willing to use 'table-pushnew, 'alist-pushnew, 'maxheap-pushnew, and so on. :-p
What other list operations do you have in mind? While 'pushnew makes sense to use with degenerate dotted lists, it's a very special case: It only clobbers the list at a finite depth, and that depth is actually zero. (Or zero in the unmodified list, anyway.)
Oh, whoops. I forgot 'pushnew actually does traverse the input list to find out whether the element is new. I was thinking of 'push. XD
I disagree with the notion of membership you're using for dotted lists. If I 'pushnew nil onto (1 2 3 . nil), I want to get (nil 1 2 3 . nil), and it won't work that way if nil is already considered to be a member.
I think the membership check you're using is like this:
(The final cdr is an element iff it isn't nil.)
If the list is...
A cons cell:
If the car is the value we're looking for, succeed. Otherwise,
continue by searching the cdr.
Nil:
Fail.
A non-cons, non-nil value:
Continue by comparing the list to the value we're looking for.
I feel we could simplify this specification by rewriting the last two cases in one of these ways, ordered from my least to most favorite:
(The final cdr is an element.)
(This breaks (pushnew nil ...) in existing Arc code.)
...
A non-cons value:
Continue by comparing the list to the value we're looking for.
(The final cdr is not an element.)
...
A non-cons value:
Fail.
(The final cdr is always nil and is not an element.)
(Arc 3.1 already uses this.)
...
Nil:
Fail.
A non-cons, non-nil value:
Raise an exception.
By using the "element iff it isn't nil" approach, you're able to use 'pushnew to traverse the simple argument lists you build as intermediate results of that 'make-br-fn implementation. But I don't know if it's worthwhile to complicate the notion of "list" just to accommodate a special case of argument lists.
"I don't know if it's worthwhile to complicate the notion of "list" just to accommodate a special case of argument lists."
Yeah I see your point. My aim was to extend the correspondence between the syntax for vararg params, rest params and lists of regular params. But those features merely match a (proper) list of args to some template. I'm entangling the template with the notion of a list. Hmm..
