Arc Forumnew | comments | leaders | submitlogin
Why did you write your programming language?
7 points by lark 4024 days ago | 15 comments
I am trying to better understand what some of the languages mentioned here do. After a cursory look, I'm unsure why each one of them was written.

  | Arc/Nu seems like a cleaned-up Arc, and faster.
  | Anarki seems like it improve Arc with the non-compatible changes.
  | Arcuid seems like a backwards compatible Arc, but transformed in C.
  | Wart seems like a Python with macros and more flexible syntax.
  | Subjective Lisp puts Arc in iOS

Why did you write these programming languages? Are you personally using them? Have you written say your website in them, or some important tool in them?

Of course there's fun and educational value in writing a programming language. You get to discover esoteric features you wouldn't have known otherwise and dig deeper. There's also unchartered territory beyond just a language. When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need.

Can you point to the exact utter breakdown of some existing language that had you throwing your computer out the window, refusing to touch one again unless there were a language that could instead do X?



11 points by akkartik 4023 days ago | link

Great question; I'm looking forward to other people's answers.

"When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need."

Building a programming language is a pretty natural itch. For a certain kind of programmer, at least. I'll give you some reasoning and rationalizations for wart, without any certainty that I can separate the two.

I've been vaguely aware of lisp almost since I was exposed to computers, back in 1996. But in India algol-style languages reigned. Basic, fortran, cobol, pascal, C/C++. A little behind the times. I took a lisp course when I moved to the US back in '99 and I could program little things[1] in common lisp after that. But I'd still stare at things like the recursive definition for reverse and wonder precisely what it was that made it so different from what I'd write in C. I periodically go back to that question, but more on that further down.

After programming mostly in C for 10 years, burning out, recovering, the usual coming-of-age stuff, I found myself in the bay area doing rails and the startup scene for a couple of years ('07-'10). I'd been interested in social software and recommendation systems since Reddit got started. I even signed an NDA with the Reddit guys to get at their code and try to build a recommendation system for them. But the lawyers took too damn long and I'd lost interest by the time they sent it to me (sometime '07). But around '08 I started thinking about recommendation systems again. Arc was out by this point[2], and it acted as a great gateway drug. Here, finally, was a lisp that helped filter out all the cruft and let me focus on how lisp was different from C, and that was ergonomic enough that I had to work to stop programming in it. At some point I looked at the logs generated by news.arc, and noticed that they were just s-exprs that could be read back in if I so desired. Hmm, crazy idea: a recommendation system that can see the recent actions in a session to inform recommendation decisions. The easiest way to do that would be to stick with lisp.

So I left rails behind and started a fresh (and my final, so far) startup in arc, to build a feedreader that didn't require understanding RSS. I never did do much with the idea of adapting to the current session, because I could never get to a point where I knew enough about a user to make any intelligent decisions. Providing a bootstrap experience that kept them coming back long enough turned out to be an eternally unsolved problem. I found out the importance of UX and how much I sucked at it. A year later the project was dead and I was working at Google. But the interest in lisp stuck, and I continued doing stuff with it, noodling on a series of little questions about its design, why it did things this way and not that. Sometimes I learned the hard way something people knew back in the '60s. Occasionally I discovered something new[3].

But I was still dissatisfied. One of the things that had bothered me all through this period was webservers and memory leaks. No matter what platform I used -- rails, python, racket, common lisp -- it seemed any non-trivial website would periodically run out of memory and die. Everywhere I looked, people dealt with this in a brutal way: by periodically restarting their servers and clearing the slate. Ugly, and it seemed to point at an endemic problem: languages spent tons of effort tweaking their garbage collectors, but programmers would mess up in some subtle way and accidentally prevent stale objects from being collected. Often this was because of interactions between multiple projects. Often there were no tools for the low-level debugging required. Or if there was, you needed to learn too much about the high-level internals of the stack you were relying on.

There were other dissatisfactions. Common lisp was a lisp-2, and its APIs were ancient and baroque[4]. I could sense that I could get used to them, but the mind rebelled at polluting my brain with all those warts. Racket was more modern and a lisp-1, but it gave up a lot of the dynamism of lisp with its phase-separation rules, hygienic macros and module system. Both had oodles of documentation, but common lisp's docs were poor and examples often wouldn't run. It had a 'standard' constraining creativity and unionizing bad APIs with lots of detailed rules, but the standard hadn't been updated in years so you were often doing stuff out of its ambit anyway (I still don't understand its condition system). It was the worst of both worlds. Racket has a much nicer documentation system, and yet it was too overwhelming for reasons I still struggle to pinpoint[5]. At some point, it seemed, the documentation was complex enough to ask, why can't I just deal with the codebase directly rather than some interface to it? How can I fork racket and throw out hygienic macros until I gain the wisdom of their need?

So eventually, after leaving arc, after trying to build arc out of common lisp macros[6], after some time agonizing over just switching to Factor[7], I bit the bullet, threw out all the dependencies, and returned to my roots in C. My goals were to create a dynamic lisp-1 with non-hygienic macros where I could define code in any order[8], and to do so with as little code as possible, in such a way that others could come after me and query the design space I had explored, asking what-if questions: what if I want to turn off this feature? Why do we implement this like so? The goal was never to build a 'real' language the way the world thinks about it. I wanted to see how far I could push a design without any black-box interfaces, where programmers had to understand the entire stack, and therefore where the entire stack had to be simple enough to fit in a single head[9], one that encouraged the combination of high- and low-level fluency necessary to track down problems like memory leaks, one where the documentation was intertwined inextricably with the implementation, so it couldn't go out of date and it could be queried interactively[10]. One that didn't spend thousands of lines of code trying to 'optimize' arbitrary code, but provided programmers the tools to measure their applications and attack bottlenecks for themselves without compromising readability, the way 'macros' provide a la carte mechanisms for reducing our own boilerplate. That's The Lisp Way.

What I have so far is a far cry from this goal. Sometimes I worry that I'm just playing in the shallows where it's fun[11] and avoiding the scary abyss. Programming languages are fun, but if I can stay disciplined wart will be more than a language. And nobody will have to ever rediscover why something is built just like so.

[1] From 2004 or so: http://akkartik.name/lisp.html

[2] After years, it felt like, at the time.

[3] http://arclanguage.org/item?id=16378; arguably this little idea is the coolest thing I've done so far.

[4] http://arclanguage.org/item?id=13664, footnote 3.

[5] Many of my initial annoyances were bugs in arc at the time (http://arclanguage.org/item?id=13616), or a result of the legacy mzscheme language (since fixed by Arc/Nu). Would I have embarked on this if the timing had been just a little different?

[6] https://github.com/akkartik/wart/tree/sbcl

[7] http://factorcode.org; awesome, awesome language. But I just liked the readability of keyword args too much to give them up.

[8] http://arclanguage.org/item?id=15587, footnote 1.

[9] VPRI is working toward a similar goal: a computer stack in orders of magnitude less code (http://www.vpri.org/html/work/ifnct.htm).

[10] I don't want to require new programmers to understand every last line of code before they can use a program. I want the program to reward curiosity, so that newcomers are able to drill down into the right bits when they run into problems, to answer questions for themselves without needing to find an expert to talk to (http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...), and to iteratively try tweaking the code with instant feedback about regressions they introduce.

[11] http://arclanguage.org/item?id=17358; http://arclanguage.org/item?id=17394; http://arclanguage.org/item?id=10692.

-----

2 points by akkartik 4015 days ago | link

One more footnote. This article catalyzed the focus on memory leaks for me when I read it back in 2011: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-.... It turned out that the memory leak was caused in a library which defined a specific method (del) that (was documented to) disabled garbage collection in some cases. You look here and the blame goes there, you look there and the blame goes here. Something is very wrong with software; who's in charge here?

-----

2 points by kinleyd 4022 days ago | link

Thanks akkartik for sharing your story. More power to you. :)

-----

2 points by lark 4015 days ago | link

Thank you for sharing this remarkable story.

-----

5 points by Pauan 4022 days ago | link

If I had to give a reason for making Arc/Nu... well, that's really complicated and involves a lot of history and backstory.

Basically, before Arc/Nu, there was ar, which was created by awwx. It was a rewrite of Arc in Racket, where the compiler was intentionally exposed to Arc. This meant you could not only access the compiler from Arc, but could actually change it!

So rather than writing the compiler in Racket, you'd write the minimum necessary to get things going, then write the rest of the compiler in Arc. This naturally made the compiler very easy to hack for Arc users.

I really liked that, but I also disliked certain things in ar. One of those is that all the compiler stuff is prefixed with "racket-", which is really verbose.

Another dissatisfaction is that although awwx did add in a module system for Arc, it was very... uh... basically you could create multiple Racket namespaces, which are completely separate from each other, and load Arc into each one.

The problem is that because they're separate, it's really difficult for them to communicate with each other. Which kinda defeats the point.

Another problem is that you had to reload Arc every time you created a new namespace, which meant ~1-2 second pauses every single time. Which is pretty ridiculous.

I felt I could do better, so I made a fork of ar and hacked away on it. But having to use the "racket-" prefix really bothered me, so I experimented with ways to get rid of it.

Eventually I decided to start over with pg's vanilla Arc 3.1, rather than ar. I fixed it so it would work in Racket rather than mzscheme, and then I did significant refactoring, making things faster, using new techniques I had discovered, etc.

I also added in a module system which was exactly the same as ar's module system, except that you could make a module that inherits from another module. This solved the "have to reload Arc all the time" problem, and also made it a bit easier to communicate between modules.

The end result was Arc/Nu. It had all the benefits of ar, all the benefits of Arc 3.1, and none of the downsides of either one.

In the end, though, Arc/Nu's module system was terrible. Yes, you could now make modules that inherit from other modules, but I kept running into all kinds of problems, both theoretical and practical. But it was the best module system I knew of.

I wrote some programs in Arc/Nu, and after many months passed, I discovered hyper-static scope.

Before I continue with this story, I'll need to explain Nulan. Nulan is a Lisp I created from scratch. It has hyper-static scope, hygienic macros, and it compiles to JavaScript and can thus be used in web browsers.

I decided to make Nulan because I wanted to experiment with cool new things like hyper-static scope, but I couldn't do that in Arc/Nu because that would break backwards compatibility with Arc 3.1. I also wanted a language for web browsers that would be better than JavaScript.

While implementing hyper-static scope in Nulan, I discovered how amazing boxes are. Seriously, they're great. Using them, you can create a module system which is easily thousands of times better than any other module system I had ever seen.

Dido posted a topic about a module system he was designing for Arc, and I replied with this: http://arclanguage.org/item?id=17408

After that, I realized that I had to change Arc/Nu to use boxes. And so I did. The module system in Arc/Nu is now amazing, and no longer based upon Racket namespaces.

As for whether I'm still using Arc/Nu, the answer is yes. I like using it for writing shell scripts, since I find Arc vastly better at that task than Python or Bash.

-----

3 points by lark 4015 days ago | link

It would be great if one could easily write in Arc as opposed to Javascript. Nulan and RainbowJS are in a position to infiltrate web-based programming through the browser.

-----

1 point by kinleyd 4022 days ago | link

Thanks Pauan, and more power to you too!

-----

5 points by dido 4021 days ago | link

First of all I don't like the fact that the present reference Arc depends on MzScheme/Racket. This makes setting it up and using it a chore, and I think this is a not insignificant factor in what hobbled wider acceptance of the language. I wanted to be able to just do a ./configure ; make ; make install and start using it, the way I can presently do with, say, Ruby. As it is you have to install a rather large Scheme interpreter (Racket 5.1.1 has over 500,000 lines of code; it might have been a little different if it were a smaller Scheme like scheme48) just to get started.

I've also wanted to have a language like Tcl or Lua that one could easily embed as a scripting language in a larger project. This is one of the things that Arcueid is designed to do: just link a single library and have an interpreter available with a relatively simple API. Arcueid is also be designed to be easily extensible via a simple foreign function interface because I don't want to use it for just programming web applications, although that is something that I would doubtless use it for a lot as well.

Writing in C (or something that compiles to C) is also probably the only way to get an Arc faster than the implementations we have today. Of course my present implementation is still somewhat slower than Arc on Racket, but then again the goal for now is to build a foundation upon which further optimisation can later be done. As always, it's better to get it to work right first, and then make it run quickly later... Eventually Arcueid will have a JIT compiler of sorts but that's a little further down the roadmap.

As for Arc, it seems like it's much better designed than any other Lisp dialect out there. The parentheses are still there, but they are used more sparingly than Scheme, which seems out to irritate you by putting extra parentheses for something so basic as a let binding when you could easily do with less and have more comprehensible code. This fact was illustrated to me about a year ago when rocketnia gave me versions of the Hanson-Lamping algorithm for dynamic-bind in Scheme and in Arc here:

http://arclanguage.org/item?id=15536

The contrast was rather startling to me. I understood the algorithm immediately on seeing the Arc code.

Another thing about other Lisps that is really irritating to me are the overly long function names like multiple-value-bind and invoke-restart-interactively and stuff like that. They make a program look rather cluttered. Arc has for the most part avoided excessively long identifiers in the core and that has been good.

Macros are something that I've also been really interested in, and something that looks like is only really possible with a Lisp. Ruby is my programming language for most of my day to day work and the simple metaprogramming it makes possible makes me wish that it could do macros too, though how it could do that without also transforming into Lisp is another matter.

-----

1 point by Pauan 4020 days ago | link

"[...] though how it could do that without also transforming into Lisp is another matter."

My experiments with Nulan have convinced me that it's easier (and better) to start with a Lisp and then add Ruby syntax which desugars to S-expressions, rather than trying to turn Ruby into a Lisp.

-----

2 points by rocketnia 4020 days ago | link

I assume by "s-expressions" we mean nested lists whose first elements are often symbols which represent operators. Apparently, Ruby already has built-in support for that code representation: http://stackoverflow.com/questions/6894763/extract-the-ast-f...

-----

1 point by Pauan 4020 days ago | link

Well then, using that it'd be possible to make Ruby macros.

-----

2 points by rocketnia 4020 days ago | link

Hmm, even if the package for doing this comes with Ruby MRI, I'm not sure there's a corresponding out-of-the-box way to take these s-expressions and get running code. But here's a five-year-old article talking about Ruby2Ruby, a library that converts these s-expressions to strings: http://www.igvita.com/2008/12/11/ruby-ast-for-fun-and-profit...

I don't think a third-party tool should count as language support, unless (and inasmuch as) the people maintaining the language regularly go out of their way to avoid breaking that tool.[1][2] This is where I draw the line, because of how it affects backwards compatibility.

Everyday language users tend to think of language updates as backwards-compatible as long as their existing code has the same behavior. But tools like eval(), JavaScript minifiers, and Ruby2Ruby place the bar higher, since their input is arbitrary language code, and it might be written in a newer language version than the tool itself. Incidentally, even a program that calls these tools will continue to work after a language update, as long as the input of these calls is all generated programmatically, rather than directly taken from the program's input.

[1] Well, I guess the term "third-party" might not apply in that case!

[2] I have no idea how much I'm actually talking about Ruby2Ruby, and this Ruby s-expression format in general, when I make this comment.

-----

2 points by lark 4015 days ago | link

I too wish Arc could be embedded. The discussion on the size of a language core is suspiciously parallel to the discussion on the size of an operating system core.

-----

1 point by kinleyd 4021 days ago | link

Since I'm on a roll here, albeit at another level entirely... thanks dido, and more power to you too!

-----

2 points by lark 4021 days ago | link

I should point out there are a few languages I missed, and possibly others:

  | Nulan seems like an that also compiles to Javascript
  | Rainbow is a Java implementation of Arc
  | RainbowJS seems like a port of Rainbow to Javascript
Sorry about that.

-----