Great question; I'm looking forward to other people's answers.
"When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need."
Building a programming language is a pretty natural itch. For a certain kind of programmer, at least. I'll give you some reasoning and rationalizations for wart, without any certainty that I can separate the two.
I've been vaguely aware of lisp almost since I was exposed to computers, back in 1996. But in India algol-style languages reigned. Basic, fortran, cobol, pascal, C/C++. A little behind the times. I took a lisp course when I moved to the US back in '99 and I could program little things[1] in common lisp after that. But I'd still stare at things like the recursive definition for reverse and wonder precisely what it was that made it so different from what I'd write in C. I periodically go back to that question, but more on that further down.
After programming mostly in C for 10 years, burning out, recovering, the usual coming-of-age stuff, I found myself in the bay area doing rails and the startup scene for a couple of years ('07-'10). I'd been interested in social software and recommendation systems since Reddit got started. I even signed an NDA with the Reddit guys to get at their code and try to build a recommendation system for them. But the lawyers took too damn long and I'd lost interest by the time they sent it to me (sometime '07). But around '08 I started thinking about recommendation systems again. Arc was out by this point[2], and it acted as a great gateway drug. Here, finally, was a lisp that helped filter out all the cruft and let me focus on how lisp was different from C, and that was ergonomic enough that I had to work to stop programming in it. At some point I looked at the logs generated by news.arc, and noticed that they were just s-exprs that could be read back in if I so desired. Hmm, crazy idea: a recommendation system that can see the recent actions in a session to inform recommendation decisions. The easiest way to do that would be to stick with lisp.
So I left rails behind and started a fresh (and my final, so far) startup in arc, to build a feedreader that didn't require understanding RSS. I never did do much with the idea of adapting to the current session, because I could never get to a point where I knew enough about a user to make any intelligent decisions. Providing a bootstrap experience that kept them coming back long enough turned out to be an eternally unsolved problem. I found out the importance of UX and how much I sucked at it. A year later the project was dead and I was working at Google. But the interest in lisp stuck, and I continued doing stuff with it, noodling on a series of little questions about its design, why it did things this way and not that. Sometimes I learned the hard way something people knew back in the '60s. Occasionally I discovered something new[3].
But I was still dissatisfied. One of the things that had bothered me all through this period was webservers and memory leaks. No matter what platform I used -- rails, python, racket, common lisp -- it seemed any non-trivial website would periodically run out of memory and die. Everywhere I looked, people dealt with this in a brutal way: by periodically restarting their servers and clearing the slate. Ugly, and it seemed to point at an endemic problem: languages spent tons of effort tweaking their garbage collectors, but programmers would mess up in some subtle way and accidentally prevent stale objects from being collected. Often this was because of interactions between multiple projects. Often there were no tools for the low-level debugging required. Or if there was, you needed to learn too much about the high-level internals of the stack you were relying on.
There were other dissatisfactions. Common lisp was a lisp-2, and its APIs were ancient and baroque[4]. I could sense that I could get used to them, but the mind rebelled at polluting my brain with all those warts. Racket was more modern and a lisp-1, but it gave up a lot of the dynamism of lisp with its phase-separation rules, hygienic macros and module system. Both had oodles of documentation, but common lisp's docs were poor and examples often wouldn't run. It had a 'standard' constraining creativity and unionizing bad APIs with lots of detailed rules, but the standard hadn't been updated in years so you were often doing stuff out of its ambit anyway (I still don't understand its condition system). It was the worst of both worlds. Racket has a much nicer documentation system, and yet it was too overwhelming for reasons I still struggle to pinpoint[5]. At some point, it seemed, the documentation was complex enough to ask, why can't I just deal with the codebase directly rather than some interface to it? How can I fork racket and throw out hygienic macros until I gain the wisdom of their need?
So eventually, after leaving arc, after trying to build arc out of common lisp macros[6], after some time agonizing over just switching to Factor[7], I bit the bullet, threw out all the dependencies, and returned to my roots in C. My goals were to create a dynamic lisp-1 with non-hygienic macros where I could define code in any order[8], and to do so with as little code as possible, in such a way that others could come after me and query the design space I had explored, asking what-if questions: what if I want to turn off this feature? Why do we implement this like so? The goal was never to build a 'real' language the way the world thinks about it. I wanted to see how far I could push a design without any black-box interfaces, where programmers had to understand the entire stack, and therefore where the entire stack had to be simple enough to fit in a single head[9], one that encouraged the combination of high- and low-level fluency necessary to track down problems like memory leaks, one where the documentation was intertwined inextricably with the implementation, so it couldn't go out of date and it could be queried interactively[10]. One that didn't spend thousands of lines of code trying to 'optimize' arbitrary code, but provided programmers the tools to measure their applications and attack bottlenecks for themselves without compromising readability, the way 'macros' provide a la carte mechanisms for reducing our own boilerplate. That's The Lisp Way.
What I have so far is a far cry from this goal. Sometimes I worry that I'm just playing in the shallows where it's fun[11] and avoiding the scary abyss. Programming languages are fun, but if I can stay disciplined wart will be more than a language. And nobody will have to ever rediscover why something is built just like so.
[5] Many of my initial annoyances were bugs in arc at the time (http://arclanguage.org/item?id=13616), or a result of the legacy mzscheme language (since fixed by Arc/Nu). Would I have embarked on this if the timing had been just a little different?
[10] I don't want to require new programmers to understand every last line of code before they can use a program. I want the program to reward curiosity, so that newcomers are able to drill down into the right bits when they run into problems, to answer questions for themselves without needing to find an expert to talk to (http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...), and to iteratively try tweaking the code with instant feedback about regressions they introduce.
One more footnote. This article catalyzed the focus on memory leaks for me when I read it back in 2011: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-.... It turned out that the memory leak was caused in a library which defined a specific method (del) that (was documented to) disabled garbage collection in some cases. You look here and the blame goes there, you look there and the blame goes here. Something is very wrong with software; who's in charge here?
