Arc Forumnew | comments | leaders | submitlogin
Issues Regarding Lisp Adoption
7 points by tokipin 4342 days ago | 27 comments
I'm curious what things hold back Lisp from being more popular.

  Choosing a Dialect/Implementation
Often mentioned is that newbies have to choose a dialect and implementation, but I don't think that is really a problem. If they're new to Lisp they probably don't care about specifics because they won't be able to gauge the differences, so they'll grab anything. Though I think Arc is an exception to this in that by both being new and focusing expressly on power it has the potential to draw more attention.

  Scary Parentheses
A more common topic is the blinding parentheses, but again I don't think they're as scary to noobs as people believe. A decade or two ago I found a page on Lisp on the net but didn't give the language a second thought because the parentheses made it look primitive. How can something so rudimentary be useful at all? It looked barely capable of basic arithmetic. Of course now that snobs left and right are saying it's the most powerful language, that assessment would be different.

  Lack of IDE's
What I think is a very large issue is the fact that there is a poor choice of IDE's available for Lisp. The console REPL is awkward and ugly. When a user first opens it up they are forced into basic, tiny-scale experimentation with the language. Writing a sizeable function or an actual program in the REPL is unreasonable.

You may say that as beginners, some simple experimentation is fitting, but I believe that if they're trying out Lisp they are at least not retarded, and trivial expressions will only entertain them for a few minutes. Furthermore, the beginner won't really see the value of the REPL as something that helps experimentation and actually allows things to be changed while running. They're just messing with basic things.

So the next step for these newbies is to write their code in a file, perhaps using a generic text editor, and then write a script or open up a console to run it. But by this step I think most beginners will be very annoyed.

I say that from experience. Maybe I'm exceptionally, incredibly lazy, but having to write a script is a large wall blocking my fun, and most of the time I decide I'd rather munch on some Cheerios. Many programmers out there are accustomed to and spoiled by IDE's, so their tolerance for these things won't be very high either.

An option is to associate lisp filetypes with the interpreter, but I doubt most people know how to do that or even think about it for that matter. (I just thought of that right now -.-) Nor is it optimal for usability.

Obviously an IDE is a good solution for these issues. The best package I have found in this regard is Lispbox, which is Emacs with lisp stuff and settings out of the box.

The problem is that Emacs is, in a similar way to the console REPL, ugly. It also has a vibe of excess -- almost as if it was developed by hundreds of programmers over the years -- which adds to the ickiness.

  Arc Plugin for Eclipse
I don't think there is a better candidate for an IDE target than Eclipse. It is very pretty. There seem to be a few Lisp plugins available for it:

I think an Arc plugin for Eclipse would go very far in helping/letting newbies get started with the language. The goal isn't to have something fancy. The fact that beginners can simply experiment in a familiar, non-ugly interface and just click Run or hit a hotkey is much, much better than other current setups.

I'm sure someone will say I should put my feet where my mouth is or something, but maybe someone is already familiar with making Eclipse plugins and can perhaps adapt an open-source one to Arc. [edit]In another topic someone linked such a thing:

If it was packaged with Eclipse and made available for download here it would be even better.

I focused on IDE's, but I'm also interested in the general question of what things can be done and not done to get more people into Lisp. Something nice I found recently from the Ruby homepage is an online interpreter/demo/tutorial for Ruby.

Something like this for Arc would be really cool.

12 points by kennytilton 4342 days ago | link

"I'm curious what things hold back Lisp from being more popular."

Inertia. Things like Java and Python were close enough to C (syntax and imperative paradigm) that moving to them was just a minor course change. Lisp feels like a U-turn, though I have argued elsewhere it is not really. Meanwhile Java had Sun behind it, giving PHBs a vital warm fuzzy. Lisp looks like the north wall of the Eiger in winter to PHBs.

The IDE problem just confirms anyone's fears if they even get that far, though Franz and Lispworks tried to address that by making unlimited-length trials of their fine IDEs available. Agreed on Emacs+Slime not being a good answer for noobs.

"I focused on IDE's, but I'm also interested in the general question of what things can be done and not done to get more people into Lisp."

Not to worry, it is happening, just step back and look at where languages are going, not the people. Steele famously describes Java as dragging people halfway from C++ to Lisp. Not sure about the fraction. :) Python moved a little more, and Ruby moved a little more. If Arc catches on it will constitute completion of the land bridge from static typing imperative languages to dynamically typed functional languages, but it does not matter because Common Lisp is starting to catch on anyway. And if you look at the ideas behind Lisp and why languages like Python and Ruby are doing so well, Lisp has already prevailed. My 2.


10 points by cchooper 4342 days ago | link

In the future, everyone will use Lisp, but they won't know it's Lisp, because it will be called Microsoft Windows Dynamic Service-Oriented Parenthesis Construction Language for .NET.


6 points by kennytilton 4342 days ago | link

heh-heh, no, we had our chance:

Mr Dussud (a respected Lisper in a former life) said MS would be happy to host a Lisp if we would just add strong static typing and lose our silly little OO package, CLOS. A couple of other things, too.

We almost ran him from the room. :)

Meanwhile, there is:

....though the name is not as good as yours.


1 point by cchooper 4342 days ago | link

I've just looked at the slides. Did he seriously call Java a dynamic language?


8 points by kennytilton 4342 days ago | link

"Did he seriously call Java a dynamic language?"

Wow, yeah. I tuned him out when I saw he wanted us to declare our variable types. I did not listen to all of the audio, but in there somewhere you will hear some nutjob asking if, given that Lisp has finally started taking over the landscape and he now wants us to run up the white flag and adopt static typing, he also thinks it is time for mainland China surrender to Taiwan. That was me.

I went up later and shook his hand for being brave enough to come into the lion's den. Strong guy. <ouch>


6 points by lojic 4342 days ago | link

Yeah, he called Ruby an untyped scripting language. Strange that it gives the following TypeError then ;)

  irb(main):003:0> x + 7
  TypeError: can't convert Fixnum into String
          from (irb):3:in `+'
          from (irb):3


3 points by DrStrangelove 4215 days ago | link

Profusion of dialects - lack of libraries. There is an inverse relationship between the success of languages and their power. C is cumbersome to write; therefore, everyone treasures the libraries that exist and takes pains to remain compatible. Java, even assuming that it made it 'halfway to Lisp', is clearly still cumbersome enough. Python is already too powerful, but it benefits from its slowness. In many problem domains (GUI, databases, numeric computing), it is way too slow, forcing you to use C-coded extensions, which again help to prevent fracturing of the user base. Where it's fast enough, you get a bazillion of half-assed libraries and frameworks. E.g. Python web programming is a mess and, compared to PHP and Java, not popular at all. (There are also limitations to the VM that matter here but I'm not going into those.)

Lisp is so powerful that it lets everyone modify it, and so everyone goes and creates their own dialect of it. It compiles to fast, native code, so no need to adopt and maintain libraries shared by a large user base.

There is a nice language called D. Unfortunately, it comes with two incompatible standard libraries. How do you like that as a new user? Well, in Lisp it's the same, only times 257. Throw in the oatmeal cum nail-clippings visual appeal (L.Wall) and you might well wonder why it is not less popular than it currently is.


11 points by KirinDave 4342 days ago | link

The elephant in the room is that Lisp's community makes it very hard to adopt Lisp for production. Scheme is not as affected, but in general you end up with several implementations, each doing now-common things like networking slightly differently, and each having a feature set that contains 90% of what you want. What's worse, if you get into non-trivial things it can be difficult to migrate from one lisp implementation to another when your priorities change and different features are required.

Compare this to most other languages–notable outliers like C/C++ excepted–have canonical implementations that world-wide communities can germinate from. This additional confusion at the outset of adoption has stopped more than one of my colleagues from learning Lisp.

There's also typically deployment issues. Because most lisp execution environments are so heavily tweaked out, a lot of times it's not clear how to deploy a piece of software. A great example of this was a web-app I received from a talented lisp consultant. His method for running the program? "Run SLIME in a Screen session and type the following commands." You can rightly argue that this was just one bad example, but I think everyone who's being honest has seen this kind of crap going on, and a clearer and more obvious way to deploy would do most lisps a lot of good.


6 points by kens1 4342 days ago | link

In my experience, the meta-issue is that there are multiple substantial barriers to Lisp adoption, and an uncertain payoff, leading to the question, "Is this really the best use of my time?"

The first substantial barrier is Emacs. Before learning Lisp, first you must learn Emacs. I've tried several times, but keep deciding it's not worth the effort. Imagine for an instant that in order to use Ruby, you need to use some strange Japanese editor, but it's really not bad once you get used to the cryptic commands and rebind your keyboard so it is usable. Do you think this might hamper the adoption of Ruby?

Next, the IDE. I'm willing to put up with Slime's 1980's styling, but I keep ending up in mysterious inferior modes that don't seem to be documented. After reading through Slime documentation, it seems I need to understand Emacs's Lisp modes first.

Then, the REPL. Sure, it's nice to be able to interact. But sometimes I need to write a self-contained script that can take part in the whole Unix environment: things like being part of a pipeline, running as a cron job, running through CGI, running as a command. Python makes this trivial; Lisp makes this difficult or impossible.

Then there's the payoff. Will Lisp actually help me solve any of my problems? That's still extremely unclear. Perl or Python are great for random problems I have, such as "Fetch a web page, pull stuff out with regular expressions, and convert it from iso-8859-1 to UTF-8" or "Analyze a pcap dump file". This is where Lisp runs into the infamous library issues. The problems Lisp solves seem to not be the problems I want to solve.

The issue of a dialect/implementation is somewhat vexing: try reading SICP and Practical Common Lisp at the same time, and discovering that everything is different. Also, the expectation that whatever implementation you pick will be the wrong one for the libraries you want is a bit scary. But overall, I'd say this isn't a fundamental barrier.

The above is meant as an honest answer to your question; I'm not interested in starting a flame war.

I am pleased with DrScheme, by the way.


7 points by jimbokun 4342 days ago | link

"But sometimes I need to write a self-contained script that can take part in the whole Unix environment: things like being part of a pipeline, running as a cron job, running through CGI, running as a command. Python makes this trivial; Lisp makes this difficult or impossible."

This is hugely under-estimated as a reason for lack of Lisp adoption. I would put this way ahead of lack of IDEs.


2 points by lojic 4341 days ago | link

"The first substantial barrier is Emacs. Before learning Lisp, first you must learn Emacs. I've tried several times, but keep deciding it's not worth the effort."

First of all, I don't think it's accurate to say that you must learn Emacs before learning Lisp. Other editors such as vi or IDEs such as ACL & LispWorks are used successfully.

On the other hand, after using IDEs for over a decade, I switched to vim for a couple of years and more recently switched to Emacs. Get the "Learning GNU Emacs" book and it will be trivial to pick up Emacs.


8 points by cchooper 4342 days ago | link

One thing that's helped Rails adoption a lot is that you can get the whole thing up and running in 3 steps: install Ruby, install Gems, get Rails. All of this is available from one page on the Rails site, so it's really easy to try it out. You don't even need a web server; they provide one for you.

Lisp in a Box is the closest thing Lisp has to this, but can anyone seriously say that asking people to learn Lisp, Emacs and SLIME at the same time is the way to get people hooked on the language?

I think what Lisp actually needs is a nice, easy install package: Scheme/Lisp, a nice HTML help/tutorial system and a simple editor, like the one that comes with Ruby (basically Notepad with syntax highlighting and top-level integration <-- this is the key feature). Perhaps you could throw in some kind of web-app package that people can start playing with (just like the one in Arc) and a package system (one as easy to use as Gems would be nice). All of this should be wrapped up in a single installer.

This way, people have a nice, gentle introduction to the language through a simple but expedient system, and can graduate to Emacs/Vi/Eclipse at their leisure.

Oh, and big, colourful, idiot-proof websites with the instructions on them are a big draw too.


5 points by jimbokun 4342 days ago | link

"I don't think there is a better candidate for an IDE target than Eclipse. It is very pretty."

Eclipse is ugly as sin.

Eclipse belongs to that school of UI design that further divides and subdivides the available space into smaller and smaller little tiles and panes until you are squinting at your code through a keyhole and distracted by all the stuff surrounding it. Yes, there are ways around this, but you have to fight the default paradigm. Most other IDEs adopt the same paradigm (to varying degrees).

Also, your whole point about IDEs does not account for the growing popularity of Python and Ruby. I know you point to an online Ruby interpreter, but these languages rose to popularity with most noobies first experiencing them through a text editor + shell/REPL environment. Stop and think about being a noobie downloading Eclipse and firing it up for the first time. What is the first step? How many steps will it take to see a useful result of your labor?

Versus an intro tutorial of:

    >>> print "Hello world"
    Hello World
Congratulations! You're now a Python programmer! (Hope I didn't mess that up.)

Sorry for the rant, but I think many IDEs are largely a cargo cult phenomenon. pg has a quote somewhere about finally figuring out what an IDE is for; something like "it generates all the code that your Lisp macros generate for you!" IDEs are largely band aids to make bad languages not hurt quite as much.


3 points by kennytilton 4342 days ago | link

No, tell us what you really think. :)

I think you are right about one thing, I freaked when I realized Python did not have quality IDEs.

otoh, clearly you have never used a quality IDE. Or you never get beyond about a hundred lines of code in your projects. Or ________? :)


3 points by lojic 4341 days ago | link

I've used Visual Studio, JBuilder, Eclipse, etc., and for the languages I used them for (C++, C#, Java), I felt the IDEs were essential. When I switched to Ruby, the same quality of IDEs weren't available, but I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE, and I actually enjoyed the lighter weight environment quite a bit.

Would I be even more productive in Ruby with an IDE? Possibly. But it's possible that more powerful languages lessen the need for an IDE. Your love of ACL for Common Lisp would be a counter data point, but it's only one data point. Well, Avi Bryant would tout the Squeak IDE as a great advantage in using Smalltalk. In my case, it's somewhat moot given the lack of IDEs for Ruby. As I develop more in Lisp, I'll be better able to judge the effectiveness of the available IDEs.

Tens of thousands of lines of Ruby haven't been a problem for me with just vim, and now Emacs, and I don't think hundreds of thousands of lines would be a problem either if designed well.


5 points by kennytilton 4341 days ago | link

"I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE"

Bingo, and that is why we do not see fancy IDEs for agile languages. The corollary being horrid languages have great IDEs because lawdy we need something!

One thing going on here is reflection. If my language has that I can toss off my own rough IDE-ish hacks in minutes and then they do exactly what I want, too, so the demand just never develops for off-the-shelf IDEs.

I work with tens of KLOC of Common Lisp at a time. I like sitting in a backtrace and using a keychord to jump to a functions definition, or sitting in the inspector and jumping to a class definition in source or with a diffeent keychord to the class in a graphical class browser. ACL integrates everything (except, strangely "who calls?", tho that is available via an unexported function), so I just sail all over my big code bases as fast as I can think, click, and keychord. Some folks, btw, feel the same about Lispworks and Slime, we need an IDE smackdown some day to comapre. :)


10 points by sacado 4342 days ago | link

have a look at DrScheme. It is a wonderfull IDE for mzscheme. As a bonus, you can even use it to run arc !


2 points by tokipin 4342 days ago | link

true, i'm not sure why it escaped my mind. probably for a couple reasons. i use Vim, which i'm very happy with and whose editing power i'm not going to be sacrificing anytime soon. i was speaking from the perspective of a common beginner. and also i just didn't like DrScheme when i opened it up. its UI and graphics could use a lot of work

however that's just nitpicky, and DrScheme fits what beginners need. how reasonable would writing an arc language thing for it be? still, if someone had to choose between creating a DrScheme language thing and an Eclipse plugin, i think Eclipse would be the best choice


3 points by soegaard 4342 days ago | link

It is actually easy to underestimate DrScheme. It's not just for beginners, but also for wizards.


1 point by tokipin 4342 days ago | link

i'm sure it is very good, i was just talking about the graphical aspects of it. maybe you're referring to my mention of Vim, but i wasn't comparing. Vim is IDE-wise completely lacking, but that's something that doesn't bother me, especially due to its superior-to-all-others editing model. and the reason i say Eclipse would be a better choice is just because it's substantially prettier and more familiar


3 points by soegaard 4342 days ago | link

For editing text Vi and Emacs are preferred by quite a few hackers.

However for editing Scheme, DrScheme is the best choice. Not because it is "graphical", but simply because it was designed with one purpose only: writing Scheme code.

It has all the standard stuff (such as: to show the documentation on an identifier just press F1), but it also contain the Scheme specific stuff, that you won't notice until you try some advanced Scheming.

Among the most impressive features is how precise the error messages are reported. This goes for both standard errors, but not the least when macros are involved. If you compare the error reports you get from a few errorneous macros in DrScheme and in a standard Scheme implementation, you'll see that you can save a lot of time if you use DrScheme.

And while at, don't forget try the macro stepper...


3 points by cooldude127 4342 days ago | link

i like drscheme in general. there is one problem with it that made it simply unusable by me. when you hit run in drscheme to reevaluate your definitions, it obliterates my entire history in the REPL. i need to have that stuff around to retry expressions.


3 points by soegaard 4342 days ago | link

That's by design. [I can dig up a reference, if you are interested]

If you want to retry expressions, stay in the REPL and use ctrl-p (one or more times) to recall previous expressions.


1 point by projectileboy 4342 days ago | link

I don't think IDEs are really a big issue. Having said that, I'm just about done with a first cut at a plugin for IntelliJ (my REPL is being flaky). I understand that Eclipse is much, much more popular, but I like IntelliJ much, much better. Plus, I suspect the intersection of Java developers interested in Arc and Java developers who use IntelliJ will be slightly higher.


3 points by Tamerlin 4340 days ago | link

A good IDE makes programming easier, but I don't think that Eclipse fits the bill in that regard. I personally prefer NetBeans for a lot of reasons (IntelliJ is much better, and the most recent versions of Visual C#/etc are getting there). I'm not talking about wizards here; there are some things that I like using wizards for, mainly things that are just tedious like creating web service clients for a provided WSDL, I'm talking about useful things like organizing and navigating source code.

A good IDE helps you with large projects, especially with multiple developers, and a bad IDE is just a glorified editor with a lot of extra clutter. Eclipse is that with a lot of extra bugginess, IMO. Given all of the assinine bugs I've found in it, I'm frankly amazed that anyone uses it, let alone uses it as a platform for their own tools (Flex?).

I'm just getting into Lisp again after being introduced to it in college before learning how to program, and I'm liking it quite a bit. I'm using JEdit and DrScheme for the most part, and it's working pretty well so far. But I'd rather use IntelliJ or Visual C# or SharpDevelop, because I'm lazy and don't want to waste time with stuff that isn't actually programming.

I think one reason that Rails has done so well is that the vast majority of the people who use it don't write much code. Most of the code that they "write" is stuff they got from someone else via one of the forums. It's stunningly easy to build a reasonable though generic-looking site in Rails without writing much code, which makes it great for newbies.

For large projects, in my experience Rails falls flat on its face. When I ditched that company, our developers were spending about as much time running unit tests as they were writing code. That ended up being about as agile as a beached trimaster with torn sails. (Rails didn't force us into that approach, nepotism did. A collection of small Rails apps would have worked pretty well.)


2 points by eds 4342 days ago | link

Some parts of the article are pretty unfounded, but there are some fragments of truth among the rubble.


1 point by cchooper 4342 days ago | link

Speaking of eclipse plug-ins, someone just posted this to another article: