I've used Visual Studio, JBuilder, Eclipse, etc., and for the languages I used them for (C++, C#, Java), I felt the IDEs were essential. When I switched to Ruby, the same quality of IDEs weren't available, but I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE, and I actually enjoyed the lighter weight environment quite a bit.
Would I be even more productive in Ruby with an IDE? Possibly. But it's possible that more powerful languages lessen the need for an IDE. Your love of ACL for Common Lisp would be a counter data point, but it's only one data point. Well, Avi Bryant would tout the Squeak IDE as a great advantage in using Smalltalk. In my case, it's somewhat moot given the lack of IDEs for Ruby. As I develop more in Lisp, I'll be better able to judge the effectiveness of the available IDEs.
Tens of thousands of lines of Ruby haven't been a problem for me with just vim, and now Emacs, and I don't think hundreds of thousands of lines would be a problem either if designed well.
"I found I was much more productive in Ruby with a simple editor than I had been with other languages plus an IDE"
Bingo, and that is why we do not see fancy IDEs for agile languages. The corollary being horrid languages have great IDEs because lawdy we need something!
One thing going on here is reflection. If my language has that I can toss off my own rough IDE-ish hacks in minutes and then they do exactly what I want, too, so the demand just never develops for off-the-shelf IDEs.
I work with tens of KLOC of Common Lisp at a time. I like sitting in a backtrace and using a keychord to jump to a functions definition, or sitting in the inspector and jumping to a class definition in source or with a diffeent keychord to the class in a graphical class browser. ACL integrates everything (except, strangely "who calls?", tho that is available via an unexported function), so I just sail all over my big code bases as fast as I can think, click, and keychord. Some folks, btw, feel the same about Lispworks and Slime, we need an IDE smackdown some day to comapre. :)