Why isn't Lisp more widespread?

Question

I am starting to learn Scheme by the SICP videos, and I would like to move to Common Lisp next.

The language seems very interesting, and most of the people writings books on it advocate that it has unequaled expressive power. CL seems to have a decent standard library.

Why is not Lisp more widespread? If it is really that powerful, people should be using it all over, but instead it is nearly impossible to find, say, Lisp job advertisements.

I hope it is not just the parenthesis, as they are not a great problem after a little while.

Answer 1

Expressiveness isn't always a positive language trait in a corporate environment. Java is extremely popular partly because its easy to learn, easy to write, and easy to read. Mediocre programmers can still be very productive in Java, even if their code is wordy and inelegant.

Furthermore, it's easy to abuse expressive languages. An expert java programmer can refactor poorly written code quickly. The more expressive the language, the more difficult understanding and refactoring horrible code becomes. LISP macros are a good example. Macros are powerful tools in the right hands. In the wrong hands, they can result in confusing and hard to debug code.

LISP is a risky choice for senior management. If things go wrong, no one is going to blame management for picking a popular object oriented language backed by a major corporation like Oracle or Microsoft. Its much easier to hire programmers with experience in popular, easy to learn languages.

Even progressive companies willing to use a more powerful language usually don't choose LISP. This is because many of the newer languages try and compromise by borrowing powerful features from LISP, while staying easy to learn for the masses. Scala and Ruby follow this model. Bad programmers can pick them up quickly and keep writing the same mediocre code that they did in Java. Good programmers can take advantage of the more advanced features to write beautiful code.

Parentheses are not the problem. Haskell is an incredibly powerful and expressive language with a syntax similar to Python or Ruby and it hasn't been widely adopted for many of the same reasons as LISP.

Despite all this, I am hoping...

Clojure has a chance of becoming popular. It runs on the JVM, has great interop with Java, and makes concurrent programming much simpler. These are all important things to many companies.

*This is my perspective as a professional JVM programmer with experience in Java, Clojure, JRuby, and Scala.

Answer 2

Why is not Lisp more widespread? If it is really that powerful, people should be using it all over,

If you believe that languages are chosen for their technical merits, you are in for a soul-crushing disappointment.

Such decisions are made based on Strippers And Steaks. Microsoft can afford them. Oracle can. Sun spent so much money hyping Java that they went bankrupt. Twice. A decentralized, heterogenous, volunteer community cannot compete with that.

but instead it is nearly impossible to find, say, Lisp job advertisements.

Funnily enough, the Lisp companies say exactly the opposite: they constantly have job openings but can't find enough people to fill them. (The same applies to Haskell, ML, O'Caml, Forth, Smalltalk, ...)

Answer 3

I have no experience in working for a real company, but I know why LISP has been hard for me to use.

First of all, this reminds me of this blog post: http://steve-yegge.blogspot.com/2006/04/lisp-is-not-acceptable-lisp.html

The main problem that I have with Lisp is the "which Lisp" question. I usually work on Linux as my main platform, but the things I make need to be compatible with Windows. That means that when I'm evaluating a technology to use, it must make my life easy when working on two radically different operating systems. I don't like this requirement, but to use it on a real project that's a requirement. Now I'll use languages that don't have very good support on Windows for my own personal side projects, but because I never get a chance to write a large software project in them, I don't have the necessary experience.

Now when I was trying to learn a functional language, I really wanted to learn Common Lisp. It seemed like the right thing to do. I started reading Practical Common Lisp as a jumping off point since I really didn't know the built-in functions and needed a project to work on in Lisp. S-expressions were beautiful and easy. All those parenthesis were incredibly beatiful to me as it was clear as day exactly what was happening in the code.

So I try to write my first program in Lisp outside of the book. I wanted a command line tool that would count lines of code and remove trivial lines from the code count. Not the most useful tool, but fun to do. It involves file access, a bit of parsing, and counting. I had implemented the same tool in Python about a week earlier.

I need to access command line arguments. Then I learn there is no standard way to get command line arguments. They're all non-standard features. It's not cross-platform at all. It mostly just gets worse from there as the language doesn't have a lot of libraries built into it. I ended up switching to Haskell and didn't get very far into Common Lisp (so my complaints may not even be valid).

This kind of non-standard thing has always been a pain for me in the past. C++ has this same problem, but with libraries like Boost you can get around those weaknesses.

It also doesn't help that the Lisp syntax for everything other than S-expressions is a little ugly.

Answer 4

IMO, it's mostly due to:

• Poor library support. Sure, there's Quicklisp now, which makes it easy to install libraries, but it doesn't compensate for them being still fairly few, and quite a few are poorly documented or not very well maintained. Compared to Python, there's a good chance that writing non-trivial Lisp application (regardless of the particular dialect) will probably still involve reinventing at least part of a wheel or two.
• Lack of familiarity with the model adopted by development tools. Most of the folks I know are fairly intimidated by having to use SLIME and Emacs, which is understandable for people used to Eclipse and Visual Studio. There are two Eclipse plugins I think, I only tried one of them a few years ago and it was pretty buggy. The whole Lisp ecosystem looks like it's stuck halfway between the days when it was part of a full-fledged Lisp-running system and the modern standard of oh-it's-another-language. Learning Lisp is not limited to learning a new language -- you also need to learn an entirely different way of working and thinking, and while this is okay if you're doing it as a hobby, it's questionable whether it's worth doing it in a business.

Things are starting to look slightly better though, especially with Clojure being around.

Answer 5

I learned LISP a billion years ago in college.

LISP, like FORTH, is great for logic. But most programming is not about logic, it's about manipulating data in boring mechanical ways. For example, at the time there as no way to right-justify numeric output.

LISP is about nested functionality, and people just don't think that way. They think in terms of DO A, B, C, D, then E. Not do A, which involves doing B and C, then D and E. THis involves a kind of concurrency which is confusing. Except for predefined tasks like "file an income tax return", people don't think concurrently, they think sequentially. This is why procedural languages are dominant today.

As a result, produral code liks Java and C can be translated into English easily. LISP code can not; no human language is structured in that way.

So it's great for problem solving, but problem solving is not very important in programming. Data entry, validation, output formatting, all of which LISP was terribly weak at.

Answer 6

I think one problem with Lisp not yet mentioned is that for a mediocre or novice programmer (like myself, I freely admit), it can be difficult to see how you turn Lisp code into a big program. It's easy to write but hard to architect. I don't think any of the concepts are particularly difficult, but the DIY mentality is so strong that I often feel at a loss where to even begin.

With an OOP language like Java or C#, you can use the type system to boost yourself towards a working model, and build off that. With Lisp (or Lua, or Javascript, for that matter) there's this notion that you can go out and do it any way you want. If you want OOP, just make your own OOP system! Except making your own OOP, or learning someone else's, is a new barrier on top of the language before you get usable programs. Plus I always feel like the OOP in Lisp or Lua isn't really there, like I could just ignore it if I really wanted, so what's the point?

In short, I think programming in Lisp requires a great deal of discipline, and that's very hard to come by. Strongly-typed languages and OOP languages both offer a kind of built in discipline, so the programmer can focus their limited reserves on finishing projects instead of tweaking the language.

EDIT: As an analogy that just struck me, it's like if you need to do some wood work and two people offer you their tool boxes. One person's tools are kind of crummy, but would basically get the job done with a bit of effort. The other person has a big bin of parts, but is promising that you can combine those parts to make the best tools you'll ever used, perfectly suited to your grip and of the best quality. You just have to build them first.

Answer 7

I've been wondering the same for long, and I even went to Lisp conferences to try to understand what is this Lisp "dark side" that keeps everyone from adopting it.

I've found no complete decent answer.

Ignorance may be the reason for the missing popularity, but what puzzles me more is that even who for sure knows about Lisp (for example Google - Peter Norvig works for them) is not using it.

The only partial explanation I come up with is that most of Lisp great ideas are now commonplace, the only really important missing one (a hugely important one IMO) is ease of metaprogramming.

Unfortunately I see no easy way to adsorb this concept for other languages because metaprogramming to be nice requires an homoiconic and regular language (I'm talking about general metaprogramming, not the dumbed-down template-only version). In other words it basically requires the Lisp approach to syntax: code is data and data is code. Writing code in a syntax-rich language that manipulates an AST is more difficult because you need to know two languages: how to write the code and how to write the AST. This is especially hard if your AST is fixed and also complex and irregular with a lot of different node types. Lisp instead has a reasonably regular (and extensible!) AST and you already normally code by directly writing the AST.

Metaprogramming is also inherently more difficult (and meta-metaprogramming even more and so on) and most programmers and managers apparently just prefer the "no one would need that" answer.

I find particularly sad that "new" languages like go ended up using text-based metaprogramming when is needed (externals code generators writing text files) and "magic" (i.e. the compiler can do what the programmers cannot do).

I think the solution to the complexity problem are powerful tools and education. The trend is apparently instead blunt tools and pretending the problem is not present.

Answer 8

For web development with Lisp dialects, there can be a bit of a chicken-and-egg problem - because few people use Lisp, hosts either don't allow it or don't make it easy, and because it's not easy, few people use it. But actually, getting Lisp running on a host can be easier than you might think, even if it requires a bit more work than their out-of-the-box PHP service does. I recently got guile Scheme apps working here with just a little effort.

Answer 9

It seems that even CL does not have very good library support. At least according to people who switched from Lisp to Python:

http://www.redmountainsw.com/wordpress/archives/reddit-switches-from-lisp-to-python

Personally, I know some Scheme and enjoy playing with it, but can't imagine doing a non-trivial project in that language.

Answer 10

Being powerful doesn't necessarily imply widespread use. Have you heard of the term 'Optimize for the common case'? Unfortunately as many have told before mediocrity if assured consistently is a lot better for people than big hits with many failures in between them.

This isn't just the case with lisp, but with a lot of technologies. A good touch over Unix text processing tools, awk, sed, perl can save you days of programming. Unfortunately I have seen people take days do these sort of tasks in other tools badly what could have done with these tools more efficiently in minutes. But if one spends all his life in eclipse he will never come to appreciate the value of these things. You may write a huge program with readability and maintainability and all that, but whats the whole point of writing such a program while not writing it could have easily done the job.

Another aspect while designing tools these days how much useful out of the box they are to use them directly to solve a problem. You can't build something too generic and then then say you will hedge all of that through libraries and frameworks. Its a little difficult to work things out that way.A good tool gels well with the environment and problem around it. That's why tools like php,perl and awk continue to remain relevant despite endless trolling and bashing, because they are too useful to throw them away and often they do a lot of work than a general language with many libraries and frameworks bolted on.

Similarly you will see languages like Java/Python are very good and I would say best for certain tasks. Python especially is very good and easy to learn and write. Also these languages do really good if your data end points are standard. Some sort of a database or a XML or data of that sort. Basically structured data.

Lisp will be there around for long time, but not necessarily widespread. As you will see each tool has its niche. And they solve certain set of problems very well out of the box. I think and I'm sure lisp is doing good in areas and problems for which its designed to solve well.

Answer 11

IMO, it's mostly a matter of poor timing: Lisp was old (and almost by definition no longer exciting) long before it became practical for most people or uses. Clojure (for one example) stands a much better chance. Its success will depend far more on being perceived as new, fashionable and cool than anything as practical as interoperation with Java (and everything else that runs on the JVM) though.