Empirical evidence for choice of programming paradigm to address a problem

Question

The C2 wiki has a discussion of Empirical Evidence for Object-Oriented Programming that basically concludes there is none beyond appeal to authority. This was last edited in 2008. Discussion here seems to bear this out: questions on whether OO is outdated, when functional programming is a bad choice and the advantages and disadvantages of AOP are all answered with contributors' opinions without reliance on evidence.

Of course, opinions of established and reputed practitioners are welcome and valuable things to have, but they're more plausible when they're consistent with experimental data. Does this evidence exist? I know that evidence-based software engineering is a thing, but can I practice it in this context? Specifically, if I have a particular problem P that I want to solve by writing software, does there exist a body of knowledge, studies and research that would let me see how the outcome of solving problems like P has depended on the choice of programming paradigm?

I know that which paradigm comes out as "the right answer" can depend on what metrics a particular study pays attention to, on what conditions the study holds constant or varies, and doubtless on other factors too. That doesn't affect my desire to find this information and critically appraise it.

It becomes clear that some people think I'm looking for a "turn the crank" solution - some sausage machine into which I put information about my problem and out of which comes a word like "functional" or "structured". This is not my intention. What I'm looking for is research into how - with a lot of caveats and assumptions that I'm not going into here but good literature on the matter would - certain properties of software vary depending on the problem and the choice of paradigm.

In other words: some people say "OO gives better flexibility" or "functional programs have fewer bugs" - (part of) what I'm asking for is the evidence of this. The rest is asking for evidence against this, or the assumptions under which these statements are true, or evidence showing that these considerations aren't important. There are plenty of opinions on why one paradigm is better than another; is there anything objective behind any of these?

Answer 1

For the previous take, see Revision 1 of this answer. However, the comments and edits to the question further clarify what the question is seeking and allow me to be more clear.

Yes, evidence based software engineering (EBSE) is a thing. There appears to be a few efforts toward EBSE databases, such as this one at Durham University and SEED, which was started by a professor at Cal Poly. All of these efforts, as well as ones discussed in a number of papers that can be found through the IEEE Xplore server or the ACM Digital Library (subscription or purchase required for many papers in both), are based on evidence-based medicine. They provide literature reviews of published empirical (observation and experiment) data. In fact, including "literature review" in a search string on any publication search yields information on most topics - over 14000 hits on the ACM and over 1000 on the IEEE database (when limited to only computing topics).

Looking at the general types of topics discussed in these EBSE databases and literature reviews, I see a common thread - they tend to be technology-independent. The emphasis appears to be mostly centered around process and methodology rather than the specific tools used to conduct software engineering.

So, this concept exists in software engineering. Academia is aware of the evidence-based concept and can successfully apply it to software engineering.

Specifically, the question addresses applying EBSE techniques to the selection of a paradigm seems difficult, due to the sheer number of variables involved, forcing assumptions to be made as well as reducing the ability to repeat the experiment or observation. It's said right in the question - "which paradigm comes out as "the right answer" can depend on what metrics a particular study pays attention to, on what conditions the study holds constant or varies, and doubtless on other factors too". Although that doesn't mean that studying which paradigm is "best" in a given situation, it makes any kind of literature review of these documents incredibly difficult to complete and still extract information across them.

There definitely isn't a "turn the crank" solution to choosing a paradigm.

Given a programming paradigm, you can find studies in the various academic and industry databases about how that paradigm influences various aspects of software development - quality, defects, efficiency, and so on - under various specific conditions, ranging from the knowledge and experience of the team to the problem domain. Any rigorous paper should clearly identify the conditions under which the data was gathered and the assumptions. The problem becomes trying to isolate the factors that make it good in each of those conditions.

Academically, there are some statements that are easy to research. For example, the statement that the functional paradigm is well suited to applications that require concurrency stems from the Church-Rosser theorem. Because of this, it's likely that a software system written in a functional language will have fewer defects related to concurrency than the same system written in a procedural or object-oriented language.

However, from a practical standpoint, a software team can't always use "the best" tool or technique for the job just because research indicates it. Although we strive for producing the highest quality software systems, we do operate within constraints. Often, I see these constraints minimized (or even removed from the equation) when discussing the effectiveness of any methodology.

As a practitioner, when I'm involved in choosing technologies to use, I try to identify the best possible tools. But then I constrain myself to what is known and well-understood by the team that I have. Going back to my previous example, if I have a team well-versed in building concurrent applications in C++ and no experience in Haskell, it doesn't make sense to propose building the system in Haskell as I likely won't be able to make schedule and budget constraints, and my quality will likely suffer due to a lack of experience in the toolchain.

To recap, evidence-based software engineering is generally a good thing that exists and literature reviews do exist and are readily available. However, there are aspects of software engineering where applying evidence-based approaches offer little value. The selection of a programming paradigm to a large scale development effort is one of these.

If you want to find out about how object-orientation addresses reusability or defects in functional programming - you'll easily find publications on those. However, I didn't find (nor would I put any amount of trust in) a publication that was able to effectively address paradigm selection across the broad range of real-world software engineering projects.

Answer 2

I've been reading The Art of Unix Programming by Eric S. Raymond. It has some very interesting historical insights about things we now take for granted. He cites some good studies from IEEE software that use empirical evidence like defect density. That might be a good source if you're looking for academic-style studies.

Even techniques like modularizing using functions were not always common practice. One of my favorite quotes from the book so far:

Dennis Ritchie encouraged modularity by telling all and sundry that function calls were really, really cheap in C. Everybody started writing small functions and modularizing. Years later we found out that function calls were still expensive on the PDP-11, and VAX code was often spending 50% of its time in the CALLS instruction. Dennis had lied to us! But it was too late; we were all hooked...

-- Steve Johnson

There are really two problems with getting too empirical, though. The first is that code quality is a very subjective thing. Code can be terrible and still be correct. People's perception of a programming paradigm is a very valid metric, because code is written for people to read as much as for computers, if not more.

The second problem is that 50% of developers have below average programming talent. It doesn't matter if your top developer is more productive using functional programming if the "rabble" struggles to write working software using it, let alone beautiful, well-architected software. Likewise with TMTOWTDI programming languages, your top developer is still going to write clean, modular code, but less talented coders write line noise due to the lack of imposed structure.

That's why I think OOP has risen to the top in popularity despite its deficiencies. It's not so restrictive that it hobbles the most talented, but its structure provides a concise way to communicate and impose good design principles on the least talented.

In our line of work we have a tendency to evaluate a solution based too much on its technical merits alone. A successful endeavor has to account for the human side of the equation as well.

Answer 3

There are programming contests that use a computer grading system and allow you to write in various languages and post all kinds of results and things. I bet they have good data for you. Here's a list of 8: http://www.makeuseof.com/tag/8-onlineprogramming-contests-challenge-win/

I imagine you can make meaningful comparisons of solutions to very simple and clear-cut problems, like sum of squares, or Fibonacci series, or drawing a straight line using the Bresenham's line algorithm. Most real-world programming tasks do not have such clear-cut goal posts and every language has its sweet spots. Much of the benefit of a language is subjective. You might find more meaningful data by surveying programmer and client happiness than by counting lines of code or numbers of defects.

I remember when I spent a half day writing one of my first Awk programs, I thought it would have taken me a whole week to do the "same" thing in Java. But that's because my Java solution would have focused on being robust where as the Awk solution was quick and dirty and required some manual tweaking on the input and output and was really throw-away when I was done. Awk and Java are both great, just not for the same things.

I guess what I'm saying is that for real world applications, comparing languages or tools in a meaningful way is extremely difficult - the old apples and oranges issue. Good luck! I'd love to see what you find out.

Answer 4

I've been studying different ways to develop software for 30+ years. There is a dearth of good published evidence on choosing a paradigm.

I put together a large searchable ASCII bibliography. This includes a lot of IEEE and ACM papers and articles. I tag the items with the type of evidence provided. Here are the most common tags:

...
133 =EXPERIMENT
177 =HISTORY
233 =IDEA
267 =SURVEY
338 =ESSAY
395 =THEORY
454 =ADVERT
491 =EXPERIENCE
500 =DEMO

Now search for PARADIGM and count the tags

  1 =ESSAY
  1 =EXPERIENCE
  1 =HISTORY
  1 =IDEA
  1 =POLEMIC
  1 =SURVEY
  1 =TALK

If you want to dig deeper, http://cse.csusb.edu/dick/lab.html and I hope it helps...

Answer 5

It seems that in many cases there isn't a corpus of research big enough or of high-enough quality to allow general conclusions to be drawn about whether one practice in software engineering is better than another. I was specifically looking for research into working in different paradigms, but the lack of availability is not limited to that arena so I'll frame my answer in a broader sense.

A paper from 2004, Evidence-Based Software Engineering by Kitchenham et al, covers quite succinctly the benefits to be derived from an evidence-based approach and the problems with implementing it in software engineering. I won't discuss the benefits side here (it's clear from the question that I would like to be able to work in this way), but the problems are relevant as an answer to the question I asked.

• firstly, if you're not a member of the ACM then you probably can't read the link above at all, which covers the first problem: not all of the extant research is actually available to practitioners.
• much software engineering practice goes on in secret as part of commercially-confidential processes so there's no visibility onto what did or didn't work for those people.
• software engineering is a skilled practice, so it's hard (not impossible, merely hard) to arrange a suitably-blinded study.
• the different parts of the software lifecycle affect each others' outcomes to an extent that's hard to control for in any experiment.
• as evinced by the discussions here, many practitioners do not see "the literature" (or the academic side of software engineering in general) as relevant to their work.

So the answer I'm after is "no", the evidence I'm looking for probably doesn't exist. I should choose my paradigm based on the existing popular criteria of what I know, what's cool and expert opinion.

Answer 6

I do not believe this type of study exists. One would believe it is not the programming paradigm that matters as much as the actual algorithm that is used. For instance given any non-trivial system one that relies on small space algorithms verse one that relies on small time algorithms would generate different metrics. The one that has the better time would most likely be deemed more valid, unless space is an issue then the inverse is true. I find it similar to paving a road. While the algorithm or recipe for making the materials is constant throughout all processes it would be possible one company thinks paving two lanes at once (one on each side of the road) is better than paving two lanes on the same side of the road at once. At the end of the day it doesn't matter as the process of making the black top is still the same, the only difference is the approach. Going back to programming, if you have a team of C developers, write the code in a procedural manner, if you have a team of Java developers write it in OO. Do not get hung up on the paradigm so much as the implementation of the algorithm. Because at the end of the day you can write Java like C and you can try to write C like Java.

UPDATE

To reply to the comment graham left me:
I assume by architecture you mean programming paradigm. If you are going to use Clojure maybe you should hire a team of Clojure programmers. However, based on a quick search Clojure is a Java based language it just so happens to be functional. Given that information I would take the Java programmers (since technically they can just write Java and it will give you the same results) or look for functional programmers such as Haskell developers. Now in terms of choosing what is best it is completely dependent on your team. I would never have a team of relational database experts organize a cloud solution for me nor would I have a team of functional programmers build out an object orient solution for me. You have to use the strengths of the team you have not the glorified vision you have in your head for what a team "should" be able to do.

Answer 7

Different paradigms lead to different solutions. Which fit is 'best' depends largely upon:

• the solution
• the development team
• the operational environment

I know of no such definitive study, and even if there was one I would not trust it.

That is the architect's job.

Replacing the architect's wisdom with the possibly irrelevant conclusions of a study is a recipe for disaster.

Note: a comment mentions deciding upon "the algorithm" and then choosing the language. Algorithms are the central structural mechanism for procedural languages. Object-oriented languages focus on classes and patterns of collaboration/communication. If you're convinced (as the architect) that an algorithm-centric solution is 'best', then stick with procedural or functional languages.

Addendum: not trusting studies is not 'superstition', it is common sense. Scientific experiments must be objective and repeatable. Software projects are highly subjective, but even worse they are unrepeatable experiments. You simply cannot implement a project X with team Y, measure the results, and then roll back time, erase memories, and re-do the same project with the same team. Information discovered or implied by studies may be useful to the architect, but it can never be definitive.