How should I organize my source tree?

Question

I am an individual developer working, largely, on web-projects (W/LAMP) and, at times, on C/C++ (non-GUI) projects of about average scale.

I often struggle with structuring my source-code tree. In fact, usually, I don't complete a project without dumping the entire tree and rearranging the pieces three-four times which really takes up a lot of effort and moreover the end result does seem like a compromise.

Sometimes, I end up with over classification of source - very long tree of folders and sub-folders. At other times, I simply end up concentrating all files in a particular folder based on the larger purpose they serve and thereby leading to 'chaotic' folders in the source.

I would want to ask:

• Are there any principles/logic/best-practices that can help me better at structuring my source tree?
• Are there any graphical/diagrammatic techniques (for eg.: DFD in case of dataflow) that can help me visualize my source tree beforehand based on the analysis of the project?
• What strategy to adopt to structure multi-media files-tree associated with the project?

About the bounty: I appreciate existing answers with the members sharing their own practices, however, I'd like to encourage more general and instructive answers (or resources) and more responses from the members.

Answer 1

I can't really give you much advice related to webprojects, but here's how I structure my tree in a programming project (mainly from a C/C++ perspective):

• /
  • src — Source files written by myself
  • ext — Contains third-party libraries
    • libname-1.2.8
      • include — Headers
      • lib — Compiled lib files
      • Donwload.txt — Contains link to download the version used
  • ide — I store project files in here
    • vc10 — I arrange project files depending on the IDE
  • bin — Compiled exe goes here
  • build — The compiler's build files
  • doc — Documentation of any kind
  • README
  • INSTALL
  • COPYING

A few notes:

1. If I'm writing a library (and I'm using C/C++) I'm going to organize my source files first in two folders called "include" and "src" and then by module. If it's an application, then I'm going to organize them just by module (headers and sources will go in the same folder).

2. Files and directories that I listed above in italics I won't add to the code repository.

Answer 2

The source tree layout should reflect the architecture; as a corollary, a well-structured architecture can lead to a well-structured source tree layout. I suggest reading up on the POSA1 Layers pattern, attempting to fit your architecture into a layered structure, then naming each of the resulting layers, and using that as a basis for your source hierarchy. Taking a common three-tier architecture as a baseline:

• presentation/webService (present a web-service interface to our business logic)
• logic/* (business logic modules go in here)
• storage/sql (back-end storage APIs here - this uses a SQL interface to store to a database)
• util/* (utility code - usable by all other layers, but that does not refer outside util, goes here)

Note that the layers do not contain code directly, but rather are strictly used to organize modules.

Within a module, I use the following sort of layout:

• <module> (path to module directly; defines modular interface)
• <module>/impl/<implName> (a specific implementation of the modular interface)
• <module>/doc (Documentation for using the module)
• <module>/tb (unit-test code for the module)

where the <module> is located in the repository according to the layer to which it belongs.

Answer 3

The Maven Standard Directory Layout is kind of specific to Java, but it may serve as a good basis for other types of projects as well.

Here is the basic structure (you could replace the java directories with php,cpp, etc):

src/main/java       Application/Library sources 
src/main/resources  Application/Library resources  
src/main/filters    Resource filter files 
src/main/assembly   Assembly descriptors 
src/main/config     Configuration files 
src/main/webapp     Web application sources 
src/test/java       Test sources 
src/test/resources  Test resources 
src/test/filters    Test resource filter files 
src/site            Site 
LICENSE.txt         Project's license 
NOTICE.txt          Notices and attributions required by libraries
README.txt          Project's readme

The structure basically breaks down to src/main and src/test then grouped by type.

Answer 4

Ideally, the organisation has a single repository, the structure of which is intended to increase engagement between engineering & business and promote reuse.

...\products\
...\products\productName\
...\products\productName\doc\

...\systems\
...\systems\systemName\
...\systems\systemName\doc\
...\systems\systemName\res\
...\systems\systemName\build\
...\systems\systemName\test\

...\library\
...\library\libraryName\
...\library\libraryName\doc\
...\library\libraryName\build\
...\library\libraryName\test\

...\devops\

products

One folder per product; helps communicate how the software supports the business.

Ideally, each "product" is little more than a configuration file indicating which systems to invoke and how they are to be configured. The doc subfolder could contain the top-level brief\spec & any promotional materiel etc...

By separating products and systems we communicate the potential of reuse to the customer-facing side of the business, and break down per-product silos. (This contrasts with the "product line" approach to the same problem)

systems

One folder per system; helps communicate the primary capabilities & opportunity/value of the contents of the repository.

1. "Configuration management" files specifying build & deployment environments.
2. System-level testing configuration (could be significant quantity).
3. Top level logic & functionality; most heavy lifting being done by library functions.

library

Reusable components invoked by various systems. Most development activities are organised around the production of libraries, rather than systems, so reuse is "baked in" to the development process.

devops

Build, Continuous Integration & other Development Automation functionality.

Conclusion

The source tree is a key piece of documentation, and shapes the approach, structure and psychology of the business' relationship with its proprietary technology.

The drivers for this approach are explained in a bit more depth in my answer to this question: https://softwareengineering.stackexchange.com/questions/43733/who-organizes-your-matlab-code/59637#59637

Answer 5

I dont really know about conventions but all of my main projects are done using Symfony Framework and i have get used to a tree structure like follows:

root/

• apps
• app_name
  • config (app specific config files)
  • lib (app specific php files)
  • modules ( modular distribution of functionality)
    • module_name
      • templates (html)
      • actions (php code)
• confing (project config files)
• lib (php code that could be use in hole project)
• model (classes that represent the project information)
  • base
• form (php files that handle forms, this could be quite difficult to achieve without symfony)
  • base (base form classes)
• web
• css
  • images
  • file.css
• js
• log (log files that might be generated)
• data (data specific information, like sql patches, or whatever)
• sql
• plugins (libraries used that could be merged with any app of the project)

If you are interested, please read the symfony documentation on the matter for further inquired (MVC and Code Organization on Symfony).

Answer 6

My recommendation is to download a variety of frameworks or engines and see how huge development teams handled their folders layout.

There are so many ways to organize files that it's better to choose one, and try to stick to it at any given project. Stick to a particular convention until completion or revamping to avoid bugs and loosing unnecesary time.

You can download Laravel, Symphony or Codeigniter frameworks for web projects to have an instant folder layout that works.

So I will try to convey a folders layout common to any development:

MVC (Model View Controller) gives a good paradigm of organization.

Root source code could be src (C++) or app (web development)

A file structure that does not have a clear objective for the classes that it groups definitely will cause confusion. It's not only to organize code, instead it can sustain auto-loaders, class factory, wrap local storage, remote storage and namespacing.

This folder structure is derived and simplified from Laravel Framework. My preference on this post is plural naming but I use singular words in my projects.

---

src/storage (models/file-storage/api/mysql/sql-lite/memcached/redis implementations)

src/repositories (A wrapper of 'storage implementations' with some storage logic, a common interface and return result convention.)

src/services | logic | entities (App bussiness logic)

src/controllers (Used on web development to route server requests to your services)

src/modules | systems (Modular systems that extend your framework general functionality. Services can use modules but not viceversa)

src/helpers (Helper or wrapper classes like eg. string manipulation. Many times this could be on libs|vendor when third party)

src/types (Named enums)

public | build | output (web or c++)

config (Setup files. YAML is becoming popular for cross-platform config files)

cache

logs

lang (en/es/ru/...)

bootstrap (Starts the framework and app)

docs (Documentation written in markdown format .md)

tests (Unit testing)

database/migrations (Create database structure from scratch)

database/seeds (Fills your database with dummy data to test)

libs | vendor (all third party software. 'libs' on C++ and 'vendor' usually on php)

assets | resources (images/sounds/scripts/json/any media)

Answer 7

What I'm trying to do for each project is similar as :

• src - source files, a folder for each namespace/package to easily retrieve files (even header files for C/C++)
• ext - for externals/third-party libraries, it is simple to add externals (such as SVN repositories). Inside, a folder for each libraries (binaries and include files)
• bin - for built binaries, could be quickly exported for release
  • inc - for C/C++ headers file (copied by IDE/makefile/etc...)
• out - for all temporarily generated files (.class, .obj etc...) and it could be ignored (for example by SVN)
• doc - for any documentation, usually generated with Doxygen
• res - by placing resources here, it's possible to separate text source files and binary resources used by the program. I don't really have specific hierarchy inside.
  • config - for some configuration files
  • drawable - for some pictures or icons

All IDE's files or makefiles are saved directly at the root if you only use one of them.

Answer 8

I like the ideas presented in this page www.javapractices.com/topic/TopicAction.do?Id=205. Basically, the recommendation is to organize your project into features(or modules, components). In addition to the reasons presented there:

1. Less cognitive load when you think about the scope of the code you are working on since you have a guarantee that any code in the feature your are working on is "feature-private".
2. There is an added sense of security when you are guaranteed that you are only modifying code for given a feature. E.g., you won't break anything other than the feature you are working on. Again this is because of the "feature-private".
3. Less cognitive load simple because there is less files you can see for a given package. I am sure everyone has seen a package that contain more than 15 files.

Note this is focused on Java packages (aka namespaces). For big projects, I do recommend, for the same reasons, dividing the project into multiple projects (as in multiple maven projects) that represents a business feature. For maven projects, I recommend this reading.

So far, the projects I was/am involved in does not follow these. There are many reasons, but here's a few:

1. The misunderstanding of Java default access modifier (most misunderstood access modifier as per this book)
2. "Argumentum ad populum": Prevailing culture of package-by-layer (probably caused by Reason #1)

I think there is a missed opportunity to prevent complexity if project source organization is not taken seriously at the start of the project as the architect Alexander said:

"As any designer will tell you, it is the first steps in a design process which count for most. The first few strokes, which create the form, carry within them the destiny of the rest." - Christopher Alexander

Depending on the size & complexity of a project, the missed opportunity to cut costs or ROI can be really big. (I'm interested to see a study to see the exact numbers for this)

Answer 9

I do something like this. Works well for a cross platform game I'm doing in my spare time. Unfortunately in work, things are much less organized...

Output                      <-- Build outputs
Docs
External
   <libname>
      Include
      Lib
Data
<ProjectName>.xcodeproj
<ProjectName>VS2010
Source
Temp                        <-- Intermediate stuff from builds and other tools
Tools

Answer 10

For my teams, we try to enforce a standard structure across projects to make is easy to find things as the team switches context and to avoid having to relearn each time through. Not all projects need all systems so we start with the minimal set.

/Source/Component/Language

/Source/Component/3rd Party/

/Documentation/Requirements

/Documentation/Design

/Tests/Automated/Unit

/Tests/Automated/ToolName

/Tests/Manual

This results in some duplication, particularily under the 3rd Party code and libraries, but at least we never forget the answer to something like "What uses the RogueWave Editor?"

Answer 11

With object orientated languages, you have the ability to build namespaces. That logical breakdown used to separate parts of the application to avoid coupling is the primary source of logical file location breakdown. Using coupling as a reason for breaking apart namespaces is a good place to start http://en.wikipedia.org/wiki/Software_package_metrics.

Others have spoken about setting up the project in relation to build, but once you get into the source itself, it's about what makes sense - just use how you logically break apart the code anyways.