Django’s Structure – A Heretic’s Eye View

Django’s Structure – A Heretic’s Eye View

The most common complaints from those first starting out with Django is “it’s too hard to understand”, or “It’s too complex”. You may even be thinking this yourself right now.

The thing is, Django isn’t hard or complex at the fundamental level. Yes, it has it’s quirks and, yes, big Django projects can be very complex beasts, but bottom line: Django is a very logically structured framework built on the easiest to learn programming language available (Python).

As an educator, I have spent a lot of time trying to work out why people find Django complex and hard to learn. Thinking on this has led me to the somewhat heretical conclusion that it’s not your fault, we’ve just been teaching it wrong.

Remember all the books and tutorials that start with “Django is a Model-View-Controller (MVC) framework…”? (This is cut and paste from one of my books by the way, so I am definitely admitting to being as guilty as anyone in this).

Stating up front that Django is an MVC framework gets one of two responses:

  1. Beginners say “What the heck is MVC? *groan*. Guess that’s one more fricking thing I have to learn!”
  2. More experienced programmers say “Aha! It’s just like Framework X.”

In both cases, they’re almost entirely wrong.

Django is a Loosely Coupled Framework

If you can indulge me a minute, and purge your brain of your favorite Three Letter Acronyms (TLAs), there’s an easier way to understand this.

First step is understanding that Django is not the result of an academic exercise, nor is it some guru’s idea of cool – it was created to solve a particular set of problems in a busy and complex news organization. At the center of this set of problems were three very different needs:

  1. The data guys (and gals) needed a common interface to be able to work with disparate data sources, formats and database software
  2. The design teams needed to be able to manage the user experience with the tools they already had (HTML, CSS, JavaScript etc.)
  3. The hard-core coders needed a framework that allowed them to rapidly deploy changes within the system that kept everyone happy

Key to making this all work was ensuring that each of these core components – data, design and business logic – could be managed independently, or to use the correct computer parlance – the framework had to employ loose coupling.

Now, it’s important to understand that I’m not trying to say Django is doing anything magic or new here, nor were the problems Django’s creators faced unique. The creators of Django are very clever guys and it’s absolutely certain that they were aware that MVC was a well established design pattern that would help solve their problems.

My point is that it’s highly unlikely any of them ever said “Hang on boys, we need to change this code because Wikipedia says a controller should … “.

You don’t need get hung up on semantics – you can safely forget about the confusing TLAs and whether Django is like Framework X and concentrate on what Django is.

Django’s architecture consists of three major parts:

  • Part 1 is a set of tools that make working with data and databases much easier
  • Part 2 is a plain-text templating system that can be used by non-programmers; and
  • Part 3 is a framework that handles communication between the user and the database and automates many of the painful parts of managing a complex website

Parts 1 and 2 are instantly relatable (see Figure 3.1):

  • Django Models are the tools we use to work with data and databases; and
  • Django Templates provide a designer friendly plain-text templating system

But what about Part 3? I hear you ask, isn’t that the controller, or a Django view?

Well, no. Which leads me to heresy #2:

A Django View is not a Controller

Check out Figure 3.1, does it look familiar?

Django MTV Pattern

Figure 3.1: The somewhat misleading Django MTV diagram.

This is one of my diagrams, but there are plenty of similar versions out there. A common way of explaining Django’s architecture in terms of MVC is to describe it as a Model-Template-View (MTV) or Model-View-Template (MVT). There’s no difference between MTV and MVT, by the way – they’re two different ways of writing exactly the same thing, which just adds to the confusion.

The misleading part of this diagram is the view. The view in Django is most often described as being equivalent to the controller in MVC, but it’s not – it’s still the view.

Figure 3.2 is a variation on Figure 3.1 to illustrate my point.

Django MTV Stack

Figure 3.2: A slighty different view of Django MTV “stack”

Note how I have drawn a line between the client and server side. Like all client/server architectures, Django uses request and response objects to communicate between the client and the server. As Django is a web framework, we’re talking about HTTP request and response objects.

So, in this simplified process, the view retrieves data from the database via the model, formats it, bundles it up in an HTTP response object and sends it to the client (browser).

In other words, the view presents the model to the client as an HTTP response. This also happens to be the exact definition of the view in MVC, or to quote Wikipedia (not the most definitive source, I know, but close enough):

“The view means presentation of the model in a particular format”

Trying to bend the definition of a Django view to fit a particular viewpoint inevitably leads to one of two things:

  1. Confused programmer puts everything in views module; or
  2. Confused programmer says “Django is too hard!”, and goes and watches TV instead

So to get away from our M’s and T’s and V’s and C’s, Figure 3.3 presents a more wholistic view of what Django’s architecture looks like.

A more wholistic view of Django's architecture

Figure 3.3: A more wholistic view of Django’s architecture

The first point of confusion we can clear up is where to put a particular function or class:

Does the function/class return a response?

  • YES – it’s a view. Put it in the views module (views.py)
  • NO – it’s not a view, it’s app logic. Put it somewhere else (somewhere_else.py)

We’ll discuss the somewhere else part in the next section of this chapter.

The next point to note is that the Django framework encapsulates the model, view logic and business logic. In some tutorials, it’s been said that the Django framework is the controller, but that isn’t true either – the Django framework can do much more than respond to user input and interact with data.

A perfect example of this extra power is Django middleware which sits between the view and the client-side. Django’s middleware performs critical security and authentication checks before the response is sent to the browser.

So, returning to the two confused responses from the beginning of the chapter:

  1. Beginners – no, you don’t have to learn about MVC because it’s more than likely going to confuse you and lead to more questions than answers
  2. Programmers – no, Django is not like Framework X and trying to think it is, is likely to confuse you and lead to more questions than answers

Now that we have got that out of the way, let’s get on and have a look at the structure of a Django project.

Django Project Structure

Django doesn’t require you to build web applications in any particular way. In fact, many billions of electrons have been sacrificed discussing the One Best Way to structure a Django project. We’re all pragmatic programmers here, so we’re not going to play that game.

Django does, however, have a default way of doing things, and there is a definite underlying logic to it that you need to understand to become a professional Django programmer.

The fundamental unit of a Django web application is a Django project. A Django project is made up one or more Django apps (Figure 3.4)

Django Project Structure

Figure 3.4: Django’s Project Structure

A Django app is a self contained package that should only do one thing. For example a blog, a membership app or an event calendar. You will notice at the bottom of Figure 3.4 there’s an extra package called Django Apps.

This is another case where Django’s logic carries right through the framework – Django itself is a collection of apps, each designed to do one thing. In the case of Django’s built-in apps, they’re all designed to make your life easier, which is a Good Thing.

While the built-in apps are invisible in your project tree, you can see them in your settings.py file:

# ...\myclub_project\myclub_site\myclub_site\settings.py

# partial listing

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

You can see that Django has added a number of apps to your project automatically. There are also many other built-in Django apps that aren’t added by default that you can add to the INSTALLED_APPS list. When you add your own apps to a Django project, you also add a link to the app configuration class to this list.

You can see this logic clearly in what Django has created for you so far. Open up your \myclub_project folder. The folder structure should look something like this:

# ...\my_clubproject

\env_myclub
\myclub_site        <= This is your Django project
    \myclub_site    <= This is a Django app
    db.sqlite3      <= Your project database
    manage.py       <= Django project management utility

Let’s examine these files and folders in more detail:

  • The env_myclub folder is where the files for your virtual environment are stored. There are lots of interesting goodies in here for advanced users, but as a beginner it’s best you leave everything inside this folder alone.
  • The outer myclub_site folder is your Django project. Django created this folder and its contents when you ran the startproject command in the last chapter. Django doesn’t care about the folder name, so you can rename it to something meaningful to you.
  • Inside the outer myclub_site folder are two files:
    • db.sqlite3. The database created when you ran the migrate command; and
    • manage.py. A command-line utility for executing Django commands from within your project.
  • The inner myclub_site folder is your Django website application. This is the one application that Django creates automatically for you. Because Django is a web framework, it assumes you are going to want a website app.

This should be making a whole lot of sense by now, but I bet there is one thing that’s still a bit confusing – the two myclub_site folders.

A very common complaint from programmers just starting out with Django is how confusing it is to know which folder they should be working in when there are two folders named the same. Django is not alone with this convention – Integrated Development Environments (IDEs) like Visual Studio create a project folder and application folder with the same name. But just because it’s common, that doesn’t mean it isn’t confusing.

As I said a a moment ago, Django doesn’t care what you name this folder – so let’s go ahead and commit heresy #3, breaking thirteen years of Django tutorial convention while we are at it, and rename the folder!

In this case, we are going to rename it to “myclub_root”.

Once you have made the change, your folder structure should go from this:

\myclub_project
    \myclub_site
        \myclub_site

To this:

\myclub_project
    \myclub_root
        \myclub_site

Now we’ve taken care of that source of confusion, let’s have a look inside the myclub_site website app Django created for us:

# \myclub_project\myclub_root\

\myclub_site
    __init.py__
    settings.py
    urls.py
    wsgi.py

Looking closer at these files:

  • The __init__.py file tells Python that this folder (your Django app) is a Python package.
  • settings.py contains the settings for your Django project. Every Django project must have a settings file. By convention, Django puts it in your website app, but it doesn’t have to live there. There are proponents for other structures and I mentioned earlier, but here we’re going to stick to the default.
  • urls.py contains project-level URL configurations. By default, this contains a single URL pattern for the admin. We will be covering URLs a bit more later in the chapter, and in great detail in Chapter [TODO].
  • wsgi.py enables WSGI compatible web servers to serve your project. We’ll discuss this file more in chapter [TODO] when we cover deployment.

Now that we’ve had a good look at the basic structure of a Django project, it’s time to take the next step and add our own Django app.

Creating your own Django Apps

You might have noticed that there is no real program code in your project so far – you have a settings file with configuration information, an almost empty URLs file and a command-line utility that launches a website that doesn’t really do anything.

This is because, to create a functioning Django website, you need to create Django applications. A Django application (or app for short) is where the work is done. Apps are one of Django’s killer features. Not only do they allow you to add functionality to a Django project without interfering with other parts of the website, but apps are designed to be portable, so you can use one app in multiple projects.

So, let’s go ahead and create our first custom Django app. Our social club website is going to have an events calendar to show upcoming events for the club, so we’re going to create a Django app called events.

Fire up your Python virtual environment, switch into the \myclub_root folder and run the command:

python manage.py startapp events

This is what your command shell output should look like:

... \myclub_project> env_myclub\scripts\activate
(env_myclub) ...> cd myclub_root
(env_myclub) ...\myclub_root> python manage.py startapp events
(env_myclub) ...\myclub_root>

Once you have created your app, you have to tell Django to install it into your project. This is easy to do – inside your settings.py file is a list named INSTALLED_APPS. This list contains all the apps that are installed in your Django project. Django comes with a few apps pre-installed, we just have to add your new events app to the list:

1 INSTALLED_APPS = [
2     'events.apps.EventsConfig',
3     'django.contrib.admin',
4     # more apps
5 ]

Inside every app, Django creates a file, apps.py, that contains a configuration class named after your app. In this case, the class is named EventsConfig. To register our app with Django, we need to point to the EventsConfig class – which is exactly what we are doing in line 2 or our modified INSTALLED_APPS list.

If you were wondering, EventsConfig by default contains a single configuration option – the name of the app (“events”).

Now let’s take a look inside the \myclub_root folder to see what Django has created for us:

\events
    \migrations
    __init__.py
    admin.py
    apps.py
    models.py
    tests.py
    views.py
  • The migrations folder is where Django stores migrations, or changes to your database. There’s nothing in here you need to worry about right now.
  • __init__.py tells Python that your events app is a package.
  • admin.py is where you register your app’s models with the Django admin application.
  • apps.py is a configuration file common to all Django apps.
  • models.py is where the models for your app are located.
  • tests.py contains test procedures that will be run when testing your app.
  • views.py is where the views for your app are located.

Now that we finally have a complete picture of a Django project, we can also answer the question from earlier in the chapter – “Well, if it’s not a view, where does it go?”

When you have code that isn’t a view, you create a new Python module (.py file) inside your app and put related functions and classes inside the file. Note the emphasis on related. If you have a bunch of functions that provide database management utilities, for example, you should put them all in one file. Functions and classes not related to database management should go in another file. You should also try to be descriptive in naming modules – after all, it’s more sensible to put your database functions in a file called db_utils.py than a file called monkeys.py

When creating new modules for your Django project, you should also consider scope. While adding custom modules to apps is far more common (and more portable), you can have project level modules (e.g., Django’s manage.py) and site level modules. In the latter case, your custom modules should go in the same folder as your settings.py file.

The last couple of points might seem blindingly obvious, but it’s important for you to understand that, while Django does have a default logic to it’s structure, nothing is cast in stone; Django is flexible and allows you to expand and modify your project structure to suit the logic of your web application.

Now that we have a thorough understanding of how Django’s projects and apps are structured, the next obvious question, given that we are building web applications is “how do we navigate a Django project?”

To answer that question, we need to check out the final piece of the Django big picture puzzle – URL configurations.

URLconfs – Django’s navigator

There’s on last piece to the Django framework puzzle – the critical communication pathway that matches a request on the client side with a project resource (the arrows between the view and the template in Figure 3.3). Like all web applications, Django uses Uniform Resource Locators (URLs) to match content with a request.

Django’s urls package provide dozens of functions and classes for working with different URL formats, name resolution, exception handling and other navigational utilities, but at its most basic, it allows you to map a URL to a function or class within your Django project.

A Django URL configuration (or URLconf for short) simply matches a unique URL with a project resource; you can think of it being like matching a person’s name with their address. Except in Django, we’re not matching a street address – we’re matching a Python path usings Python’s dot notation.

Assuming you’re not familiar with dot notation, it’s a common idiom in object-oriented programming. I like to think of the dot like a point because the dot points to something. In the case of Python, the dot operator points to the next object in the object chain.

In Django classes, the object chain is like this:

package.module.class.method

Or in the case of functions:

package.module.function.attribute

Some real-life examples:

  • forms.Form points to the Form class in the forms package.
  • events.apps.EventsConfig points to the EventsConfig class in the apps sub-package of the events package. I.e., the apps.py file in you events app.
  • django.conf.urls points to the urls package inside the conf package inside django which is also a Python package!

This can sometimes get a bit confusing, but if you remember to join the dots (sorry, bad pun there), you can usually find out what the dot operator is referring to.

With a URLconf, the path points to a function or class inside a module (.py file). Let’s look at our Django project diagram again (Figure 3.5).

Finding function with Django URLconfs

Figure 3.5: Finding functions and classes with Django URLconfs

To create a URLconf, we use the path() function. The first part of the function is the URL, so in Figure 3.5 the URL is app1/. The path() function then maps this URL to app1.views.some_view().

Assuming your site address is http://www.mycoolsite.com, in plain English we’re saying:

“When someone navigates to http://www.mycoolsite.com/app1/, run the some_view() function inside app1’s views.py file”

Note that a URL doesn’t have to map to a view – it can map to any module in your Django app. For example, you may have a set of wireless environmental sensors that post data back to the server. You could have a custom module called sensors.py that has a function or class that records the sensor data to your database, all without ever touching a view.

And that’s all there is to it. Of course, URLconfs can do a lot more than map a static URL to a function or class, but if you can understand the basics – that Django’s incredibly fast and powerful navigation system is based on the simple concept of matching a URL with a resource – then you have all you need to tie all your Django apps together into a navigable web project.

A Final Note on Writing Django Apps

A common and inevitable question arises once you get your head around Django’s basic structure:

“Where do I start? Should I start with writing my models, the URL configurations, my views? What?”

Well, here’s your final heresy for the chapter: it doesn’t matter.

Some people like to start by building all the models so they can see how the data structure looks, others prefer to build the visual layout first so they start with templates. Others might like to get the basic communication framework in place, so start with views and URLconfs. Others will just start at whatever point seems logical for the project.

Being pragmatic to the bone I am usually in the last group. I try not to get fixated on what someone else thinks is the right or the wrong way to do things and just try to find the simplest and quickest way to achieve the result I want. I also like to work incrementally starting small getting the flow right and building on that to create the complete application, so inevitably end up jumping from one element to another as the application grows.

Your brain is wired different to mine, and to every other programmer. This is a Good Thing. Just remember, an imperfect start to a project is way better than not starting at all. Do what works for you.

Chapter Summary

TODO

Mastering Django 2: Core – Table of Contents