How I Deploy Django Day-to-Day

There are a lot of ways to deploy Django so I think it is one of those topics people are really curious about how other people do it. Generally, in all deploys you need to get the latest code, run migrations, collect your static files and restart web server processes. How yo do those steps, that is the interesting part.

In todays video I go over How I deploy Django day to day, followed by some other ways I have done it. This is definitely a topic you can make as easy or complicated as you want.

Here is the link again: https://www.youtube.com/watch?v=43lIXCPMw_8?vq=hd720


5 Reasons I don't think you should use the django admin

Have you ever needed to quickly modify data or look it up. But while the task is simple the entire process of changing one value frustrates you?

Well, this is very common to me, bad UX is annoying. I quite often have to either lookup, or edit, two disparate, yet related pieces of data, and sometimes it is an exercise in frustration.

An all to common occurrence is I just needed to check the edit history of a "Company" our database. This is a simple process, go to the admin find the Company model, do a search and you have your answer.

Except, it isn't that easy in reality. Lets run through what really happened.

I went to "http://superawesomesite.com/admin/" and logged in.

I then looked at how my model options extended well below the fold of my browser so I had to scroll. No problem I'll just hit "ctrl+f" and search for it. WTF!!! where is my company model?

I then proceed to scroll down and finally find it only to remember it was pluralized. If I had searched for "Companies" I would have been good to go, grrrrr.

I click into and see the list of companies available to me. I see that there is a search feature at the top so I do a search for the relevant information. Unfortunately, I don't know the company name that is what I am trying to find out.

Except, we aren't filtering our search in the admin based on that field. So no results show up.

No problem I can do a list filter on the side. So I set the correct filters and still no luck because someone didn't set all the metadata that was supposed to be set for the company. Again grrrrr.

Finally, I abandoned the admin and opened up the django/python shell and did a query with the model, and in about 30 seconds had the record I needed. Took the id and plugged that into the django admin and I was good to go.

This was an exercise in frustration because the 30 seconds that was taken during development to decide which fields should be filtered didn't let that person imagine the field I needed. Also the fact that whomever added the company didn't add all the correct information, made it almost impossible to find the data. In the end I had to go outside this system that is hailed as one of the greatest tools for django to get the information I needed.

I propose we as a community reduce our reliance on the django admin, especially in production.

I also created a Video on the 5 reasons we shouldn't use the django admin.


Django Model Inheritance - Multi-Table Inheritance

In the last post we talked about Model Inheritance with regards to Abstract Model Classes, it was a great introduction into using OOP with your models to keep your code slim and tidy. You can read it at: Django Abstract Base Class - Model Inheritance

In this post we are going to discuss Multi-Table Inheritance. We will create a base model, that is not an abstract model, and inherit from the model. Then we are going to show you how to get and reference data from that models inheritance.

Multi Table Inheritance

When we talk about multi-table inheritance we are referring to having a top level table with all the fields that are core to that context. Then we are going to have other tables with the more specific fields. You select from the table you want, and then do a join on a relationship to get the parent table data. Since Django does this through models and code, we will talk about it in those terms.

In our example we are going to have a Ticket Model.

class Ticket(models.Model):
    title = models.CharField(max_length=255, blank=True)
    description = models.TextField(blank=True)
    name = models.CharField(max_length=255, blank=True)

This model is a good piece of code because no matter what type of ticket we ever have we are going to have a title, description, and name. This is normal for any OOP we would do as a base class. Since we have this base model we can subclass it to have more specific models with data associated with them.

class InternalTicket(Ticket):
    user = models.ForeignKey('auth.User', null=True)

class CustomerTicket(Ticket):
    product = models.TextField(blank=True)

Now we have 3 models available to us: Ticket, InternalTicket, and CustomerTicket. Each one has a title, description and name as fields. However, CustomerTicket does not have a user field and InternalTicket does not have product field. Also both InternalTicket and CustomerTicket are stored in separate database table from title, description and name. So if you looked up the tables in the database you would see a FK reference to the ticket table, id field and the field you declared. Everything else lives in the ticket table.

Getting and Creating Data

With all that said lets look at creating some data and getting it back out. Because that is really when it makes sense on how it works.

In [1]: CustomerTicket.objects.create(product='Videos', title='Odd Playback', description='doesnt play back fast enough', name='Jane Smith')
Out[1]: <CustomerTicket: CustomerTicket object>

We have created a CustomerTicket object with all of our fields set, and it succeeds. (Further proof it works)

In [2]: Ticket.objects.all()
Out[2]: [<Ticket: Ticket object>]

Here we are getting all tickets from the ticket model, which is including our CustomerTicket we create above.

In [3]: ticket = Ticket.objects.get()

In [4]: ticket.customerticket
Out[4]: <CustomerTicket: CustomerTicket object>

If we reference the customerticket property, which is the related name, it will return back the model so we can get the data from the model. We can look at that in action.

In [5]: ticket.customerticket.product
Out[5]: 'Videos'

In [6]: ticket.description
Out[6]: 'doesnt play back fast enough'

Finally, you can see the description is still on the ticket level and not the cutomerticket model.

How does it work

From the django docs:

multi-table inheritance uses an implicit OneToOneField to link the child and the parent

This means there is a one-to-one relationship between the ticket record and the corresponding subclassed models. Django just handles the translation of it from multiple tables to a single model so you don't really notice it.

CustomerTicket object

As our final look we can look at how the CustomerTicket object we created earlier references the ticket object.

n [1]: customer = CustomerTicket.objects.get()

In [2]: customer.description
Out[2]: 'doesnt play back fast enough'

As you can see django just handles the fact that description is on the model, even though the description data is in the ticket table.

In [3]: customer.ticket_ptr
Out[3]: <Ticket: Ticket object>

The .ticket_ptr property is the reference to the ticket object so if you need to actually use that model you can.

In [4]: customer.ticket_ptr_id
Out[4]: 1

ticket_ptr_id holds the id of the Ticket object in the database giving you another avenue for getting information out of the appropriate place.

Conslusion

You don't often need to use Multi-table inheritance, but when you do need to use it, it is nice that django makes it super simple to use. I have only seen a few instances where it makes sense, tickets being one, and social media posts as another. It provides great value in giving you core data, then when needed get the rest of the data. It does this while allowing you to use the models as separate models as if the inheritance didn't exist. Finally, another way to think of it is as data being used with polymorphism.


Django Abstract Base Class - Model Inheritance

Model inheritance is a very useful and powerful feature of django, but used incorrectly it can cause a lot of confusion. Lets go ahead and take a look at some of those.

Have you ever been looking through other peoples code and seen a model class Meta with abstract = True? That is the first type of inheritance we are going to look at, abstract base models. There is also multi-table inheritance, proxy models, and multiple inheritance available.

Abstract Base Classes

In this post we are going to focus on Abstract base classes. This is the easiest type of model inheritance to really understand because it works very much like normal inheritance. In OOP when you inherit from another object you get all of its members plus your own. You can modify those members or leave them the same. Abstract base classes are the same.

Here are two example classes

class Customer(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField(max_length=100)
    address = models.CharField(max_length=100)
    purchase_history = models.ForeignKey('cart.Invoice')

class Staff(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField(max_length=100)
    address = models.CharField(max_length=100)
    bio = models.TextField()
    position = models.CharField(max_length=20)

As you can see these two models are legitimate models with fields, but they do have common elements. In a lot of cases it is fine to leave them this way. However, if we use these same fields yet again it would be a good idea to use an abstract base class.

class ConactInfo(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField(max_length=100)
    address = models.CharField(max_length=100)

    class Meta:
        abstract = True

class Customer(ContactInfo):
    purchase_history = models.ForeignKey('cart.Invoice')

class Staff(ContactInfo):
    bio = models.TextField()
    position = models.CharField(max_length=20)

We now have a ConactInfo abstract base class because we added abstract = True to its Meta. This will tell django and the migration system this isn't a model we can use to store data with.

However, we can inherit from it so that each model that we subclass with it has the fields, methods, and properties of the abstract model. At this point if you did ./manage.py makemigrations it would create a migration file creating only Customer and Staff tables in the database.

Will it Work with Existing Models?

Lets say you have several existing models in your code base you want to abstract into base classes. Will it "just work"? Yes it will. Once I found out about abstract classes I went and made several then ran a makemigrations and nothing happened because to django nothing changed you just refactored you code.

Conclusion

Hopefully this gives you a place to start with doing model inheritance. It is very useful, but much like OOP don't go crazy with it and have a lot of abstract classes. You can be so abstract you have no idea what is going on.

My rule of thumb, whether good or bad, is if I have more than 3 fields repeated in only 2 models I just leave them alone. If I need to add them to a third model I will evaluate if it makes sense to use an abstract model. If I can name the abstract model and it makes sense like ContactInfo then there is enough context to create the new model, so I will. Otherwise I will put it off until the code is screaming for an abstract base class.

Stay tuned for the next post on Multi-table inheritance.

What has your experience been?


GoDjango Podcast Episode 1 - Interview with Luke Crouch

Listen in as we talk to Luke Crouch about upgrading from Django 1.4 to 1.7, MDN and discuss a potential move from a dedicated data center to AWS.


Working with Environment Variables in Python

If you look at modern code deployment practices, like in the 12 factor app, environment variables are very important for keeping secret information secret, or server specific information set for that server. All while not having long crazy settings in a settings file for every single server and computer. It helps keep things slimmer in your code along with the following quoted from the 12factorapp site:

    * Resource handles to the database, Memcached, and other backing services
    * Credentials to external services such as Amazon S3 or Twitter
    * Per-deploy values such as the canonical hostname for the deploy

What Are Environment Variables?

Environment variables are a key value pairs that can affect how a program runs. They need to be set at some point before a process is run so the process can read them in and act accordingly. A lot of times in production environments your database name and password are set as environment variables so that information does not end up in a code repository somewhere.

By relying on an environment variable your code doesn't care what a setting is because you can set that at some other point. Actually working with environment variables is really simple. All you need to start is import os and you are off to the races.

Get Environment Variable with Python:

name = os.environ.get('DATABASE_NAME')

The environ part of this call is like a dictionary of all the environment variables. So we just get the one we are after and go about our business. You might even see something like below for your database settings, in your django settings file. You are either getting the environment variable, or you are getting back an empty string.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.environ.get('DATABASE_NAME', ''),
        'USER': os.environ.get('DATABASE_USER', ''),
        'PASSWORD': os.environ.get('DATABASE_PASSWORD', ''),
        'HOST': os.environ.get('DATABASE_HOST', ''),
        'PORT': os.environ.get('DATABASE_PORT', ''),
    }
}

Running django-admin.py/manage.py with environment variables

A common use of environment variables in django is the DJANGO_SETTINGS_MODULE. It tells django-admin.py what settings file to load for a project.

DJANGO_SETTINGS_MODULES='project.settings' django-admin.py

manage.py is basically a shortcut to this command. We are setting the location of our settings.py file and running django-admin.py command. There has been a lot debate about manage.py since the above is all it really does. You can read and learn more on a ticket about it: Deprecate and then remove "shell" management command

Set environment variable in bashrc

Finally, depending on your environment you might want to setup environment variables in your bashrc or bash_profile so every time you start a command line environment they are set. To do this add something like the following to your bashrc or bash_profile

export PROGRAMMING_FOLDER='/home/user/programming/'

The export command is built into shell and sets PROGRAMMING_FOLDER to /home/user/programming for the current environment. You can verify this running env in your shell, or even env | grep PROGRAMMING to get just that environment variable.

Common Places to Set Environment Variables in Production

First link is how to set environment variables with supervisord which is a recommended way to run your server side code, generally running gunicorn or uWSGI. uWSGI is linked to because a lot of people run it outside of supervisord. However, it is recommended if you use gunicorn to use supervisord so set environment variables inside that config.

Conclusion

Environment variables are extremely useful, but can be a huge pain to work with. I really suggest playing with them in different environments to get used to working with them. When I first started setting them in production environments it was tough to first learn how to set them, and run my code. However, once I got environment variables working locally in my dev environment, and in production it became super convenient. I am super glad I took the extra effort to learn a process to work with them, and I think you will too.


Frontend Code Strategy with Django

Web development is getting more and more focused on front-end development. This leads to a lot of questions on what is the best way to handle compiling css, javascript, coffeescirpt, stylus, etc. Having been involved with a project where the frontend deals with 100+ files between stylus files, coffeescirpt files, javascript, and css. This is kind of crazy to manage without a good strategy or even knowing how to accomplish it.

How Frontend Code Works in Django

When working with django you need to consider how the code is going to go from your code base to be displayed in the browser. The normal path it takes is to live in a static directory which you edit and run locally. Then in production you do a collectstatic command and it moves everything to a folder for your webserver to pull from. With that in mind as long as the above happens you can do almost anything you want for your frontend code. Lets take a look at the 3 most common ways you can do front-end development with django.

Three Popular Ways to Write Frontend Code

Everything in the static folder

The most common way we start projects is to just stuff everything in a static file on the root of our project. We add img, css, and js folders to keep things organized. We then use the staticfiles app to use template helpers for loading in our static files without having to hard code them. This will get us a long way, and for most sites is a good approach, because lets face it most sites don't move past hobby projects.

To get this working about the only configuration you need is:

INSTALLED_APPS = (
    ...
    'django.contrib.staticfiles',
    ...
)

STATIC_URL = '/static/'
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

STATIC_ROOT = os.path.join(BASE_DIR, 'static_final')

To see it in action you can watch a video here: Staticfiles, template filters, and commands

Third Party Django Library

The next step people often move to is using a third party django app which kind of takes care of everything for you. You set in the settings what you want to use to compile and preprocess code. The django app then does all the processing, finally outputing the final information where we tell it, generally from the settings above. It is fairly simple and straight forward to use these libraries especially since they don't really break your workflow, and you get the addition of using the latest cool technologies.

The two most populare packages I have seen is django-compressor and django-pipeline, a fork of compressor. Both are fairly simple to setup and configure, but they do have their limitations and oddities. I would recommend doing a side project with each to evaluate which one you like, to determine if it is right for you.

If you want to checkout a video for django-pipline go here: Compile and compress assets with django-pipline

Front-end Build System

The final way most often seen for building front end code is to use an entire build system outside of django completely. You setup your static file locations like we did above, and then you tell the independent build system to output your files there. Generally you would use something like grunt, gulp or brunch to do this. This method gives you the most flexibility and freedom, but is the most time consuming to setup and learn. If you heavily rely on front end code then this is probably the best solution, especially for single page apps.

Also by taking the time to learn some of these tools it helps you become a better all around developer through getting you involved in other things besides just django and python. Generally these build systems are completely custom, or based on something in the node.js world. Below are some of the more popular ones:

There is also a video on this method as well using gulp check it out here: Use gulp.js to manage static assets

Conclusion

Knowing what is the best way to build your assets is hard, I use all three methods in various projects. So I kind of take a tiered approach to know when to choose what method. If it is a small site where I am going to write it once and forget about it or maybe touch it a couple of times a year I don't worry about a build system at all, not worth the effort. If I am going to touch the code once a month on average I will generally use django-pipeline. However, if I am touching the code base 2+ times a month it means I am probably touching it more than that and I need to have the most flexibility, so I will use gulp.

These are the rules I follow, but I suggest you take a look at these methods and find your own sweet spot.


Using Chart.js with Django

Chart.js Header

Chart.js is the new kid on on the block for JavaScript charts. Learn how to use them here to chart out the number of user registrations for the last 30 days.

The View

This view builds an array of people that registered on the site daily for the last 30 days.

from django.views.generic import TemplateView
from django.contrib.auth.models import User

import arrow


class AnalyticsIndexView(TemplateView):
    template_name = 'analytics/admin/index.html'

    def get_context_data(self, **kwargs):
        context = super(AnalyticsIndexView, self).get_context_data(**kwargs)
        context['30_day_registrations'] = self.thirty_day_registrations()
        return context

    def thirty_day_registrations(self):
        final_data = []

        date = arrow.now()
        for day in xrange(1, 30):
            date = date.replace(days=-1)
            count = User.objects.filter(
                date_joined__gte=date.floor('day').datetime,
                date_joined__lte=date.ceil('day').datetime).count()
            final_data.append(count)

        return final_data

The method thirty_day_registrations loops through from 1 to 30, and gets the count of registrations for that day. Then it returns that array back to the get_context_data method and assigns it to 30_day_registrations which is what we will use in our template.

The Template

The template is very basic in that it has just enough data to generate a line chart.

{% extends "base.html" %}

{% block extrahead %}
<script src="//cdnjs.cloudflare.com/ajax/libs/Chart.js/0.2.0/Chart.min.js" type="text/javascript"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/2.1.1/jquery.min.js" type="text/javascript"></script>
<script type="text/javascript">
$( document ).ready(function() {
    var data = {
        labels: ['1', '5', '10', '15', '20', '25', '30'],
        datasets: [
            {
                label: "Site Registrations in the Last 30 Days",
                fillColor: "rgba(220,220,220,0.2)",
                strokeColor: "rgba(220,220,220,1)",
                pointColor: "rgba(220,220,220,1)",
                pointStrokeColor: "#fff",
                pointHighlightFill: "#fff",
                pointHighlightStroke: "rgba(220,220,220,1)",
                data: {{ 30_day_registrations }}
            }
        ]
    };
    var ctx = document.getElementById("myChart").getContext("2d");
    var myLineChart = new Chart(ctx).Line(data);
});
</script>
{% endblock %}

{% block content %}
<canvas id="myChart" width="500" height="300"></canvas>
{% endblock %}

If we go line-by-line of the important parts this should make more sense.

<canvas id="myChart" width="500" height="300"></canvas>

This is where in the page your chart is going to show up its only real requirement is it has an id and is a canvas object.

var ctx = document.getElementById("myChart").getContext("2d");

This gets us the canvas element for us to feed data to so we can see our chart.

var myLineChart = new Chart(ctx).Line(data);

We create a Chart object and pass it our canvas element. Then we create our Line chart using a data object we are going to create.

var data = {
    labels: ['1', '5', '10', '15', '20', '25', '30'],
    datasets: [
        {
            label: "Site Registrations in the Last 30 Days",
            fillColor: "rgba(220,220,220,0.2)",
            strokeColor: "rgba(220,220,220,1)",
            pointColor: "rgba(220,220,220,1)",
            pointStrokeColor: "#fff",
            pointHighlightFill: "#fff",
            pointHighlightStroke: "rgba(220,220,220,1)",
            data: {{ 30_day_registrations }}
        }
    ]
};

This is an example data object to create a chart. Most of it is self-explanatory, but the keys parts are:

  • labels: this is outside of the datasets object.
  • datasets: you can have multiple json objects in here for more lines in chart
  • data: in this case it is just an array, which in this case we are generating in our view and outputting here.

Conclusion

Chart.js is a great library for doing charting, especially if your primary audience is using more modern browsers. I have been looking at charting libraries, but so far this has been one of the best ones of both paid and free ones. The best part is right now chart.js is free. So if you need to do some charting I would say give it a shot.


Working with JSON and Django

In this day and age of web development we work with json, a lot, especially with using web services. Unfortunately, sometimes you can forget what does what, when, and where when dealing with json. I know I have to look up the same thing over and over at times. This blog post is everything you need to deal with json "day-to-day" with Django.

JSON to Dict

One of the first things we, usually, do is convert json data that we get back from a web service call, or from a webhook, into a python datastructure.

import json

json_data = '{"hello": "world", "foo": "bar"}' 

data = json.loads(json_data)

The key element from this is the json.loads which loads a string of json and converts the data to a python dict. From there you get to use it like normal.

Dict to JSON

import json

data = {'baz': 'goo', 'foo': 'bar'}

json_data = json.dumps(data)

Normally you want to get your data into json from your dict, or model, which is just as easy using json.dumps. It converts your dict to a string of json so you can return it to in a response.

HttpResponse with JSON

There are a couple of key parts to returning json properly. You need to have a string of json, and a proper content type.

import json
from django.http import HttpResponse

def fbview(request):
    data = {'foo': 'bar', 'hello': 'world'}
    return HttpResponse(json.dumps(data), content_type='application/json')

With this you will send back json properly formatted and with the proper content type so everyone knows what is going on with the data.

JSONRequestResponseMixin from django-braces

Finally, if you are working with class based views you can use a very convenient mixin in django-braces. JSONRequestResponseMixin which gives you some built-in response methods to return your dicts and models back as json, and it handles everything else.

class APIView(CsrfExemptMixin, JsonRequestResponseMixin):
    def post(self, request, *args, **kwargs):
        data = {'foo': 'bar', 'hello': 'world'}
        return self.render_json_response(data)

As you can probably guess the render_json_response method does what the function based view above does by converting the dict to json and returning the proper response.

Conclusion

Hopefully this is a good primer and/or reminder of how you can work with json with python and django. This takes care of all your basics so you can get started with json, or do it better.


Download CSV files via CSVResponseMixin

Providing csv files to people is annoying, but writing a custom mixin for use with your views can make it super easy. Often times we have to use a custom function, of some sort, and rejigger around how our views work to get a csv file, then set header information, and finally return a response.

There is a Better Way

Below is a custom mixin you can write, or use, to return a csv file based on a list of lists.

class CSVResponseMixin(object):
    csv_filename = 'csvfile.csv'

    def get_csv_filename(self):
        return self.csv_filename

    def render_to_csv(self, data):
        response = HttpResponse(content_type='text/csv')
        cd = 'attachment; filename="{0}"'.format(self.get_csv_filename())
        response['Content-Disposition'] = cd

        writer = csv.writer(response)
        for row in data:
            writer.writerow(row)

        return response

The first few lines of code are simple. We are declaring the mixin, setting the filename, and returning the filename based on a method on the mixin. That leaves us with the render_to_csv method which takes a lsit of lists, and gives us a downloadable file.

Line by Line Explanation

response = HttpResponse(content_type='text/csv')

We are creating a Django HttpResponse object which django returns, and we are setting the content type to a csv file so the browser knows what it is.

response['Content-Disposition'] = 'attachment; filename="{0}"'.format(self.get_csv_filename())

We are setting the content disposition to be an attachment so that it forces the browser to download the file. Otherwise the browser might render it out into the browser, as text. We also are setting the filename from the get_csv_filename method call instead of using the property for a couple of reasons. First, we can override it in our implementation to be more dynamic. Second, we can mock it out in our testing.

writer = csv.writer(response)
for row in data:
    writer.writerow(row)

return response

Finally, we are creating a csv writer and using our response object. Then as we loop over our list of lists we write them out one at a time generating our csv data. From there we return our response object which causes the browser to download the csv file of data.

Example Implementation

A quick and dirty example of how to do this is quite simple. All you have to do is attach this to a url pattern, and you are good to go.

class DataView(CSVResponseMixin, View):

    def get(self, request, *args, **kwargs):
        data = (('hello', 'world'), ('foo', 'bar'))
        return self.render_to_csv(data)

The telling thing about this example is we are using the View class as our base class so it isn't some fancy generic class based view. This gives it simplicity people are after with function based views, but is more understandable and testable.

Conclusion

Just like with LastAccessMixin we are leveraging good OOP practice to give us generic functionality we want to use. This one gives us a standard way to export data as a csv file in a consistent way. If you want to use this mixin plus many other useful ones checkout django-braces. They currently have a lot of good ones, and this one will be coming in a future release.