Python: Tips on Django development

Don’t let anyone tell you this dosent scale. Instagram scaled with Django, Nginx & Postgres to 30million users in 2 years on 2 dozen Amazon EC2 instances. Read their engineering blog for detail: http://instagram-engineering.tumblr.com/

The following are as much of a reminder to me as a set of tips for you to consider or learn from. I’m sure you will not agree with everything but that is fine. That’s the idea of free speech 😉

Ok let’s get into it.

Environment

If you dont get this right off the bat you will pay for it later with delays, environment issues as your solution expands and consistent plaguing issues. You should be coding not messing around with a poorly setup environment. So get it right first go so you can reap the benefits later.

Use VM’s for development

Many folks told me to setup VirtualEnv on my Mac and go down that route. That is what I call the poor man’s development environment for those that are not capable setting up a Linux box in a VM (virtual environment) themselves and managing it. Go and grab a virtual machine app like VMWare Fusion for Mac and load the latest Ubuntu Server 64bit on it. Step by step instructions are here. You may want to replace Apache with Nginx & Gunicorn like Instagram did.

The critical difference here is that you now get to develop on the same setup that you’ll be using in production. That’s kickass alone. Thanks to Barry Allard for taking me down this route whom I met at HackerDojo.

Use 2 code editors

A light editor and a full blown IDE with intellisense and code refactoring. The light editor is great for productivity in being able to pump code out. Then run your code past a full blown IDE to clean up the edges and refactor. Development is consistent refactoring. The cleaner your code the better it is for you and other to maintain and evolve it.

My favorite are:

  • Sublime Text 2 – that’s the light editor and
  • PyCharm – great IDE with all the bells and whistles but heavy.

Don’t reinvent

When you can reuse apps. That’s one of the philosophies of Django with it’s app structure mirroring true separation of concerns essentially collecting all ingredients of one functionality in your project in one place. So, look to popular source places where you can “re-use” apps. Obviously due diligence is essential before you grab any code. Look out for total commits, last updated, comments etc. to get a sense of application maturity.

Few places to check out:

Manage your development

With a hosted wiki, issue tracker and source code version control. I’m a big fan of Mercurial DVCS (distributed version control system) and it’s project hosting companion BitBucket. Thanks to Coupons, Inc. for this experience!

If you are serious about software engineering you have to be using a DVCS. It empowers you to work independently on many feature branches with engineers without polluting core code base, versions/saves your revisions and you can always peel back to a revision like onion skins. Forces you to leave comments on each commit so others know why & what and you can do efficient business like continuous build, automated tests & releases.. just like Facebook. Great ha!

Learn about Mercurial here in my prior blog post. Git is also a DVCS and as popular if not more due to GitHub, it’s project hosting companion. Both work practically in the same manner with slight differences. Both also provide you with GUI tools for interface (SourceTree from BitBucket) if you do not like shell commands.

Use Postgres for your database

In comparison to MySQL (another popular relational database), Postgres is more reliable, has more features, more frequently updated, more powerful and flexible. It’s completely free and open source. All the beef is in 2 posts here and here. Then head over to postgresguide.com to get started.

I can also recommend you look into MongoDB where appropriate. Django v1.4 as of writing post this dosent support admin scaffolding in the NoRel 1.3 version. Admin scaffolding is superb and not only allows you to test your models but also acts as a CMS (content management system) opening up all your selected models for data management. If you think you need a software to help you manage and secure your data, then consider Couchbase.

Code

Right. Everything below is Python & Django related.

Source of knowledge

Bookmark these sites. They are great resource for your development journey.

  • djangoproject.com – main Django site with great documentation (best I’ve seen),
  • djangobook.com – A step by step how to build everything from templates to internationalized sites.
  • stackoverflow.com – A language-independent collaboratively edited question and answer site for programmers.

Separation of Concerns

http://www.muhuk.com/2010/01/developing-reusable-django-apps/

Keep to the standard. The Django stack is split vertically, not horizontally. Apps are split horizontally within, i.e. models, views, templates etc are in their seperate modules/packages/directories. This vertical splitting allows you to collect all ingredients of one functionality in your project in one place.

Models

  • Use managers for commonly accessed queries. They are the interface through which database query operations are provided to Django models.
  • Move business logic out of views and into the appropriate models as functions. This way you will be able to unit test functionality that affects your models.
  • Visualise your models to help you get a high-level Birdseye view of your models. Also great for documentation and sharing with new folks on the team. (ref code)
  • Use the admin to test your models. Better then trying to decode it via the interface.

Template tags

  • Create custom template tags to automate occurrences in code. Only data not HTML should go into these.
  • Use “as” keyword to use the results of the call elsewhere in your template.
  • Use {% MEDIA_URL %} or {{ STATIC_URL }} to reference all your static assets. Do not let Django parse the assets and make sure you use a different domain (then your site) to feed these assets through to to maximize parallel downloads.
 

Keep an eye on apache logs

Open a new terminal window and run the following command to keep a live display open of new log entries as they come in. Cancel with ctrl-c.

tail /var/log/apache2/error.log -f

Other tips

  • Always include these as your first 2 lines in every .py file.
#!/usr/bin/env python
# -*- coding: utf-8 -*-

If you have several versions of Python installed, /usr/bin/env will ensure the interpreter used is the first one on your environment’s $PATH. Last is the encoding of a Python file comes from PEP 0263 – Defining Python Source Code Encodings. Also a solution to most charsets problems.

  • Use APP_PATH(‘directory’) vs hard-coding it. You can use the following lamda expression (anonymus function).
APP_PATH = lambda *x: os.path.abspath(os.path.join(os.path.dirname(__file__), *x))
  • Always use timezone-aware datetimes when presenting to the user and naive (local) datetime for storing in the database. Unless of course you want to emulate a server house in a different geographic location without actually being housed there.

Finally, practise The Zen of Python. Guiding principles for Python’s design into 20 aphorisms.

~ Ernest

Case Study: Medlert.com on Atlassian’s OnDemand Software Development Tools

The original version of this post I first published on Medlert blog here on the September 24th, 2013. Reposting here incase readers of The Road to Silicon Valley (#TRTSV) missed it.

I am a Co-Founder & CTO of a San Francisco based startup in the health sector called Medlert. Medlert gets you the fastest medical emergency response when using our mobile apps connecting through Medlert’s emergency platform. You can learn more about Medlert here: https://www.medlert.com/medical-emergency-alarm-system-features

Like many startups in the early days, we experimented a lot with a different set of software development & collaborative tools to manage bugs, chat, project management (agile stuff), document storage, code repository etc… that long list of stuff you gotta do to keep the wheels of the machine greased up. Like many, we found the defragmentation of multiple tooling and login accounts a nightmare. More work than it’s worth. Even though most of these tools were free, the overhead was not worth it.

Good set of software development & collaborative tools need to:

  • Minimize overhead – be complementary not a headache to get things done
  • Easy to use – one consistent experience with minimal ramp up period to get staff using it
  • Integrate with a common industry project management life cycle – think Agile
  • Accessible anywhere – web based (cloud) single sign on (SSO) over a secure connection
  • Affordable for any team – especially for small teams in the early days

The search

I am an Aussie, and strangely enough I have never heard of Atlassian until few years back when working in Silicon Valley running the Coupons.com International Engineering team. There I got exposure to early Atlassian tools like JIRA, Confluence et al.. and they started to slowly grow on me. I still didn’t like the fact that we ran these tools internally and had to have people manage them. The cries of Tomcat brought shivers down my spine.

When we kicked off Medlert (The Fastest Emergency Response) I admit I did not think Atlassian. I wanted everything for free and so we ended up using host of other free tools like Asana, Pivotal Tracker, Trello, Google Docs etc etc… That quickly changed as mentioned pains of defragmentation & integration made me revisit Atlassian website, finding Atlassian OnDemand. There it was, a single hub/solution to solve our pains. A cloud based offering package with all the tools we needed in 1. Win!

Atlassian OnDemand

We started with the free month trial and today are a happy paying customer. The pricing structure is ridiculously cheap for what you get. The diagram below depicts our use of Atlassian tools with more of the Atlassian suite of tools coming online (gray circles) in the next few months.

Tools used at Medlert to get stuff done faster

How we use Atlassian OnDemand at Medlert

JIRA – tickets

JIRA is a very powerful tool for ticket creation, management & tracking wrapped in an Agile software development process (sprints, stories and epics). Complementary to it, JIRA Agile formerly “GreenHopper” allows us to manage knowledge work with an emphasis on just-in-time delivery through Kanban boards. This proves us with a very high level overview of what is in the next release, being worked on and completed.

We have about a dozen “Projects” ranging from iOS to Corporate site to Platform setup in JIRA. This allows us to separate tickets into their correct channels and gives us a high level visibility across Projects, Individual contributors, Status of tickets, type of tickets like bugs/improvements etc. My favorite are Burndown charts and Velocity charts when it comes to planning. We can see work load, progress and team’s ability to resolve commitments per sprint.

Confluence – knowledge base

Confluence is a wiki used as a knowledge base. In the early days we used Dropbox & Google Drive. As the team grew this didn’t scale too well with our needs. Confluence is a great replacement with security down to the user level for sensitive knowledge.

We now store all our documents, discuss product designs using the threaded commenting features and document everything directly in Confluence. For newbies coming on board all they have to do is point their browser to our Confluence hub to find all they need to come to speed.

With JIRA integration any ticket pasted into Confluence gets automatically recognized and connected. Very neat and allows anyone within the company to move between the knowledge base wiki and related tickets.

HipChat – chat client

HipChat is not just a chat desktop & mobile client between teams but also a communication tool for services. All those circles above send appropriate messages to their designated rooms. The beauty of this is that you can just hang out in HipChat and see the company’s activity from all the tools we use. For example:

  • JIRA’s activity stream gets sent to our “Alerts” room in HipChat.
  • Confluence activity stream also gets sent to the “Alerts” room in HipChat.
  • ZenDesk customer support requests get streamed to our “Customer Support” room.
  • NewRelic, a 3rd party server & app monitoring tool, pushes Red Alerts to our “Alerts” room too.

Now this isn’t a replacement for those tool’s notifications. The right people still get those but it helps to keep everyone in the company who is interested at anytime updated on what is going on. More on other integrations here: http://help.hipchat.com/knowledgebase/topics/10037-integrations

We also have dedicated rooms for when working with contractors and other rooms for specific topics like:

  • Firehose: when making releases and/or system updates this is where all the activity happens.
  • Platform: for platform discussions and sharing of notes,
  • Product & Marketing: for as the name implies, sharing ideas around product & marketing
  • and few others.

Bitbucket – git source control

Having previously used GitHub we found BitBucket just as good if not better due to the integration between JIRA tickets and source control.

BitBucket has a feature called “Hooks” which allows you to associate an action to a change in the repository. For example, when code is checked into a Medlert Repository, the email service fires an email to Medlert Admins informing them of the checkin with a link to Bitbucket. Same hook sends a notification to HipChat Alert room. Instant visibility of activity on the Medlert Repositories.

You can use the shell command or Atlassian’s SourceTree GUI app to manage your Bitbucket repository.

An example: a typical day in the Medlert office

Medlert staff is geographically spread. As tickets are created in JIRA (either automatically from other tools or manually) in their appropriate Projects, the project leads manage the ticket life cycle. A ticket life cycle can follow an Agile process of being scheduled into a Sprint or addressed immediately based on severity. When the Agile process kicks in, the Project lead (which could be an engineer) will work within JIRA Agile to push everything along.

As JIRA tickets are completed, watchers (anyone with a vested interest in that ticket or its creator) gets notified via email. Also a Kanban board gives further insights into progress.

As JIRA tickets and Bitbucket code repository checkins happen, all notifications are also sent to HipChat via Hooks. All Medlert employees have visibility of this activity. HipChat is great and we use it daily for communication in private and public rooms vs clogging up email.

Most Releases are done ad hock. Our fault tolerance infrastructure and release process allows us to push & reverse code using automation. All the way to the relational database which is structured around code models. As releases happen emails go out to the admins updating them of start and end builds.

Now we never push releases directly to production. There is a staging environment, a scaled down mirror of production, which is used to do UAT (User Acceptance Testing) by actual users we recruit via UserTesting.com.

Our infrastructure tools from NewRelic to AWS monitoring paint a picture of hardware & application health throughout the whole process.

Atlassian Summit

The beauty here is these tools can be used in many different ways to cater for your needs. I’m sure there are things I could improve on or change here. That I’m looking forwards to in a weeks time when I attend the Atlassian Summit to not only learn about cool new features but also find out from others how they are maximizing this investment. Lookout for me at the Summit.

I’ll be sporting a Medlert white shirt and Medlert gray jacket. Say G’day!

I hope this post has answered and provided you with insights into how Atlassian OnDemand tools can be used to help your startup move fast. Any questions please comment below.

~ Ernest