Archive for category Code

Replacing Django’s Nasty “runserver”

Have you ever tried to have more than one person view a development site using Django’s built-in development server ? Yeah, it really sucks. Apparently concurrency wasn’t high on the features list and they have stated that it never will be.

DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone through security audits or performance tests. (And that’s how it’s gonna stay. We’re in the business of making Web frameworks, not Web servers, so improving this server to be able to handle a production environment is outside the scope of Django.)

So how do we go about using something a little nicer without losing any of the auto-reload goodness and without having to setup a full blown production environment ?

There are a number of alternatives, however I’ve selected Twisted Web simply because I really like the twisted framework and due to the experience I have in using it, I am very comfortable with it. It’s a great feature-packed web server that handles concurrency (and a ton of other things) exceptionally well.

So how do we use it to serve our little Django project in a development friendly way ?

I’ve put together some code (some borrowed from other sources) and constructed a simple replacement command called “trunserver” (twisted-runserver). You can grab this code from Github. Simply install it using the standard methods, and run it with:

python manage.py trunserver [IP:PORT] [--settings=foo] [--noreload]
.

So this will start up a twisted web instance serving your Django project and just like the build-in runserver, it will automatically reload your code when it notices that your files have been modified unless –noreload has been passed.

There are a few things missing at this point, like IPv6 support and static file serving, however these are on the roadmap.

I’ll post again with some more info once it is a little more stable and an official release has been provided.

, , ,

No Comments

Natural order sorting strings with numbers

The following python code makes natural sorting sequences of lexical and numerical values a little easier. It supports any iterable containing strings which have embedded numbers. In short it would give you this:

foo1 < foo2 < foo10

instead of this:

foo1 < foo10 < foo2

As an example, if you have this sequence:

>>> seq = ['foo', 'foo1', 'foo2', 'foo10', 'foobar10', '20', '100', '1', '3', 'bar1']

a regular sort would produce this:

>>> sorted(seq)
['1', '100', '20', '3', 'bar1', 'foo', 'foo1', 'foo10', 'foo2', 'foobar10']

whereas a natural sort would produce this:

>>> natural_sort(seq)
['1', '3', '20', '100', 'bar1', 'foo', 'foo1', 'foo2', 'foo10', 'foobar10']

Here is the code:

import re

def natsort_key(item):
    chunks = re.split('(\d+(?:\.\d+)?)', item)
    for ii in range(len(chunks)):
        if chunks[ii] and chunks[ii][0] in '0123456789':
            if '.' in chunks[ii]: numtype = float
            else: numtype = int
            chunks[ii] = (0, numtype(chunks[ii]))
        else:
            chunks[ii] = (1, chunks[ii])
    return (chunks, item)

def natural_sort(seq):
    sortlist = [item for item in seq]
    sortlist.sort(key=natsort_key)
    return sortlist

, ,

No Comments

Generating a dependency graph for a PostgreSQL database

This post was mostly inspired by this one, which shows how to generate a dependency graph for a MySQL database. Here we do something similar for PostgreSQL.

This script will generate the required digraph data to pipe into graphviz dot which will generate a visual representation of dependencies in a database schema, based on foreign key constraints.

The script:


from optparse import OptionParser, OptionGroup

import psycopg2
import sys

def writedeps(cursor, tbl):
    sql = """SELECT
        tc.constraint_name, tc.table_name, kcu.column_name,
        ccu.table_name AS foreign_table_name,
        ccu.column_name AS foreign_column_name
    FROM
        information_schema.table_constraints AS tc
    JOIN information_schema.key_column_usage AS kcu ON
        tc.constraint_name = kcu.constraint_name
    JOIN information_schema.constraint_column_usage AS ccu ON
        ccu.constraint_name = tc.constraint_name
    WHERE constraint_type = 'FOREIGN KEY' AND tc.table_name = '%s'"""
    cursor.execute(sql % tbl)
    for row in cursor.fetchall():
        constraint, table, column, foreign_table, foreign_column = row
        print '"%s" -> "%s" [label="%s"];' % (tbl, foreign_table, constraint)

def get_tables(cursor):
    cursor.execute("SELECT tablename FROM pg_tables WHERE schemaname='public'")
    for row in cursor.fetchall():
        yield row[0]

def main():
    parser = OptionParser()

    group = OptionGroup(parser, "Database Options")
    group.add_option("--dbname", action="store", dest="dbname",
            help="The database name.")
    group.add_option("--dbhost", action="store", dest="dbhost",
            default="localhost",  help="The database host.")
    group.add_option("--dbuser", action="store", dest="dbuser",
            help="The database username.")
    group.add_option("--dbpass", action="store", dest="dbpass",
            help="The database password.")
    parser.add_option_group(group)

    (options, args) = parser.parse_args()

    if not options.dbname:
        print "Please supply a database name, see --help for more info."
        sys.exit(1)

    try:
        conn = psycopg2.connect("dbname='%s' user='%s' host='%s' password='%s'"
            % (options.dbname, options.dbuser, options.dbhost, options.dbpass))
    except psycopg2.OperationalError, e:
        print "Failed to connect to database,",
        print "perhaps you need to supply auth details:\n %s" % str(e)
        print "Use --help for more info."
        sys.exit(1)

    cursor = conn.cursor()

    print "Digraph F {\n"
    print 'ranksep=1.0; size="18.5, 15.5"; rankdir=LR;'
    for i in get_tables(cursor):
        writedeps(cursor, i)
    print "}"

    sys.exit(0)

if __name__ == "__main__":
    main()

You could run it as follows:

python postgres-deps.py --dbname some_database | dot -Tpng > deps.png

Note: for other options use:

python postgres-deps.py --help

That should spit out one of these:

deps

,

No Comments

Extending Python with modules written in C

Using C (or C++) to create Python modules is really quite simple, providing you know a little C of course. I recently had to do some work around getting a bunch of legacy C code talking to a newer system and thought I’d post a nice simple example of how the Python extensions work.

This code gives you a single method “do()” that will print the output of a command, passed to it as a string, to stdout and return the exit code as a python int.

Dump this into “mycmd.c”:

#include <Python.h>

static PyObject * mycmd_do(PyObject *self, PyObject *args) {
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = system(command);
    return Py_BuildValue("i", sts);
}

static PyMethodDef MyCmdMethods[] = {
    {"do", mycmd_do, METH_VARARGS, "Print output of 'cmd', return exit code."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
initmycmd(void) {
    (void) Py_InitModule("mycmd", MyCmdMethods);
}

int main(int argc, char *argv[]) {
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    initmycmd();
    return 0;
}

Great, so we have some example code now, here is how you build an importable module with it:

greg@codemine:~/code/mycmd %> cc -dynamic -g -Wall -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c mycmd.c -o mycmd.o
greg@codemine:~/code/mycmd %> cc -bundle -undefined dynamic_lookup mycmd.o -o mycmd.so

Note: Don’t forget to replace the include path above with the correct path to Python.h on your machine.

This should give you a mycmd.so on unix / linux and a mycmd.dll on windows. In the same directory, run a python interpreter and test it out.

greg@codemine:~/code/mycmd %> python
Python 2.6.3 (r263:75183, Nov  4 2009, 12:53:19)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mycmd
>>> mycmd.do('/usr/bin/false')
256
>>> mycmd.do('/usr/bin/true')
0
>>> mycmd.do('uname -a')
Darwin codemine.codelounge.int 10.2.0 Darwin Kernel Version 10.2.0: Tue Nov  3 10:37:10 PST 2009; root:xnu-1486.2.11~1/RELEASE_I386 i386
0
>>>

There is much more you can do around this, thankfully the documentation is remarkably good.

There is not much to the actual code. First, we define the C function that will handle our command “mycmd_do”. Then we set up an array of methods we want to expose to python “MyCmdMethods”. We then setup an initializer “initmycmd” to expose the module which is executed from “main” after the python initializer “Py_Initialize”.

,

No Comments

Simple HTTP POST in Java

Today I was helping a friend debug a web service they had implemented. Their side was working correctly but the developer who was trying to interface with it seemed to be running into many problems. Since they were integrating an application written in Java, I whipped up a simple test for them. All we really needed to do was to send a few variables using HTTP POST to this resource and make sure it returned exactly what we were expecting.

This uses standard libraries only, and doesn’t require anything third party. It does nothing fancy at all, just simply posts data to a URL. Hopefully you find this useful at some point.

src/postit/Main.java

/*
 * Java POST Example
 */

package postit;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.ProtocolException;
import java.net.URL;
import java.net.URLEncoder;
import java.io.DataOutputStream;
import java.io.DataInputStream;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Greg
 */
public class Main {

    /**
     * Pretend you're a script...
     */
    public static void main(String[] args) {
        final String server = "somewhere.ontheinter.net";

        URL url = null;
        try {
            url = new URL("http://" + server + "/we-expect-post/data");
        } catch (MalformedURLException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        HttpURLConnection urlConn = null;
        try {
            // URL connection channel.
            urlConn = (HttpURLConnection) url.openConnection();
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Let the run-time system (RTS) know that we want input.
        urlConn.setDoInput (true);

        // Let the RTS know that we want to do output.
        urlConn.setDoOutput (true);

        // No caching, we want the real thing.
        urlConn.setUseCaches (false);

        try {
            urlConn.setRequestMethod("POST");
        } catch (ProtocolException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        try {
            urlConn.connect();
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        DataOutputStream output = null;
        DataInputStream input = null;

        try {
            output = new DataOutputStream(urlConn.getOutputStream());
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Specify the content type if needed.
        //urlConn.setRequestProperty("Content-Type",
        //  "application/x-www-form-urlencoded");

        // Construct the POST data.
        String content =
          "name=" + URLEncoder.encode("Greg") +
          "&email=" + URLEncoder.encode("greg at code dot geek dot sh");

        // Send the request data.
        try {
            output.writeBytes(content);
            output.flush();
            output.close();
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Get response data.
        String str = null;
        try {
            input = new DataInputStream (urlConn.getInputStream());
            while (null != ((str = input.readLine()))) {
                System.out.println(str);
            }
            input.close ();
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
}

, ,

6 Comments

Simple OpenVPN Server Statistics

Ever wondered what the status of your OpenVPN server is, or wanted some simple stats ?

Here is a really simple script to give you some info. The output looks like this:

greg@leonis:~$ vpnstatus
Common Name   Virtual Address    Real Address          Sent      Received             Connected Since
greg.vpn      10.8.142.6         196.9.23.59        1.11 MB     282.50 KB    Thu Jul 16 09:07:15 2009

Firstly, ensure your server is actually saving the stats somewhere (/etc/openvpn/openvpn.status in my example):

greg@leonis:~$ grep status /etc/openvpn/server.conf
status openvpn.status

Once that is happening, drop this code into a file and make it executable.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

STATUS = "/etc/openvpn/openvpn.status"

status_file = open(STATUS, 'r')
stats = status_file.readlines()
status_file.close()

hosts = []

headers = {
    'cn':    'Common Name',
    'virt':  'Virtual Address',
    'real':  'Real Address',
    'sent':  'Sent',
    'recv':  'Received',
    'since': 'Connected Since'
}

sizes = [
    (1<<50L, 'PB'),
    (1<<40L, 'TB'),
    (1<<30L, 'GB'),
    (1<<20L, 'MB'),
    (1<<10L, 'KB'),
    (1,       'B')
]

def byte2str(size):
    for f, suf in sizes:
        if size >= f:
            break

    return "%.2f %s" % (size / float(f), suf)

for line in stats:
    cols = line.split(',')

    if len(cols) == 5 and not line.startswith('Common Name'):
        host  = {}
        host['cn']    = cols[0]
        host['real']  = cols[1].split(':')[0]
        host['recv']  = byte2str(int(cols[2]))
        host['sent']  = byte2str(int(cols[3]))
        host['since'] = cols[4].strip()
        hosts.append(host)

    if len(cols) == 4 and not line.startswith('Virtual Address'):
        for h in hosts:
            if h['cn'] == cols[1]:
                h['virt'] = cols[0]

fmt = "%(cn)-25s %(virt)-18s %(real)-15s %(sent)13s %(recv)13s %(since)25s"
print fmt % headers
print "\n".join([fmt % h for h in hosts])

,

No Comments

Run something as another user

Here is a simple way to run something on UNIX / Linux as another user, without having to resort to weird sudo incantations. The Makefile is left as an exercise for the reader.

This has only been tested on FreeBSD, Debian Linux and OpenSolaris so far.

Compile with: cc -o setuid setuid.c

#include <stdio.h>
#include <sys/types.h>
#include <pwd.h>
#include <grp.h> 

int main(int argc, const char **argv) {

    /* Check command line */
    if (argc < 3) {
        fprintf(stderr, "Usage: %s user cmd\n", argv[0]);
        return 1;
    }

    struct passwd *pw;
    pw = getpwnam(argv[1]);

    if (pw==NULL) {
        fprintf(stderr, "User not found\n");
        return 1;
    }

    if (initgroups(argv[1], pw->pw_gid)==-1) {
        perror("initgroups");
        return 1;
    }

    if (setregid(pw->pw_gid, pw->pw_gid)==0 &&
        setreuid(pw->pw_uid, pw->pw_uid)==0) {
        argv += 2;
        execvp(argv[0], argv);
        perror("exec");
        return 1;
    }

    perror("setre[gu]id");
    return 1;

}

, ,

No Comments