How to use C code in Python

… and should we do this?

Before seeing “how” to use C code from Python let’s see first “why” one may want to do this. If you are reading this article right now you probably want to do one or more of these 3 things:

1. Reusing existing C code from a Python program
2. Speed up your Python program
3. Do some low-level stuff that cannot be done directly in Python

A shortcut…

If all you want is just to speed up your Python program, then there is actually an easier way rather than writing certain parts of your program in C. You can just use PyPy instead of Python when executing your app. PyPy is an alternative implementation of the Python programming language which uses just-in-time compilation to speed up the same python code with little or no changes to your code. You can download and install it from here.

After installation, simply replace python with pypy when executing a Python script. So, instead of:

python my_awesome_program.py

do:

pypy my_awesome_program.py

As you will see in the benchmark at the end of this article, this method can make your code considerably faster and in some situations (as is the case with the simple program example shown below in this article) it can be even faster than a C implementation.

If you really want to write/re-use some C…

Next we are going to see 2 ways to use C in Python, these are:

1. Using the ctypes module to call C functions
2. Writing a custom Python module in C

The ctypes library provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

As it sounds above, this method of using ctypes may be the best choice if you want to reuse existing C code (your code or even third-party libraries without the source code). With ctypes you can do so without writing any more C. You can just call the C functions using pure Python and wrap all the C functions inside Python functions.

On the other hand, if you don’t have existing C code and want to write it now specifically to be used in Python for speeding up or doing some low-level stuff, then writing a Python module in C may be the best option. When writing a Python module in C you have access to almost all the Python built-in data types and their methods (as plain C functions). So it makes it easier to write your C program specifically for Python, and then wen you import the module in Python you don’t need to any more work to “adapt” it to Python. You just use the module as it were written in Python.

Using ctypes library to call C functions

ctypes exports a few objects that can be used to load dynamic libraries and they use different calling conventions: cdll, windll, oledll. In this article we’ll use cdll.

To load a library:

mylib = cdll.LoadLibrary('library path or name')

After a library has been loaded, we can access a function inside it this way:

my_awesome_func = mylib.my_awesome_func
my_awesome_func() # call the function

But wait… These C functions that we want to call expect C data types as parameters and return C data types. How we can make Python know how to deal with these types? Fortunately, the ctypes library also exports objects that represent C data types and can be used to convert Python variables when passing them as parameters to C functions and to inform Python about what type should expect as return value from a C function.

Here is a list with most of the available types:

• c_byte
• c_char
• c_char_p
• c_double
• c_longdouble
• c_float
• c_int
• c_int8
• c_int16
• c_int32
• c_int64
• c_long
• c_longlong
• c_short
• c_size_t
• c_ssize_t
• c_ubyte
• c_uint
• c_uint8
• c_uint16
• c_uint32
• c_uint64
• c_ulong
• c_ulonglong
• c_ushort
• c_void_p
• c_wchar
• c_wchar_p
• c_bool

And if you need a pointer of one of these types just use the POINTER(type) function like this:

pointer_to_int_type = POINTER(c_int)

To inform Python of the return type of a C function use the .restype attribute, for example:

my_awesome_func.restype = POINTER(c_int) # my_awesome_func returns a pointer to int

And to pass values to functions in the correct type just wrap them in the corresponding ctype constructor:

my_awesome_func(c_int(300)) # call function with 300 as a C int type

Let’s assume we have the following existing C code that we’d like to use in a Python program:

// fib.c

#include <stdlib.h>

unsigned long * fib(unsigned n)
{
    unsigned long * fib_arr = (unsigned long *) malloc(n * sizeof(unsigned long));
    
    if (n >= 1)
        fib_arr[0] = 0;
    if (n >= 2)
        fib_arr[1] = 1;
    
    for (unsigned i = 2; i < n; ++i)
    {
        fib_arr[i] = fib_arr[i-2] + fib_arr[i-1];
    }

    return fib_arr;
}

Quite simple, right? It’s just a function fib that returns an array with the first n fibonacci numbers. And we want to call this function in Python and show the results on the screen.

First, we need to compile this code to a shared library:

gcc -c -fpic fib.c
gcc -shared -o libfib.so fib.o

Then, to call it in Python we do:

import platform
from ctypes import *

def print_c_array(ptr, n):
    # prints a C array with a syntax similar to Python lists
    #   ptr - pointer to first element of array
    #     n - number of elements in array
    print('[', end='')
    for i in range(n):
        if i == n-1:
            print(ptr[i], end=']\n')
        else:
            print(ptr[i], end=', ')

# load our fib library and store a reference to the fib() function
fib = cdll.LoadLibrary('./libfib.so').fib
fib.restype = POINTER(c_ulong) # set return type to "pointer to unsigned long"
n = 20
res = fib(c_uint(n)) # call fib(20) and store the result in res
print_c_array(res, n)

# identify the correct C standard library depending on the OS
clib = 'msvcrt' if platform.system() == 'Windows' else 'libc.so.6'
# don't forget to free the memory of the array!
cdll.LoadLibrary(clib).free(res)

# Outputs:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]

I think the above code with the comments is quite self-explanatory, but let’s still clarify some things:

• If we return a C array as a pointer to the first element, Python doesn’t know how to print that. So we created a function for this.
• Python doesn’t keep track of the dynamic-allocated stuff that we did in C, so it won’t clean up. We need to do so by calling free() function and we need to find the proper C library for that depending on our OS.

If you want to find out more about ctypes, here is the official documentation.

Writing a custom Python module in C

So how to write such a Python module? We start with just a C file (or more files, if your module is more complex and need to split it into multiple files; but for simplicity, let’s assume we have just one C file). But this C file has to follow some special rules so that it can be compiled to a Python module.

Preprocessor directives

We need to define a macro (#define PY_SSIZE_T_CLEAN) and include the Python header file (#include <Python.h>). This header file declares (actually, it doesn’t declare itself, but includes in turn other files; but you got the idea) all the functions and data types that you need in order to communicate with the Python interpreter (work with Python objects received as parameters, create Python objects and return them, etc.). And make sure you define PY_SSIZE_T_CLEAN before including the Python header file. So, at the beginning of the file, we should have something like:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

Creating the actual functions of our module

The functions that we want to export to Python from our module should be created according to the following template:

static PyObject * func_name(PyObject *self, PyObject *args)
{
    // Parse arguments with PyArg_ParseTuple(args, ...)
    
    // Do other stuff, maybe create some Python objects like a List, Tuple, Integer object etc.
    // Return one of these objects, or maybe nothing (NULL)
}

Here, “func_name” can be anything we want but a common naming convention is to use “<module name>_<function name as it appears in Python>” as the name of our functions.

PyObject is a generic data type that can represent any object that is used in Python, such as: Integers, Floats, Lists, Tuples, etc.

These functions receives 2 parameters: self and args, both of them being pointers to PyObject. Self points to the module object, so we can use it to store an internal state of our module, or access other stuff inside our module. Args points to a tuple of positional arguments or a dictionary of keywords -> arguments. To use these arguments in our C code we need to parse them with either PyArg_ParseTuple() or PyArg_ParseTupleAndKeywords().

Positional arguments are parsed with:

PyArg_ParseTuple(args, "format_string1", &variable1, "format_string2", &variable2, ...)

Where: “format_stringN” is a string that specifies what data type to store in variableN. This code stores the arguments that are passed when the function is called in Python into variable1, variable2, … which are C variables. PyArg_ParseTuple() returns true if succeeds or false otherwise. Here you can find a complete list with format strings.

Then, to do other things that we may want to do inside the our function and work with Python objects, such as Numbers, Lists, Tuples, etc., we need to use the functions defined in Python.h which are too many to say something about each one of them here, so you can have a look in the documentation; here you can find all of these functions.

Creating the methods array

The Python interpreter doesn’t automatically know how to find in our module all the functions that we created. So we need to create a mapping “function name as we want to appear in Python” -> pointer to one of the functions in the previous section. Then, store this mapping in a variable which we will export to Python. This is how we do this:

static PyMethodDef methods_array[] = {
    {"py_name",  c_name, METH_VARARGS, "description"},
    // ...
    {NULL, NULL, 0, NULL} // Sentinel
};

Where:

• “py_name” is the name to be used in Python
• c_name is the name of our C function (a function pointer)
• then we use either METH_VARARGS – for positional arguments, or METH_VARARGS | METH_KEYWORDS – if we want to use keywords
• “description” can be any string, or NULL

Our methods_array should always have {NULL, NULL, 0, NULL} as last element.

Creating the module struct

After we created the methods_array, we need to create a variable which holds information about the module as a whole (methods_array being one of its members). We do this like this:

static struct PyModuleDef module_name = {
    PyModuleDef_HEAD_INIT,
    "module_name",
    "module description/documentation",
    -1, /* size of per-interpreter state of the module,
           or -1 if the module keeps state in global variables. */
    methods_array
};

Create the initialization function

Finally, we create the function that initializes our module by using the variable module_name from the previous step. This initialization function should be the only non-static thing in the C file. Here is how it looks like:

PyMODINIT_FUNC PyInit_module_name(void)
{
    return PyModule_Create(&module_name);
}

An example

As an example, here we will do the same fib function as previously shown in plain C, but this time we will do it as a Python module.

Here is the full code of this Python module:

// fib_c_module.c
#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject * my_c_module_fib(PyObject *self, PyObject *args)
{
    unsigned n;
    if (!PyArg_ParseTuple(args, "I", &n))
        return NULL;
    
    PyObject* fib_lst = PyList_New(0);

    if (n >= 1)
        PyList_Append(fib_lst, PyLong_FromUnsignedLong(0));
    if (n >= 2)
        PyList_Append(fib_lst, PyLong_FromUnsignedLong(1));
    
    for (unsigned i = 2; i < n; ++i)
    {
        unsigned long a = PyLong_AsUnsignedLong(PyList_GetItem(fib_lst, i-2));
        unsigned long b = PyLong_AsUnsignedLong(PyList_GetItem(fib_lst, i-1));

        PyList_Append(fib_lst, PyLong_FromUnsignedLong(a+b));
    }


    return fib_lst;
}

static PyMethodDef my_c_module_methods[] = {
    {"fib",  my_c_module_fib, METH_VARARGS, "Returns a list with the first n Fibonacci numbers."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static struct PyModuleDef my_c_module = {
    PyModuleDef_HEAD_INIT,
    "my_c_module",   /* name of module */
    "My first Python module in C", /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module,
                 or -1 if the module keeps state in global variables. */
    my_c_module_methods
};

PyMODINIT_FUNC PyInit_my_c_module(void)
{
    return PyModule_Create(&my_c_module);
}

Build the module

To build the module we use distutils Python library and create a setup.py script like this:

from distutils.core import setup, Extension

module1 = Extension(name='module_name',
                    sources=['file1.c', 'file2.c', ...],
                    include_dirs=[], # list of directories to search for C/C++ header files
                    library_dirs=[], # list of directories to search for static libraries
                    runtime_library_dirs=[], # list of directories to search for shared libraries
                    libraries=[] # list of library names)

setup (name = 'module_name',
       version = '1.0',
       description = '...',
       ext_modules = [module1])

After the setup.py script is done, we can run:

python setup.py build

to build it.

If everything worked fine and there are no compilation errors, we should have a build folder in the same parent direcrory as the setup.py file. Inside build you’ll find “lib.<your os name>” folder and inside it is the compiled Python module.

Or:

python setup.py install

to install it in the proper location.

After it has been installed we should be able to import from it using the module name that we set previously. We can also use it without installing; just after we build it, if we create a Python file in the same folder as the resulting .so (Unix) or .pyd (Windows) file, it should work to import from it using the module name that we defined (not the whole name of the compiled file).

If you are on Windows, you will need to have MS Visual Studio installed (and check the C++ option when installing, because the C/C++ compiler is needed) in order for the setup script to work.

If you want to find out more about the building process, here is the documentation.

Back to our example

Here is the setup.py file for our “my_c_module” with the fib function:

# setup.py
from distutils.core import setup, Extension

module1 = Extension('my_c_module',
                    sources = ['fib_c_module.c'])

setup (name = 'my_c_module',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])

After we build or install it, we can use the module in a Python file:

from my_c_module import fib

print(fib(20))

# Outputs:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]

Putting it all together and doing some benchmarks

Now, let’s take all 3 ways of writing our example fib() function: 1) Pure Python, 2) Python C module, 3) Pure C function called from Python, and use them in a single Python file and benchmark it to see which one is faster.

First, let’s implement the fib() function in Python:

# fib.py

def fib(n):
    fib_lst = []
    if n >= 1:
        fib_lst.append(0)
    if n >= 2:
        fib_lst.append(1)
    
    for i in range(2, n):
        fib_lst.append(fib_lst[-2]+fib_lst[-1])
    
    return fib_lst

And here we use them all:

import platform
from fib import fib as py_fib
from my_c_module import fib as c_module_fib
from ctypes import *

def print_c_array(ptr, n):
    print('[', end='')
    for i in range(n):
        if i == n-1:
            print(ptr[i], end=']\n')
        else:
            print(ptr[i], end=', ')

print('Pure Python:')
print(py_fib(20))

print('Python C Module:')
print(c_module_fib(20))

print('Pure C function called from Python:')
fib = cdll.LoadLibrary('./libfib.so').fib
fib.restype = POINTER(c_ulong)
n = 20
res = fib(c_uint(n))
print_c_array(res, n)
clib = 'msvcrt' if platform.system() == 'Windows' else 'libc.so.6'
cdll.LoadLibrary(clib).free(res)

# Outputs:
# Pure Python:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]
# Python C Module:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]
# Pure C function called from Python:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]

Now, let’s benchmark it using timeit. We are going to run each fib function for 100,000 iterations and print at the end how much it took for each one to finish.

# benchmark.py
from timeit import timeit

n_iter = 100000

py_time = timeit(
    stmt='py_fib(20)',
    setup='from fib import fib as py_fib',
    number=n_iter)

c_module_time = timeit(
    stmt='c_module_fib(20)',
    setup='from my_c_module import fib as c_module_fib',
    number=n_iter)

c_ctypes_time = timeit(
    stmt=
'''res = fib(c_uint(20))
free(res)
''',
    setup=
'''from ctypes import cdll, POINTER, c_ulong, c_uint
import platform
fib = cdll.LoadLibrary('./libfib.so').fib
fib.restype = POINTER(c_ulong)
clib = 'msvcrt' if platform.system() == 'Windows' else 'libc.so.6'
free = cdll.LoadLibrary(clib).free
''',
    number=n_iter)

print(f'Pure Python: {py_time}')

print(f'Python C Module: {c_module_time}')

print(f'Pure C function called from Python: {c_ctypes_time}')

# Outputs (results are in seconds):
# Pure Python: 0.2837742
# Python C Module: 0.0480217
# Pure C function called from Python: 0.08984860000000006

The results may vary on different runs of the script, but as I can see on my system the order is the same (from fastest to slowest): Python C Module, Pure C function called from Python, Pure Python (executed with the regular python command).

Let’s also see how much time it takes for the pure Python version to run if we execute it with PyPy:

# benchmark_pypy.py
# Execute this with "pypy script_name.py"

from timeit import timeit

n_iter = 100000

py_time = timeit(
    stmt='py_fib(20)',
    setup='from fib import fib as py_fib',
    number=n_iter)


print(f'Python PyPy: {py_time}')

# Outputs:
# Python PyPy: 0.022593

Surprisingly, it’s faster even than the module written in C. However, I don’t think this holds in general for any module you may write, but for our simple fib function that seems to be the case.

I hope you found this information useful and thanks for reading!

This article is also posted on Medium here. Feel free to have a look!