Improve your Python with C

7 min readMay 11, 2020

How to make your code runs faster with knowledge of Python wrapping and C language.

Introduction

Hello everyone! In this article you will find how to enhance your Python code productivity by using power of C language and few libraries.

But do we need it ? Why Python became nowadays (2020 y.) so popular ? I think, in Wikipedia we can find a good answer.

Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python’s design philosophy emphasizes code readability with its notable use of significant whitespace. Its language constructs and object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects. — Wikipedia

The main point here — Python is a language to write easy readable by humans code and it improves developer’s productivity by language simplicity.

So, maybe that’s all ? Python is really cool language and even young kids can learn it from the very beginning easily (of course with some efforts).

But it’s not everything about this beautiful language. Of course, like every other programming language Python has advantages and disadvantages.

Python is slow.

If you are being worked with Python more than year you can really say it. For the topic why does Python is slow there are a lot of articles, discussions etc. so the reasons of it won’t be discussed here. But in short terms I will try to show you why your Python code may be slow.

Here our journey begins. Have a good reading !

Why so slow, dude ?

The whole article code blocks will be written in Python 3.x.

Let’s begin from the simplest variable initialization+assigning and further output to console. In Python we can do it like on the block below:

a = 10
print(a)
# 10

If you are a curious person and see (or when you saw it for the first time) it might be a question appears: what is under the hood ?

At the left part of an illustration above we see simple C-lang variable initialization+assigning. In code it would be like on the code block below:

int a = 10;
printf(a);
// 10

In Python everything is an object (inherited from the object class).

First thing Python does is assigning to the PyObject class instance head property that this object is a digit with value = 10.

It means to create a variable in Python we need to write 1 row of code and under the hood 2 operations are performed.

It is a simple example of a basic operation in Python, just imaging some complex operations like creating classes, loops etc. and imaging how many “under the hood” operations will be performed there.

How to make the core of your code working faster ? We need to specify all our operations in C , wrap them beautiful in Python and we will have fast and beautiful code.

Prerequisites

First of all you will need a Python. You can install it from here: Python

Compiler of course (GCC):

Windows (follow the link)
Ubuntu(18.04): terminal usage

Install package build-essentials. It contains such things like gcc, g++, make.

sudo apt install build-essential

Mac OS: terminal usage (requires homebrew to be installed)

brew install gcc

Simple coding

Let’s begin coding ! We will start with importing library ctypes.

Simple int8 variables creation and we can continue performing Python code.

from ctypes import c_int8
a = c_int8(10)
b = c_int8(10)
print(a.value + b.value)
# 20

Here we just make our code even worse than Python. Simple operation of addition became a little nightmare. How to make it efficient ?

Let’s create a file named library.c and we will write there simple C code with function for addition of tow numbers.

//library.c
int sum (int a, int b) {
return a + b;
}

Now we need to compile or library.c code. We will make it from terminal via next command (Ubuntu/Mac OS).

We need to run this command with flag -shared, -o to specify output file as our open library and compile it with flag -fPIC (position independent code).

gcc -shared -o library.so -fPIC library.c

Now we have got a file library.so what means we have needed function to be already compiled and ready for usage. Now we need to call it from Python code.

We will need a CDLL (C-Dynamic Link Library) to load our C-code into Python. You need to specify only full path to your .so file. After creating an instance we can just refer via dot to our function name and simply call it.

from ctypes import c_int8, CDLL
import os

lib = CDLL(os.path.abspath("./library.so"))

a = c_int8(10)
b = c_int8(10)

print(lib.sum(a, b))
# 20

Great! We just make our first c-wrapping. But was it efficient ? — No.

Alright. So now you may be asking “why did we learned this useless staff ?”.

Because on simple examples like adding two numbers to each other or maybe some string manipulations this approach sucks.

Now we will learn something harder and will make some time bench marking.

Data structures and complex tasks

For example you want to create a structure. It’s simpler than class. We can user Python class API and inherit it from ctypes Structure.

import ctypes as c


class Dot(c.Structure):
    fields = {"x": c.c_int8,
              "y": c.c_int8}


# usage example
d = Dot(x=10, y=10)#also, the simple pointer wrapping will be next:
lib = c.CDLL("./...")
"""
void my_function(double a){
printf(a);
}
"""
lib.my_function.argtypes = [c.POINTER(c.c_double)]
#it's also important to define types of your functions called by #CDLL

Now let’s make a little task.

We have a 2D space. And we need to implement Dot class/structure to create there simple dots with coordinates X and Y.

Also, we need to calculate a pairwise matrix between each one element with all others. Let’s do it in Python initially. Also, we will time it.

import random
import time
class Dot:
    def __init__(self, x, y):
        self.x = x
        self.y = y


class DotFactory:
    def __init__(self, nb):
        self.points = []
        for i in range(nb):
            self.points.append(Dot(random.random(), random.random()))
        self.distances = []

    def distance_between_points(self):
        for i, a in enumerate(self.points):
            for b in self.points:
                self.distances.append(((b.x - a.x) ** 2 + (b.y - b.x) ** 2) ** 0.5)


if __name__ == '__main__':
    time_start = time.time()
    test = DotFactory(10000)
    test.distance_between_points()
    time_end = time.time()
    print("Executed for %s seconds" % (time_end-time_start))

Result: Executed for 38.43 seconds.

Well, it was awfully long.

Let’s write some C code ! We will create a file lib.c

#include <math.h>
#include <stdlib.h>

typedef struct s_point{
  double x;
  double y;
} t_point;

typedef struct s_test{
  int nb_points;
  t_point *points;
  double *distances;
} t_test;

void generate_points(t_test *test, int nb){
  t_point *points = calloc(nb + 1, sizeof(t_point));

  for (int i = 0; i < nb; i++){
    points[i].x = rand();
    points[i].y = rand();
  }
  test->points = points;
  test->nb_points = nb;
}
void distance_between_points(t_test *test){
  int nb = test->nb_points;
  double *distances = calloc(nb * nb + 1, sizeof(double));
  for (int i = 0; i < nb; i++)
    for (int j = 0; j < nb; j++)
      distances[i * nb + j] = sqrt((test->points[j].x - test->points[i].x) * (test->points[j].x - test->points[i].x)
      + (test->points[j].y - test->points[i].y) * (test->points[j].y - test->points[i].y));
  test->distances = distances;
}

Now we need to compile it once and create lib.so file.

gcc -shared -o lib.so -fPIC lib.c

And now we will write new one Python code to wrap it.

import ctypes as c
import time


class Dot(c.Structure):
    _fields_ = [('x', c.c_int), ('y', c.c_int)]


class DotFactory(c.Structure):
    _fields_ = [
        ('nb_points', c.c_int),
        ('points', c.POINTER(Dot)),
        ('distances', c.POINTER(c.c_double)),
    ]


lib = c.CDLL("./lib.so")
lib.generate_points.argtypes = [c.POINTER(DotFactory), c.c_int]
lib.distance_between_points.argtypes = [c.POINTER(DotFactory)]# IMPORTANT:
# Lines above are necessary. You need to specify types of arguments # for your functions.if __name__ == '__main__':
    time_start = time.time()
    test = {'nb_points': 0, 'points': None, 'distances': None}
    c_test = DotFactory(**test)
    factory = c.pointer(c_test)
    lib.generate_points(factory, 10000)
    lib.distance_between_points(factory)
    time_end = time.time()
    print("Executed for %s seconds" % (time_end - time_start))

Result: Executed for 1.3 seconds

It means we just speed up our code almost in 30 times !

Conclusion

Now you know how to speed up your code by using ctypes library and knowledge in C-language. You see that classic Python is really slow and may be enhanced by pure C.

Well, is it necessary to use ctypes and rewrite all core logic in c always you want to write something more difficult than “Hello world!” ?

No, it’s not necessary. Even I can tell you more. There are a lot of libraries that already made all the work that was described in this article to make your code more powerful. For example Numpy, Pandas, Scikit-learn. If you really want something to be used in production, learn more about these libraries (not only for data science purposes).

But what if you had a cool project (really cool) written in C and you want to port it to Python for the common wealth and simplicity in usage ? Well, you will need this knowledge and in much more deeper amount. It’s also possible to wrap C++ code, but you will need to wrap it in C code initially and only after this compile and wrap in Python. So increase your knowledge in C and C++ to make your programming expertise wider and stronger.

Good luck !

Bondarenko.K , machine learning engineer;