Comparing the speed of CPython, Brython, Skulpt and pypy.js

Disclaimer
I am the creator and main developer of Brython. I am aware that you might think I’m biased ! The test conditions are explained in detail below so that anyone can easily reproduce them and compare with the results presented here ; if something is wrong please post a comment and I will update this article.

Tested features
The test compares the speed of basic actions :
– assignement to an integer or a float
– augmented assignment on an integer
– building a dictionary, assigning a value to a key in a dictionary
– building a list, setting the value at a given rank
– addition of integers and strings
– casting an integer to a string
– create and run a simple function

How the test is run
The test is performed by running a single script made of all the elementary benchmarks. The source code of this script is here http://brython.info/speed/bench_str.py. The online editors or consoles are

• Brython : http://brython.info/tests/index.html (with the debug checkbox unchecked)
• Skulpt : http://www.skulpt.org/
• pypy.js : http://pypyjs.org/demo/

For Python, the script is executed on the command line.

For Brython and Skulpt, the code is copied in the editor and executed by hitting the button “run”.

For pypy.js, I tried pasting the code in a single multi-line string with :

 

>>> src = """<code>"""

but this raised an exception in the console :

 

debug: OperationError:
debug: operror-type: SyntaxError
debug: operror-value: ('EOL while scanning string literal', ('c callback', 1, 12, 'r = c.push(\'src="""import time\n', 0))
debug: OperationError:
debug: operror-type: KeyError
debug: operror-value: 'r'

So I generated a single-line version of the source code (see here http://brython.info/speed/bench_str_pypy.txt) and executed it with

 

>>> src = """<single line code>"""
>>> exec(src)

The script is run 5 times for each solution ; the result for each individual test is the geometric mean of these 5 tests.

Configuration
All the tests are run on a PC with Windows 7. The browser is Firefox version 35.0.1 ; it is restarted after the tests for a solution are done.

The tests were made on March 27, 2015 with the version available online at that date.

Results

 

                             Execution time (ms)           |    X slower than CPython
                    Cpython   Brython   pypy.js    skulpt  | Brython   pypy.js    skulpt  
assignment.py            125        14      3310      5819 |     0.11     26.57     46.70
augm_assign.py           211        44      4120      6791 |     0.21     19.56     32.24
assignment_float.py      110       508      3405      6048 |     4.63     31.04     55.13
build_dict.py            360      3490      3617     14539 |     9.70     10.05     40.41
set_dict_item.py         191        97      4820     23063 |     0.51     25.26    120.85
build_list.py            311        50      3428      6857 |     0.16     11.02     22.04
set_list_item.py         181        63      3330      8150 |     0.35     18.40     45.04
add_integers.py          249       113      4064      7691 |     0.45     16.34     30.92
add_strings.py           429       270      3972      8481 |     0.63      9.26     19.77
str_of_int.py             60       168       796      2272 |     2.81     13.36     38.15
create_function.py       240      3443      3570      8912 |    14.38     14.91     37.21
function_call.py         271      1446      3343     23341 |     5.33     12.33     86.09

Conclusion
It is hard to draw conclusions on the relative speed of implementations because it depends on the code executed. This benchmark tests very short scripts which may not be representative of real-world programs. All I can say is that with these tests, with the versions available on line on the day of the benchmark executed in the same PC and browser, Brython is generally faster than pypy.js, which is itself faster than Skulpt.

Advertisement

Translating “a+b”

The translation of a simple Python expression like a + b to Javascript gives a good example of the differences between the languages, and the choices made in Brython to be fully compliant with Python, while remaining nearly as fast as native Javascript in the most current cases.

The fist idea is to use the same Javascript operator, so to translate it to a + b.

This will work correctly, and very fast, for two types that are implemented in Brython like in Javascript : strings and integers. Strings are implemented as Javascript strings, and integers as Javascript numbers ; both types support the operator +, which returns an object of the same type (string or number).

Python lists are implemented as Javascript arrays. What happens if you use the operator + with Javascript arrays ?

> a = [1]
> b = [2]
> a + b
“12”

Javascript doesn’t concatenate the arrays like Python, it adds the string representations of the arrays.

If you take arbitrary objects, the addition will give even more surprising results :

> a = {}
> b = {}
> a + b
“[object Object][object Object]”

Javascript doesn’t throw an exception saying that the addition is not supported : like for arrays, it adds the string representation of the objects ; by default, for an object, this representation is the string "[object Object]". It can be customised by defining a method toString(), but the result will always be a Javascript string.

In Python, the operator + can be defined by the special method __add__. For a user-defined class, the addition is the result of getattr(a, '__add__')(b)

The builtin function getattr is defined in Brython, so we can use the above code as the translation of a + b, it will give the correct result in all cases.

Yes, but… for strings and integers, getting an attribute by getattr is very slow, hundreds of times slower than a + b, which gives the correct result for these types.

In the general case, the Python parser has no way of knowing the types of the variables, so it can’t generate a different code for the addition of strings and that of other types. Type checking can only be performed at run time.

This is what Brython does : a + b is translated into an expression that begins by checking if the arguments are simple types ; if so, the addition is performed by the Javascript operator + ; otherwise, it uses getattr.

Here is a simplified version of the code generated by Brython :

(typeof a.valueOf() == “number” && typeof b.valueOf() == “number”
? (typeof a == “number” && typeof b == “number”
? a+b
: new $B.$FloatClass(a+b))
: (typeof a==”string” && typeof b==”string”)
? a+b
: getattr(a,”__add__”)(b)
);

This expression uses the Javascript keyword typeof, which returns a string. It relies on the way Brython implements Python types in Javascript :

Python type	typeof obj	typeof obj.valueOf()
int	“number”	“number”
float	“object”	“number”
str	“string”	“string”

With this table, the expression should be fairly easy to understand. The only tricky case is when both operands are numbers (int or float) but one of them at least is not an integer : in this case the operation returns a float, by

new $B.$FloatClass(a+b)

The value passed to the float class constructor is a + b : how does this give the correct result, knowing that the type of at least one of the operands is “object”, not “number” ? Because when Javascript adds two objects which are not “primitive” types such as number or string, it calls their method valueOf(), and for floats, obj.valueOf() is a Javascript number, the floating point number itself.

Reference : Fake operator overloading in Javascript