Comments on: Speed up Python without effort using generator expressions

By: Daniel Lemire

Daniel Lemire — Fri, 19 Sep 2008 15:53:15 +0000

Running code through cProfile is not fair since it adds an overhead for each function call.

Run these commands instead:

echo “sum([x for x in xrange(1000000)])” > t1.py

echo “sum((x for x in xrange(1000000)))” > t2.py

python t1.py

python t2.py

By: Andres

Andres — Fri, 19 Sep 2008 14:47:22 +0000

Hi Daniel!
Reading this post I try myself to do it. The results were not the same as you, I post it here. I work on python and I think that is correct that this happens, because generators in some operations are more expensive that simple list. when you have to read a big file generators are the solution, but in other cases lists are the solution!

cProfile.run(‘sum((x for x in xrange(1000000)))’)
1000004 function calls in 1.305 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1000001 0.565 0.000 0.565 0.000 :1()
1 0.000 0.000 1.305 1.305 :1()
1 0.000 0.000 0.000 0.000 {method ‘disable’ of ‘_lsprof.Profiler’ objects}
1 0.740 0.740 1.305 1.305 {sum}

cProfile.run(‘sum([x for x in xrange(1000000)])’)
3 function calls in 0.540 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.357 0.357 0.540 0.540 :1()
1 0.000 0.000 0.000 0.000 {method ‘disable’ of ‘_lsprof.Profiler’ objects}
1 0.182 0.182 0.182 0.182 {sum}

By: Carlos

Carlos — Fri, 05 Sep 2008 16:30:36 +0000

Not only that, but expression 2 doesn’t leak a variable!

$ python
Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.
>>> i
Traceback (most recent call last):
File “”, line 1, in
NameError: name ‘i’ is not defined
>>> sum([i for i in xrange(100000)])
4999950000L
>>> i
99999
>>> sum(j for j in xrange(100000))
4999950000L
>>> j
Traceback (most recent call last):
File “”, line 1, in
NameError: name ‘j’ is not defined
>>>