Mailing List Archive

[issue11849] ElementTree memory leak
New submission from Kaifeng Zhu <cafeeee@gmail.com>:

I'm using xml.etree.ElementTree to parse large XML file, while the memory keep increasing consistently.

You can run attached test script to reproduce it. From 'top' in Linux or 'Task Manager' in Windows, the memory usage of python is not decreased as expected when 'Done' is printed.

Tested with Python 2.5/3.1 in Windows 7, and Python 2.5 in CentOS 5.3.

----------
components: XML
files: test.py
messages: 133797
nosy: Kaifeng.Zhu
priority: normal
severity: normal
status: open
title: ElementTree memory leak
type: resource usage
versions: Python 2.5, Python 3.1
Added file: http://bugs.python.org/file21669/test.py

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Florent Xicluna <florent.xicluna@gmail.com> added the comment:

Do you experience same issue with current versions of Python? (3.2 or 2.7)
The package was upgraded in latest versions.

----------
nosy: +flox

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Kaifeng Zhu <cafeeee@gmail.com> added the comment:

Yes. Just tested with Python 2.7 and 3.2 in Windows 7, the memory usage is still unexpected high after 'Done' is printed.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Florent Xicluna <florent.xicluna@gmail.com> added the comment:

I've tested a small variant of your script, on OSX.
It seems to behave correctly (with 2.5, 2.6, 2.7 and 3.1).

You can force Python to release memory immediately by calling "gc.collect()".

----------
Added file: http://bugs.python.org/file21670/issue11849_test.py

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Florent Xicluna <florent.xicluna@gmail.com> added the comment:

this is the output for 2.7.1:

$ python2.7 issue11849_test.py
*** Python 2.7.1 final
--- PID STAT TIME SL RE PAGEIN VSZ RSS LIM TSIZ %CPU %MEM COMMAND
0 2754 S+ 0:00.07 0 0 0 2441472 5372 - 0 11,7 0,1 python2.7 issue11849_test.py
1 2754 S+ 0:02.36 0 0 0 2520740 83720 - 0 100,0 2,0 python2.7 issue11849_test.py
2 2754 S+ 0:04.89 0 0 0 2596784 158888 - 0 100,0 3,8 python2.7 issue11849_test.py
3 2754 S+ 0:07.28 0 0 0 2668740 230972 - 0 100,0 5,5 python2.7 issue11849_test.py
4 2754 S+ 0:10.11 0 0 0 2740932 303200 - 0 100,0 7,2 python2.7 issue11849_test.py
5 2754 S+ 0:12.85 0 0 0 2812876 375276 - 0 98,4 8,9 python2.7 issue11849_test.py
6 2754 R+ 0:14.95 0 0 0 2885868 447740 - 0 98,9 10,7 python2.7 issue11849_test.py
7 2754 S+ 0:17.91 0 0 0 2962156 522560 - 0 99,1 12,5 python2.7 issue11849_test.py
8 2754 S+ 0:21.08 0 0 0 3034092 594620 - 0 98,3 14,2 python2.7 issue11849_test.py
9 2754 S+ 0:23.20 0 0 0 3106028 667004 - 0 100,0 15,9 python2.7 issue11849_test.py
END 2754 S+ 0:27.50 0 0 0 2551160 114480 - 0 96,3 2,7 python2.7 issue11849_test.py
GC 2754 S+ 0:27.75 0 0 0 2454904 18992 - 0 97,2 0,5 python2.7 issue11849_test.py
*** 2754 S+ 0:27.75 0 0 0 2454904 18992 - 0 3,0 0,5 python2.7 issue11849_test.py

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
kaifeng <cafeeee@gmail.com> added the comment:

Python 3.2 On Linux (CentOS 5.3)

*** Python 3.2.0 final
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 15116 pts/0 S+ 0:00 1 1316 11055 6452 0.6 python3.2 issue11849_test.py
1 15116 pts/0 S+ 0:02 1 1316 53155 47340 4.5 python3.2 issue11849_test.py
2 15116 pts/0 S+ 0:05 1 1316 91051 86364 8.3 python3.2 issue11849_test.py
3 15116 pts/0 S+ 0:08 1 1316 129067 124232 12.0 python3.2 issue11849_test.py
4 15116 pts/0 S+ 0:10 1 1316 166587 162096 15.6 python3.2 issue11849_test.py
5 15116 pts/0 S+ 0:13 1 1316 204483 198824 19.2 python3.2 issue11849_test.py
6 15116 pts/0 S+ 0:17 1 1316 242375 236692 22.8 python3.2 issue11849_test.py
7 15116 pts/0 S+ 0:19 1 1316 284383 277528 26.8 python3.2 issue11849_test.py
8 15116 pts/0 S+ 0:23 1 1316 318371 312452 30.1 python3.2 issue11849_test.py
9 15116 pts/0 S+ 0:25 1 1316 360235 353288 34.1 python3.2 issue11849_test.py
END 15116 pts/0 S+ 0:30 1 1316 393975 388176 37.4 python3.2 issue11849_test.py
GC 15116 pts/0 S+ 0:30 1 1316 352035 347656 33.5 python3.2 issue11849_test.py
*** 15116 pts/0 S+ 0:30 1 1316 352035 347656 33.5 python3.2 issue11849_test.py

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Charles-Francois Natali <neologix@free.fr> added the comment:

The "problem" is not with Python, but with your libc.
When a program - such as Python - returns memory, it uses the free(3) library call.
But the libc is free to either return the memory immediately to the kernel using the relevant syscall (brk, munmap), or keep it around just in case (to simplify).
It seems that RHEL5 and onwards tend to keep a lot of memory around, at least in this case (probably because of the allocation pattern).

To sum up, python is returning memory, but your libc is not.
You can force it using malloc_trim, see the attached patch (I'm not at all suggesting its inclusion, it's just an illustration).

Results with current code:

*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 29823 pts/0 S+ 0:00 1 1607 168176 8596 0.2 ./python /tmp/issue11849_test.py
1 29823 pts/0 S+ 0:01 1 1607 249400 87088 2.2 ./python /tmp/issue11849_test.py
2 29823 pts/0 S+ 0:03 1 1607 324080 161704 4.1 ./python /tmp/issue11849_test.py
3 29823 pts/0 S+ 0:04 1 1607 398960 235036 5.9 ./python /tmp/issue11849_test.py
4 29823 pts/0 S+ 0:06 1 1607 473356 309464 7.8 ./python /tmp/issue11849_test.py
5 29823 pts/0 S+ 0:07 1 1607 548120 384624 9.8 ./python /tmp/issue11849_test.py
6 29823 pts/0 S+ 0:09 1 1607 622884 458332 11.6 ./python /tmp/issue11849_test.py
7 29823 pts/0 S+ 0:10 1 1607 701864 535736 13.6 ./python /tmp/issue11849_test.py
8 29823 pts/0 S+ 0:12 1 1607 772440 607988 15.5 ./python /tmp/issue11849_test.py
9 29823 pts/0 S+ 0:13 1 1607 851156 685384 17.4 ./python /tmp/issue11849_test.py
END 29823 pts/0 S+ 0:16 1 1607 761712 599400 15.2 ./python /tmp/issue11849_test.py
GC 29823 pts/0 S+ 0:16 1 1607 680900 519280 13.2 ./python /tmp/issue11849_test.py
*** 29823 pts/0 S+ 0:16 1 1607 680900 519288 13.2 ./python /tmp/issue11849_test.py


Results with the malloc_trim:

*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 30020 pts/0 S+ 0:00 1 1607 168180 8596 0.2 ./python /tmp/issue11849_test.py
1 30020 pts/0 S+ 0:01 1 1607 249404 86160 2.1 ./python /tmp/issue11849_test.py
2 30020 pts/0 S+ 0:03 1 1607 324084 160596 4.0 ./python /tmp/issue11849_test.py
3 30020 pts/0 S+ 0:04 1 1607 398964 235036 5.9 ./python /tmp/issue11849_test.py
4 30020 pts/0 S+ 0:06 1 1607 473360 309808 7.9 ./python /tmp/issue11849_test.py
5 30020 pts/0 S+ 0:07 1 1607 548124 383896 9.7 ./python /tmp/issue11849_test.py
6 30020 pts/0 S+ 0:09 1 1607 622888 458716 11.7 ./python /tmp/issue11849_test.py
7 30020 pts/0 S+ 0:10 1 1607 701868 536124 13.6 ./python /tmp/issue11849_test.py
8 30020 pts/0 S+ 0:12 1 1607 772444 607212 15.4 ./python /tmp/issue11849_test.py
9 30020 pts/0 S+ 0:14 1 1607 851160 684608 17.4 ./python /tmp/issue11849_test.py
END 30020 pts/0 S+ 0:16 1 1607 761716 599524 15.3 ./python /tmp/issue11849_test.py
GC 30020 pts/0 S+ 0:16 1 1607 680776 10744 0.2 ./python /tmp/issue11849_test.py
*** 30020 pts/0 S+ 0:16 1 1607 680776 10752 0.2 ./python /tmp/issue11849_test.py

----------
nosy: +neologix

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Changes by Charles-Francois Natali <neologix@free.fr>:


----------
keywords: +patch
Added file: http://bugs.python.org/file21696/gc_trim.diff

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Antoine Pitrou <pitrou@free.fr> added the comment:

> To sum up, python is returning memory, but your libc is not.
> You can force it using malloc_trim, see the attached patch (I'm not at
> all suggesting its inclusion, it's just an illustration).

That's an interesting thing, perhaps you want to open a feature request as a separate issue?

----------
nosy: +pitrou

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
kaifeng <cafeeee@gmail.com> added the comment:

I added 'malloc_trim' to the test code and rerun the test with Python 2.5 / 3.2 on CentOS 5.3. The problem still exists.


*** Python 2.5.5 final
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 2567 pts/0 S+ 0:00 0 1 8206 4864 0.4 /home/zkf/.programs/python/bin/python issue11849_test.py
1 2567 pts/0 S+ 0:03 0 1 44558 41140 3.9 /home/zkf/.programs/python/bin/python issue11849_test.py
2 2567 pts/0 S+ 0:07 0 1 81166 77728 7.5 /home/zkf/.programs/python/bin/python issue11849_test.py
3 2567 pts/0 S+ 0:12 0 1 117798 114316 11.0 /home/zkf/.programs/python/bin/python issue11849_test.py
4 2567 pts/0 S+ 0:17 0 1 154402 150912 14.5 /home/zkf/.programs/python/bin/python issue11849_test.py
5 2567 pts/0 S+ 0:23 0 1 191018 187500 18.1 /home/zkf/.programs/python/bin/python issue11849_test.py
6 2567 pts/0 S+ 0:29 0 1 227630 224084 21.6 /home/zkf/.programs/python/bin/python issue11849_test.py
7 2567 pts/0 S+ 0:36 0 1 264242 260668 25.1 /home/zkf/.programs/python/bin/python issue11849_test.py
8 2567 pts/0 S+ 0:44 0 1 300882 297288 28.7 /home/zkf/.programs/python/bin/python issue11849_test.py
9 2567 pts/0 S+ 0:53 0 1 337230 333860 32.2 /home/zkf/.programs/python/bin/python issue11849_test.py
END 2567 pts/0 S+ 1:02 0 1 373842 370444 35.7 /home/zkf/.programs/python/bin/python issue11849_test.py
GC 2567 pts/0 S+ 1:02 0 1 373842 370444 35.7 /home/zkf/.programs/python/bin/python issue11849_test.py
*** 2567 pts/0 S+ 1:02 0 1 373714 370436 35.7 /home/zkf/.programs/python/bin/python issue11849_test.py


*** Python 3.2.0 final
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 2633 pts/0 S+ 0:00 1 1316 11051 6448 0.6 python3.2 issue11849_test.py
1 2633 pts/0 S+ 0:02 1 1316 53151 47340 4.5 python3.2 issue11849_test.py
2 2633 pts/0 S+ 0:05 1 1316 91051 85216 8.2 python3.2 issue11849_test.py
3 2633 pts/0 S+ 0:08 1 1316 128943 124228 12.0 python3.2 issue11849_test.py
4 2633 pts/0 S+ 0:11 1 1316 166803 162296 15.6 python3.2 issue11849_test.py
5 2633 pts/0 S+ 0:14 1 1316 204475 199972 19.3 python3.2 issue11849_test.py
6 2633 pts/0 S+ 0:17 1 1316 243831 238180 23.0 python3.2 issue11849_test.py
7 2633 pts/0 S+ 0:20 1 1316 284371 277532 26.8 python3.2 issue11849_test.py
8 2633 pts/0 S+ 0:23 1 1316 318187 312456 30.1 python3.2 issue11849_test.py
9 2633 pts/0 S+ 0:26 1 1316 360231 353296 34.1 python3.2 issue11849_test.py
END 2633 pts/0 S+ 0:30 1 1316 393971 388184 37.4 python3.2 issue11849_test.py
GC 2633 pts/0 S+ 0:30 1 1316 352031 347652 33.5 python3.2 issue11849_test.py
*** 2633 pts/0 S+ 0:31 1 1316 351903 347524 33.5 python3.2 issue11849_test.py

----------
versions: +Python 2.7, Python 3.2
Added file: http://bugs.python.org/file21700/issue11849_test2.py

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
kaifeng <cafeeee@gmail.com> added the comment:

Found a minor defect of Python 3.2 / 3.3: line 1676 of xml/etree/ElementTree.py
was:
del self.target, self._parser # get rid of circular references
should be:
del self.target, self._target, self.parser, self._parser # get rid of circular references

While it doesn't help this issue...

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Charles-Francois Natali <neologix@free.fr> added the comment:

> kaifeng <cafeeee@gmail.com> added the comment:
>
> I added 'malloc_trim' to the test code and rerun the test with Python 2.5 / 3.2 on CentOS 5.3.  The problem still exists.
>

Well, malloc_trim can fail, but how did you "add" it ? Did you use
patch to apply the diff ?
Also, could you post the output of a
ltrace -e malloc_trim python <test script>

For info, the sample outputs I posted above come from a RHEL6 box.

Anyway, I'm 99% sure this isn't a leak but a malloc issue (valgrind
--tool=memcheck could confirm this if you want to try, I could be
wrong, it wouldn't be the first time ;-) ).
By the way, look at what I just found:
http://mail.gnome.org/archives/xml/2008-February/msg00003.html

> Antoine Pitrou <pitrou@free.fr> added the comment:
> That's an interesting thing, perhaps you want to open a feature request as a separate issue?

Dunno.
Memory management is a domain which belongs to the operating
system/libc, and I think applications should mess with it (apart from
specific cases) .
I don't have time to look at this precise problem in greater detail
right now, but AFAICT, this looks either like a glibc bug, or at least
a corner case with default malloc parameters (M_TRIM_THRESHOLD and
friends), affecting only RHEL and derived distributions.
malloc_trim should be called automatically by free if the amount of
memory that could be release is above M_TRIM_THRESHOLD.
Calling it systematically can have a non-negligible performance impact.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
kaifeng <cafeeee@gmail.com> added the comment:

I applied your patch to Python 3.2, also I added a function call to 'malloc_trim' via ctypes, as you can see in issue11849_test2.py.

In fact I have a daemon written in Python 2.5, parsing an XML of size 10+ MB every 5 minutes, after 16+ hours running, the program finally exhausted 4 GB memory and died. I simplified the logic of the daemon and found ElementTree eats too much memory. There comes the attached test script.

BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing memory usage disappeared.


$ ltrace -e malloc_trim python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
--- SIGCHLD (Child exited) ---
*** Python 3.2.0 final
--- SIGCHLD (Child exited) ---
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
--- SIGCHLD (Child exited) ---
0 13708 pts/1 S+ 0:00 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:00 1 1316 11055 6440 0.6 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
1 13708 pts/1 S+ 0:00 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:03 1 1316 53155 47332 4.5 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
2 13708 pts/1 S+ 0:00 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:06 1 1316 91055 85204 8.2 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
3 13708 pts/1 S+ 0:01 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:10 1 1316 128947 124212 11.9 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
4 13708 pts/1 S+ 0:01 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:13 1 1316 166807 162280 15.6 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
5 13708 pts/1 S+ 0:01 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:16 1 1316 204483 198808 19.2 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
6 13708 pts/1 S+ 0:02 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:20 1 1316 242379 236672 22.8 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
7 13708 pts/1 S+ 0:02 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:23 1 1316 284383 277508 26.8 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
8 13708 pts/1 S+ 0:03 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:27 1 1316 318191 312436 30.1 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
9 13708 pts/1 S+ 0:03 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:29 1 1316 360199 353272 34.1 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
END 13708 pts/1 S+ 0:03 1 65 1742 636 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:34 1 1316 393975 388164 37.4 python3.2 Issue11849_test2.py
malloc_trim(0, 0, 0x818480a, 0x81a0114, 0xbfb6c940) = 1
--- SIGCHLD (Child exited) ---
GC 13708 pts/1 S+ 0:03 1 65 1742 648 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:35 1 1316 351871 347480 33.5 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
*** 13708 pts/1 S+ 0:03 1 65 1742 648 0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1 S+ 0:35 1 1316 351871 347480 33.5 python3.2 Issue11849_test2.py
+++ exited (status 0) +++

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Charles-Francois Natali <neologix@free.fr> added the comment:

> BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing memory usage disappeared.

If you looked at the link I posted, you'll see that lxml had some similar issues and solved it by calling malloc_trim systematically when freeing memory.
It could also be heap fragmentation, though.

To go further, it'd be nice if you could provide the output of
valgrind --tool=memcheck --leak-check=full --suppressions=Misc/valgrind-python.supp python <test script>
after uncommenting relevant lines in Misc/valgrind-python.supp (see http://svn.python.org/projects/python/trunk/Misc/README.valgrind ).
It will either confirm a memory leak or malloc issue (I still favour the later).

By the way, does

while True:
XML(gen_xml())

lead to a constant memory usage increase ?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue11849] ElementTree memory leak [ In reply to ]
Charles-Francois Natali <neologix@free.fr> added the comment:

This is definitely a malloc bug.
Test with default malloc on a Debian box:

cf@neobox:~/cpython$ ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3778 pts/2 S+ 0:00 1 1790 8245 7024 0.5 ./python ../issue11849_test.py
1 3778 pts/2 S+ 0:17 1 1790 61937 60404 4.6 ./python ../issue11849_test.py
2 3778 pts/2 S+ 0:35 1 1790 110841 108300 8.3 ./python ../issue11849_test.py
3 3778 pts/2 S+ 0:53 1 1790 159885 158540 12.2 ./python ../issue11849_test.py
4 3778 pts/2 S+ 1:10 1 1790 209369 206724 15.9 ./python ../issue11849_test.py
5 3778 pts/2 S+ 1:28 1 1790 258505 255956 19.7 ./python ../issue11849_test.py
6 3778 pts/2 S+ 1:46 1 1790 307669 304964 23.5 ./python ../issue11849_test.py
7 3778 pts/2 S+ 2:02 1 1790 360705 356952 27.5 ./python ../issue11849_test.py
8 3778 pts/2 S+ 2:21 1 1790 405529 404172 31.2 ./python ../issue11849_test.py
9 3778 pts/2 S+ 2:37 1 1790 458789 456128 35.2 ./python ../issue11849_test.py
END 3778 pts/2 S+ 3:00 1 1790 504189 501624 38.7 ./python ../issue11849_test.py
GC 3778 pts/2 S+ 3:01 1 1790 454689 453476 35.0 ./python ../issue11849_test.py
*** 3778 pts/2 S+ 3:01 1 1790 454689 453480 35.0 ./python ../issue11849_test.py
[56426 refs]


The heap is not trimmed, even after GC collection.
Now, using a smaller mmap threshold so that malloc uses mmap instead of brk:

cf@neobox:~/cpython$ MALLOC_MMAP_THRESHOLD_=1024 ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3843 pts/2 S+ 0:00 1 1790 8353 7036 0.5 ./python ../issue11849_test.py
1 3843 pts/2 S+ 0:17 1 1790 62593 59240 4.5 ./python ../issue11849_test.py
2 3843 pts/2 S+ 0:35 1 1790 112321 108304 8.3 ./python ../issue11849_test.py
3 3843 pts/2 S+ 0:53 1 1790 162313 157372 12.1 ./python ../issue11849_test.py
4 3843 pts/2 S+ 1:11 1 1790 212057 206456 15.9 ./python ../issue11849_test.py
5 3843 pts/2 S+ 1:29 1 1790 261749 255484 19.7 ./python ../issue11849_test.py
6 3843 pts/2 S+ 1:47 1 1790 311669 304484 23.5 ./python ../issue11849_test.py
7 3843 pts/2 S+ 2:03 1 1790 365485 356488 27.5 ./python ../issue11849_test.py
8 3843 pts/2 S+ 2:22 1 1790 411341 402568 31.1 ./python ../issue11849_test.py
9 3843 pts/2 S+ 2:38 1 1790 465141 454552 35.1 ./python ../issue11849_test.py
END 3843 pts/2 S+ 3:02 1 1790 67173 63892 4.9 ./python ../issue11849_test.py
GC 3843 pts/2 S+ 3:03 1 1790 9925 8664 0.6 ./python ../issue11849_test.py
*** 3843 pts/2 S+ 3:03 1 1790 9925 8668 0.6 ./python ../issue11849_test.py
[56428 refs]

Just to be sure, with ptmalloc3 malloc implementation:

cf@neobox:~/cpython$ LD_PRELOAD=../ptmalloc3/libptmalloc3.so ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3898 pts/2 S+ 0:00 1 1790 8369 7136 0.5 ./python ../issue11849_test.py
1 3898 pts/2 S+ 0:17 1 1790 62825 60264 4.6 ./python ../issue11849_test.py
2 3898 pts/2 S+ 0:34 1 1790 112641 110176 8.5 ./python ../issue11849_test.py
3 3898 pts/2 S+ 0:52 1 1790 162689 160048 12.3 ./python ../issue11849_test.py
4 3898 pts/2 S+ 1:09 1 1790 212285 209732 16.2 ./python ../issue11849_test.py
5 3898 pts/2 S+ 1:27 1 1790 261881 259460 20.0 ./python ../issue11849_test.py
6 3898 pts/2 S+ 1:45 1 1790 311929 309332 23.9 ./python ../issue11849_test.py
7 3898 pts/2 S+ 2:01 1 1790 365625 362004 27.9 ./python ../issue11849_test.py
8 3898 pts/2 S+ 2:19 1 1790 411445 408812 31.5 ./python ../issue11849_test.py
9 3898 pts/2 S+ 2:35 1 1790 465205 461536 35.6 ./python ../issue11849_test.py
END 3898 pts/2 S+ 2:58 1 1790 72141 69688 5.3 ./python ../issue11849_test.py
GC 3898 pts/2 S+ 2:59 1 1790 15001 13748 1.0 ./python ../issue11849_test.py
*** 3898 pts/2 S+ 2:59 1 1790 15001 13752 1.0 ./python ../issue11849_test.py
[56428 refs]

So the problem is really that glibc/eglibc malloc implementations don't automatically trim memory upon free (this happens if you're only allocating/deallocating small chunks < 64B that come from fastbins, but that's not the case here).
By the way, I noticed that dictionnaries are never allocated through pymalloc, since a new dictionnary takes more than 256B...

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11849>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com