Action against software patentsGnome2 LogoW3C LogoRed Hat Logo
Made with Libxml2 Logo

The XML C parser and toolkit of Gnome

Memory Management

Developer Menu
API Indexes
Related links

Table of Content:

  1. General overview
  2. Setting libxml2 set of memory routines
  3. Cleaning up after parsing
  4. Debugging routines
  5. General memory requirements

General overview

The module xmlmemory.hprovides the interfaces to the libxml2 memory system:

  • libxml2 does not use the libc memory allocator directly but xmlFree(),xmlMalloc() and xmlRealloc()
  • those routines can be reallocated to a specific set of routine, bydefault the libc ones i.e. free(), malloc() and realloc()
  • the xmlmemory.c module includes a set of debugging routine

Setting libxml2 set of memory routines

It is sometimes useful to not use the default memory allocator, either fordebugging, analysis or to implement a specific behaviour on memory management(like on embedded systems). Two function calls are available to do so:

  • xmlMemGet()which return the current set of functions in use by the parser
  • xmlMemSetup()which allow to set up a new set of memory allocation functions

Of course a call to xmlMemSetup() should probably be done before callingany other libxml2 routines (unless you are sure your allocations routines arecompatibles).

Cleaning up after parsing

Libxml2 is not stateless, there is a few set of memory structures needingallocation before the parser is fully functional (some encoding structuresfor example). This also mean that once parsing is finished there is a tinyamount of memory (a few hundred bytes) which can be recollected if you don'treuse the parser immediately:

  • xmlCleanupParser()is a centralized routine to free the parsing states. Note that itwon't deallocate any produced tree if any (use the xmlFreeDoc() andrelated routines for this).
  • xmlInitParser()is the dual routine allowing to preallocate the parsing statewhich can be useful for example to avoid initialization reentrancyproblems when using libxml2 in multithreaded applications

Generally xmlCleanupParser() is safe, if needed the state will be rebuildat the next invocation of parser routines, but be careful of the consequencesin multithreaded applications.

Debugging routines

When configured using --with-mem-debug flag (off by default), libxml2 usesa set of memory allocation debugging routines keeping track of all allocatedblocks and the location in the code where the routine was called. A couple ofother debugging routines allow to dump the memory allocated infos to a fileor call a specific routine when a given block number is allocated:

When developing libxml2 memory debug is enabled, the tests programs callxmlMemoryDump () and the "make test" regression tests will check for anymemory leak during the full regression test sequence, this helps a lotensuring that libxml2 does not leak memory and bullet proof memoryallocations use (some libc implementations are known to be far too permissiveresulting in major portability problems!).

If the .memdump reports a leak, it displays the allocation function andalso tries to give some informations about the content and structure of theallocated blocks left. This is sufficient in most cases to find the culprit,but not always. Assuming the allocation problem is reproducible, it ispossible to find more easily:

  1. write down the block number xxxx not allocated
  2. export the environment variable XML_MEM_BREAKPOINT=xxxx , the easiestwhen using GDB is to simply give the command

    set environment XML_MEM_BREAKPOINT xxxx

    before running the program.

  3. run the program under a debugger and set a breakpoint onxmlMallocBreakpoint() a specific function called when this precise blockis allocated
  4. when the breakpoint is reached you can then do a fine analysis of theallocation an step to see the condition resulting in the missingdeallocation.

I used to use a commercial tool to debug libxml2 memory problems but afternoticing that it was not detecting memory leaks that simple mechanism wasused and proved extremely efficient until now. Lately I have also used valgrindwith quite somesuccess, it is tied to the i386 architecture since it works by emulating theprocessor and instruction set, it is slow but extremely efficient, i.e. itspot memory usage errors in a very precise way.

General memory requirements

How much libxml2 memory require ? It's hard to tell in average it dependsof a number of things:

  • the parser itself should work in a fixed amount of memory, except forinformation maintained about the stacks of names and entities locations.The I/O and encoding handlers will probably account for a few KBytes.This is true for both the XML and HTML parser (though the HTML parserneed more state).
  • If you are generating the DOM tree then memory requirements will grownearly linear with the size of the data. In general for a balancedtextual document the internal memory requirement is about 4 times thesize of the UTF8 serialization of this document (example the XML-1.0recommendation is a bit more of 150KBytes and takes 650KBytes of mainmemory when parsed). Validation will add a amount of memory required formaintaining the external Dtd state which should be linear with thecomplexity of the content model defined by the Dtd
  • If you need to work with fixed memory requirements or don't need thefull DOM tree then using the xmlReaderinterfaceis probably the best way to proceed, it still allows tovalidate or operate on subset of the tree if needed.
  • If you don't care about the advanced features of libxml2 likevalidation, DOM, XPath or XPointer, don't use entities, need to work withfixed memory requirements, and try to get the fastest parsing possiblethen the SAX interface should be used, but it has known restrictions.

Daniel Veillard