I'm working on a project to parse multiple xml files concurrently in python using lxml. When I initialize the process I want my main class to do some work on the XML before it passes the etree object to the process, but I am finding that when the etree object arrives in the new process the class survives but the XML is gone from within the object and getroot() returns None.
I know that I can only pass pickable data using the queue, but is this also the case with what I pass to the process inside the 'args' field?
Here's my code:
import multiprocessing, multiprocessing.pool, time
from lxml import etree
def compute(tree):
print("Start Process")
print(type(tree)) # Returns <class 'lxml.etree._ElementTree'>
print(id(tree)) # Returns new ID 44637320 as expected
print(tree.getroot()) # Returns None
def pool_init(queue):
# see http://stackoverflow.com/a/3843313/852994
compute.queue = queue
class Main():
def __init__(self):
pass
def main(self):
tree = etree.parse('test.xml')
print(id(tree)) # Returns object ID 43998536
print(tree.getroot()) #Returns <Element SymCLI_ML at 0x29f5dc8>
self.queue = multiprocessing.Queue()
self.pool = multiprocessing.Pool(processes=1, initializer=pool_init, initargs=(self.queue,))
self.pool.apply_async(func=compute, args=(tree,))
time.sleep(10)
if __name__ == '__main__':
Main().main()
Any and all help much appreciated.
UPDATE/ANSWER
Based on the answer in the next post down I've modified it a bit and managed to get it working with a much lower memory footprint without using String IO. The etree.tostring method returns a byte array, which can be pickled, then to unpickle it the byte array can be parsed by etree.
import multiprocessing, multiprocessing.pool, time, copyreg
from lxml import etree
def compute(tree):
print("Start Process")
print(type(tree)) # Returns <class 'lxml.etree._ElementTree'>
print(tree.getroot()) # Returns <Element SymCLI_ML at 0x29f5dc8>. Success!
def pool_init(queue):
# see http://stackoverflow.com/a/3843313/852994
compute.queue = queue
def elementtree_unpickler(data):
return etree.parse(BytesIO(data))
def elementtree_pickler(tree):
return elementtree_unpickler, (etree.tostring(tree),)
copyreg.pickle(etree._ElementTree, elementtree_pickler, elementtree_unpickler)
class Main():
def __init__(self):
pass
def main(self):
tree = etree.parse('test.xml')
print(tree.getroot()) #Returns <Element SymCLI_ML at 0x29f5dc8>
self.queue = multiprocessing.Queue()
self.pool = multiprocessing.Pool(processes=1, initializer=pool_init, initargs=(self.queue,))
self.pool.apply_async(func=compute, args=(tree,))
time.sleep(10)
if __name__ == '__main__':
Main().main()
UPDATE 2
After doing some bench-marking with memory I found that passing large objects causes the objects to not be able to be cleared up by garbage collection on the main process. This probably isn't an issue at small scale, but by etree objects were in the order of multiple hundreds of MB in memory. As soon as an async task has been called with an XML object in the statement, that object cannot be cleared from memory if it is deleted from the main process, even my manually invoking garbage collection. So as a result I've reverted to closing the XML in the main process and passing the file name to the sub-process.
