I originally thought I was incorrectly passing data to my subprocesses so I asked a question How do I pass object data to other Python multiprocessing processes?. It got closed for a duplicate but after more research on the issue and implementing a Queue per the other links provided I don't think that is the problem.
Part of the data is making it to my process, a few of the other parts are not which are what I assume part of SQLAlchemy objects.
What was put into the Queue:
What I see in my process getting out of Queue:
So I believe the issue comes down to SQLAlchemy and its states. How can I pass a SQLAlchemy resultset to a process without it losing all of this data?
The sample code.
def test_mp(queue):
while True:
file = queue.get(True)
results = dbsession.query(Raw_records).limit(10).all()
inqueue = mp.Queue()
for item in results:
inqueue.put(item)
pool = mp.Pool(4, test_mp, (inqueue,))
pool.close()
pool.join()
edit: figured out the "drill_parameters_record" highlight why it isn't returning.
Had to do .options(joinedload(Raw_records.drill_parameters_record))
But the raw_xml column is still blank. It is a custom type so it is an object. I used this to create it. SQLAlchemy declaring custom UserDefinedType for XML in MSSQL
