I'm working with a Python daemon running on Python 3.7 on a Linux RedHat VM. The guts of the program are very CPU heavy so I've implemented Multiprocessing, threading was too slow. I'm required to make the program a daemon.
To overview, the daemon queries the DB and queues up all of the ids in a Queue. Then the process() function runs in each spawned process. The code works without error for 1-2 processes, but we want to run about 7. I've simplified the code below.
When I run the program with 7 processes I get the following error...
Traceback (most recent call last): File "python_daemon_multiprocessv4.py", line 77, in process cus_runid.execute("QUERY REMOVED”); File "/PATH-REMOVED/jaydebeapi/init.py", line 498, in execute self._prep = self._connection.jconn.prepareStatement(operation) jpype._jexception.java.lang.NoClassDefFoundErrorPyRaisable: java.lang.NoClassDefFoundError: com/ibm/db2/jcc/am/dg
It's an error on a db connection, again that works flawlessly with 1-2 processes. I think is is because the processes are sharing the same JVM. This program is all Python, I only use the JVM to connect to the database.
import time
import os
import jaydebeapi, sys
import multiprocessing
def bigsql_database_connection():
#creates a db connection with jaydebeapi
# Multiprocessing Variables
total_processes = 7
wait = 30
queue = multiprocessing.Queue()
# define daemon cursor
conn = bigsql_database_connection()
cus=conn.cursor()
def list_unprocessed_ids():
#returns ids to process
def process(queue):
conn = bigsql_database_connection()
while True:
try:
cus_runid=conn.cursor()
run_id_str = str(queue.get(True))
#here some db work and heavy data processing is completed
cus_runid.close()
except Exception as e:
cus_runid.close()
def daemon():
run_pool = multiprocessing.Pool(total_processes, process, (queue,))
while True:
try:
ids_to_process = list_unprocessed_ids()
if len(ids_to_process) >= 1:
for id in ids_to_process:
queue.put(str(id))
time.sleep(wait)
except Exception as e:
#handle error
return 0
daemon()
How can I give each process its own JVM so the processes don't fight over the db connection? (I'm pretty sure this is what is happening)
