apport report.py migrating to python3

Question

Was trying to migrate the project listed here Which uses multiprocessing, I get the following error:

/usr/lib/python3/dist-packages/apport/report.py:13: PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "PubMedParser.py", line 593, in _start_parser
    s = p._parse()
  File "PubMedParser.py", line 64, in _parse
    event, root = context.next()
AttributeError: '_IterParseIterator' object has no attribute 'next'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "PubMedParser.py", line 658, in <module>
    run(options.medline_path, options.clean, int(options.start), options.end, int(options.PROCESSES))
  File "PubMedParser.py", line 621, in run
    res = result.get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
AttributeError: '_IterParseIterator' object has no attribute 'next'

Is it the problem of code or is it problem in python3 (3.5.2 is the version I use)?

After implementing the suggestion of Patrick Artner more different error comes up but not sure if this is related to earlier error or not:

/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/crud.py:700: SAWarning: Column 'pubmed.tbl_xml_file.id' is marked as a member of the primary key for table 'pubmed.tbl_xml_file', but has no Python-side or server-side default generator indicated, nor does it indicate 'autoincrement=True' or 'nullable=True', and no explicit value is passed.  Primary key columns typically may not store NULL. Note that as of SQLAlchemy 1.1, 'autoincrement=True' must be indicated explicitly for composite (e.g. multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is expected for one of the columns in the primary key. CREATE TABLE statements are impacted by this change as well on most backends.
  util.warn(msg)
/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/crud.py:700: SAWarning: Column 'pubmed.tbl_xml_file.id' is marked as a member of the primary key for table 'pubmed.tbl_xml_file', but has no Python-side or server-side default generator indicated, nor does it indicate 'autoincrement=True' or 'nullable=True', and no explicit value is passed.  Primary key columns typically may not store NULL. Note that as of SQLAlchemy 1.1, 'autoincrement=True' must be indicated explicitly for composite (e.g. multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is expected for one of the columns in the primary key. CREATE TABLE statements are impacted by this change as well on most backends.
  util.warn(msg)
/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/crud.py:700: SAWarning: Column 'pubmed.tbl_xml_file.id' is marked as a member of the primary key for table 'pubmed.tbl_xml_file', but has no Python-side or server-side default generator indicated, nor does it indicate 'autoincrement=True' or 'nullable=True', and no explicit value is passed.  Primary key columns typically may not store NULL. Note that as of SQLAlchemy 1.1, 'autoincrement=True' must be indicated explicitly for composite (e.g. multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is expected for one of the columns in the primary key. CREATE TABLE statements are impacted by this change as well on most backends.
  util.warn(msg)
/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/crud.py:700: SAWarning: Column 'pubmed.tbl_xml_file.id' is marked as a member of the primary key for table 'pubmed.tbl_xml_file', but has no Python-side or server-side default generator indicated, nor does it indicate 'autoincrement=True' or 'nullable=True', and no explicit value is passed.  Primary key columns typically may not store NULL. Note that as of SQLAlchemy 1.1, 'autoincrement=True' must be indicated explicitly for composite (e.g. multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is expected for one of the columns in the primary key. CREATE TABLE statements are impacted by this change as well on most backends.
  util.warn(msg)
/usr/lib/python3/dist-packages/apport/report.py:13: PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: null value in column "id" violates not-null constraint
DETAIL:  Failing row contains (null, medline_444.xml, null, null, null, 2018-01-31 20:33:00.183484).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "PubMedParser.py", line 594, in _start_parser
    s = p._parse()
  File "PubMedParser.py", line 574, in _parse
    self.session.commit()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2243, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2369, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2333, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
    execute(statement, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in column "id" violates not-null constraint
DETAIL:  Failing row contains (null, medline_444.xml, null, null, null, 2018-01-31 20:33:00.183484).
 [SQL: 'INSERT INTO pubmed.tbl_xml_file (xml_file_name, doc_type_name, dtd_public_id, dtd_system_id, time_processed) VALUES (%(xml_file_name)s, %(doc_type_name)s, %(dtd_public_id)s, %(dtd_system_id)s, %(time_processed)s)'] [parameters: {'dtd_public_id': None, 'dtd_system_id': None, 'xml_file_name': 'medline_444.xml', 'doc_type_name': None, 'time_processed': datetime.datetime(2018, 1, 31, 20, 33, 0, 183484)}] (Background on this error at: http://sqlalche.me/e/gkpj)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "PubMedParser.py", line 659, in <module>
    run(options.medline_path, options.clean, int(options.start), options.end, int(options.PROCESSES))
  File "PubMedParser.py", line 622, in run
    res = result.get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in column "id" violates not-null constraint
DETAIL:  Failing row contains (null, medline_444.xml, null, null, null, 2018-01-31 20:33:00.183484).
 [SQL: 'INSERT INTO pubmed.tbl_xml_file (xml_file_name, doc_type_name, dtd_public_id, dtd_system_id, time_processed) VALUES (%(xml_file_name)s, %(doc_type_name)s, %(dtd_public_id)s, %(dtd_system_id)s, %(time_processed)s)'] [parameters: {'dtd_public_id': None, 'dtd_system_id': None, 'xml_file_name': 'medline_444.xml', 'doc_type_name': None, 'time_processed': datetime.datetime(2018, 1, 31, 20, 33, 0, 183484)}] (Background on this error at: http://sqlalche.me/e/gkpj)

Patrick Artner · Answer 1 · 2018-01-31T19:32:06.200

1

PendingDeprecationWarning: 
the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

says it all. The devs on Python decided to remove imp in favor or importlib - it is (yet) still in the codebase but will be removed. Go to the modules documentation and read up on how to translate imp code into importlib code.

https://docs.python.org/3/library/imp.html
Deprecated since version 3.4: The imp package is pending deprecation in favor of importlib.

This is just warning (heed it now, or fix it when you update python to a version where imp is removed).

Search SO, f.e.: Problems when converting from imp to importlib in python 3.4

The other stuff makes the program crash. Curtesy of https://stackoverflow.com/a/42742804/7505395 this might be fixable by changing

event, root = self.context.next()

to this:

event, root = context.__next__()

(untested).

edited Jan 31 '18 at 19:32

answered Jan 31 '18 at 19:23

Patrick Artner

50,409
9
43
69

I'm not sure if it resolved that error or not comes up with a different error now, – Kiran K Telukunta Jan 31 '18 at 19:34
python3/dist-packages/apport/report.py:13: PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import fnmatch, glob, traceback, errno, sys, atexit, locale, imp multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context context) File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute cursor.execute(statement, parameters) – Kiran K Telukunta Jan 31 '18 at 19:36
1

Please do not add stacktraces to comments, they are ill suited. either edit your question and add it or create a new one. As this is a different module the error stems from (sqlalchemy) you might want to create a new one with sqlalchemy as tag. (dbaccess stuff) havent used that myself in python. omit the warning, its not important and you now know why/what it is. – Patrick Artner Jan 31 '18 at 19:39

apport report.py migrating to python3

1 Answers1