flake8 ignore list isn't understood by Azure DevOps pipeline

Question

I want to capture logs that are sent to Stream, in ontology_tagger.ipynb.

Said file is 2 folders up/ back and 1 in:

from ontology_tagger.notebooks.ontology_tagger import main

I have tried various solutions without luck, from this post.

ModuleNotFoundError: No module named 'ontology_tagger.notebooks.ontology_tagger'

How do I import notebook MyCode.ipynb into testing.py, and invoke a function name?

Note: the unit test I want to work just so happens to test logging in my .ipynb file. Not to be confused with the debugging logs at the bottom.

Attempted Solution

pip install import_ipynb

and

import import_ipynb
from ontology_tagger.notebooks.ontology_tagger.ipynb import main

However, because I don't directly invoke import_ipynb, Azure DevOps throws a linting error:

#17 2.530 /home/worker/python/ontology_tagger/ontology_tagger/tests/test_ontology_tagger.py:19:1: F401 'import_ipynb' imported but unused

So, in Terminal:

flake8 --per-file-ignores="test_ontology_tagger.py:F401"

Yet, this above error still occurs :(

testing.py:

import unittest
from unittest import TestCase
import sys
import logging

from ontology_tagger.notebooks.ontology_tagger import main


class TestExample(TestCase):
    def test_logging(self):
        with self.assertLogs() as captured:
            ontology_tagger.main()  ########## HERE !
            print('captured.records: ', captured.records)
            print('list(captured.records): ', list(captured.records))
            self.assertTrue(len(captured.records) > 2)
            self.assertTrue("Started" in captured.records[0].getMessage())

            success_log_msgs = ['self.tokenizer.model_max_length >= DEFAULT_MODEL_MAX_LEN:', 'self._num_classes:']  # successful runtime logs in 'validation()' methods

            for slm in success_log_msgs:
                self.assertTrue(any(True for log in list(captured.records) if slm in log))  # If there's a subset or full occurance of log message; True
                self.assertTrue(len(captured.records) > len(success_log_msgs))  # check that there are general log messages

if __name__ == '__main__':
    unittest.main()

MyCode.ipynb:

def main():
    logger = logging.getLogger()
    streamHandler = logging.StreamHandler(sys.stdout)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    streamHandler.setFormatter(formatter)
    logger.addHandler(streamHandler)
    logger.error('Started')
    # ...
    logger.error('Finished')

Raw Build Logs on Azure DevOps:

#17 [test 5/5] RUN cd ontology_tagger && poetry run invoke deploy
#17 sha256:80a75555cf9574e61fc67c0cfb698e36af3784a913f0e5be863aa61acc4d7fc8
#17 2.600 ============================= test session starts ==============================
#17 2.600 platform linux -- Python 3.7.12, pytest-3.10.1, py-1.10.0, pluggy-1.0.0
#17 2.600 rootdir: /home/worker/python/ontology_tagger, inifile: pytest.ini
#17 2.600 collecting ... 
collecting 0 items / 1 errors                                                  
collected 0 items / 1 errors                                                   
#17 3.558 
#17 3.558 ==================================== ERRORS ====================================
#17 3.559 ________ ERROR collecting ontology_tagger/tests/test_ontology_tagger.py ________
#17 3.559 ImportError while importing test module '/home/worker/python/ontology_tagger/ontology_tagger/tests/test_ontology_tagger.py'.
#17 3.559 Hint: make sure your test modules/packages have valid Python names.
#17 3.559 Traceback:
#17 3.559 tests/test_ontology_tagger.py:19: in <module>
#17 3.559     from ontology_tagger.notebooks.ontology_tagger import main
#17 3.559 E   ModuleNotFoundError: No module named 'ontology_tagger.notebooks.ontology_tagger'
#17 3.559 !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
#17 3.559 =========================== 1 error in 0.96 seconds ============================
#17 ERROR: executor failed running [/bin/sh -c cd ontology_tagger && poetry run invoke deploy]: exit code: 2
------
 > [test 5/5] RUN cd ontology_tagger && poetry run invoke deploy:
------
executor failed running [/bin/sh -c cd ontology_tagger && poetry run invoke deploy]: exit code: 2
##[error]Bash exited with code '1'.
Finishing: Test worker

Please let me know if there's anything else I should add to post.

score 1 · Accepted Answer · answered Sep 30 '21 at 15:42

1

for per-file-ignores to work, you need a glob which matches your file or the path to that file

so if you use --per-file-ignores tests/test_ontology_tagger.py:F401 then it should work how you want

a better solution however is to use an inline ignore on the errant import:

import import_ipynb  # noqa: F401

disclaimer: I'm the current flake8 maintainer

answered Sep 30 '21 at 15:42

anthony sottile

61,815
15
148
207

I've just discovered this in-line notation. Running it now. Source: `https://flake8.pycqa.org/en/latest/user/violations.html#in-line-ignoring-errors` – StressedBoi69420 Sep 30 '21 at 15:51
I'm now getting: `ModuleNotFoundError: No module named 'import_ipynb'`. Yet, running `pip install import_ipynb` locally on my machine gives: `Requirement already satisfied`. We use `poetry`, so will need to add it and then push. – StressedBoi69420 Sep 30 '21 at 15:54
Will report back on Monday, UK Time – StressedBoi69420 Sep 30 '21 at 16:08

flake8 ignore list isn't understood by Azure DevOps pipeline

1 Answers1