4

I have multiple test files in the test folder. The structure is similar to something like this:

/test
----test_abc.py
----test_bcd.py
----test_cde.py
----conftest.py

The conftest.py contains all the spark context initialization which is necessary for running the unit test. My problem is I would like to have a test.py file which internally triggers all the test_abc.py, test_bcd.py and test_cde.py. It becomes very easy when we deal with utit_test module of python but I am not sure how to obtain this through pytest module. Do let me know if any more clarification required on the question.

The conftest.py looks something like this:

import pytest
from pyspark import SQLContext
from pyspark import SparkConf
from pyspark import SparkContext
from pyspark.streaming import StreamingContext

@pytest.fixture(scope="session")
def spark_context(request):
    conf = (SparkConf().setMaster("local[2]").setAppName("pytest-pyspark-local-testing"))
    request.addfinalizer(lambda: sc.stop())
    sc = SparkContext(conf=conf).getOrCreate()
    return sc

And one of the test_abc.py looks something like this:

import pytest
import os
from pyspark.sql import SQLContext
pytestmark = pytest.mark.usefixtures("spark_context")
def test_get_pc_browser_sql(spark_context):
    "assert something"
Al Imran
  • 882
  • 7
  • 29
talkdatatome
  • 614
  • 1
  • 10
  • 17
  • 1
    `if __name__ == '__main__': pytest.main(args=['test'])`? – hoefling Dec 10 '18 at 10:11
  • the 'test' you are referring to are the different test_*.py files? – talkdatatome Dec 10 '18 at 10:46
  • @hoefling the solution works when we put test.py outside the test folder and call it. Is there any possibility that I can place the test.py inside the test folder and try running the testfiles through test.py – talkdatatome Dec 10 '18 at 11:11

2 Answers2

1

I'd recommend just using a bash script and use that to call command line python calls. For example, in you bash script you can just write:

pytest test/

to run all tests within the test directory. You can also add all argument commands and any commands that use on the command line. Then, just execute the bash script for the specific set of files or directories to test. Look at pytest Documentation for reference.

slayer
  • 643
  • 7
  • 14
  • 1
    We don;t give external link as a solution rather explain the user your answer and may be can give link for more reference. – LOrD_ARaGOrN Dec 10 '18 at 06:15
  • 1
    My question was I want to have a master .py file where I would call all the other test_*.py files. A bash script would do my work but I dont want to do this through bash script – talkdatatome Dec 10 '18 at 06:31
  • [already answered?](https://stackoverflow.com/questions/7974849/how-can-i-make-one-python-file-run-another) – slayer Dec 10 '18 at 06:44
1

I had a similar need where I want to run only some test files and not an entire folder. In order to have pytest run several files with a single command, if you rename the files you want to run as a group, bash provides a couple of different ways to run such a group.

In this example, I want to run all tests that start with the name testpi:

  1. Gathers all matching files and puts then in a string:
$ pytest -v $(ls tests/testpi*)
  1. Pipe the output of ls to xargs to do the same:
ls tests/testpi* | xargs pytest -v
Scott
  • 1,333
  • 1
  • 14
  • 19