Recommended way to execute tests within Docker container - docker

Question

I would like to run set of specific tests within Docker container and not sure how to tackle this. Tests I want to perform are security-related, like create user(s), manage GPG keys for them and similar - which I am reluctant to run on PC running the tests.
I tried pytest-xdist/socketserver combo and also copying tests into running Docker container and use pytest-json-report to get result(s) as json saved to a volume shared with the host, but not sure this approach is good.
For now, I would settle with all tests (without mark or similar other features) are executed "remotely" (in Docker) and results are presented like everything is ran on local PC.
Don't mind writing a specific plugin, but not sure if this is a good way: do I have to make sure than, my plugin is loaded before say, pytest-xdist (or some others)? Additionally, if I use say, pytest_sessionstart in my conftest.py to build Docker image that I can then target with xdist; but my tests have also some "dependency" that I have to put within conftest.py - I cant use same conftest.py within container and in my PC running the test.
Thank you in advance

Answer 1

In case anyone else maybe have similar need, I will share what I did.
First of all, there is already an excellent pytest-json-report to export JSON results. However, I made simpler and with less functionality plugin that uses pytest_report_to_serializable directly:
import json
from socket import gethostname
def pytest_addoption(parser, pluginmanager):
parser.addoption(
'--report-file', default='%s.json' % gethostname(), help='path to JSON report'
)
def pytest_configure(config):
plugin = JsonTestsExporter(config=config)
config._json_report = plugin
config.pluginmanager.register(plugin)
def pytest_unconfigure(config):
plugin = getattr(config, '_json_report', None)
if plugin is not None:
del config._json_report
config.pluginmanager.unregister(plugin)
print('Report saved in: %s' % config.getoption('--report-file'))
class JsonTestsExporter(object):
def __init__(self, config):
self._config = config
self._export_data = {'collected': 0, 'results': []}
def pytest_report_collectionfinish(self, config, start_path, startdir, items):
self._export_data['collected'] = len(items)
def pytest_runtest_logreport(self, report):
data = self._config.hook.pytest_report_to_serializable(
config=self._config, report=report
)
self._export_data['results'].append(data)
def pytest_sessionfinish(self, session):
report_file = self._config.getoption('--report-file')
with open(report_file, 'w+') as fd:
fd.write(json.dumps(self._export_data))
Reason beyond this is that I wanted results also imported using pytest_report_from_serializable.
Simplified Dockerfile:
FROM debian:buster-slim AS builder
COPY [ "requirements.txt", "run.py", "/artifacts/" ]
COPY [ "json_tests_exporter", "/artifacts/json_tests_exporter/" ]
RUN apt-get update\
# install necesssary packages
&& apt-get install --no-install-recommends -y python3-pip python3-setuptools\
# build json_tests_exporter *.whl
&& pip3 install wheel\
&& sh -c 'cd /artifacts/json_tests_exporter && python3 setup.py bdist_wheel'
FROM debian:buster-slim
ARG USER_UID=1000
ARG USER_GID=${USER_UID}
COPY --from=builder --chown=${USER_UID}:${USER_GID} /artifacts /artifacts
RUN apt-get update\
# install necesssary packages
&& apt-get install --no-install-recommends -y wget gpg openssl python3-pip\
# create user to perform tests
&& groupadd -g ${USER_GID} pytest\
&& adduser --disabled-password --gecos "" --uid ${USER_UID} --gid ${USER_GID} pytest\
# copy/install entrypoint script and preserver permissions
&& cp -p /artifacts/run.py /usr/local/bin/run.py\
# install required Python libraries
&& su pytest -c "pip3 install -r /artifacts/requirements.txt"\
&& su pytest -c "pip3 install /artifacts/json_tests_exporter/dist/*.whl"\
# make folder for tests and results
&& su pytest -c "mkdir -p /home/pytest/tests /home/pytest/results"
VOLUME [ "/home/pytest/tests", "/home/pytest/results" ]
USER pytest
WORKDIR /home/pytest/tests
ENTRYPOINT [ "/usr/local/bin/run.py" ]
JSON exporter plugin is located in same folder as Dockerfile
run.py is as simple as:
#!/usr/bin/python3
import pytest
import sys
from socket import gethostname
def main():
if 1 == len(sys.argv):
# use default arguments
args = [
'--report-file=/home/pytest/results/%s.json' % gethostname(),
'-qvs',
'/home/pytest/tests'
]
else:
# caller passed custom arguments
args = sys.argv[1:]
try:
res = pytest.main(args)
except Exception as e:
print(e)
res = 1
return res
if __name__ == "__main__":
sys.exit(main())
requirements.txt only contains:
python-gnupg==0.4.4
pytest>=7.1.2
So basically, I can run everything with:
docker build -t pytest-runner ./tests/docker/pytest_runner
docker run --rm -it -v $(pwd)/tests/results:/home/pytest/results -v $(pwd)/tests/fixtures:/home/pytest/tests pytest-runner
Last two lines I made programatically run from Python in pytest_sessionstart(session) hook using Docker API.