Run Pytest in Gitlab-CI as User-CodePudding

I had the following gitlab-ci.yml in my python-package repository:

image: python:latest

unit-test:
    stage: test
    tags:
        - docker
    script:
        - pip install tox
        - tox

formatting-check:
    stage: test
    tags:
      - docker
    script:
      - pip install black
      - black --check .

using this tox.ini file:

[tox]
envlist = my_env

[testenv]
deps =
    -rrequirements.txt
commands =
    python -m pytest tests -s

This did work as I wanted it to.

However, then I added tests to test my code against a local Postgresql database using https://pypi.org/project/pytest-postgresql/. For this, I had to install PostgreSQL(apt -y install postgresql postgresql-contrib libpq5).

When I added this to my gitlab-ci.yml:

image: python:latest

unit-test:
    stage: test
    tags:
        - docker
    script:
        - apt -y install postgresql postgresql-contrib libpq5
        - pip install tox
        - tox

formatting-check:
    stage: test
    tags:
      - docker
    script:
      - pip install black
      - black --check .

I got the error from tox, that some module in Postgres (pg_ctl) wouldn't allow being run as the root. Log here: https://pastebin.com/fMu1JY5L

So, I must execute tox as a user, not the root.

My first idea was to create a new user (useradd) and then switch to that user, but su requires inputting a password.

From a quick google search I found out the easiest solution to creating a new user is to create a new Docker Image using Docker-in-Docker.

So, as of now I have this configuration: gitlab-ci.yml:

image: docker:19.03.12
services:
  - docker:19.03.12-dind

stages:
  - build
  - test

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

before_script:
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  - docker info

docker-build:
  stage: build
  script:
    - docker build --pull -t $CONTAINER_TEST_IMAGE .
    - docker push $CONTAINER_TEST_IMAGE

formatting-check:
  stage: test
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker run $CONTAINER_TEST_IMAGE black --check .

unit-test:
  stage: test
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker run $CONTAINER_TEST_IMAGE tox

Dockerfile:

FROM python:latest
RUN apt update
RUN apt -y install postgresql postgresql-contrib libpq5
RUN useradd -m exec_user
USER exec_user
ENV PATH "$PATH:/home/exec_user/.local/bin"
RUN pip install black tox

(I had to add ENV PATH "$PATH:/home/exec_user/.local/bin" because pip would cry about it not being in the Path)

tox.ini:

[tox]
envlist = my_env

[testenv]
deps =
    -rrequirements.txt
commands =
    python -m pytest tests -s

The job docker-build completes — the other two fail.

As for formatting-check:

$ docker run $CONTAINER_TEST_IMAGE black --check .
ERROR: Job failed: execution took longer than 1h0m0s seconds

The black command usually executes extremely fast (<1s).

As for unit-test:

/bin/sh: eval: line 120: tox: not found
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 127

I have also found, that replacing docker run $CONTAINER_TEST_IMAGE tox with docker run $CONTAINER_TEST_IMAGE python3 -m tox doesn't work. Here, python3 isn't found (which seems odd given that the base image is python:latest).

If you have any idea how to solve this issue, let me know :D

CodePudding user response：

My first idea was to create a new user (useradd) and then switch to that user, but su requires inputting a password.

This should work for your use case. Running su as root will not require a password. You could also use sudo -u postgres tox (must apt install sudo first).

As a basic working example using su (as seen here - job) using the postgres user, which is created automatically when postgres is installed.

myjob:
  image: python:3.9-slim
  script:
    - apt update && apt install -y --no-install-recommends libpq-dev postgresql-client postgresql build-essential
    - pip install psycopg2 psycopg pytest pytest-postgresql
    - su postgres -c pytest  
    # or in your case, you might use: su postgres -c tox

Alternatively, you might consider just using GitLab's services feature to run your postgres server if that's the only obstacle in your way. You can pass --postgresql-host and --postgresql-password to pytest to tell the extension to use the services.