In our Jenkins CI we are running our end2end integration tests in parallel (using 3 docker-compose run-commands). The separation works by having separate project names to each call assigned, however that results in all container become rebuild-ed (as their project names are different). Is there way to avoid that?

So what we actually need:

• build multiple containers defined in docker-compose file
• run the set of containers multiple times but completely separated - without rebuilding and with full separation

We are doing basically this:

• docker-compose build <- this builds all containers under the default project name
• docker-compose run container command_1 -p project_1 <- this re-builds all containers as no containers with the project name "project_1" exists.
• docker-compose run container command_2 -p project_2 <- as this is done in parallel, it also rebuilds
• docker-compose run container command_3 -p project_3 <- as this is done in parallel, it also rebuilds

Here is basically our docker-compose file:

version: '3'
services:
  db:
    environment:
      - POSTGRES_PASSWORD=foo
      - POSTGRES_USER=foo
      - POSTGRES_DB=foo_test
  backend:
    build:
      context: ../backend
      dockerfile: .docker/Dockerfile
    command: /workspace/.docker/start_end2end.sh
    depends_on:
      - db
      - redis
    volumes:
      - ./e2e.env:/.env
  cypress:
      ipc: host
      build: 
        context: ../tests
        dockerfile: Dockerfile
      volumes:
        - ../tests/cypress:/app/code/cypress
      depends_on:
        - backend
        - frontend
      environment:
        - CYPRESS_baseUrl=http://frontend:4200
        - CYPRESS_defaultCommandTimeout=10000
        - CYPRESS_RETRIES=3
        - NO_COLOR=1
  frontend:
    build:
      context: ../frontend
      dockerfile: .docker/Dockerfile
    command: /workspace/.docker/start_end2end.sh
    depends_on:
      - backend
    environment:
      - BACKEND_HOST=backend

And this is the relevant part from the Jenkinsfile:

stage('Building containers') {
    steps {
        sh "docker-compose -f ./shared/docker-compose.yml -f ./tests/docker-compose.override.yml build"
    }
}
parallel {
    stage('End2End Tests 1') {
        steps {
            sh "docker-compose -f ./shared/docker-compose.yml -f ./tests/docker-compose.override.yml -p ${env.JOB_NAME}_${env.BUILD_ID}_1 run --rm cypress npm run end2end:1"
        }
    }
    stage('End2End Tests 2') {
        steps {
            sh "docker-compose -f ./shared/docker-compose.yml -f ./tests/docker-compose.override.yml -p ${env.JOB_NAME}_${env.BUILD_ID}_2 run --rm cypress npm run end2end:2"
        }
    }
    stage('End2End Tests 3') {
        steps {
            sh "docker-compose -f ./shared/docker-compose.yml -f ./tests/docker-compose.override.yml -p ${env.JOB_NAME}_${env.BUILD_ID}_3 run --rm cypress npm run end2end:3"
        }
    }
}

I was not able to archive what I describe above, only:

• Don't use different project names -> no rebuilding on multiple parallel calls. But the depending containers are shared.
• Do use different project names -> rebuilding on multiple parallel calls. The containers are not shared.

CodePudding user response：

As specified in the docker compose file reference for the build section

If you specify image as well as build, then Compose names the built image with the webapp and optional tag specified in image:

build: ./dir
image: webapp:tag

So something like below will always build images for all services with the same names regardless of the current project name and use those images by default when calling docker-compose up if they already exist.

services:
  service1:
    image: my_service1:1.0.0
    build:
      context: ../service1
      dockerfile: .docker/Dockerfile
  service2:
      image: my_service2:1.0.0
      build: 
        context: ../service2
        dockerfile: Dockerfile
  service3:
      image: my_service3:1.0.0
      build:
        context: ../service3
        dockerfile: path/to/another/Dockerfile

This example can later be extended to you own needs (e.g. by passing the tag in an environment file...)