GitLab CI rules not working with extends and individual rules-CodePudding

Below are two jobs in the build stage. Default, there is set some common condition, and using extends keyword for that, ifawsdeploy. As only one of them should run, if variable $ADMIN_SERVER_IP provided then connect_admin_server should run, working that way. If no value provided to $ADMIN_SERVER_IP then create_admin_server should run, but it is not running.

.ifawsdeploy:
  rules:
    - if: '$TEST_CREATE_ADMIN && $REGION && $ROLE_ARN && $PACKAGEURL && $TEST_CREATE_ADMIN == "aws" && $SUB_PLATFORM == "aws" && $ROLE_ARN != "" && $PACKAGEURL != "" && $REGION != ""'

variables:
 TEST_CREATE_ADMIN:
   #value: aws
   description: "Platform, currently aws only"
 SUB_PLATFORM:
   value: aws
   description: "Platform, currently aws only"
 REGION:
   value: "us-west-2"
   description: "region where to deploy company"
 PACKAGEURL:
   value: "http://somerpmurl.x86_64.rpm"
   description: "company rpm file url"
 ACCOUNT_NAME:
   value: "testsubaccount"
   description: "Account name of sub account to refer in the deployment, no need to match in AWS"
 ROLE_ARN:
   value: "arn:aws:iam::491483064167:role/uat"
   description: "ROLE ARN of the user account assuming: aws sts get-caller-identity"
 tfenv_version: "1.1.9"
 DEV_PUB_KEY:
   description: "Optional public key file to add access to admin server" 
 ADMIN_SERVER_IP:
   description: "Existing Admin Server IP Address"
 ADMIN_SERVER_SSH_KEY:
   description: "Existing Admin Server SSH_KEY PEM content"
  
#export variables below will cause the terraform to use the root account instead of the one specified in tfvars file
.configure_aws_cli: &configure_aws_cli
    - aws configure set region $REGION
    - aws configure set aws_access_key_id $AWS_FULL_STS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $AWS_FULL_STS_ACCESS_KEY_SECRET
    - aws sts get-caller-identity
    - aws configure set source_profile default --profile $ACCOUNT_NAME
    - aws configure set role_arn $ROLE_ARN --profile $ACCOUNT_NAME
    - aws sts get-caller-identity --profile $ACCOUNT_NAME
    - aws configure set region $REGION --profile $ACCOUNT_NAME

.copy_remote_log: &copy_remote_log
- if [ -e outfile ]; then rm outfile; fi
- copy_command="$(cat $CI_PROJECT_DIR/scp_command.txt)"
- new_copy_command=${copy_command/"%s"/"outfile"}
- new_copy_command=${new_copy_command/"~"/"/home/ec2-user/outfile"}
- echo $new_copy_command
- new_copy_command=$(echo "$new_copy_command" | sed s'/\([^.]*\.[^ ]*\) \([^ ]*\) \(.*\)/\1 \3 \2/')
- echo $new_copy_command
- sleep 10
- eval $new_copy_command

.check_remote_log: &check_remote_log
- sleep 10
- grep Error outfile || true
- sleep 10
- returnCode=$(grep -c Error outfile) || true
- echo "Return code received $returnCode"
- if [ $returnCode -ge 1 ]; then exit 1; fi
- echo "No errors"

.prepare_ssh_key: &prepare_ssh_key
- echo $ADMIN_SERVER_SSH_KEY > $CI_PROJECT_DIR/ssh_key.pem
- cat ssh_key.pem
- sed -i -e 's/-----BEGIN RSA PRIVATE KEY-----/-bk-/g' ssh_key.pem
- sed -i -e 's/-----END RSA PRIVATE KEY-----/-ek-/g' ssh_key.pem
- perl -p -i -e 's/\s/\n/g' ssh_key.pem
- sed -i -e 's/-bk-/-----BEGIN RSA PRIVATE KEY-----/g' ssh_key.pem
- sed -i -e 's/-ek-/-----END RSA PRIVATE KEY-----/g' ssh_key.pem
- cat ssh_key.pem
- chmod 400 ssh_key.pem

connect-admin-server:
  stage: build
  allow_failure: true
  image:
    name: amazon/aws-cli:latest
    entrypoint: [ "" ]
  rules:
    - if: '$ADMIN_SERVER_IP && $ADMIN_SERVER_IP != "" && $ADMIN_SERVER_SSH_KEY && $ADMIN_SERVER_SSH_KEY != ""'
  extends:
    - .ifawsdeploy
  script:
    - TF_IN_AUTOMATION=true
    - yum update -y
    - yum install git unzip gettext jq -y
    - echo "Your admin server key and info are added as artifacts"
    # Copy the important terraform outputs to files for artifacts to pass into other jobs
    - *prepare_ssh_key
    - echo "ssh -i ssh_key.pem ec2-user@${ADMIN_SERVER_IP}" > $CI_PROJECT_DIR/ssh_command.txt
    - echo "scp -q -i ssh_key.pem %s ec2-user@${ADMIN_SERVER_IP}:~" > $CI_PROJECT_DIR/scp_command.txt
    - test_pre_command="$(cat "$CI_PROJECT_DIR/ssh_command.txt") -o StrictHostKeyChecking=no"
    - echo $test_pre_command
    - test_command="$(echo $test_pre_command | sed -r 's/(ssh )(.*)/\1-tt \2/')"
    - echo $test_command
    - echo "sudo yum install -yq $PACKAGEURL 2>&1 | tee outfile ; exit 0" | $test_command
    - *copy_remote_log
    - echo "Now checking log file for returnCode"
    - *check_remote_log
  artifacts:
    untracked: true
    when: always
    paths:
      - "$CI_PROJECT_DIR/ssh_key.pem"
      - "$CI_PROJECT_DIR/ssh_command.txt"
      - "$CI_PROJECT_DIR/scp_command.txt"
  after_script:
    - cat $CI_PROJECT_DIR/ssh_key.pem
    - cat $CI_PROJECT_DIR/ssh_command.txt
    - cat $CI_PROJECT_DIR/scp_command.txt

create-admin-server:
  stage: build
  allow_failure: false
  image:
    name: amazon/aws-cli:latest
    entrypoint: [ "" ]
  rules:
    - if: '$ADMIN_SERVER_IP != ""'
      when: never
  extends:
    - .ifawsdeploy
  script:
    - echo "admin server $ADMIN_SERVER_IP"
    - TF_IN_AUTOMATION=true
    - yum update -y
    - yum install git unzip gettext jq -y
    - *configure_aws_cli
    - aws sts get-caller-identity --profile $ACCOUNT_NAME #to check whether updated correctly or not
    - git clone "https://project-n-setup:$(echo $PERSONAL_GITLAB_TOKEN)@gitlab.com/company-oss/project-n-setup.git"
    # Install tfenv
    - git clone https://github.com/tfutils/tfenv.git ~/.tfenv
    - ln -s ~/.tfenv /root/.tfenv
    - ln -s ~/.tfenv/bin/* /usr/local/bin
    # Install terraform 1.1.9 through tfenv
    - tfenv install $tfenv_version
    - tfenv use $tfenv_version
    # Copy the tfvars temp file to the terraform setup directory
    - cp .gitlab/admin_server.temp_tfvars project-n-setup/$SUB_PLATFORM/
    - cd project-n-setup/$SUB_PLATFORM/
    - envsubst < admin_server.temp_tfvars > admin_server.tfvars
    - rm -rf .terraform || exit 0
    - cat ~/.aws/config
    - terraform init -input=false
    - terraform apply -var-file=admin_server.tfvars -input=false -auto-approve
    - echo "Your admin server key and info are added as artifacts"
    # Copy the important terraform outputs to files for artifacts to pass into other jobs
    - terraform output -raw ssh_key > $CI_PROJECT_DIR/ssh_key.pem
    - terraform output -raw ssh_command > $CI_PROJECT_DIR/ssh_command.txt
    - terraform output -raw scp_command > $CI_PROJECT_DIR/scp_command.txt
    - cp $CI_PROJECT_DIR/project-n-setup/$SUB_PLATFORM/terraform.tfstate $CI_PROJECT_DIR
    - cp $CI_PROJECT_DIR/project-n-setup/$SUB_PLATFORM/admin_server.tfvars $CI_PROJECT_DIR
  artifacts:
    untracked: true
    paths:
      - "$CI_PROJECT_DIR/ssh_key.pem"
      - "$CI_PROJECT_DIR/ssh_command.txt"
      - "$CI_PROJECT_DIR/scp_command.txt"
      - "$CI_PROJECT_DIR/terraform.tfstate"
      - "$CI_PROJECT_DIR/admin_server.tfvars"

How to fix that?

I tried the below step from suggestions on comments section.

.generalgrabclustertrigger:
  rules:
    - if: '$TEST_CREATE_ADMIN && $REGION && $ROLE_ARN && $PACKAGEURL && $TEST_CREATE_ADMIN == "aws" && $SUB_PLATFORM == "aws" && $ROLE_ARN != "" && $PACKAGEURL != "" && $REGION != ""'

.ifteardownordestroy:  # Automatic if triggered from gitlab api AND destroy variable is set
  rules:
    - !reference [.generalgrabclustertrigger, rules]
    - if: 'CI_PIPELINE_SOURCE == "triggered"'
      when: never

And included the above in extends of a job.

destroy-admin-server:
  stage: cleanup
  extends:
    - .ifteardownordestroy
  allow_failure: true
  interruptible: false

But I am getting syntax error in the .ifteardownordestroy part.

jobs:destroy-admin-server:rules:rule if invalid expression syntax

CodePudding user response：

You are overriding rules: in your job that extends .ifawsdeploy. rules: are not combined in this case -- the definition of rules: in the job takes complete precedence.

Take for example the following configuration:

.template:
  rules:
    - one
    - two

myjob:
  extends: .template
  rules:
    - a
    - b

In the above example, the myjob job only has rules a and b in effect. Rules one and two are completely ignored because they are overridden in the job configuration.

Instead of uinsg extends:, you can use !reference to preserve and combine rules. You can also use YAML anchors if you want.

create-admin-server:
  rules:
    - !reference [.ifawsdeploy, rules]
    - ... # your additional rules

If no value provided to $ADMIN_SERVER_IP then create_admin_server should run

Lastly, pay special attention to your rules:

  rules:
    - if: '$ADMIN_SERVER_IP != ""'
      when: never

In this case, there are no rules that allow the job to run ever. You either need a case that will evaluate true for the job to run, or to have a default case (an item with no if: condition) in order for the job to run.

To get the behavior you expect, you probably want your default case to be on_success:

  rules:
    - if: '$ADMIN_SERVER_IP != ""'
      when: never
    - when: on_success

CodePudding user response：

you can change your rules to :

  rules:
    - if: '$ADMIN_SERVER_IP != ""'
      when: never
    - when: always

  rules:
    - if: '$ADMIN_SERVER_IP == ""'
      when: always

I have a sample in here: try-rules-stackoverflow-72545625 - GitLab and the pipeline record Pipeline no value - GitLab, Pipeline has value - GitLab