SOLVED - recast cannot pull image from gitlab ci/cd

jhaley · January 6, 2022, 12:04am

I am about to murder someone… please help!

When I try to run recast for my analysis, it is says it not able to pull the docker image
docker pull gitlab-registry.cern.ch/sasingh/all-hadronic-vlq-t_ht:RECAST-c04fc554

I can run the workflow with yadage and I can pull the image in my terminal, but recast cannot get past trying to pull this image. I have tried everything I can think to fix it and am at my wits’ end.

Here are the contents of my yml files:
recast.yml:

name: jhaley/all-hadronic-vlq-t_ht

metadata:
  author: 'Joe Haley'
  input requirements: 'Input signal'
  short_description: 'RECAST for full Run2 single VLQ T->Ht all-hadronic analysis'

spec:
  workflow: workflow.yml

example_inputs:
  default:
    dataopts:
      initdir: '/Users/jhaley/work/RECAST/VLQhad/my-workflow'
    initdata:
      infile: 'indata/user.jfoo.26147970._000001.allhad_boosted.root'
      xsection_pb: 0.10
      camp: 'a'
      nevts: 50 # Set to -1 to run over all events

specs/workflow.yml:

stages:

- name: prep_step
  dependencies: [init]
  scheduler:
    scheduler_type: singlestep-stage
    parameters:
      input_file:         {step: init, output: infile}
      cross_section_pb:   {step: init, output: xsection_pb}
      campaign:           {step: init, output: camp}
      output_fitdir:      '{workdir}/Fit'
      nevents:            {step: init, output: nevts}
    step: {$ref: steps.yml#prepareinput}

- name: fitting_step
  dependencies: [init,prep_step]
  scheduler:
    scheduler_type: singlestep-stage
    parameters:
      local_dir:   '{workdir}'
      fit_dir:    {step: prep_step, output: output_fitdir}
    step: {$ref: steps.yml#fitting}

specs/steps.yml:

prepareinput:
  process:
    process_type: interpolated-script-cmd
    script: |
      # Set up ATLAS env to get ROOT
      source /release_setup.sh
      # Run code to create signal file for fit
      python prepare_fit_input.py '{input_file}' {cross_section_pb} '{campaign}' '{output_fitdir}' {nevents}
  environment:
    environment_type: docker-encapsulated
    image: gitlab-registry.cern.ch/sasingh/all-hadronic-vlq-t_ht
    imagetag: RECAST-c04fc554
  publisher:
    publisher_type: interpolated-pub
    publish:
      output_fitdir: '{output_fitdir}'


fitting:
  process:
    process_type: interpolated-script-cmd
    script: |
      echo $PWD 
      cd '{local_dir}' 
      echo $PWD 
      echo "Run: trex-fitter h {fit_dir}/TRexConfig.rex" 
      trex-fitter h {fit_dir}/TRexConfig.rex 
      ls -l BoostedAllhadronicVLQAnalysisRECASTFit 
  environment:
    environment_type: docker-encapsulated
    image: gitlab-registry.cern.ch/trexstats/trexfitter
    imagetag: trexfitter-00-04-15
  publisher:
    publisher_type: interpolated-pub
    publish:
      workspace: '{local_dir}/BoostedAllhadronicVLQAnalysisRECASTFit'

I have run the following for authentication:

docker login -u $USER gitlab-registry.cern.ch

I also did this, though I don’t use eos:

RECAST_USER=recasttu
RECAST_PASS=Did...(redacted}
RECAST_TOKEN=n44...(redacted)
# To pull images from a gitlab registry that $RECAST_USER has access to
eval "$(recast auth setup -a $RECAST_USER -a $RECAST_PASS -a $RECAST_TOKEN -a default)"

# To access private data that $RECAST_USER has access to on \eos
eval "$(recast auth write --basedir authdir)"

What do I need to do so that recast will be able to pull my image from gitlab???

Thanks for any help. Until then, I am totally stuck.

~Joe

feickert · January 6, 2022, 5:21am

@jhaley As your workflow is private, I’m not really able to use the YAML files that you’ve posted in any debugging sense. Luckily, I’m hoping that won’t be necessary as there might be a small confusion RE: access permissions.

You note

I have run the following for authentication:
docker login -u $USER gitlab-registry.cern.ch

but then talk about using $RECAST_USER for evaluation of credentials with recast-atlas. Unless the values of $USER and $RECAST_USER are the same then this doesn’t tell us anything.

Does the user $RECAST_USER have access to the GitLab repository that has the image registry you’re trying to pull the Docker image from? If no, then please make sure that you’ve followed the instructions related to authentication in the Introduction section and the Defining Steps section of the RECAST docs on Workflow Authoring.

If $RECAST_USER does have permissions, then this is strange. At that point I’d advise giving me permissions to your workflow repository and the image repository so we can debug.

jhaley · January 6, 2022, 6:07am

Hi @feickert ,

Sorry, I should have said that USER=jhaley, which is just my user name on gitlab, CERN, and my laptop. RECAST_USER=recasttu and I just tried doing

docker login -u $RECAST_USER gitlab-registry.cern.ch

with the corresponding password for recasttu, but the recast run fails in the same way. I will add you as a developer for the workflow repo: Cern Authentication

Note that this is for the RECAST-VLQhad branch. The master branch has the vhbb example from the tutorial, which runs as expected.

Thanks so much for the help!
~Joe

jhaley · January 6, 2022, 7:37pm

I should mention that the _packtivity log says:

2022-01-06 19:06:20,827 |  pack.prep_step.pull |   INFO | b"Error response from daemon: pull access denied for gitlab-registry.cern.ch/sasingh/all-hadronic-vlq-t_ht, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"

So it is indeed just an issue with how to get the recast job to recognize my docker login ...

I can confirm that I can access the repo when running the workflow with yadage run (slightly modified to have format for yadage), so I have permission to docker pull from the gitlab-registry, but recast doesn’t know about it and I just don’t know enough about recast to make it know.

~Joe

jhaley · January 6, 2022, 8:12pm

SOLVED!!!
It appears that the problem was that user recasttu may have permission to pull from gitlab-registry, but not any image. Since my gitlab repo is private, users with permission to that gitlab repo can pull the images, but not recasttu.
I redid all of the recast authentication with the info for my user account:

RECAST_USER=jhaley
RECAST_PASS=<my gitlab password>
RECAST_TOKEN=<my gitlab token>
# To pull images from a gitlab registry that $RECAST_USER has access to
eval "$(recast auth setup -a $RECAST_USER -a $RECAST_PASS -a $RECAST_TOKEN -a default)"

# To access private data that $RECAST_USER has access to on \eos
eval "$(recast auth write --basedir authdir)"

and then was able to run recast with my workflow.

Of course, I will eventually put my image into the central ATLAS RECAST repo, but I wanted to get everything working first, which is why I’m using the image from the private analysis gitlab for now.

Sorry for the noise! Hopefully this will be useful for others to avoid the same pitfall, though.

And thanks again @feickert for the help!
~Joe

feickert · January 6, 2022, 8:47pm

It appears that the problem was that user recasttu may have permission to pull from gitlab-registry, but not any image. Since my gitlab repo is private, users with permission to that gitlab repo can pull the images, but not recasttu

Yeah, permissions are something that can be tricky with GitLab private repositories. There are many times of course when things need to be private, so it might be worth trying to figure out some way in the docs sections that I linked to earlier to have an aside that does something along the lines of saying “STOP! Please do these verifications to ensure that your accounts have permissions set correctly.”