Issues authenticating to pull gitlab registry container

Hi,

I’m running into issues pulling a container from a gitlab registry. I’ve seen it running on my laptop (with a previously working setup), using singularity on lxplus, and testing reana.

I did the recast auth setup and have username/password/token set to values that were previously working.

When the recast workflow (or individual test) gets to pulling the image, I see this in the packtivity pull log
INFO | starting file logging for topic: pull
INFO | master: Pulling from salderwe/run_susyskimewk3l
INFO | 77e8baeda62b: Already exists
INFO | 0300be6cf65a: Already exists
INFO | 42e5a9805e42: Already exists
INFO | 37c134034d05: Pulling fs layer
INFO | 1ea103c40e2d: Pulling fs layer
INFO | 28f387c0f1b0: Pulling fs layer
INFO | 37c134034d05: Download complete
INFO | 37c134034d05: Pull complete
INFO | 1ea103c40e2d: Verifying Checksum
INFO | 1ea103c40e2d: Download complete
INFO | 1ea103c40e2d: Pull complete
INFO | 28f387c0f1b0: Download complete
INFO | 28f387c0f1b0: Pull complete
INFO | error pulling image configuration: Get https://s3.cern.ch/gitlabregistry/docker/registry/v2/blobs/sha256/dc/dc0dcb36048282beee4ded068391bf08bb4309ee1e48a33a7ca914cb7ebc7593/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=7932DD7I9ES8EWXQKV4A%2F20200930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200930T163105Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=f5876277de63a096c15e3e9e7f93eee0b184327fdc7c699bd322eeba6e283ac2: remote erro
r: tls: handshake failure

Can you advise what to test for further debugging, or which output to provide?

I also saw some “403 (Unauthorized)” errors on the same pull step, but currently cannot reproduce them. And it seems to be an unstable issue, since trying multiple times, very occasionally, I actually get a successful pull.

Using reana, the error presents as
rpc error: code = Unknown desc = unexpected http code: 403, URL: https://gitlab.cern.ch/jwt/auth?scope=repository%3Asalderwe%2Frun_susyskimewk3l%3Apull&service=container_registry

Thanks,
Sara

Was also having this issue but seems to have resolved itself for me, for now.