Sourcing setup file fails with "sh: source: not found"

RECAST
#1

Hi,

while developing the fitting step for my analysis inside RECAST I ran into the following problem (originally posted in the ATLAS Analysis Preservation) Mattermost channel:

Original question

In order to set up the fitting framework (HistFitter) I need to source some setup file. The issue is, that I see the following line in the packtivity log:

2022-01-18 13:57:03,070 | pack.packtivity_sync |   INFO | b'sh: 5: source: not found'

So somehow the code is executed with sh (and not e.g. bash), which doesn’t know the source command.
Sourcing things works in other steps, so maybe this is a problem with the docker image for this specific step? But I don’t really know how to narrow the error down.
Testing the commands interactively inside the docker image works when I run bash as a shell.
Maybe you have an idea why this is not working?
Here are is the corresponding code:

  • steps.yml in recast repo with the line which is not working: steps.yml
  • docker file for this step: Dockerfile

Things tried so far

With the help of Matthew, we already tried to following fixes

  • set the shell to bash inside the docker image with
    SHELL [ "/bin/bash", "-c" ]
    (this is also done in the base image here, so this actually shouldn’t be necessary)
  • remove the creation of the atlas user in the Dockerfile

Both of these attempts didn’t fix the issue.

Minimal working example

I’ve set up a MWE here. This example writes a bash script where an environment variable is set and then tries to source it. In the end, the value of this environment variable is used to specify the RECAST result.

This is not working if I use the atlasamglab/stats:root6.24.06 image (on which the docker container from my actual analysis is based on), but it works when I use an AnalysisBase image (e.g. atlas/analysisbase:21.2.174)

So maybe something is wrong with that specific image?

1 Like
#2

@brottler The image that you’re using as a base image happens to be Debian based

$ docker run --rm -ti atlasamglab/stats-base:root6.24.06 'cat /etc/os-release'
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

and while the default interactive shell for Debian is Bash, the default shell for Debian is dash

$ docker run --rm -ti debian:bullseye /bin/bash -c 'ls -l /bin/sh'
lrwxrwxrwx 1 root root 4 Dec 20 00:00 /bin/sh -> dash

while the default shell and interactive shell for CentOS 7 is Bash

$ docker run --rm -ti centos:7 /bin/bash -c 'ls -l /bin/sh'
lrwxrwxrwx 1 root root 4 Nov 13  2020 /bin/sh -> bash

To be assured that things will work, you can just force the default shell to be /bin/bash by adding the line

RUN ln --symbolic --force /bin/bash /bin/sh

to your Dockerfile.

This can get added as a warning note in the RECAST ATLAS docs.

Thank you for your truly excellent minimal reproducible example! It helped so much in me being able to track down what the bug was. :clap:

1 Like