How do I upload the output of my RECAST workflow back to eos after running the workflow as a CI job?

It may sometimes be desirable to automatically upload the results of your RECAST workflow back to some location on eos for safekeeping after the workflow has finished running as a CI job, eg. if you’re concerned about the limited lifetime of the output as a gitlab CI artifact. Thanks to Louis Dartmoor Corpe for bringing up this question and proposing a solution!

Two possible solutions have been suggested for this.

Using xrdcp in the same CI job

(thanks to @lheinric for suggesting this solution)

After ensuring that your user or service account has write access to the target space on eos, run kinit to set up kerberos credentials, then after the RECAST workflow has run, use xrdcp in the same CI job to upload the RECAST output:

kinit account_username@CERN.CH
eval "$(recast auth setup -a $RECAST_USER -a $RECAST_PASS -a $RECAST_TOKEN -a default)"
eval "$(recast auth write --basedir authdir)"
$(recast catalogue add $PWD)
recast run [name of workflow] --tag myrun 
xrdcp recast-myrun/path/to/file/of/interest root://

This approach is relatively straightforward to implement. It does come with the limitation that xrdcp doesn’t have functionality to copy entire directories recursively if this is desired (it can only copy one file at a time), but this could be easily overcome by eg. first converting the directory to be copied into a tarball.

Using ssh in a dedicated CI job

(thanks to Louis Dartmoor Corpe for suggesting this solution)

Save the desired RECAST output as an artifact, then write a second CI job dedicated to creating directories on eos to contain the RECAST output from the first job and copying the output to these directories via ssh.

This second solution is perhaps a bit more complex to implement than the first, but it also appears to be more flexible. Here is a sample implementation of this solution, kindly provided by Louis Dartmoor Corpe.

1 Like