Is there any alternative to copying large files from eos using `xrdcp` in my workflow?

There is an alternative for accessing root files. Suppose you currently copy the files to your machine using eg. xrdcp root://eosuser.cern.ch//path/to/root/file.root. If you don’t want or need to root file to persist on your machine, you can instead stream it directly:

root -b root://eosuser.cern.ch//path/to/root/file.root

Maybe this isn’t appropriate here (it’s more for CI than for recast), but I use something that relies less on CERN’s internal products for this:

  • I store the file in my EOS area, in a directory which is mounted as web accessible. This is super useful to have anyway (to share plots etc), but if you don’t have it you can enable it via CERN web services.
  • I use wget to download the files from there.

The advantages of this approach:

  • You can browse the files, and download them with minimal software from any machine with an internet connection. See, for example, the inputs that I use for b-tagging preprocessing.
  • If someone forks your repository there’s no additional steps for them to access the files.
  • Works with any file type. For this one rsync work as well.

Thanks Dan!

I think in the very end (once workflows get finalized), all input files will get stored in the recast area so will need kerberos credentials to access, but this approach seems like it could be super helpful while people are developing workflows and not wanting worry yet about xrootd stuff. Also, just good to know of in general!

So I’m currently using this on my local machine and have my files stored in a service account “jesjer” at the location root://eosuser.cern.ch//eos/user/j/jesjer/MPF/mc16_13TeV.361106.PowhegPythia8EvtGen_AZNLOCTEQ6L1_Zee.deriv.DAOD_JETM3.e3601_e5984_s3126_r10724_r10726_p4060 and running workflows on my local machine. I was able to access the files via streaming, but only after executing the echo "<password>" | kinit <username>@CERN.CH directive as we do for CI jobs. Is there a way around this yet?

I was able to access the files via streaming, but only after executing the echo "<password>" | kinit <username>@CERN.CH directive as we do for CI jobs. Is there a way around this yet?

(I realize this is far too many years late to be helpful to anyone in this thread but) No. XRootD uses the Kerberos V protocol for authentication. When you’re accessing your file on EOS you’re invoking XRootD to do so, and so must have already authenticated with kinit.