'recast catalogue ls' crashes

RECAST
#1

Hi,

when I use commands $(recast catalogue add $PWD) from inside of my repository and then recast catalogue ls I get output that I am not supposed to get (I guess).

/workdir # recast catalogue ls
NAME                               DESCRIPTION                                                 EXAMPLES            TAGS                
Traceback (most recent call last):
  File "/usr/bin/recast", line 8, in <module>
    sys.exit(recastatlas())
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/recastatlas/subcommands/catalogue.py", line 97, in ls
    for k, v in sorted(config.catalogue.items(), key=lambda x: x[0]):
  File "/usr/lib/python3.8/site-packages/recastatlas/config.py", line 96, in catalogue
    entry = yaml.safe_load(open(f))
  File "/usr/lib/python3.8/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python3.8/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python3.8/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/lib/python3.8/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/lib/python3.8/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "/workdir/recast.yml", line 12, column 15

But when I call recast catalogue ls before adding the current directory to RECAST catalogue, I get expected output:

/work # recast catalogue ls
NAME                               DESCRIPTION                                                 EXAMPLES            TAGS                
atlas/atlas-conf-2018-041          ATLAS MBJ                                                   default                                 
examples/checkmate1                CheckMate Tutorial Example (Herwig + CM1)                   default                                 
examples/checkmate2                CheckMate Tutorial Example (Herwig + CM2)                   default                                 
examples/rome                      Example from ATLAS Exotics Rome Workshop 2018               default,newsignal                       
testing/busyboxtest                Simple, lightweight Functionality Test                      default

I am running it from inside docker container recast/recastatlas with tag v0.1.9. This also fails in GitLab CI/CD with the same type of output. Do you have idea what could have I done wrong? I guess it is something trivial but without solving this issue, I am not able to continue. Thank you for your help!

Cheers,
Ondřej

#2

Hi @otheiner. Do you have a valid workflow in your directory to add?

This also fails in GitLab CI/CD with the same type of output. Do you have idea what could have I done wrong?

Can you share the GitLab workflow and the repository?

#3

Hi @feickert ! This is my repository right now. It runs in yadage docker container and both workflow and steps are validated by packtivity utility. Ondrej Theiner / MBJ_RECAST · GitLab

This is my workflow:

stages:
- name: production_highmass
  dependencies: [init]
  scheduler:
    scheduler_type: singlestep-stage
    parameters:
      SUBMIT_DIR: '{workdir}/submitDir_test'
      DO_JET_CLEANING: '1'
      DO_SYST: '1'
      DO_JMS: '1'
      DO_JET_TRIG_SF: '1'
      BASELINE_SELECTION_HF: '1'
      DO_NTUP: '0'
      DO_EWK: '1'
      DO_EWKBDT: '1'
      DO_BTG_COMPARISON: '0' 
      B_JET_MIN_PT: '2000'
      DRIVER: 'direct'
      BTAG_TIME_STAMP: '20193'
      DO_MUON_CORRECTION: '1'
      DO_EWK_PT40: '1'
      CONFIG: 'MultibjetsAnalysis/SUSYTools_EW.conf'
      CONFIG_LOOSE: 'MultibjetsAnalysis/SUSYTools_Lepton.conf'
      FILES: '/samples/DAOD_SUSY10.26862384._000006.pool.root.1'
      INPUT_SOURCE: 'local'
      NEVENTS: '5'
    step: {$ref: 'steps.yml#/production'}
#4

Looking at your code I think you have a YAML indentation Issue. In

you have

tests:
- name: test_production
  spec: steps.yml#/production
    parameters:
      SUBMIT_DIR: '{workdir}/submitDir_test'
...

but parameters should be at the same indent level with spec. c.f. the testing section of the docs for an example there.

#5

I fixed the indentation (which was probably issue) but there was one more problem. It seems to me that RECAST has serious problem dealing with TAB whitespace character. I bumped into this problem before and now I realised there was one whitespace of this type in recast.yml file. Now the error stack got shorter but problem persists. The error message now is following:

/workdir # $(recast catalogue add $PWD)
/workdir # recast catalogue ls
NAME                               DESCRIPTION                                                 EXAMPLES            TAGS                
Traceback (most recent call last):
  File "/usr/bin/recast", line 8, in <module>
    sys.exit(recastatlas())
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/recastatlas/subcommands/catalogue.py", line 97, in ls
    for k, v in sorted(config.catalogue.items(), key=lambda x: x[0]):
  File "/usr/lib/python3.8/site-packages/recastatlas/config.py", line 97, in catalogue
    process_entry(cfg, f, entry)
  File "/usr/lib/python3.8/site-packages/recastatlas/config.py", line 107, in process_entry
    if "toplevel" not in entry["spec"]:
TypeError: argument of type 'NoneType' is not iterable

I updated repo such that known issues should be fixed but there is still probably something more. Is there a way of validating recast.yml (similar to packtivity-validate and yadage-validate)? Because this is something I haven’t done.

#6

I don’t see this as a RECAST problem as YAML explicitly forbids tabs. So to avoid creating situations where you can’t visually detect invalid YAML from valid YAML I would setup your text editor’s config to turn all tab input into spaces.

I don’t think so as of recast-atlas v0.1.9. If this is something you think would be valuable, I would recommend making a GitHub Issue asking for it.

Your current RECAST YAML file has tabs and spaces mixed, and it is also missing some required information and has the workflow file defined wrong (e.g. extra spec/ — easy to miss!). I’d suggest finish reading through the Workflow Authoring docs, especially the Running Workflows section and open up Issues on the docs repo if things are unclear.

#7

Thank you @feickert for the help! So the problem was really that my reacst.yml was missing some items that were mandatory and also wrong path to workflow in the reacst.yml. Adding to the catalogue finally works!

1 Like