~70% of jobs from the 16.6.4.2 data11_7tev reco task 587254:

data11_7tev.00189598.physics_cosmiccalo.recon.r2358

(group production id sct, priority 750) failed repeatedly up to 9 attempts each at bnl on the rawtoesd step due to the following error (probably the problematic/empty input raw merge file?):
--------------
py:raw_to_esd        info ****************** starting raw-&gt;esd making *****************
py:athena            info including file &quot;recexcommon/recexcommon_topoptions.py&quot;
py:athena            info 'gbl.athenaservices.setfatalhandler(flag)' is deprecated
py:athena            info instead, please consider using:
py:athena            info  svcmgr.coredumpsvc.fatalhandler = flag
py:athena            info including file &quot;recexcond/recexcommon_flags.py&quot;
py:autoconfiguration    info auto-configuration key 'everything' requested. all default keys will be used.
py:autoconfiguration    info auto-configuration will procede according to the following keys:
py:autoconfiguration    info ['projectname', 'realorsim', 'fieldandgeo', 'beamtype', 'conditionstag', 'dotruth', 'inputtype', 'beamenergy', 'lumiflags', 'triggerstream']
py:inputfilepeeker    info executing   inputfilepeeker.py
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0226._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0227._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0228._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0229._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0230._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0231._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0232._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0233._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0234._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0235._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0236._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0237._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0238._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0239._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0240._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0241._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0242._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0243._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0244._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0245._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0246._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0247._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0248._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0249._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0250._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0251._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0252._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0253._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0254._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0255._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0256._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0257._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0258._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0259._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0260._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0261._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0262._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0263._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0264._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0265._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0266._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info stream_names not present in input bytestream file. giving default name 'streamraw'
py:inputfilepeeker warning unable to fill inputfilesummary from file data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb0267._sfo-all._0001.1. file is probably empty. will try again with next (if any).
py:inputfilepeeker    info extracted streams ['streamraw'] from input file 
py:inputfilepeeker warning guessing that evt_type=is_data, but this is not 100% certain. using auto-configuration is not safe if this info is wrong.
shortened traceback (most recent user call last):
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasreconstruction/16.6.4/installarea/joboptions/recjobtransforms/skeleton.rawtoesd.py&quot;, line 134, in &lt;module&gt;
    else: include( &quot;recexcommon/recexcommon_topoptions.py&quot; )
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasreconstruction/16.6.4/installarea/joboptions/recexcommon/recexcommon_topoptions.py&quot;, line 89, in &lt;module&gt;
    include ( &quot;recexcond/recexcommon_flags.py&quot; )
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/joboptions/recexcond/recexcommon_flags.py&quot;, line 235, in &lt;module&gt;
    configurefromlistofkeys(rec.autoconfiguration())
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/python/recexconfig/autoconfiguration.py&quot;, line 880, in configurefromlistofkeys
    rec.projectname=getprojectname()
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/python/recexconfig/autoconfiguration.py&quot;, line 283, in getprojectname
    from recexconfig.getcool import cool
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/python/recexconfig/getcool.py&quot;, line 32, in &lt;module&gt;
    cool=_setup()
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/python/recexconfig/getcool.py&quot;, line 19, in _setup
    lb=getlbnumber()
  file &quot;/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlasevent/16.6.4/installarea/python/recexconfig/autoconfiguration.py&quot;, line 75, in getlbnumber
    lb=lbs[0]
indexerror: list index out of range
py:athena            info leaving with code 8: &quot;an unknown exception occurred&quot;
info: read 109 error categories from file /direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlascore/16.6.4/installarea/share/atlas_error_categories.db
/direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlascore/16.6.4/installarea/share/bin/athena.py -b runargs.rawtoesd.py recjobtransforms/skeleton.rawtoesd.py rawtoesd_messagesvc_joboptions.py --drop-and-reload - exit code 8
py:rawtoesd          info execution of rawtoesd complete with exit code 8.
info: read 109 error categories from file /direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlascore/16.6.4/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlascore/16.6.4/installarea/share/atlas_error_patterns.db
info: read 58 ignore patterns from file /direct/usatlas+osg/atlas_app/atlas_rel/prod/releases/rel_16-4.7/atlascore/16.6.4/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: thu nov 17 19:02:36 2011, vmem: 94.45703125 mb, rss: 63.30859375 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,16.6.4.2 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info done scanning 391 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 43
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 0
athenalogchecker stop [rawtoesd log] wall-time: thu nov 17 19:02:36 2011, vmem: 94.8125 mb, rss: 63.5078125 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 00 secs, cpu-time: 00 hrs 00 mins 00 secs, vmem: 0.35546875 mb, rss: 0.19921875 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 11000 (exit code 10)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: thu nov 17 19:02:36 2011, vmem: 95.23046875 mb, rss: 64.0078125 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 00 hrs 02 mins 33 secs, cpu-time: 00 hrs 01 mins 46 secs, vmem: 0.90234375 mb, rss: 0.94921875 mb
===== job report produced by rawtoesd version patjobtransforms-00-03-31 =====
command=/usatlas/osg/atlas_app/atlas_rel/16.6.4/atlasproduction/16.6.4.2/installarea/share/bin/reco_trf.py inputbsfile=data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb[0226,0227,0228,0229,0230,0231,0232,0233,0234,0235,0236,0237,0238,0239,0240,0241,0242,0243,0244,0245,0246,0247,0248,0249,0250,0251,0252,0253,0254,0255,0256,0257,0258,0259,0260,0261,0262,0263,0264,0265,0266,0267]._sfo-all._0001 runnumber=189598 autoconfiguration=everything dbrelease=dbrelease-17.5.1.tar.gz amitag=r2358 --ignoreerrors=true outputntup_sctfile=ntup_sct.587254._000010.root.9
exitcode=10
errorcode=11000 (ath_failure)
fatals=0
errors=0
warnings=10
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 11000 (exit code 10)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: thu nov 17 19:02:37 2011, vmem: 95.4765625 mb, rss: 64.28515625 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 00 hrs 02 mins 55 secs, cpu-time: 00 hrs 02 mins 08 secs, vmem: 1.765625 mb, rss: 1.7890625 mb
===== job report produced by recotrf =====
command=/usatlas/osg/atlas_app/atlas_rel/16.6.4/atlasproduction/16.6.4.2/installarea/share/bin/reco_trf.py inputbsfile=data11_7tev.00189598.physics_cosmiccalo.merge.raw._lb[0226,0227,0228,0229,0230,0231,0232,0233,0234,0235,0236,0237,0238,0239,0240,0241,0242,0243,0244,0245,0246,0247,0248,0249,0250,0251,0252,0253,0254,0255,0256,0257,0258,0259,0260,0261,0262,0263,0264,0265,0266,0267]._sfo-all._0001 runnumber=189598 autoconfiguration=everything dbrelease=dbrelease-17.5.1.tar.gz amitag=r2358 --ignoreerrors=true outputntup_sctfile=ntup_sct.587254._000010.root.9
exitcode=10
errorcode=11000 (ath_failure)
fatals=0
errors=0
warnings=10
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=9852311605
rundiravailabledisk=1742881652736
workdir=/home/tmp/panda_pilot_22091_1321553134/pandajob_1360875410_1321553138
athcode=8
rundiravailabledisk=1742883848192
dbrelease=17.5.1
athacronym=an unknown exception occurred
rundiruseddisk=9850985488
atlasrelease=16.6.4.2

---------------

logfile:
http://voatlas141.cern.ch:25980/monitor/logs/701312fc-2642-485b-bd81-e75ea98e0367/tarball_pandajob_1360875410_bnl_atlas_1/athena_stdout.txt

yuri 