instructions for investigation: https://twiki.cern.ch/twiki/bin/viewauth/atlas/recbuginvestigation

a number of jobs in recjobtransformtests crashed due to this bug in rel_2 today.

example: ttbar_pu_mu0_d555

original job transform command:
/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/share/bin/reco_trf.py ami=f380 autoconfiguration=everything inputrdofile=root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000001.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000002.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000003.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000004.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000005.pool.root.2 append_preexec=rec.dotrigger=false;rec.dotruth.set_value_and_lock(false) outputaodfile=myaod.pool.root outputesdfile=myesd.pool.root outputhistfile=myhist.root outputtagfile=mytag.root 
###############################

logfile: https://atlasrtt.cern.ch/rtt/rel_2/devval/build/x86_64-slc5-gcc43-opt/offline/recjobtransformtests/mb_pu_mu0_d557/mb_pu_mu0_d557_log

excerpt:
&gt;&gt;&gt;
py:recexcommon_flags    info starting the migration to jobproperties. will soon be disabled.
traceback (most recent call last):
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasevent/rel_2/installarea/joboptions/recexcond/recexcommon_flags.py&quot;, line 320, in &lt;module&gt;
    from taurec.taurecflags import jobproperties
importerror: no module named taurec.taurecflags

error: could not instantiate taurecflags 
py:recexcommon_flags    info synchronising  old flags to new flags
py:recexcommon_flags    info inconsistency: for flag dotruth new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag dofloatingpointexception new flag: false old flag:true
py:recexcommon_flags    info modified value for flag poolrdoinput new flag: ['root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000001.pool.root.2', 'root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000002.pool.root.2', 'root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000003.pool.root.2', 'root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000004.pool.root.2', 'root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000005.pool.root.2'] old flag:['rdo.pool.root']
py:recexcommon_flags    info inconsistency: for flag docbnt new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag poolesdoutput new flag: myesd.pool.root old flag:esd.pool.root
py:recexcommon_flags    info inconsistency: for flag doaod new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag doeflow new flag: true old flag:false
py:recexcommon_flags    info inconsistency: for flag dohist new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag dotrigger new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag dowriteaod new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag evtmax new flag: -1 old flag:5
py:recexcommon_flags    info inconsistency: for flag doipatrec new flag: false old flag:true
py:recexcommon_flags    info inconsistency: for flag domonitoring new flag: true old flag:false
py:recexcommon_flags    info inconsistency: for flag amitag new flag: f380 old flag:
py:recexcommon_flags    info inconsistency: for flag dodpd new flag: true old flag:false
py:recexcommon_flags    info inconsistency: for flag dowritetag new flag: false old flag:true
oldflagcompatibility==true : deleting all standalone flags!
py:recexcommon_flags    info athenacommonflags.filesinput override input file specification
py:recexcommon_flags    info done the mapping in between the old flags and jobproperties
py:athena            info including file &quot;recexcond/recexcommon_detflags.py&quot;
py:athena            info including file &quot;recexcond/recexcommon_globalflags.py&quot;
py:jobproperty ::    info the jobproperty detdescrversion is blocked
py:muonrec           info changing default applyresilience = false
py:muonrec           info detflags.writeriopool.mdt_seton()
py:muonrec           info detflags.writeriopool.rpc_seton()
py:muonrec           info detflags.writeriopool.csc_seton()
py:muonrec           info detflags.writeriopool.tgc_seton()
py:recexcommon_flags    info trackrecordfilter alg switched off 
oldflagcompatibility==true : deleting all standalone flags! and forbidding use of old flags
py:recexcommon_topoptions    info abort on unchecked status code enabled !
py:aodfix            info running in a nightly. aodfix not scheduled.
py:aodfix            info please do manually 'rec.doapplyaodfix.set_value_and_lock(true)' to overwrite
py:configurabledb    info read module info for 5283 configurables from 9 genconfdb files
py:configurabledb    info no duplicates have been found: that's good !
eventinfomgtinit: got release version  atlasproduction-rel_2
py:iovdbsvc.conddb    info setting up conditions db access to instance oflp200
py:athena            info including file &quot;recexcond/alldet_detdescr.py&quot;
py:athena            info including file &quot;iddictdetdescrcnv/iddictdetdescrcnv_joboptions.py&quot;
py:athena            info including file &quot;tileconditions/tileconditions_joboptions.py&quot;
py:tileinfoconf.     info adding tilecablingsvc to servicemgr
py:tileinfoconf.     info adding default tilebadchantool to toolsvc
py:tileinfoconf.     info adding default tilecondtoolofc to toolsvc
py:tileinfoconf.     info adding default tilecondtoolemscale to toolsvc
py:tileinfoconf.     info adding default tilecondtoolnoisesample to toolsvc
py:tileinfoconf.     info adding default tilecondtooltiming to toolsvc
py:tileconditions_joboptions.py    info adjusting tileinfo for 7 samples
py:tileconditions_joboptions.py    info setting up cool for tilecal conditions data
py:tileinfoconf.     info changing default tilebadchantool configuration to cool source
py:tileinfoconf.     info changing default tilecondtoolemscale configuration to cool source
py:iovdbsvc.conddb    info ignoring additional request for folder /tile/ofl02/calib/cis/fit/lin
py:iovdbsvc.conddb    info ignoring additional request for folder /tile/ofl02/calib/las/lin
py:iovdbsvc.conddb    info ignoring additional request for folder /tile/ofl02/calib/ces
py:iovdbsvc.conddb    info ignoring additional request for folder /tile/ofl02/calib/ems
py:tileinfoconf.     info changing default tilecondtoolnoisesample configuration to cool source
py:iovdbsvc.conddb    info ignoring additional request for folder /tile/ofl02/noise/sample
py:tileinfoconf.     info changing default tilecondtooltiming configuration to cool source
py:tileinfoconf.     info changing default tilecondtoolofccool configuration to cool source
py:tileinfoconf.     info changing default tilecondtoolphypulsecool configuration to cool source
py:tileinfoconf.     info changing default tilecondtoolnoiseautocr configuration to cool source
py:tileconditions_joboptions.py    info adjusting tileinfo to return cell noise for opt.filter without iterations
py:athena            info including file &quot;caloidcnv/caloidcnv_joboptions.py&quot;
py:athena            info including file &quot;caloconditions/caloconditions_joboptions.py&quot;
py:athena            info including file &quot;caloconditions/larttcellmap_atlas_joboptions.py&quot;
py:athena            info including file &quot;caloconditions/calottidmap_atlas_joboptions.py&quot;
py:athena            info including file &quot;tileidcnv/tileidcnv_joboptions.py&quot;
py:athena            info including file &quot;lardetdescr/lardetdescr_joboptions.py&quot;
now initializing the lar readout geometry : standard atlas flavour
py:athena            info including file &quot;laridcnv/laridcnv_joboptions.py&quot;
py:athena            info including file &quot;lartools/lartools_joboptions.py&quot;
py:athena            info including file &quot;larconditionscommon/larconditionscommon_mc_joboptions.py&quot;
atlas-geo-16-00-00
py:jobpropertycontainer::    info  atlas-geo version 16 
py:jobpropertycontainer::    info  use global tag for all lareleccalibmc constants 
py:athena            info including file &quot;larcondathenapool/larcondathenapool_joboptions.py&quot;
py:athena            info including file &quot;larconditionscommon/laridmap_mc_joboptions.py&quot;
py:athena            info including file &quot;athenapoolcnvsvc/athenapool_joboptions.py&quot;
py:athena            info including file &quot;larcondathenapool/larcondathenapool_joboptions.py&quot;
py:jobpropertycontainer::    info  setting folder /lar/identifier/onoffidatlas with tag onoffidatlas-012
py:jobpropertycontainer::    info  setting folder /lar/identifier/febrodatlas with tag febrodatlas-005
py:athena            info including file &quot;indetbeamspotservice/beamcondsvc.py&quot;
py:athena            info including file &quot;amdcath/amdcath_joboptions.py&quot;
py:athena            info including file &quot;bfieldath/bfieldath_joboptions.py&quot;
py:athena            info including file &quot;partpropsvc/partpropsvc.py&quot;
py:athena            info including file &quot;detectorstatus/detstatussvc_conddb.py&quot;
py:athena            info including file &quot;detectorstatus/detstatussvc_tofilemetadata.py&quot;
py:aodfix            info executing addmetadata, input file type raw
old flags have been deleted
py:recexcommon_topoptions    info pool file : storing in objkeystore the list of input object directly from file
py:athena            info including file &quot;athenamonitoring/dataqualityinit_joboptions.py&quot;
py:athena            info including file &quot;athenamonitoring/dqmonflagsconfig_joboptions.py&quot;
dqmonflagsconfig_joboptions.py: monitoring environment set to tier0raw
shortened traceback (most recent user call last):
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/joboptions/recjobtransforms/skeleton.rawtoesd.py&quot;, line 134, in &lt;module&gt;
    else: include( &quot;recexcommon/recexcommon_topoptions.py&quot; )
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/joboptions/recexcommon/recexcommon_topoptions.py&quot;, line 604, in &lt;module&gt;
    treatexception(&quot;could not load athenamonitoring/dataqualityinit_joboptions.py&quot;)
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/joboptions/recexcommon/recexcommon_topoptions.py&quot;, line 602, in &lt;module&gt;
    include (&quot;athenamonitoring/dataqualityinit_joboptions.py&quot;)
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasevent/rel_2/installarea/joboptions/athenamonitoring/dataqualityinit_joboptions.py&quot;, line 12, in &lt;module&gt;
    treatexception(&quot;could not load athenamonitoring/dqmonflagsconfig_joboptions.py&quot;)
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasevent/rel_2/installarea/joboptions/athenamonitoring/dataqualityinit_joboptions.py&quot;, line 10, in &lt;module&gt;
    include(&quot;athenamonitoring/dqmonflagsconfig_joboptions.py&quot;)
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasevent/rel_2/installarea/joboptions/athenamonitoring/dqmonflagsconfig_joboptions.py&quot;, line 167, in &lt;module&gt;
    from taurec.taurecflags import jobproperties
importerror: no module named taurec.taurecflags
py:athena            info leaving with code 3: &quot;an error occurred when importing a file&quot;
info: read 113 error categories from file /afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_2/installarea/share/atlas_error_categories.db
/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_2/installarea/share/bin/athena.py -b runargs.rawtoesd.py recjobtransforms/skeleton.rawtoesd.py rawtoesd_messagesvc_joboptions.py --drop-and-reload - exit code 3
py:rawtoesd          info rawtoesd has completed running of athena with exit code 3.
info: read 113 error categories from file /afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_2/installarea/share/atlas_error_categories.db
info: read 55 error patterns from file /afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_2/installarea/share/atlas_error_patterns.db
info: read 61 ignore patterns from file /afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_2/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: tue jan 17 06:52:03 2012 cet, vmem: 337.48046875 mb, rss: 106.56640625 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,rel_2 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info done scanning 438 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 3
py:rawtoesd          info    errors  : 1
py:rawtoesd          info    fatals  : 0
athenalogchecker stop [rawtoesd log] wall-time: tue jan 17 06:52:03 2012 cet, vmem: 337.48046875 mb, rss: 106.59375 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 00 secs, cpu-time: 00 hrs 00 mins 00 secs, vmem: 0.0 mb, rss: 0.02734375 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 69999 (exit code 99)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: tue jan 17 06:52:03 2012 cet, vmem: 337.48046875 mb, rss: 106.62890625 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 00 hrs 00 mins 37 secs, cpu-time: 00 hrs 00 mins 33 secs, vmem: 0.6796875 mb, rss: 1.02734375 mb
===== job report produced by rawtoesd version unknown =====
command=/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/share/bin/reco_trf.py ami=f380 autoconfiguration=everything inputrdofile=root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000001.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000002.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000003.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000004.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000005.pool.root.2 append_preexec=rec.dotrigger=false;rec.dotruth.set_value_and_lock(false) outputaodfile=myaod.pool.root outputesdfile=myesd.pool.root outputhistfile=myhist.root outputtagfile=mytag.root
exitcode=99
errorcode=69999 (trf_unknown)
errorcategory=trf_unknown,	69999,	unknown transform error
fatals=0
errors=1
warnings=3
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 69999 (exit code 99)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: tue jan 17 06:52:03 2012 cet, vmem: 337.83203125 mb, rss: 106.95703125 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 00 hrs 00 mins 37 secs, cpu-time: 00 hrs 00 mins 33 secs, vmem: 1.28125 mb, rss: 1.40625 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_2/installarea/share/bin/reco_trf.py ami=f380 autoconfiguration=everything inputrdofile=root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000001.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000002.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000003.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000004.pool.root.2,root://eosatlas//eos/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.105001.pythia_minbias.digit.rdo.e603_s932_d557_tid364678_00/rdo.364678._000005.pool.root.2 append_preexec=rec.dotrigger=false;rec.dotruth.set_value_and_lock(false) outputaodfile=myaod.pool.root outputesdfile=myesd.pool.root outputhistfile=myhist.root outputtagfile=mytag.root
exitcode=99
errorcode=69999 (trf_unknown)
errorcategory=trf_unknown,	69999,	unknown transform error
fatals=0
errors=1
warnings=3
errorinfo
  acronym=trf_unknown (69999)
  message=could not instantiate taurecflags
  severity=error
  who=(unknown)
  producer=rawtoesd
  stage=initialise
athacronym=an error occurred when importing a file
athcode=3
rundiruseddisk=412354
rundiravailabledisk=192107888640
workdir=/pool/lsf/rtt/203939068
athcode=3
rundiravailabledisk=192108126208
dbrelease=
athacronym=an error occurred when importing a file
rundiruseddisk=193675
atlasrelease=rel_2
writing jobinfo_recotrf.xml file...
&lt;&lt;&lt;