opening a new bug. this is similar to bug #79645 but might be 
unrelated

- the tag is x100, which was supposed to fix #79645

- the crash is in muonidcombined, while it was in muoncombinedrefit

- the effect is much less dramatic, i have a total of 7 crashes
out of 4000 jobs in 4 different reconstruction tasks

data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task
data11_7tev.00177924.physics_standby.merge.raw.x100.recon.task
data11_7tev.00177924.physics_l1muon.merge.raw.x100.recon.task
data11_7tev.00177924.express_express.daq.raw.x100.recon.task

tommaso


sun mar 20 01:22:50 2011

tzex version is  $revision: 340749 $

&gt;&gt;&gt;&gt;&gt;  start of argdict 

{'--ignoreerrors': 'all',
 '--omitvalidation': 'all',
 'amitag': 'x100',
 'autoconfiguration': 'everything,conditionstag=comcond-es1p*-004-04',
 'inputbsfile': ['data11_7tev.00177912.physics_l1calo.merge.raw#data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1'],
 'maxevents': -1,
 'outputaodfile': 'data11_7tev.00177912.physics_l1calo.recon.aod.x100#data11_7tev.00177912.physics_l1calo.recon.aod.x100._lb0276._sfo-9._0001',
 'outputesdfile': 'data11_7tev.00177912.physics_l1calo.recon.esd.x100#data11_7tev.00177912.physics_l1calo.recon.esd.x100._lb0276._sfo-9._0001',
 'outputhistfile': 'data11_7tev.00177912.physics_l1calo.recon.hist.x100#data11_7tev.00177912.physics_l1calo.recon.hist.x100._lb0276._sfo-9._0001',
 'outputntup_trigfile': 'data11_7tev.00177912.physics_l1calo.recon.ntup_trig.x100#data11_7tev.00177912.physics_l1calo.recon.ntup_trig.x100._lb0276._sfo-9._0001',
 'preexec': 'indetflags.usebeamconstraint.set_value_and_lock(false);muoncombinedrecflags.docombinedmuonfit.set_value_and_lock(false)'}
&lt;&lt;&lt;&lt;&lt;  end of argdict 


sun mar 20 01:22:50 2011


stagein

[('data11_7tev.00177912.physics_l1calo.merge.raw', 'data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1', '/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw/data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1', 'data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1', 't0atlas', 'stagein', 'raw')]


sun mar 20 01:22:50 2011


-----&gt;  staging in  data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1

export stage_svcclass=t0atlas ; export stage_host=castoratlas ; umask 7 ;  rfcp /castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw/data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1 data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1
0 1619194332 bytes in 16 seconds through eth0 (in) and local (out) (98827 kb/sec)
1619194332 bytes in remote file

0 1619194332 bytes in 16 seconds through eth0 (in) and local (out) (98827 kb/sec)
1619194332 bytes in remote file

logging: connection succeeded
logging: executing...
insert into cpattempt (source,dest,retrynr,exitcode,erroracronym,startepoch,           endepoch,stdboth,filesize,server,transfertime,type,filetype,svcclass) values (:source,:dest,:retrynr,:exitcode,:eacro,:startepoch, :endepoch,:stdboth,          :filesize, :server, :transfertime, :type, :filetype, :svcclass)
{'retrynr': 1, 'svcclass': 't0atlas', 'eacro': 'ok', 'dest': 'data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1', 'startepoch': 1300580570, 'endepoch': 1300580590, 'filesize': 1619194332, 'server': '', 'source': '/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw/data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1', 'transfertime': 16, 'stdboth': '1619194332 bytes in 16 seconds through eth0 (in) and local (out) (98827 kb/sec)\n1619194332 bytes in remote file\n\n', 'type': 'stagein', 'filetype': 'raw', 'exitcode': 0}
logging: connection closed
ok 20236 20174 [(0, 1, '1619194332 bytes in 16 seconds through eth0 (in) and local (out) (98827 kb/sec)\n1619194332 bytes in remote file\n\n', 1300580570, 1300580590)]
staging in took  20 seconds ok

stagein succeeded

sun mar 20 01:23:10 2011


sun mar 20 01:23:10 2011


trans start

source /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/setuptrf.sh /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.5 atlasproduction-16.6.2.5 /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/specialsetup_tier0.sh ; python -u `which reco_trf.py` --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9845/9845929_data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.5//data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.argdict.gpickle  &amp;&gt; data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log
atlassetup(warning): test directory [/afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.5] doesn't exist
atlassetup(warning): test directory doesn't exist
    /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.5
created local authentication.xml file
created local dblookup.xml file
copied geometry db file from /afs/cern.ch/atlas/conditions/geomdb_tier0/geomdb/geomdb_sqlite
all done
unsetting frontier_server ...
setting coral_dblookup_path to  /pool/lsf/atlast0/128362072:/afs/cern.ch/atlas/software/releases/16.6.2/dbrelease/current/xmlconfig
setting coral_auth_path to  /pool/lsf/atlast0/128362072:/afs/cern.ch/atlas/tzero/prod1/inst/var/private:/afs/cern.ch/atlas/software/releases/16.6.2/dbrelease/current/xmlconfig:/afs/cern.ch/atlas/tzero/prod1/inst/var/private

trans end


sun mar 20 04:58:27 2011


tail of logfile 
&gt;&gt;&gt;
fpeauditor        warning fpe invalid in [execute] of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe divbyzero in [execute of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe invalid in [execute] of [indetprivxfinder]
fpeauditor        warning fpe divbyzero in [execute of [indetprivxfinder]
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
fpeauditor        warning fpe divbyzero in [execute of [indetalignmonmanager]
fpeauditor        warning fpe invalid in [execute] of [globalmonmanager]
fpeauditor        warning fpe divbyzero in [execute of [globalmonmanager]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #885720, run #177912 301 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #885744, run #177912 301 events processed so far  &lt;&lt;&lt;===
fpeauditor        warning fpe invalid in [execute] of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe divbyzero in [execute of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe invalid in [execute] of [indetprivxfinder]
fpeauditor        warning fpe divbyzero in [execute of [indetprivxfinder]
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
fpeauditor        warning fpe divbyzero in [execute of [indetalignmonmanager]
fpeauditor        warning fpe invalid in [execute] of [globalmonmanager]
fpeauditor        warning fpe divbyzero in [execute of [globalmonmanager]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #885744, run #177912 302 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #885724, run #177912 302 events processed so far  &lt;&lt;&lt;===
fpeauditor        warning fpe invalid in [execute] of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe divbyzero in [execute of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe invalid in [execute] of [indetprivxfinder]
fpeauditor        warning fpe divbyzero in [execute of [indetprivxfinder]
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
storegatesvc                                      warning retrieve(const): no valid proxy for object mugirlcombinedmuoncontainer  of type analysis::muoncontainer(clid 1073853171)
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
fpeauditor        warning fpe divbyzero in [execute of [indetalignmonmanager]
fpeauditor        warning fpe invalid in [execute] of [globalmonmanager]
fpeauditor        warning fpe divbyzero in [execute of [globalmonmanager]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #885724, run #177912 303 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #885664, run #177912 303 events processed so far  &lt;&lt;&lt;===
fpeauditor        warning fpe invalid in [execute] of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe divbyzero in [execute of [indettrt_standalonetrackfinder]
fpeauditor        warning fpe invalid in [execute] of [indetprivxfinder]
fpeauditor        warning fpe divbyzero in [execute of [indetprivxfinder]
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
toolsvc.muidmatchmaker                            warning  not choosing bestmatchchi2: 999999  16.832   fitchisq: 2.25693  21534.8
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
-------------------------------------------------------------------------------------
core dump from coredumpsvc on lxbrl2910.cern.ch at sun mar 20 04:58:05 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 1 (address not mapped to object)
  pid   = 64, uid = 0
  value = (2019016531, 0x7857bf53)
  vmem = 2232.11 mb
  rss  = 1822.61 mb
  total-ram = 15929.1 mb
  free-ram  = 80.8398 mb
  buffer-ram= 98.3984 mb
  total-swap= 20481.2 mb
  free-swap = 20480.9 mb
  addr  = 0x40

event counter: 304
eventid: [run,evt,lumi,time,bunchcross,detmask] = [177912,885664,276,1300507153:406948265,2233,0x1fffffffffff7]
last incident: athenaeventloopmgr:beginevent
current algorithm: muidcombined
algorithm stack: 
   athmasterseq
   athalgseq
   muidcombined
-------------------------------------------------------------------------------------

(pid=24953 ppid=24632) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 1 (address not mapped to object)
  pid    = 64, uid = 0
  value  = (2019016531, 0x7857bf53)
  addr   = 0x40
  stack  = (2, 0, (nil))

  eip: 0023:729ce9d6           eflags: 00210216
  eax: 729e1630   ebx: 729e19d8   ecx: 00000000   edx: 8994d5a0
  esi: 00000000   edi: 8994d5a0   ebp: ff9382e8   esp: ff9382b4
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ff9382b4  trap: 14/4  oldmask: 00000000   cr2: 00000040

  fpu:  control = ffff037f
        status  = ffff0120
        tag     = ffffffff
        ip      = 0010:616c8357
        data    = 002b:00000000
        state   = 00000120
    %fp0 = [0000:0000000000000000]
    %fp1 = [8000:0000000000000000]
    %fp2 = [0000:0000000000000000]
    %fp3 = [0000:0000000000000000]
    %fp4 = [4012:e315a52adef7ee97]
    %fp5 = [c009:6800f127202bd21a]
    %fp6 = [400b:000000000000d3d8]
    %fp7 = [400b:000000000000d568]

stack trace:
 0x5cdcb440 athena::debugaids::stacktrace(int) + 0x60 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5cdc6f9f athena::signal::fataldump(int, siginfo*, void*) + 0xff [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5cdc72b1 athena::signal::fatal(int, siginfo*, void*) + 0x181 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5e5fd64e coredumpsvchandler::action(int, siginfo*, void*) + 0xce [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0x729ce9d6 trk::fitparameters::fitparameters(trk::perigee const&amp;) + 0x26 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libtrkipatfitterutils.so]
 0x72984692 trk::ipatfitter::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const + 0xf2 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libtrkipatfitter.so]
 0x727edfba rec::combinedmuontrackbuilder::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const + 0x26a [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x727eb366 rec::combinedmuontrackbuilder::combinedfit(trk::track const&amp;, trk::track const&amp;, trk::track const&amp;) const + 0x846 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x727f525d rec::combinedmuontrackbuilder::indetextension(trk::track const&amp;, std::vector&lt;trk::measurementbase const*, std::allocator&lt;trk::measurementbase const*&gt; &gt; const&amp;, trk::parameterst&lt;trk::charged&gt; const*, trk::parameterst&lt;trk::charged&gt; const*, trk::parameterst&lt;trk::charged&gt; const*) const + 0x86d [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x73bdf13d rec::muidmatchmaker::indetextension(trk::track const&amp;, trk::track const&amp;, trk::track const*, trk::track const*, trk::track const*) const + 0x3dd [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0x73be5c2e rec::muidmatchmaker::execute(datavector&lt;trk::track, datavector_detail::nobase&gt; const*, datavector&lt;trk::track, datavector_detail::nobase&gt; const*, datavector&lt;trk::track, datavector_detail::nobase&gt; const*) const + 0x16ce [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0x6747304a muidcombined::execute() + 0x8da [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombined.so]
 0x5896751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x6459cba3 athsequencer::execute() + 0x1f3 [/afs/cern.ch/atlas/software/releases/16.6.2/atlascore/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x5896751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x6459cba3 athsequencer::execute() + 0x1f3 [/afs/cern.ch/atlas/software/releases/16.6.2/atlascore/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x5896751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x5e5e665f athenaeventloopmgr::executealgorithms() + 0x6f [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e5ece4e athenaeventloopmgr::executeevent(void*) + 0x83e [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e5ebb32 athenaeventloopmgr::nextevent(int) + 0x8a2 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e5e57eb athenaeventloopmgr::executerun(int) + 0x1b [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5de5da49 applicationmgr::executerun(int) + 0x79 [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
ï¿½0x5d5ed94d _zn12_global__n_1l11method_4267epvs0_rkst6vectoris0_sais0_ees0_ dictionary_dict.cpp:0 + 0x5d5ed94d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0x578ca437 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int) + 0x157 [/afs/cern.ch/atlas/software/releases/16.6.2/detcommon/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0x5e26b034 &lt;unknown function&gt;
 0x5617d512 cint::g__callfunc::execute(void*) + 0xd2 [/afs/cern.ch/atlas/software/releases/16.6.2/detcommon/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0x5590acec pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*) + 0x2c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x55910c24 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*) + 0x64 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x55910dbc pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*) + 0x6c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x5590fa92 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long) + 0x112 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
ï¿½0x55916859 pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*) methodproxy.cxx:0 + 0x55916859 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 fast_function /build/agaspar/work/python-2.6.5/python/ceval.c:3846 + 0x5a89 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call /build/agaspar/work/python-2.6.5/objects/funcobject.c:524 + 0x8137caa [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call /build/agaspar/work/python-2.6.5/objects/classobject.c:2579 + 0x8064935 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373 + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call /build/agaspar/work/python-2.6.5/objects/funcobject.c:524 + 0x8137caa [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call /build/agaspar/work/python-2.6.5/objects/classobject.c:2579 + 0x8064935 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373 + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags /build/agaspar/work/python-2.6.5/python/pythonrun.c:935 + 0x204 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags /build/agaspar/work/python-2.6.5/python/pythonrun.c:739 + 0x7a [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main /build/agaspar/work/python-2.6.5/modules/main.c:572 + 0xc56 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main /build/agaspar/work/python-2.6.5/./modules/python.c:24 + 0x32 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0050be9c __libc_start_main + 0xdc [/lib/libc.so.6]
 0x08057d51 _start + 0x8057d51 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info execution of rawtoesd complete with exit code -11.
py:rawtoesd         error transform received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/share/atlas_error_patterns.db
info: read 58 ignore patterns from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: sun mar 20 03:58:16 2011, vmem: 112.01953125 mb, rss: 66.84765625 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,16.6.2.5 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 25286 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 9158
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: sun mar 20 03:58:20 2011, vmem: 112.01953125 mb, rss: 66.84765625 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 04 secs, cpu-time: 00 hrs 00 mins 04 secs, vmem: 0.0 mb, rss: 0.0 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: sun mar 20 03:58:20 2011, vmem: 112.01953125 mb, rss: 66.875 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 03 hrs 34 mins 42 secs, cpu-time: 03 hrs 31 mins 28 secs, vmem: 0.484375 mb, rss: 0.50390625 mb
===== job report produced by rawtoesd version patjobtransforms-00-03-15 =====
command=/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9845/9845929_data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.5//data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=9
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: sun mar 20 03:58:21 2011, vmem: 112.01953125 mb, rss: 66.87890625 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 03 hrs 34 mins 42 secs, cpu-time: 03 hrs 31 mins 28 secs, vmem: 0.484375 mb, rss: 0.51953125 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.5/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9845/9845929_data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.5//data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=9
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=24953 ppid=24632) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=2191285890
rundiravailabledisk=290362523648
workdir=/pool/lsf/atlast0/128362072
athcode=8
rundiravailabledisk=290362806272
dbrelease=current
athacronym=an unknown exception occurred
rundiruseddisk=2191029514
atlasrelease=16.6.2.5
writing jobinfo_recotrf.xml file...
&lt;&lt;&lt;

ls workdir 
&gt;&gt;&gt;
total 2142140
-rw-r--r-- 1 atlast0 zp      43048 mar 20 04:57 1300580530.128362072.out
-rw-r----- 1 atlast0 zp        125 mar 20 01:22 1300580530.128362072.out.ootw
-rw-r--r-- 1 atlast0 zp         67 mar 20 04:58 9845929_data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.5.gpickle
-rw-r--r-- 1 atlast0 zp      37234 mar 20 04:58 9845929_data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.5.log
-rw-r--r-- 1 atlast0 zp       6355 mar 20 04:58 athenasummary_rawtoesd.txt
-rw-r--r-- 1 atlast0 zp       1058 mar 20 01:23 athfile-cache.ascii
-rw-r--r-- 1 atlast0 zp       5433 mar 20 01:23 authentication.xml
-rw-r----- 1 atlast0 zp 1619194332 mar 20 01:23 data11_7tev.00177912.physics_l1calo.merge.raw._lb0276._sfo-9._0001.1
-rw-r--r-- 1 atlast0 zp    2882391 mar 20 04:58 data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log
-rw-r--r-- 1 atlast0 zp  522065230 mar 20 04:57 data11_7tev.00177912.physics_l1calo.recon.esd.x100._lb0276._sfo-9._0001
-rw-r--r-- 1 atlast0 zp       4984 mar 20 01:23 dblookup.xml
-rw-r--r-- 1 atlast0 zp      45674 mar 20 04:58 dmesg_athena.txt
-rw-r--r-- 1 atlast0 zp     466191 mar 20 01:24 generated_amdb_simrec_pool
-rw-r--r-- 1 atlast0 zp   42110976 mar 20 01:23 geomdb_t0replica.sqlite
-rw-r--r-- 1 atlast0 zp       1116 mar 20 01:23 inputdictionary.pickle
-rw-r--r-- 1 atlast0 zp      42921 mar 20 04:58 jobinfo_rawtoesd.xml
-rw-r--r-- 1 atlast0 zp      42876 mar 20 04:58 jobinfo_recotrf.xml
-rw-r--r-- 1 atlast0 zp      42876 mar 20 04:58 jobinfo.xml
-rw-r--r-- 1 atlast0 zp       1398 mar 20 04:58 jobreport.gpickle
-rw-r--r-- 1 atlast0 zp      55977 mar 20 04:58 jobreport.pickle
-rw-r--r-- 1 atlast0 zp      34042 mar 20 04:58 jobreport_rawtoesd.gpickle
-rw-r--r-- 1 atlast0 zp      49232 mar 20 04:58 jobreport_rawtoesd.pickle
-rw-r--r-- 1 atlast0 zp      36157 mar 20 04:58 jobreport_rawtoesd.txt
-rw-r--r-- 1 atlast0 zp       1398 mar 20 04:58 jobreport_recotrf.gpickle
-rw-r--r-- 1 atlast0 zp      55977 mar 20 04:58 jobreport_recotrf.pickle
-rw-r--r-- 1 atlast0 zp      43171 mar 20 04:58 jobreport_recotrf.txt
-rw-r--r-- 1 atlast0 zp      43171 mar 20 04:58 jobreport.txt
-rwxr-xr-x 1 atlast0 zp        542 mar 20 01:23 last.rawtoesd
-rwxr-xr-x 1 atlast0 zp        170 mar 20 01:23 last.reco_trf
-rw-r--r-- 1 atlast0 zp        617 mar 20 01:23 last.runargs.gpickle
-rw-r--r-- 1 atlast0 zp      32143 mar 20 01:28 monitor.root
-rw-r--r-- 1 atlast0 zp     855601 mar 20 04:57 ntuple_bstoesddpd.pmon.stream
drwxr-xr-x 3 atlast0 zp       4096 mar 20 01:23 oradiag_atlast0
lrwxrwxrwx 1 atlast0 zp         57 mar 20 01:23 poolcond -&gt; /afs/cern.ch/atlas/conditions/poolcond/catalogue/poolcond
-rw-r--r-- 1 atlast0 zp        106 mar 20 01:24 poolfilecatalog.xml
-rw-r--r-- 1 atlast0 zp    2871483 mar 20 04:58 rawtoesd.log
-rw-r--r-- 1 atlast0 zp       1865 mar 20 01:23 rawtoesd_messagesvc_joboptions.py
-rwxr-xr-x 1 atlast0 zp        311 mar 20 01:23 rawtoesd_runathena
-rw-r--r-- 1 atlast0 zp        617 mar 20 01:23 runargs.rawtoesd.gpickle
-rw-r--r-- 1 atlast0 zp       1147 mar 20 01:23 runargs.rawtoesd.py
drwxr-xr-x 2 atlast0 zp       4096 mar 20 01:24 tmp
-rw-r--r-- 1 atlast0 zp        680 mar 20 04:58 transformtimer_reco_trf.pickle
-rw-r--r-- 1 atlast0 zp       3071 mar 20 01:22 wrapper.py
&lt;&lt;&lt;
&gt;&gt;&gt;&gt;&gt;  start of trf jobreport ['prodsys' key]

{'athacronym': 'an unknown exception occurred',
 'athcode': '8',
 'jobinputs': [],
 'joboutputs': [],
 'more': {'machine': 'unknown'},
 'nevents': 0,
 'trfacronym': 'trf_segfault',
 'trfcode': 60010,
 'trfexitcode': 6}
&lt;&lt;&lt;&lt;&lt;  end of trf jobreport 

[{'guid': '0a304cd4-214e-41ae-9754-729020137583',
  'checksum': 'ad:6467da09',
  'dataset': 'data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log',
  'events': 0,
  'lfn': 'data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log',
  'size': 2882391l}]


sun mar 20 04:58:28 2011


stageout


sun mar 20 04:58:28 2011


---------------------------------------------------------------------------------------------------------------------------------------
staging out  data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log # data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log
---------------------------------------------------------------------------------------------------------------------------------------

warning no storage defined, will use default (asinds)
info resolved asinds into  [{'svcclass': 't0merge', 'type': 'castor', 'basepath': '/castor/cern.ch/grid/atlas/tzero/prod1/temp', 'fileclass': 'atlas_notape', 'stagehost': 'castoratlas', 'autodirstruct': '.1/.3/.2/d'}]

-----&gt;  staging out to  {'svcclass': 't0merge', 'type': 'castor', 'basepath': '/castor/cern.ch/grid/atlas/tzero/prod1/temp', 'fileclass': 'atlas_notape', 'stagehost': 'castoratlas', 'autodirstruct': '.1/.3/.2/d'}

getdest final storageinfo =  {'svcclass': 't0merge', 'type': 'castor', 'basepath': '/castor/cern.ch/grid/atlas/tzero/prod1/temp', 'fileclass': 'atlas_notape', 'stagehost': 'castoratlas', 'autodirstruct': '.1/.3/.2/d'} 

rfmkdir -p /castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log ; nschclass atlas_notape /castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log
sun mar 20 04:58:28 2011
0 mkdir() : file exists

export stage_svcclass=t0merge ; export stage_host=castoratlas ; umask 7 ;  rfcp data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log /castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log.5
0 2882391 bytes in 1 seconds through local (in) and eth0 (out) (2814 kb/sec)
2882391 bytes in remote file

0 2882391 bytes in 1 seconds through local (in) and eth0 (out) (2814 kb/sec)
2882391 bytes in remote file

logging: connection succeeded
logging: executing...
insert into cpattempt (source,dest,retrynr,exitcode,erroracronym,startepoch,           endepoch,stdboth,filesize,server,transfertime,type,filetype,svcclass) values (:source,:dest,:retrynr,:exitcode,:eacro,:startepoch, :endepoch,:stdboth,          :filesize, :server, :transfertime, :type, :filetype, :svcclass)
{'retrynr': 1, 'svcclass': 't0merge', 'eacro': 'ok', 'dest': '/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log.5', 'startepoch': 1300593508, 'endepoch': 1300593512, 'filesize': 2882391, 'server': '', 'source': 'data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log', 'transfertime': 1, 'stdboth': '2882391 bytes in 1 seconds through local (in) and eth0 (out) (2814 kb/sec)\n2882391 bytes in remote file\n\n', 'type': 'stageout', 'filetype': 'log', 'exitcode': 0}
logging: connection closed

ok 4006 3876 [(0, 1, '2882391 bytes in 1 seconds through local (in) and eth0 (out) (2814 kb/sec)\n2882391 bytes in remote file\n\n', 1300593508, 1300593512)]

sun mar 20 04:58:32 2011

staging out took  4 seconds ok

stageout succeeded

transformation problem

report
{'prodsys': {'jobinputs': [],
             'joboutputs': [{'guid': '0a304cd4-214e-41ae-9754-729020137583',
                             'checksum': 'ad:6467da09',
                             'dataset': 'data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log',
                             'events': 0,
                             'lfn': 'data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log.5',
                             'more': {'metadata': {},
                                      'surl': '/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_l1calo/00177912/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task.tmp.log/data11_7tev.00177912.physics_l1calo.merge.raw.x100.recon.task._lb0276._sfo-9._0001.job.tmp.log.5'},
                             'size': 2882391l}],
             'more': {'num1': 0, 'num2': 0, 'txt1': ''},
             'nevents': 0,
             'trfacronym': 'trf_segfault',
             'trfcode': 60010},
 't0exe': {'cpu': 12917,
           'cpuinfo': 'cpuinfo',
           'endepoch': 1300593512,
           'erroracronym': 'trferror',
           'errorcode': none,
           'errortext': '',
           'exitcode': 0,
           'stageintime': 20,
           'stageouttime': 4,
           'startepoch': 1300580570}}

end of report

sun mar 20 04:58:32 2011


end of wrapper log file
