hi, 

one job of the release 17.0.1.2 task was crashed by segmentation fault during muidcombined algorithm. 

i will extract the log file and raw files. 

cheers, 
susumu 

+verbatim+
input file(s):
--------------------------------------------------------------------------------
stage_svcclass=atlcal
/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/physics_muons/00184130/data11_7tev.00184130.physics_muons.merge.raw/data11_7tev.00184130.physics_muons.merge.raw._lb0224._sfo-11._0001.1


tier-0 transformation command run:
--------------------------------------------------------------------------------
source /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/setuptrf.sh /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.1.2 atlasproduction-17.0.1.2 /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/specialsetup_tier0.sh ; python -u `which reco_trf.py` --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/12111/12111073_data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.3//data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.argdict.gpickle  &amp;&gt; data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.tmp.log


ami configuration tag (more details):
--------------------------------------------------------------------------------
see http://ami.in2p3.fr:8080/opencms/opencms/ami/www/referencetables/ --&gt; configurationtag --&gt; interpret --&gt; v9


logfile location:
--------------------------------------------------------------------------------
stage_svcclass=t0merge
/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_muons/00184130/data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task.tmp.log/data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.tmp.log.3


tail of the log file (300 lines):
--------------------------------------------------------------------------------
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2146994, run #184130 447 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2146994, run #184130 448 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2153886, run #184130 448 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2153886, run #184130 449 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2150675, run #184130 449 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2150675, run #184130 450 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2154932, run #184130 450 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2154932, run #184130 451 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2153836, run #184130 451 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2153836, run #184130 452 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2147832, run #184130 452 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2147832, run #184130 453 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2125878, run #184130 453 events processed so far  &lt;&lt;&lt;===
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
perfmonsvc:    info [evt: 453] cpu=30839.000 ms vmem=2140.660 mb dvmem=   4.367 mb alloc= +0 -0    0.000 kb  =&gt; [/evt/perfmonslice]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2125878, run #184130 454 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2154481, run #184130 454 events processed so far  &lt;&lt;&lt;===
toolsvc.muidsegmentregionrecoverytool             warning failed to find prds in main sector: 8
toolsvc.muidsegmentregionrecoverytool             warning failed to find prds in main sector: 8
toolsvc.mugirlcandidatetool                       warning cannot find base road
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2154481, run #184130 455 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2154678, run #184130 455 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2154678, run #184130 456 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2148625, run #184130 456 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2148625, run #184130 457 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2157275, run #184130 457 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2157275, run #184130 458 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2160549, run #184130 458 events processed so far  &lt;&lt;&lt;===
toolsvc.muidsegmentregionrecoverytool             warning failed to find prds in main sector: 6
toolsvc.mugirlcandidatetool                       warning cannot find base road
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2160549, run #184130 459 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2159011, run #184130 459 events processed so far  &lt;&lt;&lt;===
toolsvc.mugirlcandidatetool                       warning cannot find base road
576
578
580
584
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2159011, run #184130 460 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2156372, run #184130 460 events processed so far  &lt;&lt;&lt;===
toolsvc.mugirlcandidatetool                       warning cannot find base road
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2156372, run #184130 461 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2147432, run #184130 461 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2147432, run #184130 462 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2159399, run #184130 462 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2159399, run #184130 463 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2154004, run #184130 463 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2154004, run #184130 464 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2150303, run #184130 464 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
perfmonsvc:    info [evt: 464] cpu=35041.000 ms vmem=2160.184 mb dvmem=  19.523 mb alloc= +0 -0    0.000 kb  =&gt; [/evt/perfmonslice]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2150303, run #184130 465 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2156185, run #184130 465 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2156185, run #184130 466 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2151196, run #184130 466 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object antikt4topojets  of type jetcollection(clid 1162448536)
toolsvc.dqtratemontool                            warning jet collection not found
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #2151196, run #184130 467 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #2157704, run #184130 467 events processed so far  &lt;&lt;&lt;===
toolsvc.muidsegmentregionrecoverytool             warning failed to find prds in main sector: 4
toolsvc.combinedmuontrackbuilder                  warning  combined muon track is missing the tsos's describing calorimeter association 
toolsvc.combinedmuontrackbuilder                  warning  combined muon track is missing the tsos's describing calorimeter association 
toolsvc.combinedmuontrackbuilder                  warning  combined muon track is missing the tsos's describing calorimeter association 
toolsvc.combinedmuontrackbuilder                  warning  combined muon track is missing the tsos's describing calorimeter association 
toolsvc.muontracksummaryhelper                    warning number of hits in road &lt; 0 in first projection: -4, setting = 0. (nhits in first projection = -4)
toolsvc.muontracksummaryhelper                    warning number of hits in road &lt; 0 in second projection: -4, setting = 0. (nhits in second projection = -4)
toolsvc.muontracksummaryhelper                    warning number of hits in road &lt; 0 in first projection: -4, setting = 0. (nhits in first projection = -4)
toolsvc.combinedmuontrackbuilder                  warning standalonerefit: failed propagation to middletsos
-------------------------------------------------------------------------------------
core dump from coredumpsvc on lxbrb1717.cern.ch at mon jul  4 22:27:02 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 2 (invalid permissions for mapped object)
  pid   = -193956568, uid = 4159018230
  value = (1413928716, 0x5446d70c)
  vmem = 2160.18 mb
  rss  = 1788.49 mb
  total-ram = 15929.1 mb
  free-ram  = 86.957 mb
  buffer-ram= 46.5938 mb
  total-swap= 20481.2 mb
  free-swap = 20431 mb
  addr  = 0xf4707528

event counter: 468
eventid: [run,evt,lumi,time,bunchcross,detmask] = [184130,2157704,224,1309135162:532689755,2526,0xfffffffffff7]
last incident: athenaeventloopmgr:beginevent
current algorithm: muidcombined
algorithm stack: 
   athmasterseq
   athalgseq
   muidcombined
-------------------------------------------------------------------------------------

(pid=17149 ppid=17037) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 2 (invalid permissions for mapped object)
  pid    = -193956568, uid = -135949066
  value  = (1413928716, 0x5446d70c)
  addr   = 0xf4707528
  stack  = (2, 0, (nil))

  eip: 0023:f4707528           eflags: 00210296
  eax: 490a17b8   ebx: d8d21910   ecx: 17cb50bc   edx: 17ec624c
  esi: 46bb01a0   edi: 17ed0324   ebp: fffccd18   esp: fffccc1c
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: fffccc1c  trap: 14/21  oldmask: 00000000   cr2: f4707528

  fpu:  control = ffff037f
        status  = ffff0020
        tag     = ffffffff
        ip      = 0010:d8b8b534
        data    = 002b:00000000
        state   = 00000020
    %fp0 = [3ffe:bd4e3552f6458d5f]
    %fp1 = [bff9:fd1ac077b611cd1d]
    %fp2 = [bff0:6c0008773b44d7bd]
    %fp3 = [bfed:d0009c9fce68a176]
    %fp4 = [4015:36b40d76c3a9b4c9]
    %fp5 = [bfff:0000000000008000]
    %fp6 = [4005:00001ee3cf88a73f]
    %fp7 = [3fff:0000000000008000]

stack trace:
 0xf420d7a0 athena::debugaids::stacktrace(int)+0x60 ??:0   + 0x60 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf420e7bf athena::signal::fataldump(int, siginfo*, void*)+0xff ??:0   + 0xff [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf420ead1 athena::signal::fatal(int, siginfo*, void*)+0x181 ??:0   + 0x181 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf2736706 coredumpsvchandler::action(int, siginfo*, void*)+0x106 ??:0   + 0x106 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0xf4707528 vtable for hepgeom::basicvector3d&lt;double&gt;+0x8 ??:0   + 0x8 [/afs/cern.ch/atlas/software/releases/17.0.1/atlasevent/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libcaloevent.so]
 0xd77e7e58 rec::muidmatchmaker::evaluatematchproperties(rec::muonmatch&amp;) const+0x1a8 ??:0   + 0x1a8 [/afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0xd77ec0c9 rec::muidmatchmaker::execute(datavector&lt;trk::track, datamodel_detail::nobase&gt; const*, datavector&lt;trk::track, datamodel_detail::nobase&gt; const*, datavector&lt;trk::track, datamodel_detail::nobase&gt; const*) const+0x799 ??:0   + 0x799 [/afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0xe438317d muidcombined::execute()+0xe4d ??:0   + 0xe4d [/afs/cern.ch/atlas/software/releases/17.0.1/atlasreconstruction/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libmuidcombined.so]
 0xf52409dd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.1/gaudi/v22r1p2/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe79d546b athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf52409dd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.1/gaudi/v22r1p2/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe79d546b athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf52409dd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.1/gaudi/v22r1p2/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xf27f90ff athenaeventloopmgr::executealgorithms()+0x6f ??:0   + 0x6f [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf27fed16 athenaeventloopmgr::executeevent(void*)+0x6f6 ??:0   + 0x6f6 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf27fd542 athenaeventloopmgr::nextevent(int)+0x8d2 ??:0   + 0x8d2 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf27ffb49 athenaeventloopmgr::executerun(int)+0x29 ??:0   + 0x29 [/afs/cern.ch/atlas/software/releases/17.0.1/atlascore/17.0.1/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf2dec8d9 applicationmgr::executerun(int)+0x79 ??:0   + 0x79 [/afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
 0xf36dfddd _zn12_global__n_1l11method_4301epvs0_rkst6vectoris0_sais0_ees0_+0x3d ??:0   + 0xf36dfddd [/afs/cern.ch/atlas/software/releases/17.0.1/gaudi/v22r1p2/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0xf5c38a57 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int)+0x157 ??:0   + 0x157 [/afs/cern.ch/atlas/software/releases/17.0.1/lcgcmt/lcgcmt_60b/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0xf2b7d034 &lt;unknown function&gt;
 0xf6edc572 cint::g__callfunc::execute(void*)+0xd2 ??:0   + 0xd2 [/afs/cern.ch/atlas/software/releases/17.0.1/lcgcmt/lcgcmt_60b/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0xf7bb772c pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*)+0x2c ??:0   + 0x2c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/app/releases/root/5.28.00b/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bbe104 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*)+0x64 ??:0   + 0x64 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/app/releases/root/5.28.00b/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bbe29c pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*)+0x6c ??:0   + 0x6c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/app/releases/root/5.28.00b/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bbc7c3 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long)+0x153 ??:0   + 0x153 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/app/releases/root/5.28.00b/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bc3afa pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*)+0x4aa ??:0   + 0xf7bc3afa [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/app/releases/root/5.28.00b/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 pyeval_evalframeex+0x5a89 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags+0x204 /build/agaspar/work/python-2.6.5/python/pythonrun.c:935   + 0x204 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags+0x7a /build/agaspar/work/python-2.6.5/python/pythonrun.c:739   + 0x7a [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main+0xc56 /build/agaspar/work/python-2.6.5/modules/main.c:572   + 0xc56 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main+0x32 /build/agaspar/work/python-2.6.5/./modules/python.c:24   + 0x32 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x00383e9c __libc_start_main+0xdc ??:0   + 0xdc [/lib/libc.so.6]
 0x08057d51 _start+0x21 ??:0   + 0x8057d51 [/afs/cern.ch/atlas/software/releases/17.0.1/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info execution of rawtoesd complete with exit code -11.
py:rawtoesd         error transform received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/share/atlas_error_patterns.db
info: read 58 ignore patterns from file /afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: mon jul  4 20:27:17 2011, vmem: 100.80078125 mb, rss: 67.8359375 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,17.0.1.2 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 20941 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 1527
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: mon jul  4 20:27:19 2011, vmem: 100.80078125 mb, rss: 67.86328125 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 02 secs, cpu-time: 00 hrs 00 mins 02 secs, vmem: 0.0 mb, rss: 0.02734375 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: mon jul  4 20:27:19 2011, vmem: 100.80078125 mb, rss: 67.8828125 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 01 hrs 20 mins 18 secs, cpu-time: 01 hrs 17 mins 20 secs, vmem: 0.4375 mb, rss: 0.59765625 mb
===== job report produced by rawtoesd version patjobtransforms-00-03-52 =====
command=/afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/12111/12111073_data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.3//data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=3
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: mon jul  4 20:27:19 2011, vmem: 100.9453125 mb, rss: 68.04296875 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 01 hrs 20 mins 18 secs, cpu-time: 01 hrs 17 mins 20 secs, vmem: 0.58203125 mb, rss: 0.76953125 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/releases/17.0.1/atlasproduction/17.0.1.2/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/12111/12111073_data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.3//data11_7tev.00184130.physics_muons.merge.raw.v9.recon.task._lb0224._sfo-11._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=3
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=17149 ppid=17037) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=2511093660
rundiravailabledisk=205861601280
workdir=/pool/lsf/atlast0/152376876
athcode=8
rundiravailabledisk=205861859328
dbrelease=current
athacronym=an unknown exception occurred
-verbatim-