a bug which is similar to 
bug #79683: segmentation fault in muonidcombined in x100 reconstruction
but should be a different bug is seen with f351.  
f351 uses atlasproduction 16.6.2.6 but additionally uses muidcombinedtools-00-09-25-07 installed in 
/afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.6
bug #79683 should be fixed by this muidcombinedtools-00-09-25-07. 

i will extract the raw file of the event. 

susumu 

+verbatim+
input file(s):
--------------------------------------------------------------------------------
stage_svcclass=atlcal
/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/express_express/00177986/data11_7tev.00177986.express_express.merge.raw/data11_7tev.00177986.express_express.merge.raw._lb0393._sfo-all._0001.1


tier-0 transformation command run:
--------------------------------------------------------------------------------
source /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/setuptrf.sh /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.6 atlasproduction-16.6.2.6 /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/specialsetup_tier0.sh ; python -u `which reco_trf.py` --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9954/9954793_data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.3//data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.argdict.gpickle  &amp;&gt; data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.tmp.log


ami configuration tag (more details):
--------------------------------------------------------------------------------
see http://ami.in2p3.fr:8080/opencms/opencms/ami/www/referencetables/ --&gt; configurationtag --&gt; interpret --&gt; f351


logfile location:
--------------------------------------------------------------------------------
stage_svcclass=t0merge
/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/express_express/00177986/data11_7tev.00177986.express_express.merge.raw.f351.recon.task.tmp.log/data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.tmp.log.3


tail of the log file (300 lines):
--------------------------------------------------------------------------------
fpeauditor        warning fpe overflow in [execute] of [indettracktree_filteralgorithmsholder]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24390686, run #177986 172 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24391067, run #177986 172 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24391067, run #177986 173 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24393979, run #177986 173 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24393979, run #177986 174 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24396679, run #177986 174 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
mbtstimediffeventinfoalg                             info event identified as background, set bit 'mbtstimediffhalo' in eventinfo background word
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xec000000
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.indetalignmonresiduals_notrig             warning propagation of unbiased othersideparameters failed
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24396679, run #177986 175 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24397740, run #177986 175 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
mbtstimediffeventinfoalg                             info event identified as background, set bit 'mbtstimediffhalo' in eventinfo background word
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24397740, run #177986 176 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24400158, run #177986 176 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.ipatslfitter                              warning  fail track with incomplete return tsos: no trackparameters
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
calling the propertie dump
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24400158, run #177986 177 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24400905, run #177986 177 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24400905, run #177986 178 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24401507, run #177986 178 events processed so far  &lt;&lt;&lt;===
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24401507, run #177986 179 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24405023, run #177986 179 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
mbtstimediffeventinfoalg                             info event identified as background, set bit 'mbtstimediffhalo' in eventinfo background word
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xed000000
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
calling the propertie dump
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24405023, run #177986 180 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24406153, run #177986 180 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
mbtstimediffeventinfoalg                             info event identified as background, set bit 'mbtstimediffhalo' in eventinfo background word
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24406153, run #177986 181 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24406564, run #177986 181 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24406564, run #177986 182 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24409500, run #177986 182 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
mbtstimediffeventinfoalg                             info event identified as background, set bit 'mbtstimediffhalo' in eventinfo background word
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xec000000
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.dcmathsegmentmaker                        warning found multiple sectors on input, cannot process data 
toolsvc.dcmathsegmentmaker                        warning  no mdt selected, returning 0 
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24409500, run #177986 183 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24409502, run #177986 183 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
fpeauditor        warning fpe overflow in [execute] of [indettracktree]
fpeauditor        warning fpe overflow in [execute] of [indettracktree_filteralgorithmsholder]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24409502, run #177986 184 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24409798, run #177986 184 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.mctbfitter                                warning attempt to apply material corrections with q/p=0, reject track
toolsvc.mctbfitter                                warning attempt to apply material corrections with q/p=0, reject track
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
calling the propertie dump
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #24409798, run #177986 185 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #24411478, run #177986 185 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.zdcrecchanneltool                         warning zero size g0d0 at id  0xfffff800
toolsvc.zdcrecchanneltool                         warning zero size g1d0 at id  0xfffff800
toolsvc.muidmatchmaker                            warning  evaluatematchchi2: failed re-evaluation of match chi2 
-------------------------------------------------------------------------------------
core dump from coredumpsvc on lxbra5907.cern.ch at wed mar 23 15:06:22 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 1 (address not mapped to object)
  pid   = 64, uid = 0
  value = (1037709930, 0x3dda326a)
  vmem = 2288.38 mb
  rss  = 1879.73 mb
  total-ram = 15925.6 mb
  free-ram  = 81.4961 mb
  buffer-ram= 375.324 mb
  total-swap= 20481.2 mb
  free-swap = 20480.9 mb
  addr  = 0x40

event counter: 186
eventid: [run,evt,lumi,time,bunchcross,detmask] = [177986,24411478,393,1300734321:461051955,3145,0x1fffffffffff7]
last incident: athenaeventloopmgr:beginevent
current algorithm: muidcombined
algorithm stack: 
   athmasterseq
   athalgseq
   muidcombined
-------------------------------------------------------------------------------------

(pid=11982 ppid=11897) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 1 (address not mapped to object)
  pid    = 64, uid = 0
  value  = (1037709930, 0x3dda326a)
  addr   = 0x40
  stack  = (2, 0, (nil))

  eip: 0023:732f79d6           eflags: 00210216
  eax: 7330a630   ebx: 7330a9d8   ecx: 00000000   edx: 8d53eb40
  esi: 00000000   edi: 8d53eb40   ebp: ffae6f48   esp: ffae6f14
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ffae6f14  trap: 14/4  oldmask: 00000000   cr2: 00000040

  fpu:  control = ffff037f
        status  = ffff0132
        tag     = ffffffff
        ip      = 0010:61622357
        data    = 002b:00000000
        state   = 00000132
    %fp0 = [0000:0000000000000000]
    %fp1 = [0000:0000000000000000]
    %fp2 = [0000:0000000000000000]
    %fp3 = [0000:0000000000000000]
    %fp4 = [4011:ac2c783ab771e99b]
    %fp5 = [c00a:d00022c6b72693b8]
    %fp6 = [400b:000000000000d3d8]
    %fp7 = [400b:e000d17af4f7d56d]

stack trace:
 0x5cd0a440 athena::debugaids::stacktrace(int) + 0x60 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5cd05f9f athena::signal::fataldump(int, siginfo*, void*) + 0xff [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5cd062b1 athena::signal::fatal(int, siginfo*, void*) + 0x181 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5e55764e coredumpsvchandler::action(int, siginfo*, void*) + 0xce [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0x732f79d6 trk::fitparameters::fitparameters(trk::perigee const&amp;) + 0x26 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libtrkipatfitterutils.so]
 0x732ad692 trk::ipatfitter::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const + 0xf2 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libtrkipatfitter.so]
 0x73116fba rec::combinedmuontrackbuilder::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const + 0x26a [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x73114366 rec::combinedmuontrackbuilder::combinedfit(trk::track const&amp;, trk::track const&amp;, trk::track const&amp;) const + 0x846 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x7311e25d rec::combinedmuontrackbuilder::indetextension(trk::track const&amp;, std::vector&lt;trk::measurementbase const*, std::allocator&lt;trk::measurementbase const*&gt; &gt; const&amp;, trk::parameterst&lt;trk::charged&gt; const*, trk::parameterst&lt;trk::charged&gt; const*, trk::parameterst&lt;trk::charged&gt; const*) const + 0x86d [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0x7457ec3d rec::muidmatchmaker::indetextension(trk::track const&amp;, trk::track const&amp;, trk::track const*, trk::track const*, trk::track const*) const + 0x3cd [/afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0x7458a1a2 rec::muidmatchmaker::execute(datavector&lt;trk::track, datavector_detail::nobase&gt; const*, datavector&lt;trk::track, datavector_detail::nobase&gt; const*, datavector&lt;trk::track, datavector_detail::nobase&gt; const*) const + 0x8e2 [/afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libmuidcombinedtools.so]
 0x678f904a muidcombined::execute() + 0x8da [/afs/cern.ch/atlas/software/releases/16.6.2/atlasreconstruction/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libmuidcombined.so]
 0x588a751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x644f6ba3 athsequencer::execute() + 0x1f3 [/afs/cern.ch/atlas/software/releases/16.6.2/atlascore/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x588a751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x644f6ba3 athsequencer::execute() + 0x1f3 [/afs/cern.ch/atlas/software/releases/16.6.2/atlascore/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x588a751d algorithm::sysexecute() + 0x11d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x5e54065f athenaeventloopmgr::executealgorithms() + 0x6f [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e546e4e athenaeventloopmgr::executeevent(void*) + 0x83e [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e545b32 athenaeventloopmgr::nextevent(int) + 0x8a2 [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5e53f7eb athenaeventloopmgr::executerun(int) + 0x1b [/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5ddb7a49 applicationmgr::executerun(int) + 0x79 [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
ï¿½0x5d54794d _zn12_global__n_1l11method_4267epvs0_rkst6vectoris0_sais0_ees0_ dictionary_dict.cpp:0 + 0x5d54794d [/afs/cern.ch/atlas/software/releases/16.6.2/gaudi/v21r9p3a/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0x578ca437 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int) + 0x157 [/afs/cern.ch/atlas/software/releases/16.6.2/detcommon/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0x5e1c5034 &lt;unknown function&gt;
 0x5617d512 cint::g__callfunc::execute(void*) + 0xd2 [/afs/cern.ch/atlas/software/releases/16.6.2/detcommon/16.6.2/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0x5590acec pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*) + 0x2c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x55910c24 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*) + 0x64 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x55910dbc pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*) + 0x6c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x5590fa92 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long) + 0x112 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
ï¿½0x55916859 pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*) methodproxy.cxx:0 + 0x55916859 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/app/releases/root/5.26.00e_python2.6/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 fast_function /build/agaspar/work/python-2.6.5/python/ceval.c:3846 + 0x5a89 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call /build/agaspar/work/python-2.6.5/objects/funcobject.c:524 + 0x8137caa [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call /build/agaspar/work/python-2.6.5/objects/classobject.c:2579 + 0x8064935 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373 + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call /build/agaspar/work/python-2.6.5/objects/funcobject.c:524 + 0x8137caa [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call /build/agaspar/work/python-2.6.5/objects/classobject.c:2579 + 0x8064935 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373 + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call /build/agaspar/work/python-2.6.5/objects/abstract.c:2493 + 0x5c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call /build/agaspar/work/python-2.6.5/python/ceval.c:3968 + 0x2dfd [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797 + 0x80d4900 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call /build/agaspar/work/python-2.6.5/objects/methodobject.c:81 + 0x18d [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function /build/agaspar/work/python-2.6.5/python/ceval.c:3750 + 0x5469 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex /build/agaspar/work/python-2.6.5/python/ceval.c:3000 + 0x87c [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode /build/agaspar/work/python-2.6.5/python/ceval.c:547 + 0x63 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339 + 0xb4 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags /build/agaspar/work/python-2.6.5/python/pythonrun.c:935 + 0x204 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags /build/agaspar/work/python-2.6.5/python/pythonrun.c:739 + 0x7a [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main /build/agaspar/work/python-2.6.5/modules/main.c:572 + 0xc56 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main /build/agaspar/work/python-2.6.5/./modules/python.c:24 + 0x32 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x005c4e9c __libc_start_main + 0xdc [/lib/libc.so.6]
 0x08057d51 _start + 0x8057d51 [/afs/cern.ch/atlas/software/releases/16.6.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info execution of rawtoesd complete with exit code -11.
py:rawtoesd         error transform received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/share/atlas_error_patterns.db
info: read 58 ignore patterns from file /afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: wed mar 23 14:06:32 2011, vmem: 111.4453125 mb, rss: 67.0 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,16.6.2.6 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 30742 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 1047
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: wed mar 23 14:06:37 2011, vmem: 111.4453125 mb, rss: 67.0 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 04 secs, cpu-time: 00 hrs 00 mins 04 secs, vmem: 0.0 mb, rss: 0.0 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: wed mar 23 14:06:37 2011, vmem: 111.4453125 mb, rss: 67.02734375 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 00 hrs 31 mins 17 secs, cpu-time: 00 hrs 29 mins 53 secs, vmem: 0.734375 mb, rss: 0.6015625 mb
===== job report produced by rawtoesd version patjobtransforms-00-03-22 =====
command=/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9954/9954793_data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.3//data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=9
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: wed mar 23 14:06:37 2011, vmem: 111.4453125 mb, rss: 67.03515625 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 00 hrs 31 mins 17 secs, cpu-time: 00 hrs 29 mins 53 secs, vmem: 0.734375 mb, rss: 0.62109375 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/releases/16.6.2/atlasproduction/16.6.2.6/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/9954/9954793_data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.3//data11_7tev.00177986.express_express.merge.raw.f351.recon.task._lb0393._sfo-all._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=9
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=11982 ppid=11897) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=787595287
rundiravailabledisk=122673213440
workdir=/pool/lsf/atlast0/129018627
athcode=8
rundiravailabledisk=122673754112
dbrelease=current
athacronym=an unknown exception occurred

-verbatim-