hi, 

one job was aborted by segmentation fault in metrefalg. 
i will reproduce the problem. 

cheers, 
susumu 

+verbatim+
input file(s):
--------------------------------------------------------------------------------
stage_svcclass=atlcal
/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_7tev/physics_egamma/00189561/data11_7tev.00189561.physics_egamma.merge.raw/data11_7tev.00189561.physics_egamma.merge.raw._lb0327._sfo-4._0001.1


tier-0 transformation command run:
--------------------------------------------------------------------------------
source /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/setuptrf.sh /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.3.6 atlasproduction-17.0.3.6 /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/specialsetup_tier0.sh ; python -u `which reco_trf.py` --argdict=/afs/cern.ch/atlas/tzero/prod1/log2/13681/13681292_data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.3//data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.argdict.gpickle  &amp;&gt; data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.tmp.log


ami configuration tag (more details):
--------------------------------------------------------------------------------
see http://ami.in2p3.fr:8080/opencms/opencms/ami/www/referencetables/ --&gt; configurationtag --&gt; interpret --&gt; f405


logfile location:
--------------------------------------------------------------------------------
stage_svcclass=t0merge
/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_7tev/physics_egamma/00189561/data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task.tmp.log/data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.tmp.log.3


tail of the log file (300 lines):
--------------------------------------------------------------------------------
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63011207, run #189561 311 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012115, run #189561 311 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012115, run #189561 312 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63011722, run #189561 312 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63011722, run #189561 313 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63011538, run #189561 313 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63011538, run #189561 314 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012008, run #189561 314 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012008, run #189561 315 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63013206, run #189561 315 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63013206, run #189561 316 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012426, run #189561 316 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012426, run #189561 317 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012025, run #189561 317 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012025, run #189561 318 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012409, run #189561 318 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012409, run #189561 319 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012887, run #189561 319 events processed so far  &lt;&lt;&lt;===
perfmonsvc:    info [evt: 319] cpu=31044.000 ms vmem=2192.328 mb dvmem=  23.398 mb alloc= +0 -0    0.000 kb  =&gt; [/evt/perfmonslice]
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012887, run #189561 320 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63013945, run #189561 320 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63013945, run #189561 321 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012800, run #189561 321 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012800, run #189561 322 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012082, run #189561 322 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012082, run #189561 323 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63013799, run #189561 323 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63013799, run #189561 324 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014300, run #189561 324 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014300, run #189561 325 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014717, run #189561 325 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014717, run #189561 326 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014117, run #189561 326 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014117, run #189561 327 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014508, run #189561 327 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014508, run #189561 328 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63013345, run #189561 328 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63013345, run #189561 329 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014312, run #189561 329 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014312, run #189561 330 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63013170, run #189561 330 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63013170, run #189561 331 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012962, run #189561 331 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012962, run #189561 332 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014646, run #189561 332 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014646, run #189561 333 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63012960, run #189561 333 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63012960, run #189561 334 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014718, run #189561 334 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014718, run #189561 335 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014849, run #189561 335 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014849, run #189561 336 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016132, run #189561 336 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63016132, run #189561 337 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63015162, run #189561 337 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63015162, run #189561 338 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63015677, run #189561 338 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63015677, run #189561 339 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63015129, run #189561 339 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63015129, run #189561 340 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016410, run #189561 340 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63016410, run #189561 341 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63015289, run #189561 341 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63015289, run #189561 342 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63014628, run #189561 342 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63014628, run #189561 343 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016482, run #189561 343 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63016482, run #189561 344 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016157, run #189561 344 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63016157, run #189561 345 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63015687, run #189561 345 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63015687, run #189561 346 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016612, run #189561 346 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #63016612, run #189561 347 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #63016933, run #189561 347 events processed so far  &lt;&lt;&lt;===
-------------------------------------------------------------------------------------
core dump from coredumpsvc on lxbst2703.cern.ch at thu sep 22 16:00:31 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 1 (address not mapped to object)
  pid   = 64, uid = 1061514144
  value = (-1982750381, 0x89d1a153)
  vmem = 2192.33 mb
  rss  = 1817.1 mb
  total-ram = 32049.5 mb
  free-ram  = 2835.29 mb
  buffer-ram= 482.801 mb
  total-swap= 36867.8 mb
  free-swap = 36705.1 mb
  addr  = 0x40

event counter: 348
eventid: [run,evt,lumi,time,bunchcross,detmask] = [189561,63016933,327,1316522742:272214420,1067,0xfffffffffff7]
last incident: athenaeventloopmgr:beginevent
current algorithm: metrefalg
algorithm stack: 
   athmasterseq
   athalgseq
   metrefalg
-------------------------------------------------------------------------------------

(pid=9776 ppid=9672) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 1 (address not mapped to object)
  pid    = 64, uid = 1061514144
  value  = (-1982750381, 0x89d1a153)
  addr   = 0x40
  stack  = (2, 0, (nil))

  eip: 0023:e3531721           eflags: 00210282
  eax: 4ac09750   ebx: e36168d8   ecx: 0e9f7000   edx: 00000000
  esi: 00000000   edi: 5909d378   ebp: ffb818d8   esp: ffb817f8
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ffb817f8  trap: 14/4  oldmask: 00000000   cr2: 00000040

  fpu:  control = ffff037f
        status  = ffff0020
        tag     = ffffffff
        ip      = 0010:dc843266
        data    = 002b:16dea7d0
        state   = 00000020
    %fp0 = [3ff9:40005e35c05bd8c0]
    %fp1 = [0000:0000000000000000]
    %fp2 = [3ffe:c8003e540a40d970]
    %fp3 = [3fff:0000000000008000]
    %fp4 = [4009:f0007791f040cbcf]
    %fp5 = [400a:9000c64925ccd192]
    %fp6 = [400a:9000c64925ccd192]
    %fp7 = [400a:9000c64925ccd192]

stack trace:
 0xf41747a0 athena::debugaids::stacktrace(int)+0x60 ??:0   + 0x60 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf41757bf athena::signal::fataldump(int, siginfo*, void*)+0xff ??:0   + 0xff [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf4175ad1 athena::signal::fatal(int, siginfo*, void*)+0x181 ??:0   + 0x181 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf268b006 coredumpsvchandler::action(int, siginfo*, void*)+0x106 ??:0   + 0x106 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0xe3531721 mettrackselectortool::select(rec::trackparticle const*, caloclustercontainer const*)+0xe1 ??:0   + 0xe1 [/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/i686-slc5-gcc43-opt/lib/libmissingetlib.so]
 0xe350279a metrefinedclustertool::executeco(caloclustercontainer const*, missinget*, missingetcomposition*)+0x60a ??:0   + 0x60a [/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/i686-slc5-gcc43-opt/lib/libmissingetlib.so]
 0xe3505b59 metrefinedclustertool::execute(missingetcomposition*)+0x259 ??:0   + 0x259 [/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/i686-slc5-gcc43-opt/lib/libmissingetlib.so]
 0xe34c1170 metrefinedtoolbase::exectool(missingetcomposition*)+0x60 ??:0   + 0x60 [/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/i686-slc5-gcc43-opt/lib/libmissingetlib.so]
 0xe355b9d2 metrefinedalgorithm::execute()+0x4d2 ??:0   + 0x4d2 [/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/i686-slc5-gcc43-opt/lib/libmissingetlib.so]
 0xf51aaabd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.3/gaudi/v22r1p4/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe79206fb athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf51aaabd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.3/gaudi/v22r1p4/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe79206fb athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf51aaabd algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.3/gaudi/v22r1p4/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xf274f90f athenaeventloopmgr::executealgorithms()+0x6f ??:0   + 0x6f [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf2755e66 athenaeventloopmgr::executeevent(void*)+0x6f6 ??:0   + 0x6f6 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf2753d52 athenaeventloopmgr::nextevent(int)+0x8d2 ??:0   + 0x8d2 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf2756359 athenaeventloopmgr::executerun(int)+0x29 ??:0   + 0x29 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf2d5b2f9 applicationmgr::executerun(int)+0x79 ??:0   + 0x79 [/afs/cern.ch/atlas/software/releases/17.0.3/gaudi/v22r1p4/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
 0xf363bf2d _zn12_global__n_1l11method_4304epvs0_rkst6vectoris0_sais0_ees0_+0x3d ??:0   + 0xf363bf2d [/afs/cern.ch/atlas/software/releases/17.0.3/gaudi/v22r1p4/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0xf5c29a57 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int)+0x157 ??:0   + 0x157 [/afs/cern.ch/atlas/software/releases/17.0.3/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0xf2ad8034 &lt;unknown function&gt;
 0xf6ed8572 cint::g__callfunc::execute(void*)+0xd2 ??:0   + 0xd2 [/afs/cern.ch/atlas/software/releases/17.0.3/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0xf7bb475c pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*)+0x2c ??:0   + 0x2c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bbb134 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*)+0x64 ??:0   + 0x64 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bbb2cc pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*)+0x6c ??:0   + 0x6c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bb97f3 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long)+0x153 ??:0   + 0x153 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7bc0b2a pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*)+0x4aa ??:0   + 0xf7bc0b2a [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 pyeval_evalframeex+0x5a89 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags+0x204 /build/agaspar/work/python-2.6.5/python/pythonrun.c:935   + 0x204 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags+0x7a /build/agaspar/work/python-2.6.5/python/pythonrun.c:739   + 0x7a [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main+0xc56 /build/agaspar/work/python-2.6.5/modules/main.c:572   + 0xc56 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main+0x32 /build/agaspar/work/python-2.6.5/./modules/python.c:24   + 0x32 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x00383e9c __libc_start_main+0xdc ??:0   + 0xdc [/lib/libc.so.6]
 0x08057d51 _start+0x21 ??:0   + 0x8057d51 [/afs/cern.ch/atlas/software/releases/17.0.3/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -11.
py:rawtoesd         error athena received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.3.6/installarea/share/atlas_error_categories.db
info: read 53 error patterns from file /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.3.6/installarea/share/atlas_error_patterns.db
info: read 61 ignore patterns from file /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.3.6/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: thu sep 22 16:00:42 2011 cest, vmem: 101.81640625 mb, rss: 68.37109375 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,17.0.3.6 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 19182 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 318
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: thu sep 22 16:00:43 2011 cest, vmem: 101.81640625 mb, rss: 68.38671875 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 01 secs, cpu-time: 00 hrs 00 mins 01 secs, vmem: 0.0 mb, rss: 0.015625 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: thu sep 22 16:00:43 2011 cest, vmem: 101.81640625 mb, rss: 68.40625 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 01 hrs 06 mins 16 secs, cpu-time: 01 hrs 04 mins 51 secs, vmem: 0.44921875 mb, rss: 0.546875 mb
===== job report produced by rawtoesd version unknown =====
command=/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log2/13681/13681292_data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.3//data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=8
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: thu sep 22 16:00:43 2011 cest, vmem: 101.96875 mb, rss: 68.58203125 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 01 hrs 06 mins 16 secs, cpu-time: 01 hrs 04 mins 51 secs, vmem: 0.6015625 mb, rss: 0.77734375 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/releases/17.0.3/atlasproduction/17.0.3.6/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log2/13681/13681292_data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.3//data11_7tev.00189561.physics_egamma.merge.raw.f405.recon.task._lb0327._sfo-4._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=8
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=9776 ppid=9672) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=1282457563
rundiravailabledisk=1101254995968
workdir=/pool/lsf/atlast0/171700176
athcode=8
rundiravailabledisk=1101255323648
dbrelease=current
athacronym=an unknown exception occurred
-verbatim-