rtt: 17.x.0-val (rel_4) 32 bit dbg: many jobs with timeouts in antikt6towerjets

investigation still ongoing but this is seen in at least 4 jobs:
mb_pu_mu0_d557
mb_pu_mu0_d557_wtrigtruth
zee_pu_mu0_d557
zee_pu_mu0_d557_wtrigtruth

logfile snippet:
----------------
core dump from coredumpsvc on lxbrb1402.cern.ch at thu aug 18 22:35:43 2011

event counter: 1
eventid: [run,evt,lumi,time,bunchcross,detmask] = [106046,35101,0,0:0,0,0x0]
last incident: athenaeventloopmgr:beginevent
reason: timeout  (3600000 msec) reached
current algorithm: antikt6towerjets
algorithm stack: 
   athmasterseq
   athalgseq
   antikt6towerjets
-------------------------------------------------------------------------------------

(pid=10634 ppid=9394) received fatal signal 6 (aborted)
signal context:
  signo  = 6, errno = 0, code = -6 (*unknown reason*)
  pid    = 10634, uid = 24039
  value  = (-2147361333, 0x8001ddcb)
  stack  = (2, 0, (nil))

  eip: 0023:ffffe410           eflags: 00200296
  eax: 00000000   ebx: 0000298a   ecx: 0000298a   edx: 00000006
  esi: ffc4c44c   edi: 00bc6ff4   ebp: ffc4c3ac   esp: ffc4c394
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ffc4c394  trap: 13/0  oldmask: 00000200   cr2: 00000000

  fpu:  control = ffff037f
        status  = ffff0000
        tag     = ffffffff
        ip      = 0010:00ab3d6e
        data    = 002b:ffc4ae30
        state   = 00000000
    %fp0 = [0000:0000000000000000]
    %fp1 = [0000:0000000000000000]
    %fp2 = [0000:0000000000000000]
    %fp3 = [0000:0000000000000000]
    %fp4 = [0000:0000000000000000]
    %fp5 = [0000:0000000000000000]
    %fp6 = [4013:0000000040008061]
    %fp7 = [4009:0000000000009a00]

stack trace:
 0x5c4e71ba stacktrace ../src/sealdebug.cxx:736   + 0x7a [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libcxxutils.so]
 0x5c4e5f18 fataldump ../src/sealsignal.cxx:1397   + 0x214 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libcxxutils.so]
 0x5c4e616e fatal ../src/sealsignal.cxx:898   + 0x1dc [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libcxxutils.so]
 0xffffe600 &lt;unknown function&gt;
 0xffffe410 &lt;unknown function&gt;
 0x00a9adf0 gsignal  + 0x50 [/lib/libc.so.6]
 0x00a9c701 abort  + 0x101 [/lib/libc.so.6]
 0x5c4af6d0 abortjob ../src/algorithmtimer.cxx:283   + 0x56a [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4b1302 operator() /afs/cern.ch/sw/lcg/external/boost/1.44.0_python2.6/i686-slc5-gcc43-opt/include/boost-1_44/boost/bind/mem_fn_template.hpp:49   + 0x4e [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4b155a operator()&lt;boost::function&lt;void ()()&gt;, boost::_mfi::mf0&lt;boost::function&lt;void ()()&gt;, athena::algorithmtimer&gt;, boost::_bi::list0&gt; /afs/cern.ch/sw/lcg/external/boost/1.44.0_python2.6/i686-slc5-gcc43-opt/include/boost-1_44/boost/bind/bind.hpp:243   + 0x4c [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4b15b3 operator() /afs/cern.ch/sw/lcg/external/boost/1.44.0_python2.6/i686-slc5-gcc43-opt/include/boost-1_44/boost/bind/bind_template.hpp:20   + 0x47 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4b15ec invoke /afs/cern.ch/sw/lcg/external/boost/1.44.0_python2.6/i686-slc5-gcc43-opt/include/boost-1_44/boost/function/function_template.hpp:153   + 0x28 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4b4fff operator() /afs/cern.ch/sw/lcg/external/boost/1.44.0_python2.6/i686-slc5-gcc43-opt/include/boost-1_44/boost/function/function_template.hpp:1013   + 0x87 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0x5c4affe5 onalarmsignal ../src/algorithmtimer.cxx:72   + 0xbe [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenakernel.so]
 0xffffe600 &lt;unknown function&gt;
 0x7ab890d6 virtual thunk to trig::mcbunchcrossingtool::isfilled(unsigned int) const ../src/mcbunchcrossingtool.cxx:226   + 0x0 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlastrigger/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libtrigbunchcrossingtool.so]
 0x7a2146aa initializeevent ../src/jetcalibsequence.cxx:86   + 0xdc [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libjetcalibtoolslib.so]
 0x7a240dd2 execute ../src/jetcalibrationtool.cxx:38   + 0x78 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libjetcalibtoolslib.so]
 0x6cbc82f6 toplevel_execute ../src/jetalgtoolbase.cxx:36   + 0xb8 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libjetreclib.so]
 0x6cbaab56 executetool ../src/jetalgorithm.cxx:242   + 0x22 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libjetreclib.so]
 0x6cbadb39 execute ../src/jetalgorithm.cxx:167   + 0x1e5 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libjetreclib.so]
 0x59ed9588 sysexecute ../src/lib/algorithm.cpp:602   + 0x278 [/afs/cern.ch/atlas/software/builds/nightlies/devval/gaudi/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudikernel.so]
 0x68403ad8 execute ../src/athsequencer.cxx:157   + 0x25c [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudisequencer.so]
 0x59ed9588 sysexecute ../src/lib/algorithm.cpp:602   + 0x278 [/afs/cern.ch/atlas/software/builds/nightlies/devval/gaudi/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudikernel.so]
 0x68403ad8 execute ../src/athsequencer.cxx:157   + 0x25c [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudisequencer.so]
 0x59ed9588 sysexecute ../src/lib/algorithm.cpp:602   + 0x278 [/afs/cern.ch/atlas/software/builds/nightlies/devval/gaudi/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudikernel.so]
 0x5c16746d executealgorithms ../src/athenaeventloopmgr.cxx:595   + 0x8d [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenaservices.so]
 0x5c1697b1 executeevent ../src/athenaeventloopmgr.cxx:682   + 0x7f3 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenaservices.so]
 0x5c16b44a nextevent ../src/athenaeventloopmgr.cxx:907   + 0xf58 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenaservices.so]
 0x5c167990 executerun ../src/athenaeventloopmgr.cxx:753   + 0x30 [/afs/cern.ch/atlas/software/builds/nightlies/devval/atlascore/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libathenaservices.so]
 0x5b5f1245 executerun ../src/applicationmgr/applicationmgr.cpp:866   + 0xab [/afs/cern.ch/atlas/software/builds/nightlies/devval/gaudi/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudisvc.so]
 0x5a8c3bc3 method_4307 ../i686-slc5-gcc43-dbg/dict/gaudikernel/dictionary_dict.cpp:5751   + 0x5a8c3bc3 [/afs/cern.ch/atlas/software/builds/nightlies/devval/gaudi/rel_4/installarea/i686-slc5-gcc43-dbg/lib/libgaudikerneldict.so]
 0x58999aba method_stub_with_context /build/bellenot/spi/i686-slc5-gcc43-dbg/root/cint/cintex/src/cintfunctional.cxx:320   + 0x1c9 [/afs/cern.ch/atlas/offline/external/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-dbg/lib/libcintex.so]
 0x5bbd1034 &lt;unknown function&gt;
 0x5623ff49 execute cint/cint/src/callfunc.cxx:440   + 0x18b [/afs/cern.ch/atlas/offline/external/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-dbg/lib/libcint.so]
 0x558ef0b7 execint include/callfunc.h:98   + 0x2b [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558ebf52 execute /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/executors.cxx:229   + 0x24 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558f5cc6 callfast /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/methodholder.cxx:94   + 0x4c [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558f5e79 callsafe /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/methodholder.cxx:118   + 0xa1 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558facd2 execute /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/methodholder.cxx:627   + 0x104 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558f72bf operator() /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/methodholder.cxx:690   + 0x23f [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x558fe100 mp_call /build/bellenot/spi/i686-slc5-gcc43-dbg/root/bindings/pyroot/src/methodproxy.cxx:476   + 0x558fe100 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-dbg/root/lib/libpyroot.so]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call inlined at python/ceval.c:3773 in pyeval_evalframeex python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 fast_function inlined at python/ceval.c:3771 in pyeval_evalframeex python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode python/ceval.c:547   + 0x63 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod inlined at python/pythonrun.c:1325 in pyrun_fileexflags python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call objects/methodobject.c:81   + 0x18d [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function inlined at python/ceval.c:2412 in pyeval_evalframeex python/ceval.c:3750   + 0x5469 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call inlined at python/ceval.c:3773 in pyeval_evalframeex python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode python/ceval.c:547   + 0x63 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod inlined at python/pythonrun.c:1325 in pyrun_fileexflags python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call objects/methodobject.c:81   + 0x18d [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 call_function inlined at python/ceval.c:2412 in pyeval_evalframeex python/ceval.c:3750   + 0x5469 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d do_call inlined at python/ceval.c:3773 in pyeval_evalframeex python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode python/ceval.c:547   + 0x63 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 run_mod inlined at python/pythonrun.c:1325 in pyrun_fileexflags python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags python/pythonrun.c:935   + 0x204 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags python/pythonrun.c:739   + 0x7a [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main modules/main.c:572   + 0xc56 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main ./modules/python.c:24   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x00a87e9c __libc_start_main  + 0xdc [/lib/libc.so.6]
 0x08057d51 _start ??:0   + 0x8057d51 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -6.
py:rawtoesd         error athena received signal 6. exit code reset to athena exit code 8.

command:
--------
/afs/cern.ch/atlas/software/builds/nightlies/devval/atlasreconstruction/rel_4/installarea/share/bin/reco_trf.py --uploadtoami=1.0 ami=f380 autoconfiguration=everything inputrdofile=root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.106046.pythiazee_no_filter.digit.rdo.e603_s932_d557_tid364679_00/rdo.364679._000001.pool.root.1,root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.106046.pythiazee_no_filter.digit.rdo.e603_s932_d557_tid364679_00/rdo.364679._000002.pool.root.1,root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.106046.pythiazee_no_filter.digit.rdo.e603_s932_d557_tid364679_00/rdo.364679._000003.pool.root.1,root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.106046.pythiazee_no_filter.digit.rdo.e603_s932_d557_tid364679_00/rdo.364679._000004.pool.root.1,root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/phys-rig/pileupsamples/valid1.106046.pythiazee_no_filter.digit.rdo.e603_s932_d557_tid364679_00/rdo.364679._000005.pool.root.1 append_preexec=rec.dotrigger=false;rec.dotruth.set_value_and_lock(false) outputaodfile=myaod.pool.root outputesdfile=myesd.pool.root outputhistfile=myhist.root


logfile:
--------
https://atlasrtt.cern.ch/rtt/rel_4/devval/build/i686-slc5-gcc43-dbg/offline/recjobtransformtests/zee_pu_mu0_d557/zee_pu_mu0_d557_log

please consult:
---------------
https://twiki.cern.ch/twiki/bin/viewauth/atlas/recbuginvestigation

camille (valshift)