rtt:17.0.x (rel_2), out of memory error for athenaapp_all_rdo

logfile snippet:
----------------
toolsvc.dqtbackgroundmon                          warning   could not retrieve lucid_rawdata, lucid hit histograms will not be filled for this event!
athenasummarysvc     fatal out of memory: saving summary ...
toolsvc.hltcalo                                     error unknown exception caught, while filling histograms
toolsvc.hltcalofeb                                  error unknown exception caught, while filling histograms
storegatesvc                                      warning retrieve(const): no valid proxy for object tiledigitscnt  of type tiledigitscontainer(clid 2925)
toolsvc.tilembtsmon                               warning can't find tiledigitscontainer with name tiledigitscnt in tds
storegatesvc                                      warning retrieve(const): no valid proxy for object free  of type lardigitcontainer(clid 2711)
toolsvc.lardigitmon                               warning can't retrieve lardigitcontainer with key free
storegatesvc                                      warning retrieve(default): no valid proxy for default object 
 of type rpcsectorlogiccontainer(clid 1298668563)
toolsvc.rpclv1rawdatasectorlogic                     info cannot retrieve the rpcsectorlogiccontainer
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = allcalo
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):2802 allcalo
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = truthevent
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):133273 truthevent
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = trigindettracktruthmap
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):78130186 trigindettracktruthmap
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = hltresult_ef
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):107385089 hltresult_ef
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = hltresult_l2
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):107385089 hltresult_l2
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_bjet_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_bjet_efid
athenapoolconverter                                 error failed to create persistent object: t/p converter: 22siclusterontrackcnv_p1 called recursively! -&gt; enable the recursive flag
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_bphysics_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_bphysics_efid
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_electron_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_electron_efid
athenapoolconverter                                 error failed to create persistent object: t/p converter: 28trt_driftcircleontrackcnv_p1 called recursively! -&gt; enable the recursive flag
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_fullscan_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_fullscan_efid
athenapoolconverter                                 error failed to create persistent object: t/p converter: 28trt_driftcircleontrackcnv_p1 called recursively! -&gt; enable the recursive flag
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_muon_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_muon_efid
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = hlt_indettrigtrackslimmer_tau_efid
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 hlt_indettrigtrackslimmer_tau_efid
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = mooretracks
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 mooretracks
athenapoolconverter                                 error failed to create persistent object: t/p converter: 28trt_driftcircleontrackcnv_p1 called recursively! -&gt; enable the recursive flag
athenapoolconverter                                 error createrep failed, key = mugirlrefittedtracks
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 mugirlrefittedtracks
athenapoolconverter                                 error failed to create persistent object: t/p converter: 28trt_driftcircleontrackcnv_p1 called recursively! -&gt; enable the recursive flag
athenapoolconverter                                 error createrep failed, key = mutagimotracks
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 mutagimotracks
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = muidextrapolatedtracks
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 muidextrapolatedtracks
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = tracks
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117682415 tracks
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = antikt4lctopojetsmomentmap
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117927529 antikt4lctopojetsmomentmap
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = antikt4topoemjetsmomentmap
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117927529 antikt4topoemjetsmomentmap
athenapoolconverter                                 error failed to create persistent object: std::bad_alloc
athenapoolconverter                                 error createrep failed, key = antikt4towerjetsmomentmap
streamesd.streamesdtool                             error could not create rep for dataobject (clid/key):117927529 antikt4towerjetsmomentmap
streamesd                                           fatal 
streamesd                                           error 
athalgseq                                           fatal 
athalgseq                                           error 
athmasterseq                                        fatal 
athmasterseq                                        error 
could not open /dev/null!
could not open /dev/null!
traceback (most recent call last):
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/share/bin/athena-app.py&quot;, line 143, in &lt;module&gt;
    retcode=main(options)
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/share/bin/athena-app.py&quot;, line 47, in main
    if app.exitstate != exitcodes.all_ok:
memoryerror
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

(pid=5745 ppid=3727) received fatal signal 6 (aborted)
signal context:
  signo  = 6, errno = 0, code = -6 (*unknown reason*)
  pid    = 5745, uid = 24039
  value  = (0, 0x200000000)
  stack  = (2, 0, (nil))

  80000000 00000000 01010101 01010101 08000000 00000000 06020000 00000000
  9025c505 ab2a0000 01000000 00000000 00000000 00000000 507b4010 132b0000
  71160000 00000000 71160000 00000000 80803110 132b0000 b078007a ff7f0000
  06000000 00000000 00000000 00000000 ffffffff ffffffff 0878007a ff7f0000
  6502a3e1 3e000000 06020000 00000000 33000000 00000000 00000000 00000000
  0d000000 00000000 00000000 00000000 00000000 00000000 8075007a ff7f0000
  233f9e16 132b0000 e01e9e16 132b0000 f8ffffff ffffffff 01000000 00000000
  10000000 00000000 1b1b0100 00000000 00000000 00000000 f878007a ff7f0000

stack trace:
 0x2aaaadc9fcbc athena::debugaids::stacktrace(int)+0x4c ??:0   + 0x4c [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x2aaaadc9dec2 athena::signal::fataldump(int, siginfo*, void*)+0x152 ??:0   + 0x152 [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x2aaaadc9e1d4 athena::signal::fatal(int, siginfo*, void*)+0x194 ??:0   + 0x194 [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x3ee260eb10 __restore_rt ??:0   + 0x3ee260eb10 [/lib64/libpthread.so.0]
 0x3ee1a30265 gsignal+0x35 ??:0   + 0x35 [/lib64/libc.so.6]
 0x3ee1a31d10 abort+0x110 ??:0   + 0x110 [/lib64/libc.so.6]
 0x2b13169e4024 ()+0x2b13169e4024 ??:0   + 0x114 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2b13169e2466 ()+0x2b13169e2466 ??:0   + 0x2b13169e2466 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2b13169e2493 ()+0x2b13169e2493 ??:0   + 0x2b13169e2493 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2b13169e257a ()+0x2b13169e257a ??:0   + 0x2b13169e257a [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2aaaada769a1 athenasummarysvc::newhandler()+0x301 ??:0   + 0x301 [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/x86_64-slc5-gcc43-opt/lib/libathenaservices.so]
 0x2b13169e298a ()+0x2b13169e298a ??:0   + 0x3a [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2b1312a7af0d _zl23g__setup_memfunctpythonv+0x3de ??:0   + 0x2ad [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x2b1312a7b2fd _zl23g__setup_memfunctpythonv+0x7ce ??:0   + 0x1d [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x2b1312a75c54 pyroot::utility::mapoperatorname(std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, bool)+0x394 ??:0   + 0x44 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x2b1312a7ca7e _zl44g__setup_memfuncpyrootclcltpyrootapplicationv+0x30e ??:0   + 0x2b1312a7ca7e [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00496742 pyeval_evalframeex+0x2c32 /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2c32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f2244 function_call+0x1b4 /build/agaspar/work/python-2.6.5/objects/funcobject.c:532   + 0x4f2244 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049590a pyeval_evalframeex+0x1dfa /build/agaspar/work/python-2.6.5/python/ceval.c:4063   + 0x1dfa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f213d function_call+0xad /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x4f213d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00492df6 pyeval_callobjectwithkeywords+0x56 /build/agaspar/work/python-2.6.5/python/ceval.c:3619   + 0x56 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004b9007 py_finalize+0xc7 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1718   + 0xc7 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00414a20 py_main+0x4f0 /build/agaspar/work/python-2.6.5/modules/main.c:596   + 0x4f0 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x3ee1a1d994 __libc_start_main+0xf4 ??:0   + 0xf4 [/lib64/libc.so.6]
 0x004141d9 _start+0x29 ??:0   + 0x4141d9 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -6.
py:rawtoesd         error athena received signal 6. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/share/atlas_error_patterns.db
info: read 61 ignore patterns from file /afs/cern.ch/atlas/software/builds/nightlies/17.0.x/atlascore/rel_2/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: tue aug 16 11:54:43 2011 cest, vmem: 286.94140625 mb, rss: 95.37109375 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,rel_2 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 56020 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 255
py:rawtoesd          info    errors  : 59
py:rawtoesd          info    fatals  : 1

logfile:
--------
https://atlasrtt.cern.ch/rtt/rel_2/17.0.x/build/x86_64-slc5-gcc43-opt/offline/recperftests/athenaapp_all_rdo/athenaapp_all_rdo_log

please consult:
---------------
https://twiki.cern.ch/twiki/bin/viewauth/atlas/recbuginvestigation

camille, valshifter