toolsvc.hltcalo; error unknown exception caught, while filling histograms; for 17.0.x.y (x86_64-slc5-gcc43-opt)

cheers michael 
(valshifter)

coral/services/connectionservice info user session with sessionid=b96bddce-d837-11e0-97ff-0030487c8db0 will be ended on connectionid=b96bd87e-d837-11e0-97ff-0030487c8db0
coral/services/connectionservice info connection to service &quot;atlf/()&quot; with connectionid=b96bd87e-d837-11e0-97ff-0030487c8db0 will be disconnected
caught standard c++ exception: std::bad_alloc from initialize.
toolsvc.hltcalo                                     error unknown exception caught, while filling histograms
toolsvc.hltcalofeb                                  error unknown exception caught, while filling histograms
algerrorauditor                                     error illegal return code: algorithm hltmonmanager reported an error, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
trtelectronmonmanager                               fatal  standard std::exception is caught 
trtelectronmonmanager                               error std::bad_alloc
algerrorauditor                                     error illegal return code: algorithm trtelectronmonmanager reported an error, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
algerrorauditor                                     error illegal return code: algorithm trtelectronmonmanager reported a fatal, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
athalgseq                                           fatal  standard std::exception is caught 
athalgseq                                           error std::exception
algerrorauditor                                     error illegal return code: algorithm athalgseq reported an error, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
algerrorauditor                                     error illegal return code: algorithm athalgseq reported a fatal, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
athmasterseq                                        fatal  standard std::exception is caught 
athmasterseq                                        error std::exception
algerrorauditor                                     error illegal return code: algorithm athmasterseq reported an error, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
algerrorauditor                                     error illegal return code: algorithm athmasterseq reported a fatal, but returned a statuscode &quot;success&quot;
error policy described in https://twiki.cern.ch/twiki/bin/view/atlas/reportingerrors
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
traceback (most recent call last):
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y/atlasproduction/rel_2/installarea/joboptions/athenacommon/runbatch.py&quot;, line 11, in &lt;module&gt;
    theapp.run()     # runs until theapp.evtmax events reached
  file &quot;/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y/atlasproduction/rel_2/installarea/python/athenacommon/appmgr.py&quot;, line 623, in run
    sc = self.gethandle()._evtpro.executerun( nevt )
runtimeerror: (file &quot;&quot;, line 0) std::exception (c++ exception)

(pid=27885 ppid=24514) received fatal signal 6 (aborted)
signal context:
  signo  = 6, errno = 0, code = -6 (*unknown reason*)
  pid    = 27885, uid = 24039
  value  = (0, 0x200000000)
  stack  = (2, 0, (nil))

  ffffffff 00000000 01010101 01010101 08000000 00000000 06020000 00000000
  80b0c51c 00000000 00000000 00000000 50cf9f1e ab2a0000 c0f75577 e72a0000
  ed6c0000 00000000 ed6c0000 00000000 80305773 e72a0000 80a6a26e ff7f0000
  06000000 00000000 00000000 00000000 ffffffff ffffffff d8a5a26e ff7f0000
  6502e33c 38000000 06020000 00000000 33000000 00000000 00000000 00000000
  0d000000 00000000 00000000 00000000 00000000 00000000 50a3a26e ff7f0000
  230f977a e72a0000 e0ee967a e72a0000 f8ffffff ffffffff 01000000 00000000
  10000000 00000000 1b1b0100 00000000 00000000 00000000 c8a6a26e ff7f0000

stack trace:
 0x2ae77eb09cbc athena::debugaids::stacktrace(int)+0x4c ??:0   + 0x4c [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x2ae77eb13732 athena::signal::fataldump(int, siginfo*, void*)+0x152 ??:0   + 0x152 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x2ae77eb13a44 athena::signal::fatal(int, siginfo*, void*)+0x194 ??:0   + 0x194 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/x86_64-slc5-gcc43-opt/lib/libcxxutils.so]
 0x383da0eb70 __restore_rt ??:0   + 0x383da0eb70 [/lib64/libpthread.so.0]
 0x383ce30265 gsignal+0x35 ??:0   + 0x35 [/lib64/libc.so.6]
 0x383ce31d10 abort+0x110 ??:0   + 0x110 [/lib64/libc.so.6]
 0x2ae77a971024 ()+0x2ae77a971024 ??:0   + 0x114 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2ae77a96f466 ()+0x2ae77a96f466 ??:0   + 0x2ae77a96f466 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2ae77a96f493 ()+0x2ae77a96f493 ??:0   + 0x2ae77a96f493 [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2ae77a96f57a ()+0x2ae77a96f57a ??:0   + 0x2ae77a96f57a [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2aaaadd44c41 athenasummarysvc::newhandler()+0x301 ??:0   + 0x301 [/afs/cern.ch/atlas/software/releases/17.0.3/atlascore/17.0.3/installarea/x86_64-slc5-gcc43-opt/lib/libathenaservices.so]
 0x2ae77a96f98a ()+0x2ae77a96f98a ??:0   + 0x3a [/afs/cern.ch/sw/lcg/external/gcc/4.3.5/x86_64-slc5/lib64/libstdc++.so.6]
 0x2ae776e8421c g__cpp_setup_tagtableg__base1+0x92c ??:0   + 0x14c [/afs/cern.ch/atlas/software/releases/17.0.3/lcgcmt/lcgcmt_60c/installarea/x86_64-slc5-gcc43-opt/lib/libcore.so]
 0x2ae776e87e79 _zl26g__setup_memfunctvirtualpsv+0xcb ??:0   + 0x119 [/afs/cern.ch/atlas/software/releases/17.0.3/lcgcmt/lcgcmt_60c/installarea/x86_64-slc5-gcc43-opt/lib/libcore.so]
 0x2ae7769f30b2 ()+0x2ae7769f30b2 ??:0   + 0x12 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x2ae776a203d6 ()+0x2ae776a203d6 ??:0   + 0x316 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x2ae776a1b3ca ()+0x2ae776a1b3ca ??:0   + 0x5a [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00e/x86_64-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0049888c pyeval_evalframeex+0x4d7c /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x4d7c [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00499097 pyeval_evalframeex+0x5587 /build/agaspar/work/python-2.6.5/python/ceval.c:3836   + 0x5587 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f213d function_call+0xad /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x4f213d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00420b00 instancemethod_call+0x130 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x420b00 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0046deb5 slot_tp_init+0x85 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5633   + 0x46deb5 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0046418e type_call+0x7e /build/agaspar/work/python-2.6.5/objects/typeobject.c:740   + 0x46418e [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00496742 pyeval_evalframeex+0x2c32 /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2c32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f213d function_call+0xad /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x4f213d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049590a pyeval_evalframeex+0x1dfa /build/agaspar/work/python-2.6.5/python/ceval.c:4063   + 0x1dfa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004987e8 pyeval_evalframeex+0x4cd8 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x4cd8 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004987e8 pyeval_evalframeex+0x4cd8 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x4cd8 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a3a2 pyeval_evalcode+0x32 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004ba9aa pyrun_fileexflags+0xaa /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xaa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00491d42 builtin_execfile+0x1d2 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x491d42 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049888c pyeval_evalframeex+0x4d7c /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x4d7c [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f213d function_call+0xad /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x4f213d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00420b00 instancemethod_call+0x130 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x420b00 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0046e591 slot_tp_call+0x81 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x46e591 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00496742 pyeval_evalframeex+0x2c32 /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2c32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a3a2 pyeval_evalcode+0x32 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004ba9aa pyrun_fileexflags+0xaa /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xaa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00491d42 builtin_execfile+0x1d2 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x491d42 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049888c pyeval_evalframeex+0x4d7c /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x4d7c [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004f213d function_call+0xad /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x4f213d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00420b00 instancemethod_call+0x130 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x420b00 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0046e591 slot_tp_call+0x81 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x46e591 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004197b8 pyobject_call+0x68 /build/agaspar/work/python-2.6.5/objects/abstract.c:2492   + 0x68 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00496742 pyeval_evalframeex+0x2c32 /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2c32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a3a2 pyeval_evalcode+0x32 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004ba9aa pyrun_fileexflags+0xaa /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xaa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x00491d42 builtin_execfile+0x1d2 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x491d42 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049888c pyeval_evalframeex+0x4d7c /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x4d7c [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a2a7 pyeval_evalcodeex+0x927 /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x927 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x0049a3a2 pyeval_evalcode+0x32 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004ba9aa pyrun_fileexflags+0xaa /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xaa [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004bac9d pyrun_simplefileexflags+0x20d /build/agaspar/work/python-2.6.5/python/pythonrun.c:935   + 0x20d [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x004150a3 py_main+0xb73 /build/agaspar/work/python-2.6.5/modules/main.c:572   + 0xb73 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
 0x383ce1d994 __libc_start_main+0xf4 ??:0   + 0xf4 [/lib64/libc.so.6]
 0x004141d9 _start+0x29 ??:0   + 0x4141d9 [/afs/cern.ch/sw/lcg/external/python/2.6.5/x86_64-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -6.
py:rawtoesd         error athena received signal 6. exit code reset to athena exit code 8.