hi,

athenaapp_all_bs encountered a seg fault in 17.0.x.y-val rel_3.

instructions for investigating reco bugs: https://twiki.cern.ch/twiki/bin/viewauth/atlas/recbuginvestigation

original job transform command:
/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y-val/atlasproduction/rel_3/installarea/share/bin/rawtoesd_trf.py --useathenaapp=true inputbsfile=/afs/cern.ch/atlas/project/pmb/data/data11_7tev.00182726.physics_jettauetmiss.merge.raw._lb0215._sfo-5._0001.1_1000evts maxevents=250 autoconfiguration=everything --athenaopts=--stdcmalloc preexec=from@perfmoncomps.perfmonflags@import@jobproperties@as@pmjp,,pmjp.perfmonflags.domallocmonitoring=true,,pmjp.perfmonflags.dopostprocessing=true,,pmjp.perfmonflags.dosemidetailedmonitoringfullprint=true outputdqmonitorfile=mymon.root outputesdfile=myesd.pool.root outputntup_muoncalibfile=myntup_muoncalib.root outputdraw_zmumufile=mydraw_zmumu.data outputdraw_zeefile=mydraw_zee.data 

log file: https://atlasrtt.cern.ch/rtt/rel_3/17.0.x.y-val/build/i686-slc5-gcc43-opt/offline/recperftests/athenaapp_all_bs/athenaapp_all_bs_log

snippet:
--------------------
core dump from coredumpsvc on lxbsu2915.cern.ch at wed nov 16 04:29:30 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 2 (invalid permissions for mapped object)
  pid   = 1661609840, uid = 16777228
  value = (-4750168, 0xffb784a8)
  vmem = 2066.38 mb
  rss  = 1728.03 mb
  total-ram = 32049.5 mb
  free-ram  = 12943.6 mb
  buffer-ram= 485.492 mb
  total-swap= 36867.8 mb
  free-swap = 36836.4 mb
  addr  = 0x630a2770

event counter: 2
eventid: [run,evt,lumi,time,bunchcross,detmask] = [182726,477339,215,1306529256:10782280,660,0xfffffffffff7]
last incident: athenaeventloopmgr:beginevent
-------------------------------------------------------------------------------------

(pid=21767 ppid=21344) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 2 (invalid permissions for mapped object)
  pid    = 1661609840, uid = 16777228
  value  = (-4750168, 0xffb784a8)
  addr   = 0x630a2770
  stack  = (2, 0, (nil))

  eip: 0023:630a2770           eflags: 00210206
  eax: 630a2098   ebx: 771f523c   ecx: 00000001   edx: 1221b962
  esi: ffb78828   edi: 409985f8   ebp: ffb78838   esp: ffb7878c
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ffb7878c  trap: 14/21  oldmask: 00000000   cr2: 630a2770

  fpu:  control = ffff037f
        status  = ffff0132
        tag     = ffffffff
        ip      = 0010:76fc0d60
        data    = 002b:ffb78868
        state   = 00000132
    %fp0 = [8000:0000000000000000]
    %fp1 = [c002:5b9d89e0e8289fd8]
    %fp2 = [400b:8d9c6fa435489abc]
    %fp3 = [3ffd:e38e8e3838e3e38e]
    %fp4 = [3ffc:00002169daa2c90f]
    %fp5 = [3ffe:6800f9def333b504]
    %fp6 = [3ffa:c0002168daa2c90f]
    %fp7 = [3ffa:c0002168daa2c90f]

stack trace:
 0x5aca27a0 athena::debugaids::stacktrace(int)+0x60 ??:0   + 0x60 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5aca37bf athena::signal::fataldump(int, siginfo*, void*)+0xff ??:0   + 0xff [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5aca3ad1 athena::signal::fatal(int, siginfo*, void*)+0x181 ??:0   + 0x181 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0x5aa1d006 coredumpsvchandler::action(int, siginfo*, void*)+0x106 ??:0   + 0x106 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0x630a2770 classid_traits&lt;lvl1::jemhits&gt;::typename()::s_name ??:0   + 0x0 [/afs/cern.ch/atlas/software/releases/17.0.5/atlasevent/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libtrigt1caloeventlib.so]
 0x76fc0d8b emefficienciesmontool::triggertoweranalysis()+0xbb ??:0   + 0xbb [/afs/cern.ch/atlas/software/releases/17.0.5/atlastrigger/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libtrigt1calomonitoring.so]
 0x76fc16c1 emefficienciesmontool::fillhistograms()+0x881 ??:0   + 0x881 [/afs/cern.ch/atlas/software/releases/17.0.5/atlastrigger/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libtrigt1calomonitoring.so]
 0x62a6e3cf managedmonitortoolbase::fillhists()+0x4bf /build/atnight/localbuilds/nightlies/17.0.x.y-val-prod/atlasproduction/rel_0/control/athenamonitoring/cmt/../src/managedmonitortoolbase.cxx:807   + 0x4bf [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y-val/atlasproduction/rel_3/installarea/i686-slc5-gcc43-opt/lib/libathenamonitoringlib.so]
 0x62a9d7fa athenamonmanager::execute()+0x56a /afs/cern.ch/atlas/software/builds/gaudi/v22r1p5/installarea/include/gaudikernel/statuscode.h:105   + 0x56a [/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y-val/atlasproduction/rel_3/installarea/i686-slc5-gcc43-opt/lib/libathenamonitoringlib.so]
 0x5831c98d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.5/gaudi/v22r1p5/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x624a474b athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x5831c98d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.5/gaudi/v22r1p5/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x624a474b athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0x5831c98d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.5/gaudi/v22r1p5/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0x5aae200f athenaeventloopmgr::executealgorithms()+0x6f ??:0   + 0x6f [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5aae85c6 athenaeventloopmgr::executeevent(void*)+0x6f6 ??:0   + 0x6f6 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5aae64b2 athenaeventloopmgr::nextevent(int)+0x8d2 ??:0   + 0x8d2 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5aae8ab9 athenaeventloopmgr::executerun(int)+0x29 ??:0   + 0x29 [/afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0x5a5c1619 applicationmgr::executerun(int)+0x79 ??:0   + 0x79 [/afs/cern.ch/atlas/software/releases/17.0.5/gaudi/v22r1p5/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
 0x58bc502d _zn12_global__n_1l11method_4303epvs0_rkst6vectoris0_sais0_ees0_+0x3d ??:0   + 0x58bc502d [/afs/cern.ch/atlas/software/releases/17.0.5/gaudi/v22r1p5/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0x579878b7 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int)+0x157 ??:0   + 0x157 [/afs/cern.ch/atlas/software/releases/17.0.5/lcgcmt/lcgcmt_60d/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0x59a98034 &lt;unknown function&gt;
 0x560f8422 cint::g__callfunc::execute(void*)+0xd2 ??:0   + 0xd2 [/afs/cern.ch/atlas/software/releases/17.0.5/lcgcmt/lcgcmt_60d/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0x558f157c pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*)+0x2c ??:0   + 0x2c [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00g/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x558f7ea4 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*)+0x64 ??:0   + 0x64 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00g/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x558f803c pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*)+0x6c ??:0   + 0x6c [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00g/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x558f659b pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long)+0x14b ??:0   + 0x14b [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00g/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x558fd7b2 pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*)+0x4a2 ??:0   + 0x558fd7b2 [/afs/cern.ch/sw/lcg/app/releases/root/5.28.00g/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 pyeval_evalframeex+0x5a89 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 pyeval_evalframeex+0x5a89 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags+0x204 /build/agaspar/work/python-2.6.5/python/pythonrun.c:935   + 0x204 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags+0x7a /build/agaspar/work/python-2.6.5/python/pythonrun.c:739   + 0x7a [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main+0xc56 /build/agaspar/work/python-2.6.5/modules/main.c:572   + 0xc56 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main+0x32 /build/agaspar/work/python-2.6.5/./modules/python.c:24   + 0x32 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x00337e9c __libc_start_main+0xdc ??:0   + 0xdc [/lib/libc.so.6]
 0x08057d51 _start+0x21 ??:0   + 0x8057d51 [/afs/cern.ch/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -11.
py:rawtoesd         error athena received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/share/atlas_error_categories.db
info: read 53 error patterns from file /afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/share/atlas_error_patterns.db
info: read 61 ignore patterns from file /afs/cern.ch/atlas/software/releases/17.0.5/atlascore/17.0.5/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: wed nov 16 04:29:40 2011 cet, vmem: 99.11328125 mb, rss: 67.17578125 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,rel_3 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 18713 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 28
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: wed nov 16 04:29:41 2011 cet, vmem: 99.24609375 mb, rss: 67.25 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 01 secs, cpu-time: 00 hrs 00 mins 01 secs, vmem: 0.1328125 mb, rss: 0.07421875 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: wed nov 16 04:29:41 2011 cet, vmem: 99.24609375 mb, rss: 67.2734375 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 00 hrs 06 mins 06 secs, cpu-time: 00 hrs 04 mins 10 secs, vmem: 0.74609375 mb, rss: 0.67578125 mb
===== job report produced by rawtoesd version unknown =====
command=/afs/cern.ch/atlas/software/builds/nightlies/17.0.x.y-val/atlasproduction/rel_3/installarea/share/bin/rawtoesd_trf.py --useathenaapp=true inputbsfile=/afs/cern.ch/atlas/project/pmb/data/data11_7tev.00182726.physics_jettauetmiss.merge.raw._lb0215._sfo-5._0001.1_1000evts maxevents=250 autoconfiguration=everything --athenaopts=--stdcmalloc preexec=from@perfmoncomps.perfmonflags@import@jobproperties@as@pmjp,,pmjp.perfmonflags.domallocmonitoring=true,,pmjp.perfmonflags.dopostprocessing=true,,pmjp.perfmonflags.dosemidetailedmonitoringfullprint=true outputdqmonitorfile=mymon.root outputesdfile=myesd.pool.root outputntup_muoncalibfile=myntup_muoncalib.root outputdraw_zmumufile=mydraw_zmumu.data outputdraw_zeefile=mydraw_zee.data
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=10
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=21767 ppid=21344) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
workdir=/pool/lsf/rtt/188203339
athcode=8
rundiravailabledisk=1151684972544
dbrelease=
athacronym=an unknown exception occurred
rundiruseddisk=7954271
atlasrelease=rel_3
writing jobinfo_rawtoesd.xml file...
------------

dhiman (valshift)
