hi, 

tier-0 configuration was changed to use atlasproduction 17.0.2.10. 
as a validation test data11_cos.00187749.physics_cosmiccalo.merge.raw is processed with v41 tag. this v41 tag is identical to f397 tag. 
17 out of 69 jobs are crashed by muoncombinedrefit. 
one log file is below. i will extract the crash events. 

cheers, 
susumu 

+verbatim+
input file(s):
--------------------------------------------------------------------------------
stage_svcclass=atlcal
/castor/cern.ch/grid/atlas/tzero/prod1/perm/data11_cos/physics_cosmiccalo/00187749/data11_cos.00187749.physics_cosmiccalo.merge.raw/data11_cos.00187749.physics_cosmiccalo.merge.raw._lb0007._sfo-all._0001.1


tier-0 transformation command run:
--------------------------------------------------------------------------------
source /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/setuptrf.sh /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/patches/atlasproduction-17.0.2.10 atlasproduction-17.0.2.10 /afs/cern.ch/atlas/tzero/prod1/inst/projects/data11/setup/specialsetup_tier0.sh ; python -u `which reco_trf.py` --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/13228/13228075_data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.1//data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.argdict.gpickle  &amp;&gt; data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.tmp.log


ami configuration tag (more details):
--------------------------------------------------------------------------------
see http://ami.in2p3.fr:8080/opencms/opencms/ami/www/referencetables/ --&gt; configurationtag --&gt; interpret --&gt; v41


logfile location:
--------------------------------------------------------------------------------
stage_svcclass=t0merge
/castor/cern.ch/grid/atlas/tzero/prod1/temp/data11_cos/physics_cosmiccalo/00187749/data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task.tmp.log/data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.tmp.log.1


tail of the log file (300 lines):
--------------------------------------------------------------------------------
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.taucellcorrectiontool                     warning could not retrieve vxcontainer &lt;vxprimarycandidate&gt; during initialize() 
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
eflowcaloobjectbuilder                            warning  primary vertex container (vxprimarycandidate) not found in storegate
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #4463, run #187749 108 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #4470, run #187749 108 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
mooreextrtpamcreator                              warning  caloenergy missing from extrapolated track
mooreextrtpamcreator                              warning  caloenergy missing from extrapolated track
toolsvc.trackdepositincalotool                    warning failure in extrapolating to solenoid surface.
toolsvc.trackdepositincalotool                    warning failure in extrapolating to solenoid surface.
toolsvc.trackdepositincalotool                    warning failure in extrapolating to solenoid surface.
toolsvc.trackdepositincalotool                    warning failure in extrapolating to solenoid surface.
toolsvc.trackdepositincalotool                    warning failure in extrapolating to solenoid surface.
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt6towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt4towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.taucellcorrectiontool                     warning could not retrieve vxcontainer &lt;vxprimarycandidate&gt; during initialize() 
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
eflowcaloobjectbuilder                            warning  primary vertex container (vxprimarycandidate) not found in storegate
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #4470, run #187749 109 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #4472, run #187749 109 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt6towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt4towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.taucellcorrectiontool                     warning could not retrieve vxcontainer &lt;vxprimarycandidate&gt; during initialize() 
metrefalg.metrefjet                               warning failed to retrieve jet moment, applying scale factor == 1.
metrefalgmuid.metrefjetmuid                       warning failed to retrieve jet moment, applying scale factor == 1.
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
metrefalg_em.metrefjet_em                         warning failed to retrieve jet moment, applying scale factor == 1.
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
eflowcaloobjectbuilder                            warning  primary vertex container (vxprimarycandidate) not found in storegate
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #4472, run #187749 110 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #4524, run #187749 110 events processed so far  &lt;&lt;&lt;===
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt6towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
antikt4towerjets.applyem:offset_origin_etajes_s...warning could not retrieve vxcontainer during initialize(): vxprimarycandidate
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.taucellcorrectiontool                     warning could not retrieve vxcontainer &lt;vxprimarycandidate&gt; during initialize() 
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
eflowcaloobjectbuilder                            warning  primary vertex container (vxprimarycandidate) not found in storegate
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  done processing event #4524, run #187749 111 events processed so far  &lt;&lt;&lt;===
athenaeventloopmgr                                   info   ===&gt;&gt;&gt;  start processing event #4528, run #187749 111 events processed so far  &lt;&lt;&lt;===
dataproxy                                         warning accessdata:  iconversionsvc ptr not set
toolsvc.emtrackisolationtool                      warning vertex container vxprimarycandidate not found in storegate!
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.egammaisolationmvatoptool                 warning vertex container vxprimarycandidate not found in storegate!
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
storegatesvc                                      warning retrieve(const): no valid proxy for object vxprimarycandidate  of type vxcontainer(clid 9327453)
toolsvc.tracktocaloselection                         info track summary tool retrieved
toolsvc.tracktocaloselection                         info retrieved tool magneticfieldtool = publictoolhandle('trk::magneticfieldtool/indetmagfield')
mooreextrtpamcreator                              warning  caloenergy missing from extrapolated track
mooreextrtpamcreator                              warning  caloenergy missing from extrapolated track
toolsvc.configuredforallstationsdomutaggedsegme...   info initialisation started     
toolsvc.configuredforallstationsdomutaggedsegme...   info ================================
toolsvc.configuredforallstationsdomutaggedsegme...   info =proprieties are     
toolsvc.configuredforallstationsdomutaggedsegme...   info ================================
-------------------------------------------------------------------------------------
core dump from coredumpsvc on lxbrb1810.cern.ch at thu aug 25 23:03:57 2011

caught signal 11(segmentation fault). details: 
  errno = 0, code = 2 (invalid permissions for mapped object)
  pid   = -221148888, uid = 3218607408
  value = (1340987434, 0x4fedd82a)
  vmem = 2023.12 mb
  rss  = 1641.36 mb
  total-ram = 15929.1 mb
  free-ram  = 6690.11 mb
  buffer-ram= 344.969 mb
  total-swap= 20481.2 mb
  free-swap = 20481.2 mb
  addr  = 0xf2d18928

event counter: 112
eventid: [run,evt,lumi,time,bunchcross,detmask] = [187749,4528,7,1313855740:621673565,1685,0x1ffffcffffff7]
last incident: athenaeventloopmgr:beginevent
current algorithm: muoncombinedrefit
algorithm stack: 
   athmasterseq
   athalgseq
   muoncombinedrefit
-------------------------------------------------------------------------------------

(pid=25331 ppid=25273) received fatal signal 11 (segmentation fault)
signal context:
  signo  = 11, errno = 0, code = 2 (invalid permissions for mapped object)
  pid    = -221148888, uid = -1076359888
  value  = (1340987434, 0x4fedd82a)
  addr   = 0xf2d18928
  stack  = (2, 0, (nil))

  eip: 0023:f2d18928           eflags: 00010296
  eax: 47e8ac30   ebx: dc491824   ecx: 00000084   edx: 5456baf8
  esi: 54ea3ea0   edi: 00000000   ebp: ff941ac8   esp: ff941aac
   ds: 002b        es: 002b        fs: 0000        ss: 002b

  signal esp: ff941aac  trap: 14/21  oldmask: 00000000   cr2: f2d18928

  fpu:  control = ffff037f
        status  = ffff4020
        tag     = ffffffff
        ip      = 0010:f278d4ea
        data    = 002b:00000000
        state   = 00004020
    %fp0 = [0000:0000000000000000]
    %fp1 = [3ffd:9800f177fce9c393]
    %fp2 = [c005:8e8e2d676ea58bcc]
    %fp3 = [8000:0000000000000000]
    %fp4 = [4006:6882851b8648a91c]
    %fp5 = [4009:5b7f24ff5e08ecd2]
    %fp6 = [400a:8aef63fa34dcff17]
    %fp7 = [0000:0000000000000000]

stack trace:
 0xf2f4e7a0 athena::debugaids::stacktrace(int)+0x60 ??:0   + 0x60 [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf2f4f7bf athena::signal::fataldump(int, siginfo*, void*)+0xff ??:0   + 0xff [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf2f4fad1 athena::signal::fatal(int, siginfo*, void*)+0x181 ??:0   + 0x181 [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libcxxutils.so]
 0xf3140796 coredumpsvchandler::action(int, siginfo*, void*)+0x106 ??:0   + 0x106 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xffffe600 &lt;unknown function&gt;
 0xf2d18928 vtable for hepgeom::basicvector3d&lt;double&gt;+0x8 ??:0   + 0x8 [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libgeomodelkernel.so]
 0xdc446273 trk::globalchi2fitter::calculatetrackparameters(trk::gxftrajectory&amp;, bool) const+0xd13 ??:0   + 0xd13 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libtrkglobalchi2fitter.so]
 0xdc447e1c trk::globalchi2fitter::runiteration(trk::gxftrajectory&amp;, int, tmatrixtsym&lt;double&gt;&amp;, tvectort&lt;double&gt;&amp;, tdecomplu&amp;, bool&amp;) const+0xbc ??:0   + 0xbc [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libtrkglobalchi2fitter.so]
 0xdc44c5b5 trk::globalchi2fitter::myfit(trk::gxftrajectory&amp;, trk::parameterst&lt;trk::charged&gt; const&amp;, bool, trk::particlehypothesis) const+0xb15 ??:0   + 0xb15 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libtrkglobalchi2fitter.so]
 0xdc4504d1 trk::globalchi2fitter::backupcombinationstrategy(trk::track const&amp;, trk::track const&amp;, std::vector&lt;trk::materialeffectsontrack, std::allocator&lt;trk::materialeffectsontrack&gt; &gt;&amp;) const+0x17c1 ??:0   + 0x17c1 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libtrkglobalchi2fitter.so]
 0xdc4545e9 trk::globalchi2fitter::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const+0x7f9 ??:0   + 0x7f9 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libtrkglobalchi2fitter.so]
 0xd91b2ae5 rec::outwardscombinedmuontrackbuilder::fit(trk::track const&amp;, trk::track const&amp;, bool, trk::particlehypothesis) const+0x75 ??:0   + 0x75 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libmuidtrackbuilder.so]
 0xe3f04892 refitmuon::execute()+0x2842 ??:0   + 0x2842 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasreconstruction/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libcombinedmuonrefit.so]
 0xf4cb3a6d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.2/gaudi/v22r1p3/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe77a66fb athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf4cb3a6d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.2/gaudi/v22r1p3/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xe77a66fb athsequencer::execute()+0x25b ??:0   + 0x25b [/afs/cern.ch/atlas/software/releases/17.0.2/atlascore/17.0.2/installarea/i686-slc5-gcc43-opt/lib/libgaudisequencer.so]
 0xf4cb3a6d algorithm::sysexecute()+0x11d ??:0   + 0x11d [/afs/cern.ch/atlas/software/releases/17.0.2/gaudi/v22r1p3/installarea/i686-slc5-gcc43-opt/lib/libgaudikernel.so]
 0xf31a962f athenaeventloopmgr::executealgorithms()+0x6f ??:0   + 0x6f [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf31afb86 athenaeventloopmgr::executeevent(void*)+0x6f6 ??:0   + 0x6f6 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf31ada72 athenaeventloopmgr::nextevent(int)+0x8d2 ??:0   + 0x8d2 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf31b0079 athenaeventloopmgr::executerun(int)+0x29 ??:0   + 0x29 [/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/i686-slc5-gcc43-opt/lib/libathenaservices.so]
 0xf3838289 applicationmgr::executerun(int)+0x79 ??:0   + 0x79 [/afs/cern.ch/atlas/software/releases/17.0.2/gaudi/v22r1p3/installarea/i686-slc5-gcc43-opt/lib/libgaudisvc.so]
 0xf41a4f2d _zn12_global__n_1l11method_4304epvs0_rkst6vectoris0_sais0_ees0_+0x3d ??:0   + 0xf41a4f2d [/afs/cern.ch/atlas/software/releases/17.0.2/gaudi/v22r1p3/installarea/i686-slc5-gcc43-opt/lib/libgaudikerneldict.so]
 0xf5cd1a57 root::cintex::method_stub_with_context(root::cintex::stubcontext_t*, g__value*, char const*, g__param*, int)+0x157 ??:0   + 0x157 [/afs/cern.ch/atlas/software/releases/17.0.2/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-opt/lib/libcintex.so]
 0xf3544034 &lt;unknown function&gt;
 0xf6f80572 cint::g__callfunc::execute(void*)+0xd2 ??:0   + 0xd2 [/afs/cern.ch/atlas/software/releases/17.0.2/lcgcmt/lcgcmt_60c/installarea/i686-slc5-gcc43-opt/lib/libcint.so]
 0xf7c5c75c pyroot::trootobjectbyvalueexecutor::execute(cint::g__callfunc*, void*)+0x2c ??:0   + 0x2c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7c63134 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::callsafe(void*)+0x64 ??:0   + 0x64 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7c632cc pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::execute(void*)+0x6c ??:0   + 0x6c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7c617f3 pyroot::tmethodholder&lt;pyroot::tscopeadapter, pyroot::tmemberadapter&gt;::operator()(pyroot::objectproxy*, _object*, _object*, long)+0x153 ??:0   + 0x153 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0xf7c68b2a pyroot::(anonymous namespace)::mp_call(pyroot::methodproxy*, _object*, _object*)+0x4aa ??:0   + 0xf7c68b2a [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/app/releases/root/5.28.00e/i686-slc5-gcc43-opt/root/lib/libpyroot.so]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dc329 pyeval_evalframeex+0x5a89 /build/agaspar/work/python-2.6.5/python/ceval.c:3846   + 0x5a89 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d4900 builtin_execfile+0x1d0 /build/agaspar/work/python-2.6.5/python/bltinmodule.c:797   + 0x80d4900 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x081391bd pycfunction_call+0x18d /build/agaspar/work/python-2.6.5/objects/methodobject.c:81   + 0x18d [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dbd09 pyeval_evalframeex+0x5469 /build/agaspar/work/python-2.6.5/python/ceval.c:3750   + 0x5469 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08137caa function_call+0xaa /build/agaspar/work/python-2.6.5/objects/funcobject.c:524   + 0x8137caa [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08064935 instancemethod_call+0x145 /build/agaspar/work/python-2.6.5/objects/classobject.c:2579   + 0x8064935 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080b29e3 slot_tp_call+0x83 /build/agaspar/work/python-2.6.5/objects/typeobject.c:5373   + 0x80b29e3 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x0805d88c pyobject_call+0x5c /build/agaspar/work/python-2.6.5/objects/abstract.c:2493   + 0x5c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080d969d pyeval_evalframeex+0x2dfd /build/agaspar/work/python-2.6.5/python/ceval.c:3968   + 0x2dfd [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080ddc8c pyeval_evalcodeex+0x87c /build/agaspar/work/python-2.6.5/python/ceval.c:3000   + 0x87c [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080dddd3 pyeval_evalcode+0x63 /build/agaspar/work/python-2.6.5/python/ceval.c:547   + 0x63 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe044 pyrun_fileexflags+0xb4 /build/agaspar/work/python-2.6.5/python/pythonrun.c:1339   + 0xb4 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe3a4 pyrun_simplefileexflags+0x204 /build/agaspar/work/python-2.6.5/python/pythonrun.c:935   + 0x204 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x080fe69a pyrun_anyfileexflags+0x7a /build/agaspar/work/python-2.6.5/python/pythonrun.c:739   + 0x7a [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08058d46 py_main+0xc56 /build/agaspar/work/python-2.6.5/modules/main.c:572   + 0xc56 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x08057e42 main+0x32 /build/agaspar/work/python-2.6.5/./modules/python.c:24   + 0x32 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
 0x00a07e9c __libc_start_main+0xdc ??:0   + 0xdc [/lib/libc.so.6]
 0x08057d51 _start+0x21 ??:0   + 0x8057d51 [/afs/cern.ch/atlas/software/releases/17.0.2/sw/lcg/external/python/2.6.5/i686-slc5-gcc43-opt/bin/python]
py:rawtoesd          info rawtoesd has completed running of athena with exit code -11.
py:rawtoesd         error athena received signal 11. exit code reset to athena exit code 8.
info: read 109 error categories from file /afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/share/atlas_error_categories.db
info: read 52 error patterns from file /afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/share/atlas_error_patterns.db
info: read 61 ignore patterns from file /afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/share/atlas_error_ignore.db
athenalogchecker start [rawtoesd log] wall-time: thu aug 25 23:04:07 2011 cest, vmem: 108.0 mb, rss: 68.30859375 mb
py:rawtoesd          info scanning athena logfile rawtoesd.log assuming atlas release all,17.0.2.10 ...
py:rawtoesd          info athena initialise()...
py:rawtoesd          info athena execute()...
py:rawtoesd          info done scanning 12735 lines of file rawtoesd.log. summary:
py:rawtoesd          info    ignored : 0
py:rawtoesd          info    warnings: 2214
py:rawtoesd          info    errors  : 0
py:rawtoesd          info    fatals  : 2
athenalogchecker stop [rawtoesd log] wall-time: thu aug 25 23:04:09 2011 cest, vmem: 108.0 mb, rss: 68.30859375 mb
athenalogchecker report [rawtoesd log] wall-time: 00 hrs 00 mins 02 secs, cpu-time: 00 hrs 00 mins 02 secs, vmem: 0.0 mb, rss: 0.0 mb
py:rawtoesd          info jobtransform completed for rawtoesd with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [rawtoesd] wall-time: thu aug 25 23:04:09 2011 cest, vmem: 108.0 mb, rss: 68.328125 mb
py:rawtoesd          info transform timer report [rawtoesd] wall-time: 00 hrs 09 mins 59 secs, cpu-time: 00 hrs 08 mins 34 secs, vmem: 0.43359375 mb, rss: 0.515625 mb
===== job report produced by rawtoesd version unknown =====
command=/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/13228/13228075_data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.1//data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=10
writing jobinfo_rawtoesd.xml file...
esdok is  false
skipping esd-&gt;esd step...
skipping esd-&gt;aod step...
skipping esd-&gt;dpd step...
skipping aod-&gt;dpd step...
skipping aod-&gt;tag step...
skipping monitoring file merging step...
py:reco_trf          info jobtransform completed for reco_trf with error code 60010 (exit code 6)
py:rawtoesd          info transform timer stop [reco_trf] wall-time: thu aug 25 23:04:09 2011 cest, vmem: 108.1484375 mb, rss: 68.515625 mb
py:rawtoesd          info transform timer report [reco_trf] wall-time: 00 hrs 09 mins 59 secs, cpu-time: 00 hrs 08 mins 34 secs, vmem: 0.58203125 mb, rss: 0.75 mb
===== job report produced by recotrf =====
command=/afs/cern.ch/atlas/software/releases/17.0.2/atlasproduction/17.0.2.10/installarea/share/bin/reco_trf.py --argdict=/afs/cern.ch/atlas/tzero/prod1/log1/13228/13228075_data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.1//data11_cos.00187749.physics_cosmiccalo.merge.raw.v41.recon.task._lb0007._sfo-all._0001.job.argdict.gpickle
exitcode=6
errorcode=60010 (trf_segfault)
errorcategory=trf_segfault,	60010,	segmentation fault (2 times)
fatals=2
errors=0
warnings=10
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=caught signal 11(segmentation fault). details:
  severity=fatal
  producer=rawtoesd
  stage=execute
errorinfo
  acronym=trf_segfault (60010) segmentation fault
  message=(pid=25331 ppid=25273) received fatal signal 11 (segmentation fault)
  severity=fatal
  producer=rawtoesd
  stage=execute
athacronym=an unknown exception occurred
athcode=8
rundiruseddisk=402659027
rundiravailabledisk=137467867136
workdir=/pool/lsf/atlast0/165240871
athcode=8
rundiravailabledisk=137468149760
dbrelease=current
athacronym=an unknown exception occurred

-verbatim-