hi, one reprocessing job failed in es cloud with message:

athena ran out of memory exe: trf_unknown | &quot;out of memory: saving summary ...&quot; | &quot;failed to create persistent object: std::bad_alloc&quot; | &quot;createrep failed, key = tracks&quot; | &quot;could not create rep for dataobject (clid/key):117682415 tracks&quot; | &quot;streamobjects failed.&quot; | &quot;pool commitan
in:   data11_7tev.00187457.physics_background.merge.raw out: data11_7tev.00187457.physics_background.recon.hist.r271


log file extracts:

mismatch | trf error code mismatch: exeerrorcode = 69999, transexitcode = 99- job report produced by recotrf -
command=/software/atlas/prod/releases/rel_17-04/atlasproduction/17.0.3.3/installarea/share/bin/reco_trf.py inputbsfile=data11_7tev.00187457.physics_background.merge.raw._lb0385._sfo-all._0001.1 runnumber=187457 autoconfiguration=everything,conditionstag=comcond-blkp*-005-06 dbrelease=16090101._0187457.tar.gz dbrelease=dbrelease-16.9.1.1.tar.gz amitag=r2713 outputesdfile=esd.512548._000244.pool.root.22 outputhistfile=hist.512548._000244.root.22 --extraignorefilters=trigbsextraction_error_ignore.db
exitcode=99
errorcode=69999 (trf_unknown)
errorcategory=trf_unknown,	69999,	unknown transform error (58 times)
fatals=7
errors=10
warnings=8
errorinfo
  acronym=trf_unknown (69999)
  message=out of memory: saving summary ...
  severity=fatal
  who=athenasummarysvc
  producer=rawtoesd
  stage=execute
  backtrace=file &quot;/software/atlas/prod/releases/rel_17-04/atlasproduction/17.0.3.3/installarea/python/pyjobtransformscore/trf.py&quot;, line 2041, in runjob
              maxevents = maxeventsarg.value()
  message=failed to create persistent object: std::bad_alloc
  severity=error
  who=athenapoolconverter
  message=createrep failed, key = tracks
  message=could not create rep for dataobject (clid/key):117682415 tracks
  who=streamesd.streamesdtool
  message=streamobjects failed.
  who=streamesd
errorinfo (count=2)
  message=pool commitandhold failed 0xb3d69d8
  who=poolsvc
  message=commitoutput failed to commitandhold outputstream.
  who=athenapoolcnvsvc
  message=unable to commit output esd.512548._000244.pool.root.22
  message=commitoutput failed.
  message=terminating event processing loop due to errors
  who=athenaeventloopmgr
  message=athenapoolconverter createrep failed to get token, key = streamesd
  message=could not create rep for dataobject (clid/key):167728019 streamesd
  who=streamesd_fh.streamesd_fhtool
  message=athenapoolconverter createrep failed to get token, key = incompletelumiblocks
  who=strea

- walltime -


the link the the failed job is:

http://panda.cern.ch/server/pandamon/query?job=*&amp;cloud=es&amp;type=production&amp;jobstatus=failed&amp;hours=1&amp;processingtype=reprocessing&amp;workinggroup=reprocessing

would you please have a look?

cheers,
marisa (adcos shifter)
