Hitachi Vantara Pentaho Community Forums
Results 1 to 1 of 1

Thread: xml file Performance issue

  1. #1
    Join Date
    Dec 2012
    Posts
    29

    Default xml file Performance issue

    Hi All,


    i have to generate many xml files.
    Infact, need to generate about a million xml files ,one for each member.
    I am looping member ids in a job to generate xml and generating the files.
    It caused heap size error "java.lang.OutOfMemoryError: Java heap space".
    So i increased the setting in spoon.sh OPT =-Xmx1024m . But it didnt work out.
    So i tried to run it in a job executer step.
    Now the job is working fine.
    But it is taking one hour to generate 1000 xml file.

    Is there any way to improve the performance ?
    Each file will take about 2-3 seconds to generate. And 1 million record means
    a million selects to Db,i million file generation. My transformation have table input steps,add xml ,join xml, replace in string,text file out put steps.

    i am not able to take the whole records like 'select * from members' because the xml file will not generate for one member.(Add xml file cannot do for one member data alone)
    Any other logic to generate the xmls?
    Any help would be great!!!


    Thanks,
    Vineetha P
    Last edited by Plearner; 12-14-2018 at 07:26 AM. Reason: more info

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.