Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Performance Suggestions & Java out of Memory Error

  1. #1
    Join Date
    Dec 2008
    Posts
    12

    Default Performance Suggestions & Java out of Memory Error

    Hi,

    I am newbee to PDI.I created a transformation which pulls the data from source system using SQL statement and creates a XML file.This source system has around 3 millions of records.Currently I am testing using Kettle with JVM setting of 1024 MB.I have end up with the following error:

    "Java out of Memory Error"

    Please help to know the approach or an alternative to come out of this issue(or Memory settings).

    Thanks,
    Sashi

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    show us the transformation

  3. #3
    Join Date
    Dec 2008
    Posts
    12

    Default

    Please find the attached transformation.This transformations creates only 35000 currently.As per your suggestion I would like to change this number.

    Thanks.
    Attached Files Attached Files

  4. #4
    Join Date
    Dec 2008
    Posts
    12

    Default

    Hi Matt, I attached the required transfromation in my previous mail.By any chance did you got a chance to look into it.

    Thanks,
    Sashi

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    I never said I was going to look at it.
    But now that I did... <sigh>

    Code:
    select DBMS_XMLQuery.GetXML(DBMS_XMLQuery.newContext
    ('select CAST(MULTISET(SELECT addr.add_street,addr.add_street2,addr.add_building ,addr.add_apartment ,addr.add_city ,s.lu_code ,addr.add_zip ,loc.l_code ,cau.lu_code FROM sjs.address addr,sjs.v_states s,sjs.locations loc,sjs.v_caution_type cau,sjs.sjs_agency sa WHERE (i."AgencyORI" = sa.agcy_ori) AND addr.stateid = s.lu_id (+) AND addr.l_id = loc.l_id (+) and sa.add_id=addr.add_id AND addr.cautionid = cau.lu_id (+)) as xmlapi_addresses_type) "AgcyAddresses",(select agcy_phone_number from sjs.SJS_AGENCY sa where sa.agcy_ori=i."AgencyORI") AgcyPhonNumber,i.*,''GREEN'' as SecFlag from xmlapi_incident_view  i where rownum<35000')) as "XML" from dual
    It seems to be that the only place that this transformation can run out of memory is in the Oracle JDBC driver.

  6. #6
    Join Date
    Apr 2007
    Posts
    2,009

    Default

    urgh, really, getting the data from the db as xml? why??

  7. #7
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    That's all the transformation does: read with the above statement and write to another database. That's why.

  8. #8
    Join Date
    Dec 2008
    Posts
    12

    Default

    We are generating XML from db for transporting to another remote location.it's a final step of our process.

    Thanks,

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.