Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Massive Load Data (6 files of 800,000 records)

  1. #1
    Join Date
    Mar 2010
    Posts
    28

    Default Massive Load Data (6 files of 800,000 records)

    Good morning.

    I have installed PDI in a Linux virtual machine with a MySQL database server.

    I'm loading a serie of files of about 800,000 records.
    The first three I could load the files correctly. With the following is not able to finish loading and the server leaves me hanging. Starts loading correctly but gradually slows down load the table until a time that does not advance, also uses all the physical memory of the virtual machine.
    Is there any way that the memory is emptied out every so often to get to spend it not all?

    Greetings.

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    So first of all: that's not a massive data load.
    Then: how are you loading the data, with the MySQL Bulk loader step?
    Finally: what version of MySQL are you using? I remember a problem with index space being used up on the MySQL side while bulk loading. We created the "Bulk size" option for that.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.