Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Pentaho JSON Input : GC Overhead Error

  1. #1

    Question Pentaho JSON Input : GC Overhead Error

    Hi Folks,

    I am trying to load a json file of 400MB using petaho DI version 6. But getting GC overhead error. This error occurs when pentaho trying to load the whole file in one go. I have also tried to load the file by increasing the -Xmx and permsize but unfortunately that didn't work.

    Is there requirement to break the file ?

    Can anyone please help me with this.


    thanks
    gnish

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Why break up the file?

    We don't know about your system configuration, but generally, it's a bad idea to exchange single objects that large.
    So long, and thanks for all the fish.

  3. #3

    Default

    How much is the preferred configuration for a json file of size 400 MB ?

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Memory footprint depends very much on the data structure and the parser.
    I don't know about FastJsonReader, but with Python's json parser (for example) you could very well end up with 8GB RAM for your 440MB JSON object.
    So long, and thanks for all the fish.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.