Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Pentaho goes idle on Vertica bulk loader

  1. #1

    Default Pentaho goes idle on Vertica bulk loader

    Hello, everyone.
    For the past couple of days we are having problems with ETL process. Specifically it's based around Vertica Bulk Loader component.
    Background:
    So the main issue is that Vertica Bulk loader generates COPY statement to Vertica database which after some time fails with an error: [Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 4311521, Free = 216564 (Limit = 21834206, Used = 21617642) (queueing threshold)]. However, the session still appears to be active even though SQL statement already failed and Pentaho job is also appears to be running infinitely. We tried closing the session on the database side which resulted in no active session on the database side, but Pentaho job (java) was still running on the server (CentOS).
    With the help of other colleague we managed to get thread dump of idle process, attachment - js1.zip.

    The strange thing with all this is that we had the same Vertica error on Table output component and job part just failed and everything continued as designed in the workflow. But Vertica Bulk Loader just goes to idle.
    I don't know how to dig deeper, because I am not such a Linux and Java expert. That's why I'm here

    Transformation is basically table input and Vertica Bulk Loader (some other components are in the middle like table lookup).

    Hope for your help.

    Thank you.

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Your JDBC driver needs more memory it seems.

    If you're loooking for improvements to the Vertica-Bulk-Loader step, you'll have to turn to file a bug report - if no one already did that.

    Also, version numbers are relevant.
    So long, and thanks for all the fish.

  3. #3

    Default

    Hi.
    PDI version is 7.1.
    Why do you think it is because of memory ? As I said - the problem here is that pentaho goes to idle, not the vertica resources error.
    I am not sure if I am looking for improvements to the vertica bulk loader step, I just want to understand why it hangs after fail.

    Thank you.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Quote Originally Posted by pavels.derjagins View Post
    [Vertica][VJDBC](3587) ... Memory(KB) Exceeded: Requested = 4311521, Free = 216564 (Limit = 21834206, Used = 21617642)
    For me the step idling forever is an - albeit poor - consequence of the memory exhaustion, not vice versa.
    But I could be wrong, of course.
    Good luck.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.