Hitachi Vantara Pentaho Community Forums
Results 1 to 10 of 10

Thread: use kitchen to launch jobs but waiting between calls

  1. #1

    Question use kitchen to launch jobs but waiting between calls

    Hello,

    Is possible to use kitchen to launch jobs but waiting to the previous execution to finish?

    Something like:

    Code:
    :NEXTITEM1
    set PENTAHO_DI_JAVA_OPTIONS="-Xmx1024m" "-XX:MaxPermSize=256m"
    call kitchen.bat /rep:"REPO" /job:"Job1"...........................
    IF %ERRORLEVEL% == 0 goto NEXTITEM2
    else goto QUIT
    
    :NEXTITEM2
    set PENTAHO_DI_JAVA_OPTIONS="-Xmx2048m" "-XX:MaxPermSize=256m"
    call kitchen.bat /rep:"REPO" /job:"Job2"...........................
    IF %ERRORLEVEL% == 0 goto NEXTITEM3
    else goto QUIT
    
    :NEXTITEM3
    set PENTAHO_DI_JAVA_OPTIONS="-Xmx512m" "-XX:MaxPermSize=256m"
    call kitchen.bat /rep:"REPO" /job:"Job3"...........................
    IF %ERRORLEVEL% == 0 goto FINISHJOBS
    else goto QUIT
    
    
    :QUIT
    exit
    I want to set specific memory settings for some jobs but if all are called at the same time, the memory is going to end quickly.

    Currently running 5.3.0.0-213 under Win2k8R2

    Any clue?

    Thanks!

  2. #2
    Join Date
    Aug 2016
    Posts
    290

    Default

    You're free to script in your environment any way you like. How to make an execution script to run programs sequentially is a topic for a different forum on a different site.

    You could also pack your sequential jobs inside a root job and execute the root job.

  3. #3

    Default

    Hello Sparkles,


    I don't know how to react to your answer, are you saying: "Go to google and don't bother us here with your questions?" or I get wrong?


    I don't think asking this on this forum, where are PDI related questions is wrong. All of us come here to ask for help and professional advices. Anyway, don't worry, I already figured out how to accomplish this idea of dynamic memory assignment to processes.


    Thank you.


    Quote Originally Posted by Sparkles View Post
    You're free to script in your environment any way you like. How to make an execution script to run programs sequentially is a topic for a different forum on a different site.

    You could also pack your sequential jobs inside a root job and execute the root job.

  4. #4
    Join Date
    Aug 2016
    Posts
    290

    Default

    Sorry, can you explain more in detail:

    Do you want to run jobs sequentially inside a root job? Or do you want to make a script to run kitchen sequentially with different jobs as argument? Either way will only run a job once and in sequence. This will not use memory for all jobs at the same time

  5. #5

    Default

    Hi Sparkles,

    Yes, running different jobs with different arguments, in this case the max memory.

    Batch 1:
    Job 1: Catalogs update, run with a 512 mb max memory
    Job 2: Get FTP source files, run with 256 mb max memory
    Job 3: Update all the 'heavy' fact tables, run with 10gb max memory

    Batch 2:
    Job 1: The rest of 'non-heavy' fact tables, run with 4gb max memory
    Job 2: some other things, run with 512 mb max memory

    The idea is to distribute the memory acording the needed for every job / batch.






    Quote Originally Posted by Sparkles View Post
    Sorry, can you explain more in detail:

    Do you want to run jobs sequentially inside a root job? Or do you want to make a script to run kitchen sequentially with different jobs as argument? Either way will only run a job once and in sequence. This will not use memory for all jobs at the same time

  6. #6
    Join Date
    Nov 2013
    Posts
    382

    Default

    Most probably I don't understand the problem ... call command will pause the script until the end of the job, so your proposed syntax seems all you need. Where is the mistery?

  7. #7
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    From what I recall, the kitchen.bat command spawns a handful of other processes, which don't necessarily wait for the command to return.

    You can reverse engineer the PDI scripts, and call the underlying commands.

    Your script, as presented in post#1 looks right to me... what's the overall issue?

  8. #8

    Default

    Hello gutlez,

    At the begining of the thread, I just ask for advice for how to accomplish this idea, now is resolved with the script in the same way as my example.

    Thanks!

    Quote Originally Posted by gutlez View Post
    From what I recall, the kitchen.bat command spawns a handful of other processes, which don't necessarily wait for the command to return.

    You can reverse engineer the PDI scripts, and call the underlying commands.

    Your script, as presented in post#1 looks right to me... what's the overall issue?

  9. #9
    Join Date
    Aug 2016
    Posts
    290

    Default

    Out of curiousity, what langauge is that script in?

  10. #10
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by Sparkles View Post
    Out of curiousity, what langauge is that script in?
    It appears to be MS Batch (AKA .bat file)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.