Hitachi Vantara Pentaho Community Forums
Results 1 to 1 of 1

Thread: How to run jobs on carte cluster via http request(hostname:port/kettle/executeJob)?

  1. #1
    Join Date
    Nov 2017
    Posts
    1

    Question How to run jobs on carte cluster via http request(hostname:port/kettle/executeJob)?

    My PDI version is 7.1

    I had set up a carte cluster (1 master and 2 slave2).
    I created 3 "Slave server" and 1 "Kettle cluster schemas" in spoon.
    I created a "Run configuration" and set "Romote", "Location - Clustered".

    I ran the job in spoon with "Environment Type -- Local" and it ran on the cluster successfully(There were connection infos in master and handle infos in slaves).

    But when I ran with kitchen command or carte http request(hostname: port/kettle/executeJob), it ran as standalone (just run in master node).

    It means that I can run jobs on carte cluster in spoon(local), but it not work in kitchen and http request.

    How can I r
    un jobs on carte cluster via http request?

    1. ran in spoon with "Enviroment Type -- Local"

    master output:

    2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
    Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 04:47:10 - sortcluster111 (master) - Dispatching started for transformation [sortcluster111 (master)]
    Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 04:47:10 - output.0 - Connected to database [102] (commit=1000)
    2017/11/28 04:47:10 - input.0 - Finished reading query, closing connection.
    2017/11/28 04:47:10 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
    2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40001], reading from server Dynamic slave [kettleslave02:8083]
    2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40000], reading from server Dynamic slave [kettleslave01:8082]

    2017/11/28 04:47:10 - output.0 - Finished processing (I=47, O=47, R=0, W=47, U=0, E=0)


    slave01 output:
    2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
    Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 04:47:10 - sortcluster111 (cluster1ynamic slave [kettleslave01:8082]) - Dispatching started for transformation [sortcluster111 (cluster1ynamic slave [kettleslave01:8082])]
    2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
    2017/11/28 04:47:10 - sort.0 - Finished processing (I=24, O=0, R=0, W=24, U=0, E=0)

    slave02 output:
    2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
    2017/11/28 04:47:09 - General - Unable to connect to the repository with name 'Mysqlrep'
    2017/11/28 04:47:10 - sortcluster111 (cluster1ynamic slave [kettleslave02:8083]) - Dispatching started for transformation [sortcluster111 (cluster1ynamic slave [kettleslave02:8083])]
    2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
    2017/11/28 04:47:10 - sort.0 - Finished processing (I=23, O=0, R=0, W=23, U=0, E=0)

    2. ran with http request

    http://kettlemaster01:8080/kettle/ex...s1&level=debug

    master output : ( has no output in slaves)
    2017/11/28 22:17:55 - trans1 - Start of job execution
    2017/11/28 22:17:55 - trans1 - exec(0, 0, START.0)
    2017/11/28 22:17:55 - START - Starting job entry
    2017/11/28 22:17:55 - trans1 - trans1
    2017/11/28 22:18:05 - trans1 - Triggering heartbeat signal for trans1 at every 10 seconds
    2017/11/28 22:18:05 - trans1 - Starting entry [SQL]
    2017/11/28 22:18:05 - trans1 - exec(1, 0, SQL.0)
    2017/11/28 22:18:05 - SQL - Starting job entry
    2017/11/28 22:18:05 - 102 - New database connection defined
    Tue Nov 28 22:18:05 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 22:18:05 - SQL - Running SQL :truncate table bddia_statistic_field_tmp;
    2017/11/28 22:18:05 - 102 - launch DDL statement:
    2017/11/28 22:18:05 - 102 - truncate table bddia_statistic_field_tmp
    2017/11/28 22:18:05 - 102 - 1 statement executed
    2017/11/28 22:18:05 - 102 - Connection to database closed!
    2017/11/28 22:18:05 - trans1 - Starting entry [sorttrans]
    2017/11/28 22:18:05 - trans1 - exec(2, 0, sorttrans.0)
    2017/11/28 22:18:05 - sorttrans - Starting job entry
    2017/11/28 22:18:05 - sorttrans - Opening transformation: [file:///root/cluster.ktr]
    2017/11/28 22:18:05 - sorttrans - Loading transformation from XML file [file:///root/cluster.ktr]
    2017/11/28 22:18:05 - sorttrans - Starting transformation...(file=${Internal.Entry.Current.Directory}/cluster.ktr, name=sorttrans, repinfo=null)
    2017/11/28 22:18:05 - sorttrans - Using run configuration [cluster config]
    2017/11/28 22:18:05 - sorttrans - Using legacy execution engine
    2017/11/28 22:18:05 - sortcluster111 - Transformation is pre-loaded.
    2017/11/28 22:18:05 - sortcluster111 - nr of steps to run : 3 , nr of hops : 2
    2017/11/28 22:18:05 - sortcluster111 - Dispatching started for transformation [sortcluster111]
    2017/11/28 22:18:05 - sortcluster111 - Nr of arguments detected:0
    2017/11/28 22:18:05 - sortcluster111 - This is not a replay transformation
    2017/11/28 22:18:05 - sortcluster111 - I found 3 different steps to launch.
    2017/11/28 22:18:05 - sortcluster111 - Allocating rowsets...
    2017/11/28 22:18:05 - sortcluster111 - Allocating rowsets for step 0 --> input
    2017/11/28 22:18:05 - sortcluster111 - prevcopies = 1, nextcopies=1
    2017/11/28 22:18:05 - sortcluster111 - Transformation allocated new rowset [input.0 - sort.0]
    2017/11/28 22:18:05 - sortcluster111 - Allocated 1 rowsets for step 0 --> input
    2017/11/28 22:18:05 - sortcluster111 - Allocating rowsets for step 1 --> sort
    2017/11/28 22:18:05 - sortcluster111 - prevcopies = 1, nextcopies=1
    2017/11/28 22:18:05 - sortcluster111 - Transformation allocated new rowset [sort.0 - output.0]
    2017/11/28 22:18:05 - sortcluster111 - Allocated 2 rowsets for step 1 --> sort
    2017/11/28 22:18:05 - sortcluster111 - Allocating rowsets for step 2 --> output
    2017/11/28 22:18:05 - sortcluster111 - Allocated 2 rowsets for step 2 --> output
    2017/11/28 22:18:05 - sortcluster111 - Allocating Steps & StepData...
    2017/11/28 22:18:05 - sortcluster111 - Transformation is about to allocate step [input] of type [TableInput]
    2017/11/28 22:18:05 - sortcluster111 - Step has nrcopies=1
    2017/11/28 22:18:05 - input.0 - distribution activated
    2017/11/28 22:18:05 - input.0 - Starting allocation of buffers & new threads...
    2017/11/28 22:18:05 - input.0 - Step info: nrinput=0 nroutput=1
    2017/11/28 22:18:05 - input.0 - output rel. is 1:1
    2017/11/28 22:18:05 - input.0 - Found output rowset [input.0 - sort.0]
    2017/11/28 22:18:05 - input.0 - Finished dispatching
    2017/11/28 22:18:05 - sortcluster111 - Transformation has allocated a new step: [input].0
    2017/11/28 22:18:05 - sortcluster111 - Transformation is about to allocate step [sort] of type [SortRows]
    2017/11/28 22:18:05 - sortcluster111 - Step has nrcopies=1
    2017/11/28 22:18:05 - sort.0 - distribution activated
    2017/11/28 22:18:05 - sort.0 - Starting allocation of buffers & new threads...
    2017/11/28 22:18:05 - sort.0 - Step info: nrinput=1 nroutput=1
    2017/11/28 22:18:05 - sort.0 - Got previous step from [sort] #0 --> input
    2017/11/28 22:18:05 - sort.0 - input rel is 1:1
    2017/11/28 22:18:05 - sort.0 - Found input rowset [input.0 - sort.0]
    2017/11/28 22:18:05 - sort.0 - output rel. is 1:1
    2017/11/28 22:18:05 - sort.0 - Found output rowset [sort.0 - output.0]
    2017/11/28 22:18:05 - sort.0 - Finished dispatching
    2017/11/28 22:18:05 - sortcluster111 - Transformation has allocated a new step: [sort].0
    2017/11/28 22:18:05 - sortcluster111 - Transformation is about to allocate step [output] of type [TableOutput]
    2017/11/28 22:18:05 - sortcluster111 - Step has nrcopies=1
    2017/11/28 22:18:05 - output.0 - distribution activated
    2017/11/28 22:18:05 - output.0 - Starting allocation of buffers & new threads...
    2017/11/28 22:18:05 - output.0 - Step info: nrinput=1 nroutput=0
    2017/11/28 22:18:05 - output.0 - Got previous step from [output] #0 --> sort
    2017/11/28 22:18:05 - output.0 - input rel is 1:1
    2017/11/28 22:18:05 - output.0 - Found input rowset [sort.0 - output.0]
    2017/11/28 22:18:05 - output.0 - Finished dispatching
    2017/11/28 22:18:05 - sortcluster111 - Transformation has allocated a new step: [output].0
    2017/11/28 22:18:05 - sortcluster111 - This transformation can be replayed with replay date: 2017/11/28 22:18:05
    2017/11/28 22:18:05 - sortcluster111 - Initialising 3 steps...
    2017/11/28 22:18:05 - input.0 - Released server socket on port 0
    2017/11/28 22:18:05 - sort.0 - Released server socket on port 0
    2017/11/28 22:18:05 - 102 - New database connection defined
    2017/11/28 22:18:05 - output.0 - Released server socket on port 0
    2017/11/28 22:18:05 - 102 - New database connection defined
    Tue Nov 28 22:18:05 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    Tue Nov 28 22:18:05 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 22:18:05 - input.0 - Connected to database...
    2017/11/28 22:18:05 - output.0 - Connected to database [102] (commit=1000)
    2017/11/28 22:18:05 - 102 - Auto commit off
    2017/11/28 22:18:05 - sortcluster111 - Step [input.0] initialized flawlessly.
    2017/11/28 22:18:05 - sortcluster111 - Step [sort.0] initialized flawlessly.
    2017/11/28 22:18:05 - sortcluster111 - Step [output.0] initialized flawlessly.
    2017/11/28 22:18:05 - input.0 - Starting to run...
    2017/11/28 22:18:05 - input.0 - SQL query : SELECT id,alias,parent_id,field_order,type,attr,condition_oper,statistics,unit,statistics_table_id,col,default_oper,col_source,subject_id,sub_default_oper FROM bddia_statistic_field WHERE statistics = 0
    2017/11/28 22:18:05 - sort.0 - Starting to run...
    2017/11/28 22:18:05 - output.0 - Starting to run...
    2017/11/28 22:18:05 - sortcluster111 - Transformation has allocated 3 threads and 2 rowsets.
    2017/11/28 22:18:05 - input.0 - Signaling 'output done' to 1 output rowsets.
    2017/11/28 22:18:05 - input.0 - Finished reading query, closing connection.
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 1
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 2
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 3
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 4
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 5
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 6
    2017/11/28 22:18:05 - 102 - Connection to database closed!
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 7
    2017/11/28 22:18:05 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 8
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 9
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 10
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 11
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 12
    ........
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 46
    2017/11/28 22:18:05 - sort.0 - Start dump buffer to disk, memory = 0%, buffer size: 47
    2017/11/28 22:18:05 - sort.0 - The number of binary string to data type conversions done in this sort block is 0
    2017/11/28 22:18:05 - sort.0 - Signaling 'output done' to 1 output rowsets.
    2017/11/28 22:18:05 - sort.0 - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0)
    2017/11/28 22:18:05 - output.0 - Prepared statement : INSERT INTO bddia_statistic_field_tmp (id, alias, parent_id, field_order, type, attr, condition_oper, statistics, unit, statistics_table_id, col, default_oper, col_source, subject_id, sub_default_oper) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    2017/11/28 22:18:05 - 102 - Commit on database connection [102]
    2017/11/28 22:18:05 - output.0 - Signaling 'output done' to 0 output rowsets.
    2017/11/28 22:18:05 - 102 - Commit on database connection [102]
    2017/11/28 22:18:05 - 102 - Connection to database closed!
    2017/11/28 22:18:05 - output.0 - Finished processing (I=0, O=47, R=47, W=47, U=0, E=0)
    2017/11/28 22:18:05 - sortcluster111 - searching for annotations
    2017/11/28 22:18:05 - sortcluster111 - no annotations found
    2017/11/28 22:18:05 - trans1 - Starting entry [finish]
    2017/11/28 22:18:05 - trans1 - exec(3, 0, finish.0)
    2017/11/28 22:18:05 - finish - Starting job entry
    2017/11/28 22:18:05 - trans1 - Finished job entry [finish] (result=[true])
    2017/11/28 22:18:05 - trans1 - Finished job entry [sorttrans] (result=[true])
    2017/11/28 22:18:05 - trans1 - Finished job entry [SQL] (result=[true])
    2017/11/28 22:18:05 - trans1 - Job execution finished

    3. ran with kitchen

    kitchen.sh -rep=Mysqlrep -user=admin -pass=admin -job trans1

    master output : ( has no output in slaves)
    2017/11/28 04:10:19 - trans1 - Starting entry [sorttrans]
    2017/11/28 04:10:19 - sorttrans - Loading transformation from repository [sortcluster111] in directory [/]
    2017/11/28 04:10:19 - sorttrans - Using run configuration [cluster config]
    2017/11/28 04:10:19 - sorttrans - Using legacy execution engine
    2017/11/28 04:10:19 - sortcluster111 - Dispatching started for transformation [sortcluster111]
    Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2017/11/28 04:10:19 - output.0 - Connected to database [102] (commit=1000)
    2017/11/28 04:10:19 - input.0 - Finished reading query, closing connection.
    2017/11/28 04:10:19 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
    2017/11/28 04:10:19 - sort.0 - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0)
    2017/11/28 04:10:19 - output.0 - Finished processing (I=0, O=47, R=47, W=47, U=0, E=0)
    2017/11/28 04:10:19 - trans1 - Starting entry [finish]
    2017/11/28 04:10:19 - trans1 - Finished job entry [finish] (result=[true])
    2017/11/28 04:10:19 - trans1 - Finished job entry [sorttrans] (result=[true])
    2017/11/28 04:10:19 - trans1 - Finished job entry [SQL] (result=[true])
    2017/11/28 04:10:19 - trans1 - Job execution finished
    2017/11/28 04:10:19 - Kitchen - Finished!
    2017/11/28 04:10:19 - Kitchen - Start=2017/11/28 04:10:00.586, Stop=2017/11/28 04:10:19.739
    2017/11/28 04:10:19 - Kitchen - Processing ended after 19 seconds.

    Regardes
    John
    Last edited by sev_y; 11-28-2017 at 11:50 PM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.