Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Sqoop Import - Can Transfer into HDFS but Can't Run Hive

  1. #1
    Join Date
    Mar 2016
    Posts
    3

    Default Sqoop Import - Can Transfer into HDFS but Can't Run Hive

    Hello,

    I'm trying to run a Sqoop Import that will import data from MySQL into HDFS, create a Hive table for the imported data, then move the imported data into that Hive table. The data import to HDFS succeeded but the Hive portion fails. Any ideas?

    Thanks!

    Info:
    Data Integration 6.0 on Win7 x64
    Cloudera Hadoop 5.4 on Centos 6.5
    JDK 7u55 on cluster and kettle machine


    Sqoop import:
    --connect "jdbc:mysql://x.x.x.x:3306/medical?defaultFetchSize=500&useCursorFetch=true" --hive-import --hive-table default.medical --password xxxx --query "SELECT patient_key FROM patients WHERE $CONDITIONS" --split-by patient_key --target-dir /user/xxxx/pentahosqoop --username xxxx


    Error:
    Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - Encountered IOException running import job: java.io.IOException: Cannot run program "hive": CreateProcess error=2, The system cannot find the file specified



    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - fs.default.name is deprecated. Instead, use fs.defaultFS
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Running Sqoop version: 1.4.5-cdh5.4.0
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Setting your password on the command-line is insecure. Consider using -P instead.
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Using Hive-specific delimiters for output. You can override
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - delimiters with --fields-terminated-by, etc.
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Preparing to use a MySQL streaming resultset.
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Beginning code generation
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Executing SQL statement: SELECT patient_key FROM patients WHERE (1 = 0)
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Executing SQL statement: SELECT patient_key FROM patients WHERE (1 = 0)
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - Executing SQL statement: SELECT patient_key FROM patients WHERE (1 = 0)
    2016/04/08 12:23:38 - Sqoop Import - 2016/04/08 12:23:38 - $HADOOP_MAPRED_HOME is not set
    2016/04/08 12:23:40 - Sqoop Import - 2016/04/08 12:23:40 - Writing jar file: \tmp\sqoop-xxxxx\compile\5b7182d75b508dc43717262b67afc91c\QueryResult.jar
    2016/04/08 12:23:40 - Sqoop Import - 2016/04/08 12:23:40 - Beginning query import.
    2016/04/08 12:23:40 - Sqoop Import - 2016/04/08 12:23:40 - mapred.jar is deprecated. Instead, use mapreduce.job.jar
    2016/04/08 12:23:41 - Sqoop Import - 2016/04/08 12:23:41 - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    2016/04/08 12:23:41 - Sqoop Import - 2016/04/08 12:23:41 - SQOOP_HOME is unset. May not be able to find all job dependencies.
    2016/04/08 12:23:42 - Sqoop Import - 2016/04/08 12:23:42 - Connecting to ResourceManager at /172.16.200.100:8032
    2016/04/08 12:23:42 - Sqoop Import - 2016/04/08 12:23:42 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2016/04/08 12:23:43 - Sqoop Import - 2016/04/08 12:23:43 - Using read commited transaction isolation
    2016/04/08 12:23:43 - Sqoop Import - 2016/04/08 12:23:43 - BoundingValsQuery: SELECT MIN(patient_key), MAX(patient_key) FROM (SELECT patient_key FROM patients WHERE (1 = 1) ) AS t1
    2016/04/08 12:23:44 - Sqoop Import - 2016/04/08 12:23:44 - number of splits:4
    2016/04/08 12:23:44 - Sqoop Import - 2016/04/08 12:23:44 - fs.default.name is deprecated. Instead, use fs.defaultFS
    2016/04/08 12:23:44 - Sqoop Import - 2016/04/08 12:23:44 - Submitting tokens for job: job_1457971636283_2045
    2016/04/08 12:23:44 - Sqoop Import - 2016/04/08 12:23:44 - Submitted application application_1457971636283_2045
    2016/04/08 12:23:45 - Sqoop Import - 2016/04/08 12:23:45 - The url to track the job: http://xxxxxxx:8088/proxy/application_1457971636283_2045/
    2016/04/08 12:23:45 - Sqoop Import - 2016/04/08 12:23:45 - Running job: job_1457971636283_2045
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Job job_1457971636283_2045 running in uber mode : false
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - map 100% reduce NaN%
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Job job_1457971636283_2045 completed successfully
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Transferred 8.6846 KB in 18.6269 seconds (477.4289 bytes/sec)
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Retrieved 2000 records.
    2016/04/08 12:24:00 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:00 - Fri Apr 08 12:24:00 EDT 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Executing SQL statement: SELECT patient_key FROM patients WHERE (1 = 0)
    2016/04/08 12:24:00 - Sqoop Import - 2016/04/08 12:24:00 - Executing SQL statement: SELECT patient_key FROM patients WHERE (1 = 0)
    2016/04/08 12:24:01 - Sqoop Import - 2016/04/08 12:24:01 - Loading uploaded data into Hive
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - Encountered IOException running import job: java.io.IOException: Cannot run program "hive": CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.Runtime.exec(Runtime.java:617)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.Runtime.exec(Runtime.java:528)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.util.Executor.exec(Executor.java:76)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:386)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:339)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:240)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    2016/04/08 12:24:01 - Sqoop Import - at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    2016/04/08 12:24:01 - Sqoop Import - at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:49)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.common.CommonSqoopShim.runTool(CommonSqoopShim.java:44)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.access$001(ClassPathModifyingSqoopShim.java:41)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim$1.call(ClassPathModifyingSqoopShim.java:80)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim$1.call(ClassPathModifyingSqoopShim.java:77)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.runWithModifiedClassPathProperty(ClassPathModifyingSqoopShim.java:64)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.runTool(ClassPathModifyingSqoopShim.java:77)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.hadoop.shim.cdh54.delegating.DelegatingSqoopShim.runTool(DelegatingSqoopShim.java:41)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.di.job.entries.sqoop.AbstractSqoopJobEntry.executeSqoop(AbstractSqoopJobEntry.java:293)
    2016/04/08 12:24:01 - Sqoop Import - at org.pentaho.di.job.entries.sqoop.AbstractSqoopJobEntry$1.run(AbstractSqoopJobEntry.java:258)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.Thread.run(Thread.java:745)
    2016/04/08 12:24:01 - Sqoop Import - Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.ProcessImpl.create(Native Method)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    2016/04/08 12:24:01 - Sqoop Import - at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
    2016/04/08 12:24:01 - Sqoop Import - ... 23 more
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - Got exception running Sqoop: java.lang.RuntimeException: java.io.IOException: Cannot run program "hive": CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - java.lang.RuntimeException: java.io.IOException: Cannot run program "hive": CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:616)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:49)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.common.CommonSqoopShim.runTool(CommonSqoopShim.java:44)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.access$001(ClassPathModifyingSqoopShim.java:41)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim$1.call(ClassPathModifyingSqoopShim.java:80)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim$1.call(ClassPathModifyingSqoopShim.java:77)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.runWithModifiedClassPathProperty(ClassPathModifyingSqoopShim.java:64)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.ClassPathModifyingSqoopShim.runTool(ClassPathModifyingSqoopShim.java:77)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.hadoop.shim.cdh54.delegating.DelegatingSqoopShim.runTool(DelegatingSqoopShim.java:41)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.di.job.entries.sqoop.AbstractSqoopJobEntry.executeSqoop(AbstractSqoopJobEntry.java:293)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.pentaho.di.job.entries.sqoop.AbstractSqoopJobEntry$1.run(AbstractSqoopJobEntry.java:258)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.Thread.run(Thread.java:745)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - Caused by: java.io.IOException: Cannot run program "hive": CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.Runtime.exec(Runtime.java:617)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.Runtime.exec(Runtime.java:528)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.util.Executor.exec(Executor.java:76)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:386)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:339)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:240)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - ... 15 more
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.ProcessImpl.create(Native Method)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
    2016/04/08 12:24:01 - Sqoop Import - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : 2016/04/08 12:24:01 - ... 23 more

    2016/04/08 12:24:01 - pentahosqooptest - Finished job entry [Sqoop Import] (result=[false])
    2016/04/08 12:24:01 - pentahosqooptest - Job execution finished
    2016/04/08 12:24:01 - Spoon - Job has ended.

  2. #2

    Default

    I have not yet used Sqoop in Pentaho but from the second line it could be that you have not set the sqoop config file correctly?
    $SQOOP_CONF_DIR has not been set in the environment

  3. #3
    Join Date
    Mar 2016
    Posts
    3

    Default

    That seems like a good lead. I don't get that warning when I run sqoop through the CLI, only in Pentaho. Any ideas on how to set $SQOOP_CONF_DIR? Maybe in yarn-site.xml or mapred-site.xml classpath?

  4. #4

    Default

    Like I can see you are running a Cloudera distribution right? I would suggest to make all the config modification from there.

    The actual error above seems to happen because hive cannot be found. Have you check if hive is installed? If yes, where does the path 'echo ${HIVE_HOME}' points to?

    You can set environment variables with the export command (eg. export HIVE_HOME=/hadoop/hive). Variables then should be stored in '/etc/environment' or '/etc/.bashrc'.

    For your case check out the following ones: HADOOP_HOME, SQOOP_HOME and HIVE_HOME

    Here you can find a similar thread that can help you out for the missing hive error.

  5. #5
    Join Date
    Apr 2015
    Posts
    1

    Default

    I have the same isue.
    Could you solve this?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.