Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Sqoop Issue with PDI

  1. #1
    Join Date
    Jan 2015
    Posts
    3

    Default Sqoop Issue with PDI

    I am trying to run this simple sqoop command using PDI and it seems to be hanging when I run it through PDI but works fine when I do it within my cluster. I have been able to copy files using the copy command but I have not been able to use Sqoop for some reason, could someone point me to possible reasons for this?

    sqoop import --connect jdbc:mysql://10.3.57.53:3306/hwscrub --username root --password test --verbose --table account --target-dir hdfs:/10.3.57.54/test34567


    It seems to bet stuck at the following point

    2015/01/26 08:53:47 - Sqoop Import - 2015/01/26 08:53:47 - mapred.jar is deprecated. Instead, use mapreduce.job.jar
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - SQOOP_HOME is unset. May not be able to find all job dependencies.
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - Connecting to ResourceManager at /172.28.104.198:8032
    2015/01/26 08:53:56 - Sqoop Import - 2015/01/26 08:53:56 - Using read commited transaction isolation
    2015/01/26 08:53:56 - Sqoop Import - 2015/01/26 08:53:56 - BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `account`
    2015/01/26 08:53:57 - Sqoop Import - 2015/01/26 08:53:57 - number of splits:2
    2015/01/26 08:53:57 - Sqoop Import - 2015/01/26 08:53:57 - fs.default.name is deprecated. Instead, use fs.defaultFS
    2015/01/26 08:53:58 - Sqoop Import - 2015/01/26 08:53:58 - Submitting tokens for job: job_1422050346255_0009
    2015/01/26 08:53:59 - Sqoop Import - 2015/01/26 08:53:59 - Submitted application application_1422050346255_0009
    2015/01/26 08:53:59 - Sqoop Import - 2015/01/26 08:53:59 - Running job: job_1422050346255_0009

  2. #2
    Join Date
    Jan 2015
    Posts
    3

    Default

    Just to give some more context, I have not been able to get it to run on Talend either. I am using the CDH 5.2 with Yarn shim and I believe I have set the xml files correctly the yarn-site.xml file is is set with the correct IP address for the resource manager and and the mapred-site.xml is with the correct job history address and port (10020). Is there anything else that I should check? I tried to run a Pig job and it is also getting stuck as well.



    Quote Originally Posted by chiragHW View Post
    I am trying to run this simple sqoop command using PDI and it seems to be hanging when I run it through PDI but works fine when I do it within my cluster. I have been able to copy files using the copy command but I have not been able to use Sqoop for some reason, could someone point me to possible reasons for this?

    sqoop import --connect jdbc:mysql://10.3.57.53:3306/hwscrub --username root --password test --verbose --table account --target-dir hdfs:/10.3.57.54/test34567


    It seems to bet stuck at the following point

    2015/01/26 08:53:47 - Sqoop Import - 2015/01/26 08:53:47 - mapred.jar is deprecated. Instead, use mapreduce.job.jar
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - SQOOP_HOME is unset. May not be able to find all job dependencies.
    2015/01/26 08:53:53 - Sqoop Import - 2015/01/26 08:53:53 - Connecting to ResourceManager at /172.28.104.198:8032
    2015/01/26 08:53:56 - Sqoop Import - 2015/01/26 08:53:56 - Using read commited transaction isolation
    2015/01/26 08:53:56 - Sqoop Import - 2015/01/26 08:53:56 - BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `account`
    2015/01/26 08:53:57 - Sqoop Import - 2015/01/26 08:53:57 - number of splits:2
    2015/01/26 08:53:57 - Sqoop Import - 2015/01/26 08:53:57 - fs.default.name is deprecated. Instead, use fs.defaultFS
    2015/01/26 08:53:58 - Sqoop Import - 2015/01/26 08:53:58 - Submitting tokens for job: job_1422050346255_0009
    2015/01/26 08:53:59 - Sqoop Import - 2015/01/26 08:53:59 - Submitted application application_1422050346255_0009
    2015/01/26 08:53:59 - Sqoop Import - 2015/01/26 08:53:59 - Running job: job_1422050346255_0009

  3. #3
    Join Date
    Jan 2015
    Posts
    3

    Default

    I am able to now get it to work. It seems that just editing the yarn-site.xml and mapred-site.xml file was not enough, I actually copied over all of the configuration xml files from my Hadoop cluster into my cdh shim and now it finally works. There are probably so many moving parts with node manager, resource manger, etc... that having the exact configuration file will be the best option it seems.




    Quote Originally Posted by chiragHW View Post
    Just to give some more context, I have not been able to get it to run on Talend either. I am using the CDH 5.2 with Yarn shim and I believe I have set the xml files correctly the yarn-site.xml file is is set with the correct IP address for the resource manager and and the mapred-site.xml is with the correct job history address and port (10020). Is there anything else that I should check? I tried to run a Pig job and it is also getting stuck as well.

  4. #4

    Default

    Hi chiragHW,

    I recently ran Sqoop using Pentaho DI 5.4 and it hangs. Sqoop runs in cmd prompt outside. I am using HDP 2.2 installed in Windows Server 2012. I believe the issue is with the yarn.application.classpath and I tried to map all the paths mentioned with the corresponding ones in Windows, but no luck. I tried copying all the configuration files from cluster to hdp shim(from your comment), but that did not do the trick. Can you suggest a solution for me?

    Thanks.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.