Sqoop导入到hdfs

【自取】最近整理的,有需要可以领取学习:

1.注意win下直接复制进linux 改一下–等

csharp;gutter:true; sqoop-list-databases --connect jdbc:mysql://122.206.79.212:3306/ --username root -P</p> <pre><code> 先看一下是什么数据库,发现有些数据库,可以查询数据库就可以导入,很奇怪。<details><summary>[En]</summary>First take a look at what database, found that some databases, can query the database can be imported, very strange.</details> 2.导入到hdfs ;gutter:true;
sqoop import –connect jdbc:mysql://122.206.79.212:3306/dating –username root –password 123456 –table t_rec_top –driver com.mysql.jdbc.Driver

数据库端口号、帐户名、密码,那张表不需要添加驱动程序,那么它就不指定导入到HDFS的位置,必须有一个默认位置。

[En]

The database port number, account name, password, that table does not need to add a driver, then it does not specify where to import to hdfs, there must be a default location.

您可以看到,只有映射任务,而没有减去任务

[En]

You can see that there are only map tasks but no reduce tasks

csharp;gutter:true; Warning: /home/hxsyl/Spark_Relvant/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail.</p> <p>Please set $HCAT_HOME to the root of your HCatalog installation.</p> <p>Warning: /home/hxsyl/Spark_Relvant/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail.</p> <p>Please set $ACCUMULO_HOME to the root of your Accumulo installation.</p> <p>17/03/15 11:05:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 17/03/15 11:05:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.</p> <p>17/03/15 11:05:12 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.</p> <p>17/03/15 11:05:12 INFO manager.SqlManager: Using default fetchSize of 1000 17/03/15 11:05:12 INFO tool.CodeGenTool: Beginning code generation 17/03/15 11:05:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.<em> FROM t_rec_top AS t WHERE 1=0 17/03/15 11:05:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.</em> FROM t_rec_top AS t WHERE 1=0 17/03/15 11:05:13 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hxsyl/Spark_Relvant/hadoop-2.6.4/share/hadoop/mapreduce Note: /tmp/sqoop-hxsyl/compile/ddeeb02cdbd25cddc2662317b89c80f1/t_rec_top.java uses or overrides a deprecated API.</p> <p>Note: Recompile with -Xlint:deprecation for details.</p> <p>17/03/15 11:05:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hxsyl/compile/ddeeb02cdbd25cddc2662317b89c80f1/t_rec_top.jar 17/03/15 11:05:18 INFO mapreduce.ImportJobBase: Beginning import of t_rec_top SLF4J: Class path contains multiple SLF4J bindings.</p> <p>SLF4J: Found binding in [jar:file:/home/hxsyl/Spark_Relvant/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hxsyl/Spark_Relvant/hbase-1.2.4/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.</p> <p>SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/03/15 11:05:19 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 17/03/15 11:05:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM t_rec_top AS t WHERE 1=0 17/03/15 11:05:21 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 17/03/15 11:05:21 INFO client.RMProxy: Connecting to ResourceManager at CentOSMaster/192.168.58.180:8032 17/03/15 11:05:28 INFO db.DBInputFormat: Using read commited transaction isolation 17/03/15 11:05:28 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(id), MAX(id) FROM t_rec_top 17/03/15 11:05:28 INFO mapreduce.JobSubmitter: number of splits:1 17/03/15 11:05:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489547007191_0001 17/03/15 11:05:30 INFO impl.YarnClientImpl: Submitted application application_1489547007191_0001 17/03/15 11:05:31 INFO mapreduce.Job: The url to track the job: http://CentOSMaster:8088/proxy/application_1489547007191_0001/ 17/03/15 11:05:31 INFO mapreduce.Job: Running job: job_1489547007191_0001 17/03/15 11:05:48 INFO mapreduce.Job: Job job_1489547007191_0001 running in uber mode : false 17/03/15 11:05:48 INFO mapreduce.Job: map 0% reduce 0% 17/03/15 11:06:06 INFO mapreduce.Job: map 100% reduce 0% 17/03/15 11:06:07 INFO mapreduce.Job: Job job_1489547007191_0001 completed successfully 17/03/15 11:06:07 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=127058 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=99 HDFS: Number of bytes written=21 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=13150 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=13150 Total vcore-milliseconds taken by all map tasks=13150 Total megabyte-milliseconds taken by all map tasks=13465600 Map-Reduce Framework Map input records=1 Map output records=1 Input split bytes=99 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=183 CPU time spent (ms)=1200 Physical memory (bytes) snapshot=107761664 Virtual memory (bytes) snapshot=2069635072 Total committed heap usage (bytes)=30474240 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=21 17/03/15 11:06:07 INFO mapreduce.ImportJobBase: Transferred 21 bytes in 46.7701 seconds (0.449 bytes/sec) 17/03/15 11:06:07 INFO mapreduce.ImportJobBase: Retrieved 1 records.</p> <pre><code> ![](https://johngo-pic.oss-cn-beijing.aliyuncs.com/articles/rewrite/387014-20170315111657166-1003045547.png) ![](https://johngo-pic.oss-cn-beijing.aliyuncs.com/articles/rewrite/387014-20170315111819557-1527999495.png) 创建一个user/yonhumig的目录,其中t_rec_top里就是我们的数据,不过没有标头,可以看出只是以m,表示map任务就结束了 ![](https://johngo-pic.oss-cn-beijing.aliyuncs.com/articles/rewrite/387014-20170315112009370-1759948483.png) wc00是配置文件 ;collapse:true;;gutter:true;
"AS 1
"License"); 1
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. 1
(the 1
–> 3
2.0 1

Original: https://www.cnblogs.com/hxsyl/p/6553033.html
Author: 加拿大小哥哥
Title: Sqoop导入到hdfs

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/6895/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

发表回复

登录后才能评论
免费咨询
免费咨询
扫码关注
扫码关注
联系站长

站长Johngo!

大数据和算法重度研究者!

持续产出大数据、算法、LeetCode干货,以及业界好资源!

2022012703491714

微信来撩,免费咨询:xiaozhu_tec

分享本页
返回顶部