在执行flume任务的时候报了错,如下:
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.ExceptionInInitializerError
at org.apache.hadoop.tracing.SpanReceiverHost.get(SpanReceiverHost.java:79)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
at java.lang.Runtime.addShutdownHook(Runtime.java:211)
at org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:47)
找了一下,说是将flume的lib下的低版本guava-11.0.2.jar换成Hadoop下的高版本guava-27.0-jre.jar
然后找到本环境下的flume、hadoop找到guava.jar文件发现连个版本相同!?
最后找到的原因是:hdfs上的目标文件目录不存在,需要先创建
[hadoop@master lib]$ hdfs dfs -ls /tmp/flume
[hadoop@master lib]$ cd /usr/local/src/flume/
[hadoop@master flume]$ ll
总用量 136
drwxr-xr-x. 2 hadoop hadoop 62 2月 20 14:31 bin
-rw-r--r--. 1 hadoop hadoop 69856 5月 9 2015 CHANGELOG
drwxr-xr-x. 2 hadoop hadoop 147 2月 20 14:37 conf
-rw-r--r--. 1 hadoop hadoop 6172 5月 9 2015 DEVNOTES
drwxr-xr-x. 10 hadoop hadoop 4096 5月 12 2015 docs
drwxr-xr-x. 2 hadoop hadoop 4096 2月 20 14:31 lib
-rw-r--r--. 1 hadoop hadoop 25903 5月 9 2015 LICENSE
-rw-r--r--. 1 hadoop hadoop 249 5月 9 2015 NOTICE
-rw-r--r--. 1 hadoop hadoop 1779 5月 9 2015 README
-rw-r--r--. 1 hadoop hadoop 1585 5月 9 2015 RELEASE-NOTES
-rw-rw-r--. 1 hadoop hadoop 507 2月 20 14:38 simple-hdfs-flume.conf
drwxr-xr-x. 2 hadoop hadoop 68 2月 20 14:31 tools
[hadoop@master flume]$ flume-ng agent --conf-file simple-hdfs-flume.conf --name a1
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including Hive libraries found via (/usr/local/src/hive) for Hive access
+ exec /usr/local/src/java/bin/java -Xmx20m -cp '/usr/local/src/flume/lib/*:/usr/local/src/hadoop/etc/hadoop:/usr/local/src/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/src/hadoop/share/hadoop/common
......................................
[hadoop@master flume]$ hdfs dfs -ls /tmp/flume #查看hdfs上传输的flume日志文件
Found 67 items
-rw-r--r-- 3 hadoop supergroup 1568 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575970
-rw-r--r-- 3 hadoop supergroup 1643 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575971
-rw-r--r-- 3 hadoop supergroup 1707 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575972
-rw-r--r-- 3 hadoop supergroup 1864 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575973
-rw-r--r-- 3 hadoop supergroup 1732 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575974
-rw-r--r-- 3 hadoop supergroup 1365 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575975
-rw-r--r-- 3 hadoop supergroup 1379 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575976
-rw-r--r-- 3 hadoop supergroup 1376 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575977
-rw-r--r-- 3 hadoop supergroup 1707 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575978
-rw-r--r-- 3 hadoop supergroup 1864 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575979
-rw-r--r-- 3 hadoop supergroup 814 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575980
-rw-r--r-- 3 hadoop supergroup 1665 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575981