Spark2.X集群运行模式

rn

启动

先把这三个文件的名字改一下

配置slaves

配置spark-env.sh

export JAVA_HOME=/opt/modules/jdk1.8.0_60
export SCALA_HOME=/opt/modules/scala-2.11.8


SPARK_MASTER_HOST=bigdata-pro02.kfk.com
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1g
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081

SPARK_CONF_DIR=/opt/modules/spark-2.2.0-bin/conf

将spark 配置分发到其他节点并修改每个节点特殊配置

scp -r spark-2.2.0-bin bigdata-pro01.kfk.com:/opt/modules/

scp -r spark-2.2.0-bin bigdata-pro03.kfk.com:/opt/modules/

​​ http://bigdata-pro02.kfk.com:8080/​​

在浏览器打开这个页面

 客户端测试

bin/spark-shell --master spark://bigdata-pro02.kfk.com:7077

执行一个job

 点进去看看

bin/spark-submit --master spark://bigdata-pro02.kfk.com:7077 --deploy-mode cluster /opt/jars/sparkStu.jar  file:///opt/datas/stu.txt

可以看到报错了！！！！

我们应该使用这个模式

 启动一下yarn

​​http://bigdata-pro01.kfk.com:8088/cluster​​

 我们就把HADOOP_CONF_DIR配置近来

 其他两个节点也一样。

再次运行，还是报错了

[kfk@bigdata-pro02 spark-2.2.0-bin]$  bin/spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/03/15 10:13:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/15 10:13:21 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/03/15 10:15:00 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2509)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:909)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:901)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:97)
    at $line3.$read$$iw$$iw.<init>(<console>:15)
    at $line3.$read$$iw.<init>(<console>:42)
    at $line3.$read.<init>(<console>:44)
    at $line3.$read$.<init>(<console>:48)
    at $line3.$read$.<clinit>(<console>)
    at $line3.$eval$.$print$lzycompute(<console>:7)
    at $line3.$eval$.$print(<console>:6)
    at $line3.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
    at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
    at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
    at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
    at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
    at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
    at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
    at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
    at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
    at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
    at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
    at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
    at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:98)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
    at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
    at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
    at org.apache.spark.repl.Main$.doMain(Main.scala:70)
    at org.apache.spark.repl.Main$.main(Main.scala:53)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/03/15 10:15:00 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
18/03/15 10:15:00 WARN MetricsSystem: Stopping a MetricsSystem that is not running
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
  at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2509)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:909)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:901)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:97)
  ... 47 elided
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.

 我们来修改这个配置文件yarn-site.xml

加上这两项

<property>
          <name>yarn.nodemanager.pmem-check-enabled</name>
          <value>false</value>
     </property>
     
      <property>
          <name>yarn.nodemanager.vmem-check-enabled</name>
          <value>false</value>
     </property>

  其他两个节点的yarn-site.xml也是一样，这里我就不多说了。或者是我们把节点2的这个文件分发给另外两个节点也是可以的。

不过分发之前先把yarn停下来

还有一点细节一定要注意，报这个错误其实原因有很多的，不单单是说内存不够的问题，内存不够只是其中一个原因，还有一个细节我们容易漏掉的就jdk版本一定要跟spark-env.sh的一致

尤其要注意hadoop里面的这两个文件

我这里是以其中一个节点来说明，其他两个节点的hadoop配置文件也是这样修改，因为我们之前的hadoop是用jdk1.7版本的，spark改用1.8版本了，所以关于hadoop的所有配置文件有关配置jdk的都某要改成1.8

 我们再次启动yarn

启动spark(由于考虑到spark比较消耗内存，我就把spark的master切换到节点1去了，因为节点1我给他分配了4G内存)

记得修改spark-env.sh文件（3个节点都改）

 进行分组求和

退出

用submit模式跑一下

 可以看到报错了

[kfk@bigdata-pro01 spark-2.2.0-bin]$ bin/spark-submit --class com.spark.test.Test --master yarn --deploy-mode cluster /opt/jars/sparkStu.jar  file:///opt/datas/stu.txt 
18/03/16 12:12:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/16 12:12:40 INFO Client: Requesting a new application from cluster with 3 NodeManagers
18/03/16 12:12:40 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (16384 MB per container)
18/03/16 12:12:40 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/03/16 12:12:40 INFO Client: Setting up container launch context for our AM
18/03/16 12:12:40 INFO Client: Setting up the launch environment for our AM container
18/03/16 12:12:40 INFO Client: Preparing resources for our AM container
18/03/16 12:12:45 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/03/16 12:12:47 INFO Client: Uploading resource file:/tmp/spark-edc616a1-10bf-4175-9d7c-91a2430844f8/__spark_libs__6050155581866596916.zip -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0003/__spark_libs__6050155581866596916.zip
18/03/16 12:12:54 INFO Client: Uploading resource file:/opt/jars/sparkStu.jar -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0003/sparkStu.jar
18/03/16 12:12:54 INFO Client: Uploading resource file:/tmp/spark-edc616a1-10bf-4175-9d7c-91a2430844f8/__spark_conf__6419799297331143395.zip -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0003/__spark_conf__.zip
18/03/16 12:12:54 INFO SecurityManager: Changing view acls to: kfk
18/03/16 12:12:54 INFO SecurityManager: Changing modify acls to: kfk
18/03/16 12:12:54 INFO SecurityManager: Changing view acls groups to: 
18/03/16 12:12:54 INFO SecurityManager: Changing modify acls groups to: 
18/03/16 12:12:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(kfk); groups with view permissions: Set(); users  with modify permissions: Set(kfk); groups with modify permissions: Set()
18/03/16 12:12:54 INFO Client: Submitting application application_1521167375207_0003 to ResourceManager
18/03/16 12:12:55 INFO YarnClientImpl: Submitted application application_1521167375207_0003
18/03/16 12:12:56 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:12:56 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1521173574957
     final status: UNDEFINED
     tracking URL: http://bigdata-pro01.kfk.com:8088/proxy/application_1521167375207_0003/
     user: kfk
18/03/16 12:12:57 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:12:58 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:12:59 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:00 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:01 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:02 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:03 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:04 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:05 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:06 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:07 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:08 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:09 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:10 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:11 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:12 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:13 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:14 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:15 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:16 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:17 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:18 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:19 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:20 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:21 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:22 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:23 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:24 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:25 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:26 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:27 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:28 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:29 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:30 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:31 INFO Client: Application report for application_1521167375207_0003 (state: ACCEPTED)
18/03/16 12:13:32 INFO Client: Application report for application_1521167375207_0003 (state: FAILED)
18/03/16 12:13:32 INFO Client: 
     client token: N/A
     diagnostics: Application application_1521167375207_0003 failed 2 times due to AM Container for appattempt_1521167375207_0003_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://bigdata-pro01.kfk.com:8088/proxy/application_1521167375207_0003/Then, click on links to logs of each attempt.
Diagnostics: File does not exist: hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0003/__spark_libs__6050155581866596916.zip
java.io.FileNotFoundException: File does not exist: hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0003/__spark_libs__6050155581866596916.zip
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1521173574957
     final status: FAILED
     tracking URL: http://bigdata-pro01.kfk.com:8088/cluster/app/application_1521167375207_0003
     user: kfk
Exception in thread "main" org.apache.spark.SparkException: Application application_1521167375207_0003 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1104)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1150)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/03/16 12:13:32 INFO ShutdownHookManager: Shutdown hook called
18/03/16 12:13:32 INFO ShutdownHookManager: Deleting directory /tmp/spark-edc616a1-10bf-4175-9d7c-91a2430844f8
[kfk@bigdata-pro01 spark-2.2.0-bin]$

我们在idea把sparkStu的源码打开

 改一下这里

把包完之后我们把这个包再次上传（为了保险，我把3个节点都上传了，可能我比较SB）

先把原来的包干掉

 现在上传

再跑一次

 可以看到成功了

[kfk@bigdata-pro01 spark-2.2.0-bin]$ bin/spark-submit --class com.spark.test.Test --master yarn --deploy-mode cluster /opt/jars/sparkStu.jar  file:///opt/datas/stu.txt 
18/03/16 16:46:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/16 16:46:10 INFO Client: Requesting a new application from cluster with 3 NodeManagers
18/03/16 16:46:10 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (16384 MB per container)
18/03/16 16:46:10 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/03/16 16:46:10 INFO Client: Setting up container launch context for our AM
18/03/16 16:46:10 INFO Client: Setting up the launch environment for our AM container
18/03/16 16:46:10 INFO Client: Preparing resources for our AM container
18/03/16 16:46:14 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/03/16 16:46:19 INFO Client: Uploading resource file:/tmp/spark-43f281a9-034a-424b-8032-d6d00addfff6/__spark_libs__8012713420631475441.zip -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0004/__spark_libs__8012713420631475441.zip
18/03/16 16:46:29 INFO Client: Uploading resource file:/opt/jars/sparkStu.jar -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0004/sparkStu.jar
18/03/16 16:46:29 INFO Client: Uploading resource file:/tmp/spark-43f281a9-034a-424b-8032-d6d00addfff6/__spark_conf__8776342149712582279.zip -> hdfs://ns/user/kfk/.sparkStaging/application_1521167375207_0004/__spark_conf__.zip
18/03/16 16:46:29 INFO SecurityManager: Changing view acls to: kfk
18/03/16 16:46:29 INFO SecurityManager: Changing modify acls to: kfk
18/03/16 16:46:29 INFO SecurityManager: Changing view acls groups to: 
18/03/16 16:46:29 INFO SecurityManager: Changing modify acls groups to: 
18/03/16 16:46:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(kfk); groups with view permissions: Set(); users  with modify permissions: Set(kfk); groups with modify permissions: Set()
18/03/16 16:46:29 INFO Client: Submitting application application_1521167375207_0004 to ResourceManager
18/03/16 16:46:30 INFO YarnClientImpl: Submitted application application_1521167375207_0004
18/03/16 16:46:31 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:31 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1521189989993
     final status: UNDEFINED
     tracking URL: http://bigdata-pro01.kfk.com:8088/proxy/application_1521167375207_0004/
     user: kfk
18/03/16 16:46:32 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:33 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:35 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:36 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:37 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:38 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:39 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:40 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:41 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:42 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:43 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:44 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:45 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:46 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:47 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:48 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:49 INFO Client: Application report for application_1521167375207_0004 (state: ACCEPTED)
18/03/16 16:46:50 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:50 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.86.152
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1521189989993
     final status: UNDEFINED
     tracking URL: http://bigdata-pro01.kfk.com:8088/proxy/application_1521167375207_0004/
     user: kfk
18/03/16 16:46:51 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:52 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:53 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:55 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:56 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:57 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:58 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:46:59 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:00 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:01 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:02 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:03 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:04 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:05 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:06 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:07 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:08 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:09 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:10 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:11 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:12 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:13 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:14 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:15 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:16 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:17 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:18 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:19 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:20 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:21 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:22 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:23 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:24 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:25 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:26 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:27 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:28 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:29 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:30 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:31 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:32 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:33 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:34 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:35 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:36 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:37 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:38 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:39 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:40 INFO Client: Application report for application_1521167375207_0004 (state: RUNNING)
18/03/16 16:47:41 INFO Client: Application report for application_1521167375207_0004 (state: FINISHED)
18/03/16 16:47:41 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.86.152
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1521189989993
     final status: SUCCEEDED
     tracking URL: http://bigdata-pro01.kfk.com:8088/proxy/application_1521167375207_0004/A
     user: kfk
18/03/16 16:47:41 INFO ShutdownHookManager: Shutdown hook called
18/03/16 16:47:41 INFO ShutdownHookManager: Deleting directory /tmp/spark-43f281a9-034a-424b-8032-d6d00addfff6
[kfk@bigdata-pro01 spark-2.2.0-bin]$

在这里我补充一下，我们能看见终端打印这么多日志，是因为修改了这个文件