Spark_rdd_Source-CFANZ编程社区

文章目录

1.From Memory
2.From File
3.From File

1.From Memory

object RDDTest01 {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setAppName("rdd").setMaster("local")
    val sc = new SparkContext(sparkConf)

    val rdd1 = sc.parallelize(Seq(1, 2, 3, 4))
    val rdd2 = sc.makeRDD(Seq(1, 2, 3, 4))

    rdd2.collect().foreach(println)
    rdd1.collect().foreach(print)

    sc.stop()
  }

}

Spark_rdd_Source_spark
这里的makeRDD和parallelize没有区别, make底层用的就是parallelize函数

2.From File

object RDDTest02 {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setAppName("rdd").setMaster("local")
    val sc = new SparkContext(sparkConf)

    val value1 = sc.textFile("E:\\atguigu05\\spark\\spark-core\\src\\main\\resources\\1.txt")

    value1.collect().foreach(println)

    sc.stop()
  }

}

Spark_rdd_Source_Spark_02

3.From File

object RDDTest03 {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setAppName("rdd").setMaster("local")
    val sc = new SparkContext(sparkConf)

    // 路径可以为目录
    sc.textFile("data").collect().foreach(println)
    sc.wholeTextFiles("data").collect().foreach(println)


    sc.stop()
  }

}

Spark_rdd_Source_Spark_03
第二个方法返回了完整路径