如果运行出现以下报错:
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.ClassTag$.Int()Lscala/reflect/ManifestFactory$IntManifest;
是Scala版本不匹配造成,请参见:
Spark3.2教程(前置)关于Spark3.2.0与Scala版本的坑
在IDEA中开发:
package com.alan import org.apache.spark.{SparkConf, SparkContext} object Test1 { def main(args: Array[String]): Unit = { //创建SparkContexta //本地运行 // val conf=new SparkConf().setMaster("local[2]").setAppName("WordCount") //提交运行 val conf = new SparkConf().setAppName("WordCound"); val sc = new SparkContext(conf) //加载文件 val file = sc.textFile("hdfs://hp301:9000/ee/b.txt") //val file=sc.textFile("file:///d:/test/words.txt") //处理 val flatFile = file.flatMap(item => item.split(" ")) val flatFileMap = flatFile.map(item => (item, 1)) val aggCount = flatFileMap.reduceByKey((curr, agg) => curr + agg) //得到结果 aggCount.foreach(item => println(item)) } }
注意:本次是从HDFS中取数据进行计算,如果没有配置Hadoop,可以改成从Linux本地读取。
配置pom.xml中打包插件,为了将依赖都打入包中:
<bulid> <!--指定源码包和测试包的位置--> <sourceDirectory>src/main/scala</sourceDirectory> <plugins> <!-- 该插件用于将 Scala 代码编译成 class 文件 --> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <executions> <execution> <!-- 声明绑定到 maven 的 compile 阶段 --> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> <configuration> <scalaVersion>${scala.version}</scalaVersion> <args> <arg>-target:jvm-1.8</arg> </args> </configuration> </execution> </executions> </plugin> <!--maven打包的插件--> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <configuration> <archive> <manifest> <mainClass>com.alan.Test1</mainClass> </manifest> </archive> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
双击package命令,执行Maven打包:
将打好的包上传到linux的/apps文件夹下,执行命令:
spark-submit --master spark://hp301:7077 /apps/test_s2-1.0-SNAPSHOT-jar-with-dependencies.jar
在Web UI中可以看到结果: