本文主要是介绍自定义聚合函数(统计每种行为的触发次数排名前三的商品id),对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
package SparkSQL.fun.project
import org.apache.spark.SparkConf
import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types.{DataType, DataTypes, StructField, StructType}
import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
/**
* 统计每种行为的触发次数排名前三的商品id
*/
object BehaviorCode2 {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setAppName("project01").setMaster("local[*]")
val session = SparkSession.builder().config(sparkConf).getOrCreate()
val map = Map("mode"->"dropMalformed","inferSchema"->"true")
val frame = session.read.options(map).csv("G:\\shixunworkspace\\sparkcode\\src\\main\\java\\SparkSQL\\fun\\project\\b.csv")
// "userId", "goodsId", "categoryId", "behavior", "time"
import session.implicits._
val frame1: Dataset[UserBehaviorBean] = frame.map(row => {
UserBehaviorBean(row.getInt(0), row.getInt(1),
row.getInt(2), row.getString(3), row.getInt(4))
})
val frame3 = frame1.toDF("userId", "goodsId", "categoryId", "behavior", "time")
frame3.createTempView("tmp")
val frame2 = session.sql("select behavior, goodsId, count(*) count from tmp group by behavior, goodsId")
frame2.show()
frame2.createTempView("tmp1")
val frame4 = session.sql("select behavior, goodsId, count, row_number() over(partition by behavior, goodsId order by count) rn from tmp1")
frame4.show()
frame4.createTempView("temp2")
val frame5 = session.sql("select behavior, goodsId, count, rn from temp2 where rn <= 3")
frame5.show()
session.stop()
}
}
这篇关于自定义聚合函数(统计每种行为的触发次数排名前三的商品id)的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!