云计算

一、Flink-1.13.6源码编译运行

本文主要是介绍一、Flink-1.13.6源码编译运行,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
1、概述

本节演示如何在本地编译、运行 Flink 源码。

技术有限,欢迎各位大佬在评论区批评指正。

2、版本说明
名称 版本
flink 1.13.6
jdk 1.8
Maven 3.2.5
操作系统 Mac
3、编译 Flink 源码

1)从 github 下载 Flink 源码

git clone https://github.com/apache/flink

2)切换到目标分支

git checkout remotes/origin/release-1.13
3)编译源码
mvn clean install -DskipTests -Dfast -Dcheckstyle.skip=true

问题:出现某些依赖找不到

解决:单独下载问题 Jar 包,并 install 到本地仓库

问题:依赖下载慢、依赖下载失败

解决:vpn

4)IDEA启动 Standalone 集群

1.配置准备

a)在 flink-dist 模块下 bin 同级目录 创建 log 目录、lib 目录

b)把 resource 目录中的 flink-conf.yaml 文件拷贝到 conf 目录

c)把 target 里的flink-dist.jar包放到 lib 目录,还有四个 log 相关的 jar 包和 zookeeper 相关的 jar 包

file

2.启动 JobManager

启动类:

flink-runtime 模块下
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint#main

IDEA 启动参数配置:

VM参数:
-Xmx1073741824
-Xms1073741824
-XX:MaxMetaspaceSize=268435456
-Dlog.file=/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/log/flink-standalonesession.log
-Dlog4j.configuration=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlogback.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/logback-console.xml
-classpath
/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-csv-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-json-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-shaded-zookeeper-3.4.14.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table-blink_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-1.2-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-core-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-slf4j-impl-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/target/flink-dist_2.11-1.13.6.jar:::

运行参数:
--configDir /Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf
--executionMode cluster
-D jobmanager.memory.off-heap.size=134217728b
-D jobmanager.memory.jvm-overhead.min=201326592b
-D jobmanager.memory.jvm-metaspace.size=268435456b
-D jobmanager.memory.heap.size=1073741824b
-D jobmanager.memory.jvm-overhead.max=201326592b

3.启动 TaskManager

启动类:

flink-runtime 模块下
org.apache.flink.runtime.taskexecutor.TaskManagerRunner#main

IDEA 启动参数配置:

VM 参数:
-XX:+UseG1GC
-Xmx536870902
-Xms536870902
-XX:MaxDirectMemorySize=268435458
-XX:MaxMetaspaceSize=268435456
-Dlog.file=/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/log/flink-taskexecutor.log
-Dlog4j.configuration=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlogback.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/logback-console.xml
-classpath
/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-csv-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-json-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-shaded-zookeeper-3.4.14.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table-blink_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-1.2-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-core-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-slf4j-impl-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/target/flink-dist_2.11-1.13.6.jar:::

运行参数:
--configDir /Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf
-D taskmanager.memory.network.min=134217730b
-D taskmanager.cpu.cores=1.0
-D taskmanager.memory.task.off-heap.size=0b
-D taskmanager.memory.jvm-metaspace.size=268435456b
-D external-resources=none
-D taskmanager.memory.jvm-overhead.min=201326592b
-D taskmanager.memory.framework.off-heap.size=134217728b
-D taskmanager.memory.network.max=134217730b
-D taskmanager.memory.framework.heap.size=134217728b
-D taskmanager.memory.managed.size=536870920b
-D taskmanager.memory.task.heap.size=402653174b
-D taskmanager.numberOfTaskSlots=1
-D taskmanager.memory.jvm-overhead.max=201326592b

问题:运行代码后,打印日志控制台无输出

解决:增加控制台日志配置,在 flink 项目中有 log4j-console.properties 和 logback-console.xml,在 IDEA 启动参数中传入日志配置

-Dlog.file=/Users/*/Desktop/flink-1.13.6/log/flink-standalonesession.log 
-Dlog4j.configuration=file:/Users/*/Desktop/flink-1.13.6/conf/log4j-console.properties 
-Dlog4j.configurationFile=file:/Users/*/Desktop/flink-1.13.6/conf/log4j-console.properties 
-Dlogback.configurationFile=file:/Users/*/Desktop/flink-1.13.6/conf/logback-console.xml
5)打开 Flink WebUI 进行验证
localhost:8081

file

4、修改源码

1)打包源码

问题:打包源码时,出现某个 Module 找不到

解决:优先打包报错的 Module,并 install 到本地仓库即可

2)修改源码后,重新打包,并运行验证

问题:修改源码后运行无效果,如增加标准输出不显示

解决:每次修改需要重新打包,因为我们运行的是flink-dist模块下的flink-dist_2.11-1.13.6.jar

file

这篇关于一、Flink-1.13.6源码编译运行的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!