背景
当我在使用IDEA本地连接服务器spark服务时,虽然拷贝了hive-site.xml,提供了相应的依赖,仍然报错:org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism
依赖如下:
<dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>3.1.2</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>3.1.2</version> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>8.0.25</version> </dependency> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>3.1.2</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>3.1.2</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-auth</artifactId> <version>3.2.0</version> </dependency> </dependencies>
分析
在远程访问hadoop时,需要提供认证信息
https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html
https://blog.csdn.net/csr_hema/article/details/8147590
措施
在Maven添加如下依赖:
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-auth</artifactId> <version>3.2.0</version> </dependency>