C/C++教程

17、【SparkSQL】org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism异常

本文主要是介绍17、【SparkSQL】org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism异常,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
  • 背景
    当我在使用IDEA本地连接服务器spark服务时,虽然拷贝了hive-site.xml,提供了相应的依赖,仍然报错:org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism
    依赖如下:

      <dependencies>
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-core_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-sql_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>mysql</groupId>
              <artifactId>mysql-connector-java</artifactId>
              <version>8.0.25</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.hive</groupId>
              <artifactId>hive-exec</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-hive_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-auth</artifactId>
              <version>3.2.0</version>
          </dependency>
      </dependencies>
    
    
  • 分析
    在远程访问hadoop时,需要提供认证信息

    https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html
    https://blog.csdn.net/csr_hema/article/details/8147590

  • 措施
    在Maven添加如下依赖:

            <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-auth</artifactId>
              <version>3.2.0</version>
          </dependency>
    
这篇关于17、【SparkSQL】org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism异常的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!