Client cannot authenticate via:[TOKEN, KERBEROS]

Question

I'm using YarnClient to programmatically start a job. The cluster i'm running on has been kerberos-ized.

Normal map reduce jobs submitted via "yarn jar examples.jar wordcount..." work.

The job i'm trying to submit programmatically, does not. I get this error:

14/09/04 21:14:29 ERROR client.ClientService: Error happened during application submit: Application application_1409863263326_0002 failed 2 times due to AM Container for appattempt_1409863263326_0002_000002 exited with exitCode: -1000 due to: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "yarn-c1-n1.clouddev.snaplogic.com/10.184.28.108"; destination host is: "yarn-c1-cdh.clouddev.snaplogic.com":8020; .Failing this attempt.. Failing the application. 14/09/04 21:14:29 ERROR client.YClient: Application submission failed

The code looks something like this:

ClientContext context = createContextFrom(args);
YarnConfiguration configuration = new YarnConfiguration();
YarnClient yarnClient = YarnClient.createYarnClient();
yarnClient.init(configuration);
ClientService client = new ClientService(context, yarnClient, new InstallManager(FileSystem.get(configuration)));
LOG.info(Messages.RUNNING_CLIENT_SERVICE);
boolean result = client.execute();

I had thought that perhaps adding something to the effect of:

yarnClient.getRMDelegationToken(new Text(InetAddress.getLocalHost().getHostAddress()));

Could perhaps assuage my woes, but that doesn't seem to help either. Any help would be greatly appreciated.

score 2 · Answer 1 · edited Dec 17 '16 at 13:11

2

Alright, well after hours and hours and hours we have this figured out. For all following generations of coders, forever plagued by hadoop's lack of documentation:

You must grab the tokens from UserGroupInformation object with a call to get credentials. Then you must set the tokens on the ContainerLaunchContext.

edited Dec 17 '16 at 13:11

Termininja

6,620
12
48
49

answered Sep 16 '14 at 17:24

Eric Fulton

87
1
1
6

10

A code snippet showing how to do this would be really nice. I'm struggling with this issue and it is not obvious how to do what you say. – quux00 Oct 03 '14 at 17:37
1

This is related: https://groups.google.com/a/cloudera.org/forum/#!topic/scm-users/D0qbrngp59g – Robert Metzger Dec 18 '14 at 15:57
2

The link seems broken – zjffdu Dec 19 '20 at 16:06

score 1 · Answer 2 · answered Jul 05 '18 at 12:08

The same error with incompatible hadoop artifact versions.

Working example:

public static final String CONF_CORE_SITE = "/etc/hadoop/conf/core-site.xml";
public static final String CONF_HDFS_SITE = "/etc/hadoop/conf/hdfs-site.xml";

/**
 * Pick the config files from class path
 */
private static Configuration getHdfsConfiguration() throws IOException {
    Configuration configuration = new Configuration();

    configuration.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
    configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());

    File hadoopCoreConfig = new File(CONF_CORE_SITE);
    File hadoopHdfsConfig = new File(CONF_HDFS_SITE);

    if (! hadoopCoreConfig.exists() || ! hadoopHdfsConfig.exists()) {
        throw new FileNotFoundException("Files core-site.xml or hdfs-site.xml are not found. Check /etc/hadoop/conf/ path.");
    }

    configuration.addResource(new Path(hadoopCoreConfig.toURI()));
    configuration.addResource(new Path(hadoopHdfsConfig.toURI()));

    //Use existing security context created by $ kinit
    UserGroupInformation.setConfiguration(configuration);
    UserGroupInformation.loginUserFromSubject(null);
    return configuration;
}

pom.xml

<properties>
  <hadoop.version>2.6.0</hadoop.version>
  <hadoop.release>cdh5.14.2</hadoop.release>
</properties>


<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-core</artifactId>
  <version>${hadoop.version}-mr1-${hadoop.release}</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-hdfs</artifactId>
  <version>${hadoop.version}-${hadoop.release}</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>${hadoop.version}-${hadoop.release}</version>
</dependency>

score 0 · Answer 3 · answered Jul 31 '17 at 19:29

You will also get this error if you are using the actual name node instead of logical URI of HA in any of the hdfs path.

This is because if it finds a namenode uri instead of logical uri then it will create non-HA file system which will try to use simple UGI instead of kerberos UGI.

Client cannot authenticate via:[TOKEN, KERBEROS]

3 Answers3

Linked