1

I have been trying to make Apache Knox work on a kerberised Cloudera Cluster. I downloaded the zip containing Knox and installed it. I also made changes specific to my cluster on the sandbox.xml file. However, when i run cURL command i get 404 not found error. Has someone, successfully managed to run Knox ? If yes, please assist

Shashank S
  • 161
  • 4
  • 18

1 Answers1

0

you can take a look at the Knox documentation here for details on WebHdfs service that Knox supports.

Getting back to your 404 issue, it means that Knox could not find your service, this is most likely because

  1. The service definition was missing from your topology file (looks like you added it)
  2. The service url that was used returned 404.

Also make sure you have enabled WebHdfs service and it is up and running on the port you expect it to refer cloudera documentation. The example ports listed on Knox guide might be different for your installation.

If you can share Knox logs that would be helpful. Also, take a look at Secure Clusters on documentation of how to configure Knox with kerberized clusters.

Good Luck !

Sandeep More
  • 655
  • 1
  • 6
  • 22
  • Hi Sandeep, thanks for your inputs. All the necessary topology files look good as per the documentation. Also, WebHDFS is enabled. The documentation on Knox seems insufficient for a AD + Kerberos integration on a non HDP cluster which makes it really hard for me to make it work.Do you have any document which refers to the installation on a Cloudera cluster? – Shashank S Jun 24 '18 at 23:50
  • Knox documentation should work for any Hadoop distribution, the documentation for secure clusters - https://knox.apache.org/books/knox-1-0-0/user-guide.html#Secure+Clusters should be enough for setting up Knox with Kerberized Hadoop clusters. Can you share Knox logs (gateway.log) this will help us track down the issue, whether it is with a setup or with service etc. You could either share the log snippet here or on the Knox mailing list. Also, if you see Knox documentation is targeted towards a specific distribution can you open a JIRA so we can rectify that. – Sandeep More Jun 25 '18 at 15:05
  • Hi Sandeep, attached is the gateway.log. Please let me know if you find something worth trying. Thanks : – Shashank S Jun 26 '18 at 01:29
  • Hi Sandeep, attached is the gateway.log. Please let me know if you find something worth trying.2018-06-25 12:55:51,797 ERROR knox.gateway (GatewayServer.java:handleCreateDeployment(922)) - Failed to deploy topology azlchdedwpoc0001: org.apache.knox.gateway.deploy.DeploymentException: Failed to contribute provider. Role: authentication Name: kerberos. Please check the topology for errors in name and role and that the provider is on the classpath. – Shashank S Jun 26 '18 at 04:08
  • I see, Knox does not have "kerberos" authentication provider, as a result of which Knox is failing to deploy the topology. Try using the Shiro LDAP authentication example: https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044f811dee40a/gateway-release/home/templates/sample.xml#L22 and start the demo ldap server (for testing only) that is shipped with Knox. here i am assuming that you have already done the secure cluster setup. – Sandeep More Jun 26 '18 at 13:31
  • Hey Sandeep, would you have any idea on this ? WEBHDFS||||access|uri|/gateway/test/webhdfs/v1/?op=GETHOMEDIRECTORY|success|Response status: 401 – Shashank S Jun 27 '18 at 02:12
  • As i can see from other blogs, Knox is not showing me the computed userDN. Do you know what could be the reason? – Shashank S Jun 27 '18 at 04:03
  • Looks like we are getting closed, 401 is better than 404. 1. Start the demo ldap server (/bin/ldap.sh start) 2. use the default user guest to access webhdfs (curl -k -i -u guest:guest-password "https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS” ). – Sandeep More Jun 27 '18 at 13:49
  • Here I am assuming you are using the out-of-the-box shiro ldap authentication. I also think it would be a better idea to move the discussion to Knox mailing list. – Sandeep More Jun 27 '18 at 13:50
  • 1
    Hi Sandeep, I was able to fix this. I have set up Knox running on a Kerberised Cloudera Cluster. Anyone looking for help can let me know. – Shashank S Jun 29 '18 at 07:11
  • Great, good to know ! you might want to answer this question with the steps and accept it so it will be easier for others to find the solution. – Sandeep More Jun 29 '18 at 13:29