How to formulate curl -XHEAD request in java

Question

I am using elasticsearch Java API and trying to make a curl call in order to find out whether a document exists in my index or not. This is how it is done in the command line. As far as I can tell from the posts here, I should either use HttpURLConnection java class or apache httpclient to send the curl request in java. My request should be like :

curl -i -XHEAD http://localhost:9200/indexName/mappingName/docID.

There are actually many questions on how to send curl requests via java, but the answers are not that explanatory - thus I am not sure how to configure the request parameters for the curl head request. So far, I have reproduced this answer from Ashay and it doesn't work.

Does anybody send curl calls in elasticsearch's java API and can explain how to do it?

Here is my code and the error I get is "java.net.MalformedURLException: no protocol"

import org.apache.commons.codec.binary.Base64;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;

String encodedURL = URLEncoder.encode(new StringBuilder()
    .append("http://").append(serverName).append(":9200/") // elasticsearch port
    .append(indexName).append("/").append(mappingName).append("/")
    .append(url).toString(), "UTF-8"); // docID is a url
System.out.print("encodedURL : " + encodedURL + "\n");

URL url = new URL(new StringBuilder().append(encodedURL).toString());
System.out.print("url "+ url.toString() + "\n");

HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("X-Requested-With", "Curl");
connection.setRequestMethod("HEAD");

String userpass = new StringBuilder().append(username).append(":").append(password).toString();
String basicAuth = new StringBuilder().append("Basic ").append(new String(new Base64().encode(userpass.getBytes()))).toString();
connection.setRequestProperty("Authorization", basicAuth);
String inputLine;
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));

P.S. the document ids of this index are urls and that is why I need to encode them. On the other hand, I am not sure whether I should encode the full http request or not.

SubOptimal · Answer 1 · 2016-05-07T09:15:01.597

Following snippet could be a starting point.

String serverName = "localhost";
String indexName = "index_name";
String mappingName = "mapping_name";
String docId = "FooBarId";

String username = "JohnDoe";
String password = "secret";

String requestURL = String.format("http://%s:9200/%s/%s/%s",
        serverName,
        indexName,
        mappingName,
        docId
);
System.out.println("requestURL: " + requestURL);

URL url = new URL(requestURL);
System.out.println("URL: " + url);

HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("X-Requested-With", "Curl");
connection.setRequestMethod("HEAD");

String credentials = username + ":" + password;

Base64.Encoder encoder = Base64.getEncoder();
String basicAuth = "Basic " + encoder.encodeToString(credentials.getBytes());

connection.setRequestProperty("Authorization", basicAuth);

connection.getHeaderFields()
        .entrySet()
        .forEach((Entry<String, List<String>> t) -> {
            System.out.printf("%-20s : %s%n", t.getKey(), t.getValue());
        });

Using requestURL = "http://localhost:9200"; with a default elasticsearch installation would return

requestURL: http://localhost:9200
URL: http://localhost:9200
null                 : [HTTP/1.1 200 OK]
Content-Length       : [0]
Content-Type         : [text/plain; charset=UTF-8]

add Maybe you can try something similar to following steps. Amend them to your needs. Maybe you can skip the first step.

index something

curl -XPUT "http://localhost:9200/books/book/1" -d'
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}'

query it from commandline

curl -X GET http://localhost:9200/books/book/1

output

{"_index":"books","_type":"book","_id":"1","_version":1,"found":true,"_source":
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}}

query using the above Java snippet

String serverName = "localhost";
String indexName = "books";
String mappingName = "book";
String docId = "1";
String requestURL = String.format("http://%s:9200/%s/%s/%s",
        serverName,
        indexName,
        mappingName,
        docId
);
System.out.println("requestURL: " + requestURL);
URL url = new URL(requestURL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.getHeaderFields()
        .entrySet()
        .forEach((Entry<String, List<String>> t) -> {
            System.out.printf("%-20s : %s%n", t.getKey(), t.getValue());
        });
try (InputStream inputStream = connection.getInputStream()) {
    for (int i = inputStream.read(); i > -1; i = inputStream.read()) {
        System.out.print((char) i);
    }
}

output

requestURL: http://localhost:9200/books/book/1
null                 : [HTTP/1.1 200 OK]
Content-Length       : [184]
Content-Type         : [application/json; charset=UTF-8]
{"_index":"books","_type":"book","_id":"1","_version":1,"found":true,"_source":
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}}

The example use a default elasticsearch installation.

Depending on what you really want to achieve. You might be better use the elasticsearch TransportClient.

import java.net.InetAddress;
import org.elasticsearch.action.get.GetRequestBuilder;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.transport.InetSocketTransportAddress;

public class GetDemo {

    public static void main(String[] args) throws Exception {
        InetAddress hostAddr = InetAddress.getByName("localhost");
        InetSocketTransportAddress socketAddr =
                new InetSocketTransportAddress(hostAddr, 9300);
        try (Client client = TransportClient.builder().build()
                .addTransportAddress(socketAddr)) {
            GetRequestBuilder request = client.prepareGet("books", "book", "1");
            GetResponse response = request.execute().actionGet();
            response.getSource()
                    .forEach((k, v) -> System.out.printf("%-6s: %s%n", k, v));
        }
    }
}

output

...
year  : 1978
author: Douglas Adams
title : The Hitchhikers Guide to the Galaxy

Thanks for the reply. Your example code yields a "null=[HTTP/1.1 400 Bad Request] Content-Length=[145] Content-Type=[text/plain; charset=UTF-8]" error for all docIds. The only changes I made were to replace localhost with our server name and also use "Base64.encodeBase64(credentials.getBytes());" because Base64.Encoder is private — KLaz, May 04 '16 at 13:05
@KonstantinaLazaridou [Base64.Encoder](http://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html) is a `public static class` in Java 8. Have you checked your generated `url`and the `basicAuth`? As `HTTP 400` is for `The request could not be understood by the server due to malformed syntax.` — SubOptimal, May 04 '16 at 13:29
yes, the url seems fine and the user info are also correct. Is it possible that the server uses different encryption system than Base64? I also commended the authorization property (in case it was not necessary), but still the same error. I also don't know whether `.setRequestProperty("X-Requested-With", "Curl");` is syntactically correct. — KLaz, May 04 '16 at 20:00
@Try to break down the problem. **1)** Enter the url you want to access in a browser. **a)** Is the url valid. **b)** Do you get a request pop-up for user/password. **2)** If the url is working assign it as a fixed string to `requestURL` in the above example. Another question is your curl example working without an user/password? If so. Why do you believe you need it in the java code? — SubOptimal, May 07 '16 at 08:03

How to formulate curl -XHEAD request in java

1 Answers1