Read timed out on Spring RestTemplate call

Question

I have an API which returns a json it is of type GET method. Since it is GET, when I open the URL in browser it works fine and renders the json, however, upon using RestTemplate to retrieve the json it fails.

Could you please provide a way to read below API.

API URL: https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY

Spring Boot Rest Template Call:

final String uri = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY";
RestTemplate restTemplate = new RestTemplate();
Map result = restTemplate.getForObject(uri, Map.class);

Error:

java.net.SocketTimeoutException: Read timed out
    at java.base/java.net.SocketInputStream.socketRead0(Native Method) ~[na:na]
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[na:na]
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) ~[na:na]
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823) ~[na:na]
    at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[na:na]
    at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[na:na]
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) ~[na:na]

Answer 1

You could use the WebClient from webflux instead:

- Add the dependency

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

- Create the POJO

@Data
public class Root {
    private Records records;
    private Filtered filtered;

    @Data
    public static class PE {
        private int strikePrice;
        private String expiryDate;
        private String underlying;
        private String identifier;
        private int openInterest;
        private int changeinOpenInterest;
        private double pchangeinOpenInterest;
        private int totalTradedVolume;
        private double impliedVolatility;
        private double lastPrice;
        private double change;
        private double pChange;
        private int totalBuyQuantity;
        private int totalSellQuantity;
        private int bidQty;
        private double bidprice;
        private int askQty;
        private double askPrice;
        private double underlyingValue;
    }

    @Data
    public static class CE {
        private int strikePrice;
        private String expiryDate;
        private String underlying;
        private String identifier;
        private int openInterest;
        private int changeinOpenInterest;
        private int pchangeinOpenInterest;
        private int totalTradedVolume;
        private int impliedVolatility;
        private int lastPrice;
        private double change;
        private double pChange;
        private int totalBuyQuantity;
        private int totalSellQuantity;
        private int bidQty;
        private double bidprice;
        private int askQty;
        private double askPrice;
        private double underlyingValue;
    }

    @Data
    public static class Datum {
        private int strikePrice;
        private String expiryDate;
        private PE PE;
        private CE CE;
    }

    @Data
    public static class Records {
        private List<String> expiryDates;
        private List<Datum> data;
        private String timestamp;
        private double underlyingValue;
        private List<Integer> strikePrices;
    }

    @Data
    public static class Filtered {
        //TODO
    }
}

- Let WebClient make the call

@SpringBootApplication
public class MultipleConfigurationPropertiesApplication {

    public static void main(String[] args) {
        SpringApplication.run(MultipleConfigurationPropertiesApplication.class, args);
    }

    @Bean
    CommandLineRunner commandLineRunner() {
        return args -> {
            WebClient client = WebClient.builder()
                    .baseUrl("https://www.nseindia.com")
                    .exchangeStrategies(ExchangeStrategies.builder()
                            .codecs(configurer -> configurer
                                    .defaultCodecs()
                                    .maxInMemorySize(16 * 1024 * 1024))
                            .build())
                    .build();

            Mono<Root> result = client.get()
                    .uri("/api/option-chain-indices?symbol=NIFTY").accept(MediaType.APPLICATION_JSON)
                    .retrieve()
                    .bodyToMono(Root.class);

            System.out.println(result.block());
        };
    }
}

You have to tune the buffer size as the request result is huge.

Also, you might want to consider using non-blocking/stream solution to handle this.

Answer 2

I tested the API GET request in a browser moments ago, and the results are highly nested. You are attempting to retrieve the results into a Map object, but you are not specifying the types for Key or Value, the latter of which is likely a Map or List itself.

First, determine if you can retrieve the raw JSON results into a String.

ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);

If this works, then you will know that the GET request is working via RestTemplate. If not, you can troubleshoot the default timeout settings and adjust based on how long you observe the network response to take in the browser, for instance.

Assuming you can retrieve the raw JSON results into a String, then your next step will be to analyze the "shape" of the JSON data returned. If you want the JSON results converted into a POJO, you will have to create a class (or set of classes) that match the "shape" of the JSON data.

records:
    
  expiryDates   […]
  data  […]
  timestamp "31-Aug-2020 15:30:00"
  underlyingValue   11387.5
  strikePrices  […]

filtered:   
  data  […]
  CE    
    totOI   367314
    totVol  4988131
  PE    
    totOI   261696
    totVol  5501580

Answer 3

Read timeouts like this occur when you reach the max period of inactivity between consecutive data packets. You'll have to provide a read timeout configured ClientHttpRequestFactory to your RestTemplate when you initialize it. Here is one example of doing this.

You may also want to consider making your RestTemplate into a Bean if you have to make more of these long running requests. This way you don't have to configure this again each time.