2

I have been using StormCrawler to crawl websites. As https protocol, I set default https protocol in StormCrawler. However, when I crawl some websites I am receiving below exception:

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_131]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_131]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_131]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[?:1.8.0_131]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_131]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_131]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_131]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_131]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_131]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:1.8.0_131]
... 20 more

Is there any mechanism to automatically download certificate and set the crawler and how should I set config of the crawler?

Julien Nioche
  • 4,772
  • 1
  • 22
  • 28
isspek
  • 133
  • 1
  • 11

1 Answers1

0

This problem is not specific to StormCrawler. This answer explains that you can either import the certificates by hand which is not really an option unless you are crawling that site specifically. Another option is to disable certificate validation. This would require modifying the protocol implementation but should be doable.

Have you tried the OKHttp implementation? It might behave differently than the Apache HttClient one. See okhttp wiki.

Julien Nioche
  • 4,772
  • 1
  • 22
  • 28