We are trying to use the minio-java client that uses okhttp client for uploading object to buckets. Currently the server that we use supports only server authentication and not mutual tls and this mainly means that we have to verify the certificates that the server provides with given CA certificate. For that reason, we did create the following method to simply pass ca certificate file in trustedstore.
private OkHttpClient addCertificates(OkHttpClient httpClient,
Path certificatesDir) throws CertificateException, KeyStoreException, NoSuchAlgorithmException, IOException, UnrecoverableKeyException, KeyManagementException
{
// TODO: remove printing of certificates
String certContents = Files.readString(certificatesDir);
log.debug("certificate contents: {}", certContents);
Collection<? extends Certificate> certificates = null;
try (FileInputStream fis = new FileInputStream(certificatesDir.toFile().getAbsolutePath()))
{
certificates = CertificateFactory.getInstance("X.509").generateCertificates(fis);
}
if (certificates == null || certificates.isEmpty())
{
throw new IllegalArgumentException("expected non-empty set of trusted certificates");
}
char[] password = "password".toCharArray();
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(null, password);
int index = 0;
for (Certificate certificate : certificates)
{
String certificateAlias = Integer.toString(index++);
keyStore.setCertificateEntry(certificateAlias, certificate);
}
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
sslContext.init(null, trustManagers, null);
SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
return httpClient.newBuilder().sslSocketFactory(sslSocketFactory, (X509TrustManager) trustManagers[0]).build();
}
It would be great to help us understand how the minio seperate ca/client certificates with the "enableExternalCertificates" below in MinioClient.java that requires one single file for SSL certificates.
private OkHttpClient enableExternalCertificates(OkHttpClient httpClient, String filename)
throws GeneralSecurityException, IOException {
Collection<? extends Certificate> certificates = null;
try (FileInputStream fis = new FileInputStream(filename)) {
certificates = CertificateFactory.getInstance("X.509").generateCertificates(fis);
}
if (certificates == null || certificates.isEmpty()) {
throw new IllegalArgumentException("expected non-empty set of trusted certificates");
}
char[] password = "password".toCharArray(); // Any password will work.
// Put the certificates a key store.
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
// By convention, 'null' creates an empty key store.
keyStore.load(null, password);
int index = 0;
for (Certificate certificate : certificates) {
String certificateAlias = Integer.toString(index++);
keyStore.setCertificateEntry(certificateAlias, certificate);
}
// Use it to build an X509 trust manager.
KeyManagerFactory keyManagerFactory =
KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keyStore, password);
TrustManagerFactory trustManagerFactory =
TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
final KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
final TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);
SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
return httpClient
.newBuilder()
.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustManagers[0])
.build();
}
Currently, we do have the following exceptions that I am not sure if those are related to certificates.
--- connect exception ---
java.net.ConnectException: Failed to connect to object-storage/10.10.10.10:9000 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:248) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) at okhttp3.RealCall.execute(RealCall.java:92) at io.minio.MinioClient.execute(MinioClient.java:635) at io.minio.MinioClient.getRegion(MinioClient.java:805) at io.minio.MinioClient.execute(MinioClient.java:568) at io.minio.MinioClient.executeHead(MinioClient.java:837) at io.minio.MinioClient.bucketExists(MinioClient.java:2209) at com.ericsson.sc.s3c.S3MinioClientHandler.bucketExists(S3MinioClientHandler.java:201) at com.ericsson.sc.s3c.S3Agent.checkBucket(S3Agent.java:80) at com.ericsson.sc.s3c.S3Agent.lambda$uploadFileToBucket$3(S3Agent.java:57) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at com.ericsson.sc.s3c.S3Agent.lambda$uploadFileToBucket$4(S3Agent.java:51) at io.reactivex.internal.operators.completable.CompletableFromAction.subscribeActual(CompletableFromAction.java:35) at io.reactivex.Completable.subscribe(Completable.java:2309) at io.reactivex.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.drain(FlowableConcatMapCompletable.java:253) at io.reactivex.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.onNext(FlowableConcatMapCompletable.java:118) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableOnBackpressureLatest$BackpressureLatestSubscriber.drain(FlowableOnBackpressureLatest.java:129) at io.reactivex.internal.operators.flowable.FlowableOnBackpressureLatest$BackpressureLatestSubscriber.onNext(FlowableOnBackpressureLatest.java:68) at io.reactivex.internal.operators.flowable.FlowableThrottleLatest$ThrottleLatestSubscriber.drain(FlowableThrottleLatest.java:221) at io.reactivex.internal.operators.flowable.FlowableThrottleLatest$ThrottleLatestSubscriber.onNext(FlowableThrottleLatest.java:119) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.util.NotificationLite.accept(NotificationLite.java:224) at io.reactivex.internal.operators.flowable.FlowableReplay$BoundedReplayBuffer.replay(FlowableReplay.java:855) at io.reactivex.internal.operators.flowable.FlowableReplay$ReplaySubscriber.onNext(FlowableReplay.java:388) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:175) at io.reactivex.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:364) at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.onSuccess(SingleSubscribeOn.java:68) at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:56) at io.reactivex.Single.subscribe(Single.java:3603) at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89) at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578) at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.net.ConnectException: Connection timed out (Connection timed out) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ... 64 common frames omitted
--- socket exception ---
java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at okio.Okio$2.read(Okio.java:140) at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) at okio.RealBufferedSource.read(RealBufferedSource.java:47) at okhttp3.internal.http1.Http1Codec$AbstractSource.read(Http1Codec.java:363) at okhttp3.internal.http1.Http1Codec$UnknownLengthSource.read(Http1Codec.java:507) at okio.Buffer.writeAll(Buffer.java:1135) at okio.RealBufferedSource.readString(RealBufferedSource.java:199) at okhttp3.ResponseBody.string(ResponseBody.java:176) at io.minio.MinioClient.execute(MinioClient.java:663) at io.minio.MinioClient.getRegion(MinioClient.java:805) at io.minio.MinioClient.putObject(MinioClient.java:4584) at io.minio.MinioClient.putObject(MinioClient.java:2726) at io.minio.MinioClient.uploadObject(MinioClient.java:2890) at com.ericsson.sc.s3c.S3MinioClientHandler.uploadFile(S3MinioClientHandler.java:237) at com.ericsson.sc.s3c.S3Agent.lambda$uploadFileToBucket$3(S3Agent.java:60) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at com.ericsson.sc.s3c.S3Agent.lambda$uploadFileToBucket$4(S3Agent.java:51) at io.reactivex.internal.operators.completable.CompletableFromAction.subscribeActual(CompletableFromAction.java:35) at io.reactivex.Completable.subscribe(Completable.java:2309) at io.reactivex.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.drain(FlowableConcatMapCompletable.java:253) at io.reactivex.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.onNext(FlowableConcatMapCompletable.java:118) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableOnBackpressureLatest$BackpressureLatestSubscriber.drain(FlowableOnBackpressureLatest.java:129) at io.reactivex.internal.operators.flowable.FlowableOnBackpressureLatest$BackpressureLatestSubscriber.onNext(FlowableOnBackpressureLatest.java:68) at io.reactivex.internal.operators.flowable.FlowableThrottleLatest$ThrottleLatestSubscriber.drain(FlowableThrottleLatest.java:221) at io.reactivex.internal.operators.flowable.FlowableThrottleLatest$ThrottleLatestSubscriber.onNext(FlowableThrottleLatest.java:119) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.util.NotificationLite.accept(NotificationLite.java:224) at io.reactivex.internal.operators.flowable.FlowableReplay$BoundedReplayBuffer.replay(FlowableReplay.java:855) at io.reactivex.internal.operators.flowable.FlowableReplay$ReplaySubscriber.onNext(FlowableReplay.java:388) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableDoOnEach$DoOnEachSubscriber.onNext(FlowableDoOnEach.java:92) at io.reactivex.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:175) at io.reactivex.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:364) at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.onSuccess(SingleSubscribeOn.java:68) at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:56) at io.reactivex.Single.subscribe(Single.java:3603) at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89) at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578) at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
I think that the first exception after the check if the bucket exists is maybe related with the resolve of the endpoint-hostname "object-storage" but still we do have ip address and all other requests like bucket create have "connection reset" exception maybe due to the first exception because we do still use the same client. However, we did notice that new clients sometime generate the "connection reset" exception directly.
According to minio, the filename used correspond to environmental variable SSL_CERT_FILE and it correspond to path to TLS certificate file (e.g. self-signed TLS certificate). Also, if we do not generate our own custom http-client, MinioClient.Builder.build() will create default HTTP client object with such TLS certificate loaded using enableExternalCertificates() above. As for the http client staff, I guess that this forum is most appropriate.
Any suggestions will be appreciated.
