openshift v3 online pro volume and memory limit issues

Question

I am trying to run an sonatype/nexus3 on openshift online v3 pro. If I just use the web console to create a new app from image it assigns it only 512Mi and it dies with OOM. It did get created though and logged a lot of java output before it died of out of memory. When using the web console there doesnt appear a way to set the memory on the image. When I try to edited the yaml of the pod it doesn't let me edited the memory limit.

Reading the docs about memory limits it suggests that I can run with this:

oc run nexus333 --image=sonatype/nexus3 --limits=memory=750Mi

Then it doesn't even start. It dies with:

{kubelet ip-172-31-59-148.ec2.internal} Error: Error response from daemon: {"message":"create c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59: mkdir /var/lib/docker/volumes/c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59: permission denied"}

More information from oc get events:

FIRSTSEEN   LASTSEEN   COUNT     NAME                KIND                    SUBOBJECT                     TYPE      REASON                        SOURCE                                    MESSAGE
16m         16m        1         nexus333-1-deploy   Pod                                                   Normal    Scheduled                     {default-scheduler }                      Successfully assigned nexus333-1-deploy to ip-172-31-50-97.ec2.internal
16m         16m        1         nexus333-1-deploy   Pod                     spec.containers{deployment}   Normal    Pulling                       {kubelet ip-172-31-50-97.ec2.internal}    pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
16m         16m        1         nexus333-1-deploy   Pod                     spec.containers{deployment}   Normal    Pulled                        {kubelet ip-172-31-50-97.ec2.internal}    Successfully pulled image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
15m         15m        1         nexus333-1-deploy   Pod                     spec.containers{deployment}   Normal    Created                       {kubelet ip-172-31-50-97.ec2.internal}    Created container
15m         15m        1         nexus333-1-deploy   Pod                     spec.containers{deployment}   Normal    Started                       {kubelet ip-172-31-50-97.ec2.internal}    Started container
15m         15m        1         nexus333-1-rftvd    Pod                                                   Normal    Scheduled                     {default-scheduler }                      Successfully assigned nexus333-1-rftvd to ip-172-31-59-148.ec2.internal
15m         14m        7         nexus333-1-rftvd    Pod                     spec.containers{nexus333}     Normal    Pulling                       {kubelet ip-172-31-59-148.ec2.internal}   pulling image "sonatype/nexus3"
15m         10m        19        nexus333-1-rftvd    Pod                     spec.containers{nexus333}     Normal    Pulled                        {kubelet ip-172-31-59-148.ec2.internal}   Successfully pulled image "sonatype/nexus3"
15m         15m        1         nexus333-1-rftvd    Pod                     spec.containers{nexus333}     Warning   Failed                        {kubelet ip-172-31-59-148.ec2.internal}   Error: Error response from daemon: {"message":"create 3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: mkdir /var/lib/docker/volumes/3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: permission denied"}

I am not sure why if I use the web console I cannot assign more memory. I am not sure why running it with oc run dies with the mkdir error. Can anyone tell me how to run sonatype/nexus3 on openshift online pro?

You wouldn't normally use ``oc run`` to deploy an existing container image in OpenShift. You would use ``oc new-app`` as it creates the deployment config and other bits which keeps it all running. The ``oc run`` command is very low level. When you have the deployment config, you can edit resources for it in web console using the drop down menu on deployment page. — Graham Dumpleton, Oct 24 '17 at 00:30
You may want to work through tutorials at https://learn.openshift.com to get a better feel for how to deploy applications to OpenShift. — Graham Dumpleton, Oct 24 '17 at 00:30
I tried `oc run` is that the documentation has it at https://docs.openshift.com/container-platform/3.6/dev_guide/compute_resources.html#specifying-compute-resources-via-cli. I already have nexus3 running on CDK/minishift as I am reading the docs. — simbo1905, Oct 24 '17 at 06:23
@GrahamDumpleton I edited the question to reference the docs which refer to `oc run`. — simbo1905, Oct 24 '17 at 06:37

tm1701 · Answer 1 · 2018-08-29T15:42:12.637

Looking in the documentation I see that it is a Java VM solution.

When using Java 8, memory usage can be DRAMATICALLY IMPROVED using only the following 2 runtime Java VM options:

... "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap" ...

I just deployed my container (Spring Boot JAR) that consumed over 650 MB RAM. With just these two (new) options RAM consumption dropped to just 270 MB!!!

So, with these 2 runtime settings all OOM's are left far behind! Enjoy!

simbo1905 · Answer 2 · 2017-10-24T14:39:01.280

Okay the mkdir /var/lib/docker/volumes/ permission denied seems to be that the image needs a /nexus-data mount and that is refused. I saw that by deploying from the web console (dies with OOM) but the edit yaml for the created pod to see the generated volume mount.

Creating the image with the following yaml using cat nexus3_pod.ephemeral.yaml | oc create -f - with the volume mount and explicit memory settings the container will now start up:

apiVersion: "v1"
kind: "Pod"
metadata:
  name: "nexus3"
  labels:
    name: "nexus3"
spec:
  containers:
    -
      name: "nexus3"
      resources:
        requests:
          memory: "1200Mi" 
        limits:
          memory: "1200Mi" 
      image: "sonatype/nexus3"
      ports:
        -
          containerPort: 8081
          name: "nexus3"
      volumeMounts:
        - mountPath: /nexus-data
          name: nexus3-1
  volumes:
    - emptyDir: {}
      name: nexus3-1

Notes

The mage sets -Xmx1200m as documented at sonatype/docker-nexus3. So if you assign memory less than 1200Mi it will crash with OOM when the heap grows over the limit. You may as well set requested and max to be the max heap side anything.
When the allocated memory was too low it crashed die just as it was setting up the DB which corrupted the db log which meant it then got in a crash loop "couldn't load 4 byte from 0 byte file" when I recreated it with more memory. It seems that with an emptyDir the files hang around between crash restarts and memory changes (that's documented behaviour I think). I had to recreate a pod with a different name to get a clean emptyDir and assigned memory of 1200Mi to get it to all start.

score 0 · Answer 3 · answered Oct 24 '17 at 01:24

0

You may want to also follow along with the tutorial that is in the OpenShift docs https://docs.openshift.com/online/dev_guide/app_tutorials/maven_tutorial.html

I have had success deploying this in OpenShift Online Pro

answered Oct 24 '17 at 01:24

SteveS

407
2
11

Thanks that had a lot of helpful information but that documentation is for nexus2 `sonatype/nexus` and it's page at https://hub.docker.com/r/sonatype/nexus/ points to `sonatype/nexus3` as the latest version. Looking at the docs the data storage persistent name is different `/sonatype-work` not `/nexus-data` so the commands for mounting pvc will need to be different. In this case we need nuget and raw repositories for `dotnet` core 2.0 which was released may 2017. So we are aiming at the latest nexus version. – simbo1905 Oct 24 '17 at 06:05
I tried the docs nexus2 comes up, I can login, when I click a button to create a new repo it crashes with OOM. Looking at the console it was defaulted to 512Mi so I will have to increase that. – simbo1905 Oct 24 '17 at 14:07
Aha seems that nexus is greedy. nexus3 as I had allocated 1Gi but the JVM options set in the image set the `-Xmx1200m` so if the defaults given to the created pod/container are smaller it will it the limit and be die with OOM. – simbo1905 Oct 24 '17 at 14:31

openshift v3 online pro volume and memory limit issues

3 Answers3