-1

We run Selenium (JAVA) with headless Chrome. This all takes about 300 MB RAM. VPS has 1 GB RAM but selenium "crushes":

Exception in thread "pool-1-thread-115" org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.

It seems there is not enough free RAM:
$ free -m results in:

      total        used        free      shared  buff/cache   available
  Mem: 1024         940           0         821          83           0

Why is that? Full processes list is below:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2 195028  2748 ?        Ss   Jun20  16:36 /usr/lib/systemd/systemd --system --deseriali
root         2  0.0  0.0      0     0 ?        S    Jun20   0:00 [kthreadd/568406]
root         3  0.0  0.0      0     0 ?        S    Jun20   0:06 [khelper/5684067]
root         4  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root         5  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root         6  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root         7  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root         8  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root         9  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        10  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        11  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        12  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        13  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        14  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        15  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        16  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        17  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        18  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        19  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        20  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        21  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        22  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        23  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        24  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        25  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        26  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        27  0.0  0.0      0     0 ?        S    Jun20   0:00 [rpciod/56840679]
root        28  0.0  0.0      0     0 ?        S    Jun20   0:00 [nfsiod/56840679]
root       110  0.0  0.4  38520  4256 ?        Ss   Jun20  13:38 /usr/lib/systemd/systemd-journald
dbus       165  0.0  0.1  58228  1332 ?        Ss   Jun20  14:56 /usr/bin/dbus-daemon --system --address=syste
root       167  0.0  0.0  74408   732 ?        Ss   Jun20   0:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root       170  0.0  0.1  26704  1488 ?        Ss   Jun20   7:24 /usr/lib/systemd/systemd-logind
root       173  0.0  0.0  74408   732 ?        S    Jun20   0:00 /usr/sbin/saslauthd -m /run/saslauthd -a pam
root       190  0.0  0.1  22728  1104 ?        Ss   Jun20   1:00 /usr/sbin/crond -n
root       191  0.0  0.0   6468   132 tty2     Ss+  Jun20   0:00 /sbin/agetty --noclear tty2 linux
root       192  0.0  0.0  25860   376 ?        Ss   Jun20   0:00 /usr/sbin/atd -f
root       193  0.0  0.0   6468   132 tty1     Ss+  Jun20   0:00 /sbin/agetty --noclear --keep-baud console 11
mysql      345  0.1  5.9 1299292 62404 ?       Sl   16:38   0:01 /usr/sbin/mysqld --daemonize --pid-file=/var/
root       480  0.0  0.0  27120   256 ?        Ss   Jun20   0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run
root       482  0.1  2.2 440080 23080 ?        Ssl  Jun20  87:46 /usr/bin/python -s /usr/bin/fail2ban-server -
nobody     541  0.0  0.2 198844  2636 ?        SLs  Jun20   1:02 proftpd: (accepting connections)
exim       700  0.0  0.1 119948  1828 ?        Ss   Jul03  18:01 /usr/sbin/exim -bd -q1h
root      1145  0.0  1.0 522640 11328 ?        Ss   Jun20   1:55 /usr/sbin/httpd -DFOREGROUND
root      1298  0.0  0.1 112876  1368 ?        Ss   Jun20   2:37 /usr/sbin/sshd -D
root      1306  0.0  0.4  49972  4244 ?        Ss   Jun20   0:00 nginx: master process /usr/sbin/nginx -c /etc
apache    5622  0.0  1.4 538492 15008 ?        S    Jul10   0:38 php-fpm: pool apache
apache    5623  0.0  0.5 341420  5756 ?        S    Jul10   0:00 php-fpm: pool www
apache    5624  0.0  0.5 341420  5756 ?        S    Jul10   0:00 php-fpm: pool www
apache    5625  0.0  0.5 341420  5756 ?        S    Jul10   0:00 php-fpm: pool www
apache    5626  0.0  0.5 341420  5760 ?        S    Jul10   0:00 php-fpm: pool www
apache    5627  0.0  0.5 341420  5760 ?        S    Jul10   0:00 php-fpm: pool www
apache    5651  0.0  0.9  52808  9456 ?        S    Jul10  18:11 nginx: worker process
exim      6036  0.0  0.3 137268  3276 ?        S    16:57   0:00 /usr/sbin/exim -bd -q1h
root      6037  0.0  0.1  51708  1732 pts/0    R+   16:57   0:00 ps axu
apache    6575  0.0  1.6 538352 17420 ?        S    Jul10   0:37 php-fpm: pool apache
apache    6578  0.0  1.4 538424 15336 ?        S    Jul10   0:37 php-fpm: pool apache
igor      9281  0.0  1.5 432016 16392 ?        S    Jul26   0:06 /opt/php74/bin/php-cgi php
root      9399  0.0  0.0  16172   692 ?        Ss   Jun25   0:50 /usr/sbin/dovecot
dovecot   9415  0.0  0.0   9700   500 ?        S    Jun25   2:25 dovecot/anvil
root      9506  0.0  0.0   9832   652 ?        S    Jun25   0:00 dovecot/log
root      9508  0.0  0.1  13040  1516 ?        S    Jun25   0:02 dovecot/config
apache   12135  0.0  0.7 279260  7916 ?        S    Jul26   0:05 /usr/sbin/httpd -DFOREGROUND
igor     12252  0.0  1.5 432016 16392 ?        S    Jul26   0:12 /opt/php74/bin/php-cgi php
igor     12257  0.0  1.5 432016 16388 ?        S    Jul26   0:11 /opt/php74/bin/php-cgi php
root     13747  0.0  0.1  72608  1776 ?        S    09:33   0:00 /usr/sbin/CROND -n
root     13750  0.0  0.0   9516   716 ?        Ss   09:33   0:00 /bin/sh -c    /opt/php71/bin/php -c /usr/loca
root     13752  0.0  0.9 310668 10060 ?        S    09:33   0:15 /opt/php71/bin/php -c /usr/local/mgr5/addon/r
apache   14429  0.0  1.0 522640 10828 ?        S    09:50   0:00 /usr/sbin/httpd -DFOREGROUND
apache   14430  0.0  1.0 522640 10800 ?        S    09:50   0:00 /usr/sbin/httpd -DFOREGROUND
igor     14434  0.0  4.8 521224 51156 ?        S    09:50   0:22 /opt/php74/bin/php-cgi php
apache   14453  0.0  1.0 522640 10812 ?        S    09:51   0:00 /usr/sbin/httpd -DFOREGROUND
apache   14462  0.0  1.0 522640 10812 ?        S    09:51   0:00 /usr/sbin/httpd -DFOREGROUND
apache   14469  0.0  1.0 522640 10812 ?        S    09:51   0:00 /usr/sbin/httpd -DFOREGROUND
igor     14473  0.0  5.0 521236 52760 ?        S    09:51   0:13 /opt/php74/bin/php-cgi php
igor     14475  0.0  4.6 521240 48688 ?        S    09:51   0:17 /opt/php74/bin/php-cgi php
root     15155  0.0  0.5 341428  6060 ?        Ss   Jul01   1:51 php-fpm: master process (/etc/php-fpm.conf)
dovecot  19219  0.0  0.1  34592  1532 ?        S    Jul28   0:09 dovecot/auth
root     19403  0.0  0.3 135560  3260 ?        Ss   Jul23   6:23 /usr/local/mgr5/sbin/ihttpd
apache   19595  0.0  1.6 538444 17576 ?        S    Jul14   0:18 php-fpm: pool apache
root     20388  0.0  1.8 635764 19132 ?        Sl   Jul23   3:44 bin/core core
apache   23214  0.0  1.5 536332 16500 ?        S    Jul13   0:22 php-fpm: pool apache
root     27932  0.0  0.3 155288  3192 ?        Ss   Jul29   0:00 sshd: root@pts/0
root     27943  0.0  0.1  11780  1568 pts/0    Ss   Jul29   0:00 -bash
root     30284  0.0  3.7 1392544 39328 ?       Sl   03:37   0:30 bin/core ispmgr
root     30540  0.0  0.3 153152  3204 ?        Ss   16:31   0:00 sshd: root@pts/1
root     30573  0.0  0.1  11884  1620 pts/1    Ss+  16:31   0:00 -bash
named    31996  0.0  5.4 252516 57548 ?        Ssl  Jul29   0:00 /usr/sbin/named -u named -c /etc/named.conf
root     32492  0.0  0.0  43100   428 ?        Ss   Jul03   0:00 /usr/lib/systemd/systemd-udevd
[root@134-0-117-214 ~]# ps -eo pid,cmd,%mem,%cpu --sort=-%mem | head
  PID CMD                         %MEM %CPU
  345 /usr/sbin/mysqld --daemoniz  5.9  0.1
31996 /usr/sbin/named -u named -c  5.4  0.0
14473 /opt/php74/bin/php-cgi php   5.0  0.0
14434 /opt/php74/bin/php-cgi php   4.8  0.0
14475 /opt/php74/bin/php-cgi php   4.6  0.0
30284 bin/core ispmgr              3.7  0.0
  482 /usr/bin/python -s /usr/bin  2.2  0.1
20388 bin/core core                1.8  0.0
19595 php-fpm: pool apache         1.6  0.0
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Igor Savinkin
  • 5,669
  • 8
  • 37
  • 69

1 Answers1

0

ThreadPoolExecutor Class

As per the documentation of ThreadPoolExecutor Class an ExecutorService that executes each submitted task using one of possibly several pooled threads, normally configured using Executors factory methods. Thread pools address two different problems: they usually provide improved performance when executing large numbers of asynchronous tasks, due to reduced per-task invocation overhead, and they provide a means of bounding and managing the resources, including threads, consumed when executing a collection of tasks. Each ThreadPoolExecutor also maintains some basic statistics, such as the number of completed tasks. To be useful across a wide range of contexts, this class provides many adjustable parameters and extensibility hooks.

When a new task is submitted in method execute(Runnable), and fewer than corePoolSize threads are running, a new thread is created to handle the request, even if other worker threads are idle. If there are more than corePoolSize but less than maximumPoolSize threads running, a new thread will be created only if the queue is full. By setting corePoolSize and maximumPoolSize the same, you create a fixed-size thread pool. By setting maximumPoolSize to an essentially unbounded value such as Integer.MAX_VALUE, you allow the pool to accommodate an arbitrary number of concurrent tasks. Most typically, core and maximum pool sizes are set only upon construction, but they may also be changed dynamically using setCorePoolSize(int) and setMaximumPoolSize(int).

This usecase

Your code trials would have helped to construct a more canonical answer. However, loosely structured ThreadPool can suffer from this error if the program that creates the thread but can't handle an uncaught exception from that thread. Possibly you have implemented FixedThreadPool but haven't implemented it's exception handling properly. As a result, thread may be dying depending on the way you submit the task to ExeuctorService.

Solution

If you are using FixedThreadPool, fixed number of threads have to maintained to avoid thread death. Else if you use execute() instead of submit(), thread may die in case of unhandled exceptions.

You can find a relevant detailed discussion in Chrome crashes after several hours while multiprocessing using Selenium through Python

tl; dr

Some relevant discussions:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352