8

I need to implement anti-virus on-access scanning solution for files inside docker containers using open-source software. Clamav On-Access works fine but have some requirements and limitations:

  • require CAP_SYS_ADMIN capability for working inside a container
  • needs to be run per-container, not per-host
  • require 850Mb resident memory for signatures in each running container, even small one

Does this limitation - "fanotify not working for container events when watching from host", really exists or I just misconfigured ClamAV? I have no deep knowledge how fanotify works with namespaces, but it looks like kernel limitation to me.

UPDATE: Are there any workarounds for this limitation? Adding /var/lib/docker/overlay2/container_id/merged is one option, because of dynamic container nature clamd.conf needs to be updated on every container event. But even with added path ClamAV doesn't detect malicious files in the containers.

Running ClamAV per-container creates huge memory overhead, especially for small containers.

Links collection:

mva
  • 384
  • 1
  • 5
  • 12
  • 2
    Could you run the AV on the host, and expose on-host volumes from the containers you want to scan? – halfer Feb 04 '20 at 16:27
  • 1
    @halfer, that's exactly what I'm trying to do now, run one clamd in inotify mode for /var/lib/docker/overlay2 and second clamd in fanotify mode for / at host – mva Feb 05 '20 at 15:50

3 Answers3

4

I have a solution with a patched ClamAV.

  1. Must use ClamAV < 0.102.0 because of the splitting of scanning and detection: detected files can't be scanned because the path is observed from the container point of view
  2. OnAccessMount doesn't work because you have to list each mount in ClamAV config then restart and docker creates mounts on the fly
  3. Must use overlayfs not LVM so ClamAV can access the mount
  4. OnAccessIncludePath doesn't work because the file and folder enumeration method doesn't traverse file systems (doesn't scan beyond mount for path specified)

I was able to get OnAccessIncludePath working with a patch I posted to clamav-devel mailing list: https://lists.gt.net/clamav/devel/77347#77347.

I ended up with one process using fanotify for static mounts and one using inotify to monitor /var/lib/docker ephemeral mounts. Having 2 instances is still much better than 1 per container. I did a fair bit of load testing and have had the patch in production since about the time I mailed the list.

Sophos didn't work for me but I gave up pretty quickly.

user44127
  • 56
  • 1
  • thank you for the answer, works for me, do you have an idea, how your patch can be integrated with more recent clamav versions? – mva Feb 09 '20 at 07:05
  • btw, why do we need two clamav daemons and cannot run just one with watching / recursively? – mva Feb 10 '20 at 15:16
  • 1
    ClamAV 0.102.0 would have to pass info from clamscan to clamd using something other than path (inode entry ID?). – user44127 Feb 11 '20 at 16:06
  • 1
    The OnAccessIncludePath and OnAccessMountPath are mutually exclusive and OnAccessMountPath supersedes OnAccessIncludePath globally per process not by path. OnAccessPrevention only works with OnAccessMountPath (fanotify). Using fanotify is more efficient, doesn't use up inotify watches and is less likely to miss a file. – user44127 Feb 11 '20 at 16:09
  • 1
    looks like the opposite - https://lists.gt.net/clamav/users/74107: From the documentation here: https://github.com/Cisco-Talos/clamav-devel/blob/dev/0.101/docs/UserManual/Usage.md#On-access-Scanning Watch your entire filesystem only using the clamd.conf OnAccessMountPath option. While this will disable on-access prevention, it will avoid potential system lockups caused by fanotify’s blocking functionality. To use OnAccessPrevention, you'll need to use OnAccessIncludePath instead of OnAccessMountPath. – mva Feb 12 '20 at 11:40
  • Right the entire filesystem but each docker container has a different filesystem. You have to tell ClamAV about that filesystem but you won't know the path ahead of time. – user44127 Feb 13 '20 at 15:21
1

Yes, fanotify only monitors events in the mount namespace that it is running in.

Douglas Leeder
  • 52,368
  • 9
  • 94
  • 137
  • thank you for the answer @Douglas Leeder, I've tested Sophos Antivirus as well, it is able to detect viruses in container when running on host. Unfortunately, unable to download it from official site due to some restrictions. – mva Feb 03 '20 at 19:44
0

Now fanotify can monitor events across the filesystem, regardless of the mount namespaces. You need to use the flag FAN_MARK_FILESYSTEM alongwith FAN_MARK_ADD (FAN_MARK_ADD | FAN_MARK_FILESYSTEM). From the fanotify_mark man page here is the snippet:

FAN_MARK_FILESYSTEM (since Linux 4.20)
       Mark the filesystem specified by pathname.  The filesystem
       containing pathname will be marked.  All the contained
       files and directories of the filesystem from any mount
       point will be monitored.
surajd
  • 1,627
  • 13
  • 13