Our fist example is an easy one: detecting an attacker running an interactive shell in any of your containers. This alert is included in the default rule set. Let's trigger it first and then we can look the rule definition.

Run any container on your Docker host, Nginx for example:

docker run -d -P --name example1 nginx

docker ps

Now spawn an interactive shell:

docker exec -it example1 bash

You can play around a little if you want, then execute exit to leave the container shell.

If we tail the log file with tail /var/log/falco_events.log we should be able to read:

   17:13:24.357351845: Notice A shell was spawned in a container with an attached terminal (user=root example1 (id=604aa46610dd) shell=bash parent=<NA> cmdline=bash terminal=34816)

This is the specific /etc/falco/falco_rules.yaml rule that fired the event:

   - rule: Terminal shell in container
     desc: A shell was spawned by a program in a container with an attached terminal.
     condition: >
       spawned_process and container
       and shell_procs and proc.tty != 0
     output: "A shell was spawned in a container with an attached terminal (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty)"
     priority: NOTICE
     tags: [container, shell]

This is a rather complex rule, don't worry if you don't fully understand every section at this moment but we can identify a rule name, description, some trigger conditions, event output with some context-aware variables provided by Falco like %proc.name or %container.info, the priority and some tags.

Docker best practices recommend running just one process per container. This has a significant security impact because you know exactly which processes you expect to see running.

You know that your Nginx containers should only be executing the nginx process. Anything else is an indicator of an anomaly.

Let's download a new version of the configuration file for this example:

sudo -s cd /etc/falco curl https://raw.githubusercontent.com/katacoda-scenarios/sysdig-scenarios/master/sysdig-falco/assets/falco_rules_step3.yaml -o falco_rules.yaml

Now, pay attention to the following rule in the file:

   # Our nginx containers should only be running the 'nginx' process
   - rule: Unauthorized process on nginx containers
     desc: There is a process running in the nginx container that is not described in the template
     condition: spawned_process and container and container.image startswith nginx and not proc.name in (nginx)
     output: Unauthorized process (%proc.cmdline) running in (%container.id)
     priority: WARNING

Let's dissect the triggering conditions for this rule:

• spawned_process (a macro: a new process was executed (execve-ed) succesfully)
• container (a macro: the namespace where it was executed belongs to a container and not the host)
• container.image startswith nginx (a container property: the image name so you can have an authorized process lists for each one)
• not proc.name in (nginx) (the list of allowed processes names)

To apply the new configuration file we will restart the Sysdig Falco container: docker restart falco.

Now we need to run a new Nginx container:

docker run -d -P --name example2 nginx

And run anything in the example2 container, ls for example:

docker exec -it example2 ls

If we look at the log with tail /var/log/falco_events.log you will be able to read:

   18:38:43.364877988: Warning Unauthorized process (ls ) running in (604aa46610dd)

Good catch! The Sysdig Falco notification shows that recognized an unexpected process.

Container immutability means that running containers are exactly the same, they don't have any changes in the software running from the image and any user data is in a externally mounted volume. Let's trigger an alarm when any process tries to write to a non data directory.

Let's download a new version of the configuration file for this example:

sudo -s cd /etc/falco curl https://raw.githubusercontent.com/katacoda-scenarios/sysdig-scenarios/master/sysdig-falco/assets/falco_rules_step4.yaml -o falco_rules.yaml

Pay attention to the macro that defines the write-allowed directories that we customized for Nginx:

   - macro: user_data_dir
     condition: evt.arg[1] startswith /userdata or evt.arg[1] startswith /var/log/nginx or evt.arg[1] startswith /var/run/nginx

Right below is the rule for this step:

   - rule: Write to non user_data dir
     desc: attempt to write to directories that should be immutable
     condition: open_write and container and not user_data_dir
     output: "Writing to non user_data dir (user=%user.name command=%proc.cmdline file=%fd.name)"
     priority: ERROR

Let's take a look at the open_write macro used above:

   - macro: open_write
   condition: (evt.type=open or evt.type=openat) and evt.is_open_write=true and fd.typechar='f'

These conditions are based on the Sysdig system call filters, in this case we filter open or openat system calls, open mode write and for file descriptors that are files. If you want to learn more about Sysdig filters, take our Sysdig: container troubleshooting and visibility course or check out these Sysdig examples.

To apply the new configuration file we will restart the Sysdig Falco container: docker restart falco.

Now, you can spawn a new container and try this rule:

docker run -d -P --name example3 nginx docker exec -it example3 bash mkdir /userdata touch /userdata/foo # Shouldn't trigger this rule touch /usr/foo # But this will do

exit the container and look at the log file tail /var/log/falco_events.log. You will find these two events:

   21:15:01.998703651: Error Writing to non user_data dir (user=root command=bash  file=/dev/tty)
   21:15:58.476945006: Error Writing to non user_data dir (user=root command=touch /usr/foo file=/usr/foo)

The first event is because running an interactive shell writes to /dev/tty, this is normal and expected. The second event is where Falco detected an anomalous file write to /usr.

Containers usually have a defined and non often changing set of mountpoints. If a container tries to mount a host directory or file outside the allowed directory set, either someone trying to breakout the container or a team member giving excesive visibility to the container.

Let's download a new version of the configuration file for this example:

sudo -s cd /etc/falco curl https://raw.githubusercontent.com/katacoda-scenarios/sysdig-scenarios/master/sysdig-falco/assets/falco_rules_step5.yaml -o falco_rules.yaml

This is the rule that watches for sensitive mounts inside containers:

- rule: Launch Sensitive Mount Container
  desc: >
    Detect the initial process started by a container that has a mount from a sensitive host directory
    (i.e. /proc). Exceptions are made for known trusted images.
  condition: evt.type=execve and proc.vpid=1 and container and sensitive_mount and not trusted_containers
  output: Container with sensitive mount started (user=%user.name command=%proc.cmdline %container.info)
  priority: INFO
  tags: [container, cis]

The macro sensitive_mount includes the forbidden directories. By default it just watches /proc but in our configuration file we have modified to include /mnt as well.

- macro: sensitive_mount
  condition: (container.mount.dest[/proc*] != "N/A" or container.mount.dest[/mnt*] != "N/A")

To apply the new configuration file we will restart the Sysdig Falco container: docker restart falco.

Now, you can spawn a new container and try to mount /mnt:

docker run -d -P --name example4 -v /mnt:/tmp/mnt alpine

If we look at the log with tail /var/log/falco_events.log you will be able to read:

13:32:41.070491862: Informational Container with sensitive mount started (user=root command=sh -g daemon off; example4 (id=c46fa3bf0651))

The Sysdig Falco notification shows that it detected a sensitive mount.

Falco has a synthetic event generator that shows off all the capabilities of the default ruleset. This is great to fully understand all the capabilities of Sysdig Falco.

Let's pull and launch the event generator:

docker pull sysdig/falco-event-generator docker run -d --name falco-event-generator sysdig/falco-event-generator

If we look at the log with tail -f /var/log/falco_events.log you'll see lots of suspicious activity detected, as that container simulates all sorts of typicall container break-in and break-out attempts:

19:00:55.362191761: Error File created below /dev by untrusted program (user=root command=event_generator  file=/dev/created-by-event-generator-sh)
19:00:56.365043165: Notice Database-related program spawned process other than itself (user=root program=sh -c ls > /dev/null parent=mysqld)
19:00:57.367928872: Warning Sensitive file opened for reading by non-trusted program (user=root name=event_generator command=event_generator  file=/etc/shadow)
19:00:59.370589147: Error File below known binary directory renamed/removed (user=root command=event_generator  operation=rename file=<NA> res=0 oldpath=/bin/true newpath=/bin/true.event-generator-sh )
...