Limiting the size of the lists in rule files

[konstantin-921]

Describe the bug

Following this article - https://sysdig.com/blog/how-to-identify-malicious-ip-activity-using-falco/ I configured a rule to track malicious IPs. The rule works when an incoming file from malicious IPs less than 500 KB is used. But the rule doesn't work when too many IPs (more than 2 MB) are used.

How to reproduce it

To reproduce the error you just need to follow this article - https://sysdig.com/blog/how-to-identify-malicious-ip-activity-using-falco/. But I made some adjustments to the proposed script, so in the presented form it did not work as it should. Therefore, I will describe it step by step:

Step 1: Writing the Falco rule

– rule: Malicious IPs
 desc: Detect connections to/from a malicious IP
 condition: (inbound_outbound) and fd.sip in (malicous_ip_list) or fd.cip in (malicous_ip_list)
 output: >
   Suspicious connection to/from a malicious IP detected (command=%proc.cmdline connection=%fd.name user=%user.name container_id=%container.id)
 priority: WARNING
 tags: [network]

To complete this step, we will write this rule to a file under/etc/falco/malicious_ips_rule.yaml.

Step 2: Generate a constantly updated malicious IP list

• create new file /etc/falco/malicous_ips_list.yaml.
• create the file get_malicious_ips.sh with this script for getting Malicious IPs:

#!/bin/bash
/usr/bin/curl --compressed https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt 2>/dev/null | grep -v "#" | grep -v -E "s[1-5]$" |  cut -f 1  | sed "s/.*/'\"&\"',/g" | tr '\n' ' ' | sed "s/, $//" | sed 's/.*/- list: malicous_ip_list\n  items: [&]/' > /etc/falco/malicious_ips_list.yaml

/usr/sbin/service falco restart

Step 3: Update Falco configuration (/etc/falco/falko.yml) like this:

rules_file:
 - /etc/falco/malicious_ips_rule.yaml
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco_rules.local.yaml
 - /etc/falco/rules.d
 - /etc/falco/malicious_ips_list.yaml

Step 4: Testing!

• start get_malicious_ips.sh script
• try to use any IP from the malicous_ips_list.yaml frome the same node, for example:

curl http://198.98.58.250

Expected behaviour

I was expecting new logs in /var/log/syslog.log file related to an attempt to connect to malicious IPs. But that didn't happen.

But if I use an incoming file with malicious IPs of smaller size, e.g. using a slightly different address in a script like this https://raw.githubusercontent.com/stamparm/ipsum/master/levels/2.txt, everything will work fine and we can see the new logs in the /var/log/syslog.log.

Environment

• Falco version: 0.29.1
• System info:

{
  "machine": "x86_64",
  "nodename": "kub-node",
  "release": "4.15.0-137-generic",
  "sysname": "Linux",
  "version": "#141-Ubuntu SMP Fri Feb 19 13:46:27 UTC 2021"
}

• OS:

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel:
  Linux kub-node1-fl 4.15.0-137-generic Rule fixes for dragent. #141-Ubuntu SMP Fri Feb 19 13:46:27 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Installation method:
  APT package

Additional context

I did not find any information in the documentation about the limits for the lists used in the rules

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.