Tracee-eBPF performance and scalability

[rafaeldtinoco]

This is the thread I'll be discussing tracee-ebpf performance and scalability.

Testing Environment

• 24 x Intel Xeon CPU E5-2640 @ 2.50 GHz
• 197 GB in memory (total)
• 2 NUMA nodes with 12 CPU cores each:
  • available: 2 nodes (0-1)
    • node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17
      • node 0 size: 96377 MB
      • node 0 free: 76710 MB
    • node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23
      • node 1 size: 96764 MB
      • node 1 free: 73952 MB

[rafaeldtinoco]

TEST 01 (2 full CPUs with heavy SYS time)

Limits of the measurement methodology

$ dd if=/dev/zero | pv -terb -a | dd of=/dev/null bs=4MB
3.91GiB 0:00:09 [ 484MiB/s] [ 445MiB/s]

NOTES:

• This measuring methodology can handle up to ~440 MB/s of throughput
• If the throughput being piped is higher than 440 MB/s, we are limited by the methodology.

Workload used for ALL measurements

• 2 x CPUs doing 100% CPU utilization (1 on each NUMA domain: cpus 17 and 23) :
  • (while true; do ps -ef > /dev/null; done)
• This is the equivalent of 2/24 CORES used
  • (or 8% of total node CPUs utilization)
• Output format is ALWAYS GOB
• All tests are a 60 sec run of tracee-ebpf

Tracee eBPF (only) throughput and scalability

sudo taskset -c 0 \
	cpulimit -l <percentage> -- \
	./dist/tracee-ebpf \
	-o format:gob \
	-o option:parse-arguments \
	-o option:detect-syscall \
	--trace event=$(./dist/tracee-rules --list-events) | \
	pv -terb -a | \
	dd of=/dev/null bs=4MB

tracee-ebpf in a single CPU CORE limiting it to 10% of 2.50 GHz:

Average 1 MB/s.

Lost: 48785 + 454018 + 838236 + 835271 + 1093976 + 1101768 + 890532 +
818976 Events

tracee-ebpf in a single CPU CORE limiting it to 25% of 2.50 GHz:

Average: 2.36 MB/s

Lost: 40462 + 253072 + 393037 + 380545 + 370103 + 329832 + 404033 + 371003 +
313105 + 324980 + 364524 + 366516 + 332446 + 382711 + 404028 + 314142 +
309329 + 374641 + 309831 + 361672 + 306427 Events

tracee-ebpf in a single CPU CORE limiting it to 50% of 2.50 GHz:

Average: 4.10 MB/s

Lost: 21858 + 114352 + 200374 + 192206 + 167655 + 225435 + 168040 + 220803 +
180560 + 202014 + 183895 + 195182 + 170930 + 171484 + 187388 + 188977 +
166214 + 184037 + 181991 + 181070 + 165850 + 167314 + 183697 + 185872 +
174529 + 179152 + 167435 + 184135 + 167130 + 180365 + 190930 + 209195 +
181833 + 179324 + 166100 + 224126 + 184653 + 201916 + 165679 Events

tracee-ebpf in a single CPU CORE of 2.50 GHz (100%):

Average: 7.40 MB/s

Lost: 47208 + 101892 + 95254 + 94501 + 79530 + 78862 + 83956 + 86353 +
84205 + 96393 + 85175 + 88227 + 81155 + 106265 + 82378 + 103538 + 82354 +
104686 + 83019 + 105579 + 82111 + 104307 + 83244 + 102887 + 84116 +
108559 + 62266 + 83251 + 84742 + 81201 + 88778 + 93138 + 82824 + 103247 +
87131 + 110396 + 83372 + 104072 + 83776 + 106093 + 82892 + 107527 + 82703 +
82470 + 82227 + 84915 + 83576 + 86413 + 84244 + 86475 + 82062 + 88295 +
83344 + 86868 + 82041 + 84790 + 80480 + 73824 + 78165 + 87015 + 93394 +
96862 + 85904 + 89443 + 84559 + 85574 + 84812 + 86712 + 86420 + 84508
Events

tracee-ebpf in 2 CPU CORE(s) of 2.50 GHz (no limit):

Average: 9.34 MB/s

Lost: 16508 + 59386 + 75777 + 65316 + 64549 + 67670 + 64070 + 55012 +
59277 + 53405 + 63206 + 63502 + 58698 + 62329 + 59705 + 60076 + 58157 +
59108 + 58324 + 59033 + 56963 + 59149 + 57801 + 58137 + 57902 + 58762 +
56681 + 58467 + 59072 + 60455 + 57934 + 54966 + 59138 + 60021 + 58624 +
60667 + 58405 + 60851 + 59319 + 57810 + 57625 + 58357 + 57051 + 56662 +
57952 + 51018 + 59462 + 50106 + 61323 + 61880 + 56005 + 58432 + 57604 +
57662 + 57902 + 58433 + 55240 + 58266 + 57824 + 59928 + 57719 + 60037 +
57350 + 56872 + 59709 + 59069 + 57465 + 58877 + 55688 + 59679 + 57802 +
57707 + 58254 + 60182 + 60031 + 61552 + 57115 + 57804 + 56376 + 54978 +
56477 + 55137 + 66687 + 65994 + 62037 + 63364 + 58095 + 58701 + 57497
Events

tracee-ebpf in 3 CPU CORE(s) of 2.50 GHz (no limit):

Average: 7.93 MB/s (performance is degraded when comparing to 2 CPUs)

Results

Tracee-eBPF has reached its maximum throughput when using 2 full CPUs (9.34 MB/s). It had its performance degraded when trying to use more CPUs. The maximum throughput wasn't enough to guarantee the perf-buffer pressure relief and, with selected workload, it had many event losses.

[mtcherni95]

Hello :)

Great insight! I agree with you that we still have a long way to go, probably tracee-ebpf and tracee-rules might go over an architectural rethinking in order to sustain more load.

However, I must say that after implementing a lab tool (let's that call it Simulator) that uses Tracee as a library (both rules and ebpf) and measuring lost events, and throughput we do see improvements in perfomance between v0.6.5 and main branch.
@NDStrahilevitz run a benchmark using Simulator on 6.5 (on the left)and main (on the right):

I do agree though that there is still margin of improvement for sustain higher throughputs, but I do see a significant improvement between the two versions.

[NDStrahilevitz]

It's important to note that we're measuring the size of the payload on the right, and not the full Event structure (we're using #1493 in our tests) which is smaller (since it only contains a pointer to the payload). If we measure the envelope size we get a similar throughput between the two.
Nevertheless - the improvement in lost events is significant.

[rafaeldtinoco]

@NDStrahilevitz

Forget about tracee-rules now.

I'm puzzled on why you're not loosing events in the 2nd example. If you have the same throughput as I do (~8MB/s), why aren't you loosing events and I am ?

I can only mimic that behavior - of not losing events - if I put a cache right after the perf-buffer consumer. As soon as the cache is full, I start losing events.

For how long did you run your tests ? Could the channel buffering still be caching ? Isn't that what is happening for you ? I'm really curious to understand how come you're not loosing events...

[NDStrahilevitz]

@rafaeldtinoco

That's what I explained in my comment, that throughput is misleading. the actual size of the event passed here is 0.25 of the original payload, so the throughput should be 8.32*0.25=2.08mb/s, nothing close to yours. As I said the throughput in the two examples is similar, we're just able to handle it after all the optimizations in main.

[rafaeldtinoco]

@NDStrahilevitz Oh, its clear now .. thanks for explaining it!