Sudden high CPU load

[Tony-MF]

Hi,

I lately get super high CPU load like this.

It seem it's coming from the app container.

The app container logs are full of:

18:01:14.467 [error] #PID<0.4713.0> running PlausibleWeb.Endpoint (connection #PID<0.4712.0>, stream id 1) terminated
Server: stats.mydomain.com:80 (http)
Request: POST /api/event
** (exit) an exception was raised:
** (RuntimeError) could not find persistent term for endpoint PlausibleWeb.Endpoint. Make sure your endpoint is started and note you cannot access endpoint functions at compile-time
(plausible 0.0.1) deps/phoenix/lib/phoenix/endpoint.ex:544: PlausibleWeb.Endpoint.persistent!/0
(plausible 0.0.1) deps/phoenix/lib/phoenix/endpoint.ex:584: PlausibleWeb.Endpoint.script_name/0
(plausible 0.0.1) lib/plausible_web/endpoint.ex:1: PlausibleWeb.Endpoint.call/2
(plug_cowboy 2.7.2) lib/plug/cowboy/handler.ex:11: Plug.Cowboy.Handler.init/2
(cowboy 2.12.0) /app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
(cowboy 2.12.0) /app/deps/cowboy/src/cowboy_stream_h.erl:306: :cowboy_stream_h.execute/3
(cowboy 2.12.0) /app/deps/cowboy/src/cowboy_stream_h.erl:295: :cowboy_stream_h.request_process/3
(stdlib 6.0) proc_lib.erl:329: :proc_lib.init_p_do_apply/3

The clickhouse container logs are ok I think:

2025.01.19 17:55:38.539129 [ 1 ] {} Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9005 (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

Any idea what it might be?

[Tony-MF]

Claude helped me. It was the docker proxy probably.

These two steps fixed the issue:

# Edit /etc/docker/daemon.json
{
  "userland-proxy": false,
  "iptables": true
}

# Add to /etc/security/limits.conf
*         soft    nofile      65535
*         hard    nofile      65535