Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Initial errors/failures observed when testing AbuseCH integration in Agentless #42815

Open
qcorporation opened this issue Feb 20, 2025 · 4 comments
Open

[Filebeat] Initial errors/failures observed when testing AbuseCH integration in Agentless #42815

qcorporation opened this issue Feb 20, 2025 · 4 comments
Labels
Agent Filebeat Filebeat :integrations needs_team Indicates that the issue/PR needs a Team:* label

Comments

@qcorporation
Copy link

qcorporation commented Feb 20, 2025 •
edited
Loading

Please include configurations and logs if available.
elastic-agent-diagnostics-2025-02-20T20-36-47Z-00.zip

For confirmed bugs, please report:

Version:

9.0.0-beta1

Operating System:

Agentless Deployment

Steps to Reproduce:

  1. Instantiate a cloud security elastic stack instance on staging against v9.0.0
  2. Install a modified ti_abusech integration with agentless enabled using elastic-package to the instance
  3. Create/Install an AbuseCH integration in Agentless
  4. After a few minutes when logs have been ingested, capture an agent diagnostic
  5. note errors within agent logs
{"log.level":"error","@timestamp":"2025-02-20T20:05:47.646Z","message":"fatal error: concurrent map read and map write","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"ecs.version":"1.6.0"}

Running it through the agent analyzer synposis looks like this:

2025-02-20T20:05:47.68Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:05:47.68Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:05:47.68Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:06:44.379Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:06:44.379Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:06:44.379Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:07:21.605Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:07:21.605Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:07:21.605Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:09:12.822Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:12.823Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:12.823Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:57.277Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '491' exited with code '2'

and

2025-02-20T20:07:43.807Z: Error fetching data for metricset http.json: error making http request: Get "http://unix/inputs": dial unix /agentless/data/tmp/WDSbrgjXxHKnME7Z0CSyI1xJnirO2oRp.sock: connect: connection refused
2025-02-20T20:10:41.879Z: Error fetching data for metricset http.json: error making http request: Get "http://unix/stats": dial unix /agentless/data/tmp/WDSbrgjXxHKnME7Z0CSyI1xJnirO2oRp.sock: connect: connection refused
@qcorporation qcorporation added :integrations Agent Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label labels Feb 20, 2025
@botelastic
Copy link

botelastic bot commented Feb 20, 2025

This issue doesn't have a Team:<team> label.

@efd6
Copy link
Contributor

efd6 commented Feb 20, 2025

It looks like multiple goroutines are holding on to a configuration value.

fatal error: concurrent map read and map write

goroutine 88 [running]:
github.com/elastic/go-ucfg.(*fields).get(...)
	github.com/elastic/[email protected]/ucfg.go:249
github.com/elastic/go-ucfg.namedField.GetValue({{0xaaaaf83bf28e?, 0x8?}}, 0x40032340a0, {0xaaaafa20b6b8, 0x4003381560})
	github.com/elastic/[email protected]/path.go:189 +0x70
github.com/elastic/go-ucfg.cfgPath.GetValue({{0x400202c360, 0x1, 0x1}, {0xaaaaf8053510, 0x1}}, 0x4003381560, 0x40032340a0)
	github.com/elastic/[email protected]/path.go:176 +0x124
github.com/elastic/go-ucfg.reifyGetField(0x4003381560, {0x40032340a0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}}, {0xaaaaf83bf28e, 0x7}, ...)
	github.com/elastic/[email protected]/reify.go:315 +0xd0
github.com/elastic/go-ucfg.reifyStruct(0x40032340a0, {0xaaaaf9e62220?, 0x400323e000?, 0x0?}, 0x4003381560)
	github.com/elastic/[email protected]/reify.go:292 +0x390
github.com/elastic/go-ucfg.reifyInto(0x40032340a0, {0xaaaaf8ce2da0?, 0x400323e000?, 0xaaaaf9d23600?}, 0x4003381560)
	github.com/elastic/[email protected]/reify.go:177 +0x1c8
github.com/elastic/go-ucfg.(*Config).Unpack(0x4003381560, {0xaaaaf8ce2da0, 0x400323e000}, {0x4002445980?, 0x0?, 0x0?})
	github.com/elastic/[email protected]/reify.go:155 +0xc0
github.com/elastic/elastic-agent-libs/config.(*C).Unpack(...)
	github.com/elastic/[email protected]/config/config.go:159
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewClients(0x4003381560, {0xaaaaf710115c, 0x8})
	github.com/elastic/beats/v7/libbeat/esleg/eslegclient/connection.go:208 +0x108
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnectedClient({0xaaaafa1bb4a8, 0x40022d2f50}, 0xaaaaf7162f9e?, {0xaaaaf710115c?, 0x0?})
	github.com/elastic/beats/v7/libbeat/esleg/eslegclient/connection.go:255 +0x3c
github.com/elastic/beats/v7/libbeat/statestore/backend/es.(*store).configure(0x4003316380, {0xaaaafa1bb4a8, 0x40022d2f50}, 0x4003381560)
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:309 +0x178
github.com/elastic/beats/v7/libbeat/statestore/backend/es.(*store).loop(0x4003316380, {0xaaaafa1bb4a8, 0x40022d2f50}, 0x0?, 0x0?, 0x40000bd200)
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:337 +0xb0
created by github.com/elastic/beats/v7/libbeat/statestore/backend/es.openStore in goroutine 1
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:85 +0x2c4

goroutine 1 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:44
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0x4001eab340, 0x4001d8c008)
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:496 +0x180c
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0x4001d8c008, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:749 +0x6d0
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x40024bfbd8, 0x4001dbfc30)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:200 +0xe0
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:201 +0x24
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0x4001a58600?, {0x40018e11e0?, 0x4?, 0xaaaaf70f1cda?})
	github.com/elastic/beats/v7/libbeat/cmd/run.go:36 +0x50
github.com/spf13/cobra.(*Command).execute(0x4002068e08, {0x40018e1040, 0x1a, 0x1a})
	github.com/spf13/[email protected]/command.go:989 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x4001a3a308)
	github.com/spf13/[email protected]/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:1041
main.main()
	github.com/elastic/beats/v7/x-pack/agentbeat/main.go:27 +0x20

goroutine 35 [select]:
go.opencensus.io/stats/view.(*worker).start(0x400105f300)
	[email protected]/stats/view/worker.go:292 +0x88
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	[email protected]/stats/view/worker.go:34 +0x98

goroutine 10 [select]:
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata(0x4002f43ce0)
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/providers.go:195 +0x3a8
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1()
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:98 +0x28
sync.(*Once).doSlow(0x4001e040a0?, 0x4001e04098?)
	sync/once.go:74 +0x100
sync.(*Once).Do(...)
	sync/once.go:65
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init(0x0?)
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:97 +0x40
created by github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.New in goroutine 1
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:87 +0x294

goroutine 31 [chan receive]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run(0x40022d2490, 0x4001e93690)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:51 +0x90
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer in goroutine 1
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x1d8

goroutine 67 [select]:
github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.(*cache).cleanup(0x4002f76300)
	github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata/cache.go:79 +0xb4
created by github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.newCache in goroutine 1
	github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata/cache.go:44 +0xdc

goroutine 30 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run(0x40022d2460)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:161 +0x228
github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:91 +0x54
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer in goroutine 1
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:89 +0x174

goroutine 70 [IO wait]:
internal/poll.runtime_pollWait(0xffff3c2527a8, 0x72)
	runtime/netpoll.go:345 +0xa0
internal/poll.(*pollDesc).wait(0x3?, 0x10?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x4002f55480)
	internal/poll/fd_unix.go:611 +0x250
net.(*netFD).accept(0x4002f55480)
	net/fd_unix.go:172 +0x28
net.(*UnixListener).accept(0x4002f63d01?)
	net/unixsock_posix.go:172 +0x20
net.(*UnixListener).Accept(0x4002f76ae0)
	net/unixsock.go:260 +0x2c
net/http.(*Server).Serve(0x4002576960, {0xaaaafa19c540, 0x4002f76ae0})
	net/http/server.go:3260 +0x2a8
net/http.Serve(...)
	net/http/server.go:2799
github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1({0xaaaafa19c540, 0x4002f76ae0})
	github.com/elastic/beats/v7/libbeat/api/server.go:72 +0x118
created by github.com/elastic/beats/v7/libbeat/api.(*Server).Start in goroutine 1
	github.com/elastic/beats/v7/libbeat/api/server.go:70 +0x118

goroutine 32 [select]:
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0x40022d2780)
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:166 +0x1b4
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1()
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:139 +0x54
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter in goroutine 1
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:137 +0x2f0

goroutine 82 [chan receive]:
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
	github.com/elastic/[email protected]/service/service.go:50 +0x4c
created by github.com/elastic/elastic-agent-libs/service.HandleSignals in goroutine 1
	github.com/elastic/[email protected]/service/service.go:49 +0x154

goroutine 61 [syscall]:
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x30
os/signal.loop()
	os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1 in goroutine 1
	os/signal/signal.go:151 +0x28

goroutine 84 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.func1()
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:309 +0x30
created by github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run in goroutine 1
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:308 +0x3c8

goroutine 85 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x4003202180?, 0x4002f5de40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f81e0, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2e10, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 86 [select]:
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*updateChan).Recv(0x4003312210, {0xaaaafa1bb4a8, 0x40022d2f00})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:151 +0x1b0
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*updateWriter).run(0x400216dae0, {0xaaaafa1bb4a8, 0x40022d2f00})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:87 +0x54
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.newUpdateWriter.func1({0xaaaafa1bb4a8?, 0x40022d2f00?})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:65 +0x34
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 87 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x400320dde8?, 0x4002f5ce48)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*cleaner).run(0x40020f81f8, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2eb0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/clean.go:50 +0xa0
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/manager.go:134 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 89 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x400321de40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8220, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2fa0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 90 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x400321ce40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8238, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2ff0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 91 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f61e40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8250, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3090, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 92 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x40000ece40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8268, {0xffff3c056b60, 0x40022d2e60}, 0x40022d30e0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 93 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f60e40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8280, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3130, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 94 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f5ee40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8298, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3180, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 95 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x40000ede40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f82b0, {0xffff3c056b60, 0x40022d2e60}, 0x40022d31d0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 96 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f5ae40)
	github.com/elastic/[email protected]/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f82e8, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3220, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/[email protected]/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/[email protected]/unison/taskgroup.go:160 +0xf8

goroutine 97 [select]:
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run(0x4001afc360)
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:164 +0x1cc
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1()
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:122 +0x54
created by github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start in goroutine 1
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:120 +0xb4

goroutine 98 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b00, {0xaaaafa1bb4a8, 0x40022d3450})
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 99 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b30, {0xaaaafa1bb4a8, 0x40022d34a0})
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 100 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b70, {0xaaaafa1bb4a8, 0x40022d34f0})
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 63 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 102
	google.golang.org/[email protected]/stream.go:392 +0xb2c

goroutine 102 [semacquire]:
sync.runtime_Semacquire(0x4000183180?)
	runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0x40019b89a0)
	sync/waitgroup.go:116 +0x74
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip(0x4001e0a700)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:501 +0x1c4
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0x4003321f78)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:406 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin.func1()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:435 +0xb4
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin in goroutine 1
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:427 +0x70

goroutine 103 [semacquire]:
sync.runtime_Semacquire(0x4000183880?)
	runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0x40019b8b40)
	sync/waitgroup.go:116 +0x74
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip(0x4001e0a700, 0x4001afc8a0)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:841 +0x1ec
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1.1()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:781 +0x24
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0x4003322f70)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:406 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:780 +0xc8
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions in goroutine 1
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:772 +0x94

goroutine 104 [select]:
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan(0x4002edd0e0, {0xaaaafa1bb4a8, 0x40022d39f0})
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:461 +0x98
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:287 +0x12c

goroutine 105 [runnable]:
runtime.CallersFrames(...)
	runtime/symtab.go:77
go.uber.org/zap/internal/stacktrace.Capture(0x5, 0x0)
	go.uber.org/[email protected]/internal/stacktrace/stack.go:107 +0x158
go.uber.org/zap.(*Logger).check(0x4003330d00, 0x0, {0x40033366c0, 0x118})
	go.uber.org/[email protected]/logger.go:379 +0x540
go.uber.org/zap.(*Logger).Check(...)
	go.uber.org/[email protected]/logger.go:222
go.uber.org/zap.(*SugaredLogger).log(0x40020f8730, 0x0, {0xaaaaf723322b?, 0x24?}, {0x4003378cb0?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
	go.uber.org/[email protected]/sugar.go:354 +0xb8
go.uber.org/zap.(*SugaredLogger).Infof(...)
	go.uber.org/[email protected]/sugar.go:204
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
	github.com/elastic/[email protected]/logp/logger.go:156
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/elasticsearch.go:57 +0x204
github.com/elastic/beats/v7/libbeat/outputs.Load({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/output_reg.go:109 +0xd4
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createOutput(0x4001d8c008, {0xaaaafa201280, 0x4003330800}, {{0x40032bc410, 0xd}, 0x4003381560})
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1505 +0x198
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload.func1({0xaaaafa201280?, 0x4003330800?})
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:179 +0x68
github.com/elastic/beats/v7/libbeat/publisher/pipeline.loadOutput({0x400223ae00?, 0x40022f0100?, 0x4001e93690?, 0x40018e1ba0?}, 0x40033794a8?)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:139 +0x12c
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload(0x4001e8e480, 0x4003386100, 0x40020520a0)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:177 +0xbc
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater.(*Beat).makeOutputReloader.func2(0x4003386100)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1392 +0xd0
github.com/elastic/beats/v7/libbeat/common/reload.ReloadableFunc.Reload(0x40032ba510?, 0x40020399ff?)
	github.com/elastic/beats/v7/libbeat/common/reload/reload.go:226 +0x28
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reloadOutput(0x4002edd0e0, 0x40032a8540)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:784 +0x270
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload(0x4002edd0e0, 0x4003289e40)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:646 +0x7d0
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0x4002edd0e0)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:547 +0x6c4
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:295 +0x1f0

goroutine 106 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.(*signalWait).AddChan.waitChannel.func17()
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:74 +0x24
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add.func1()
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:51 +0x2c
created by github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add in goroutine 1
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:50 +0x7c

goroutine 64 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readHeaderClient(0x4002b9f310, {0x40031f5210, 0x5, 0x5})
	google.golang.org/[email protected]/internal/transport/transport.go:199 +0x78
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadHeader(0x4002b9f310, {0x40031f5210?, 0x40033386c8?, 0x4003284360?})
	google.golang.org/[email protected]/internal/transport/transport.go:145 +0x3c
google.golang.org/grpc/internal/transport.(*transportReader).ReadHeader(0x4001a79ba0, {0x40031f5210?, 0x40020b1c00?, 0x100004001bbd305?})
	google.golang.org/[email protected]/internal/transport/transport.go:614 +0x2c
google.golang.org/grpc/internal/transport.(*Stream).ReadHeader(0x4003284360, {0x40031f5210, 0x5, 0x5})
	google.golang.org/[email protected]/internal/transport/transport.go:557 +0xa4
google.golang.org/grpc.(*parser).recvMsg(0x40031f5200, 0x6400000)
	google.golang.org/[email protected]/rpc_util.go:659 +0x44
google.golang.org/grpc.recvAndDecompress(0x40031f5200, 0x4003284360, {0x0, 0x0}, 0x6400000, 0x0, {0x0, 0x0}, 0x0)
	google.golang.org/[email protected]/rpc_util.go:823 +0x60
google.golang.org/grpc.recv(0x4003359b88?, {0xffff3436f648, 0xaaaaffdbf340}, 0xaaaaf06a8160?, {0x0?, 0x0?}, {0xaaaaf9bc3200, 0x4001db5490}, 0x4003359c28?, 0xaaaaf088a4bc?, ...)
	google.golang.org/[email protected]/rpc_util.go:916 +0x6c
google.golang.org/grpc.(*csAttempt).recvMsg(0x4002ee8820, {0xaaaaf9bc3200, 0x4001db5490}, 0x58?)
	google.golang.org/[email protected]/stream.go:1129 +0x250
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x10?)
	google.golang.org/[email protected]/stream.go:969 +0x28
google.golang.org/grpc.(*clientStream).withRetry(0x4002f7a480, 0x4003359e40, 0x4003359e30)
	google.golang.org/[email protected]/stream.go:773 +0x4e8
google.golang.org/grpc.(*clientStream).RecvMsg(0x4002f7a480, {0xaaaaf9bc3200?, 0x4001db5490?})
	google.golang.org/[email protected]/stream.go:968 +0xf4
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentCheckinV2Client).Recv(0x4001e93e40)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/proto/elastic-agent-client_grpc.pb.go:109 +0x58
github.com/elastic/elastic-agent-client/v7/pkg/client/chunk.RecvExpected({0xffff3c056c58, 0x4001e93e40})
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/chunk/expected.go:108 +0x48
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func1()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:484 +0xd0
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 102
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:480 +0x138

goroutine 109 [IO wait]:
internal/poll.runtime_pollWait(0xffff3c2526b0, 0x72)
	runtime/netpoll.go:345 +0xa0
internal/poll.(*pollDesc).wait(0x4003212380?, 0x400329ca80?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4003212380, {0x400329ca80, 0x2a80, 0x2a80})
	internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x4003212380, {0x400329ca80?, 0xffff3c199ed8?, 0x4001ea5428?})
	net/fd_posix.go:55 +0x28
net.(*conn).Read(0x40020f8588, {0x400329ca80?, 0x4003287938?, 0xaaaaf014981c?})
	net/net.go:185 +0x34
crypto/tls.(*atLeastReader).Read(0x4001ea5428, {0x400329ca80?, 0x0?, 0x4001ea5428?})
	crypto/tls/conn.go:806 +0x40
bytes.(*Buffer).ReadFrom(0x4001afb430, {0xaaaafa10a000, 0x4001ea5428})
	bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x4001afb188, {0xaaaafa107520, 0x40020f8588}, 0x4003287a10?)
	crypto/tls/conn.go:828 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x4001afb188, 0x0)
	crypto/tls/conn.go:626 +0x35c
crypto/tls.(*Conn).readRecord(...)
	crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0x4001afb188, {0x4003362000, 0x8000, 0xaaaaf014981c?})
	crypto/tls/conn.go:1370 +0x168
bufio.(*Reader).Read(0x4003350ba0, {0x4001d689e0, 0x9, 0xaaaaf01a0490?})
	bufio/bufio.go:241 +0x1b4
io.ReadAtLeast({0xaaaafa1061d8, 0x4003350ba0}, {0x4001d689e0, 0x9, 0x9}, 0x9)
	io/io.go:335 +0xa0
io.ReadFull(...)
	io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x4001d689e0, 0x9, 0x5fe38fa?}, {0xaaaafa1061d8?, 0x4003350ba0?})
	golang.org/x/[email protected]/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x4001d689a0)
	golang.org/x/[email protected]/http2/frame.go:501 +0x78
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x40033386c8, 0x4003350c00)
	google.golang.org/[email protected]/internal/transport/http2_client.go:1645 +0x154
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 101
	google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1698

goroutine 62 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400331b6c0, 0x1)
	google.golang.org/[email protected]/internal/transport/controlbuf.go:412 +0x140
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x400219dc00)
	google.golang.org/[email protected]/internal/transport/controlbuf.go:575 +0x7c
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	google.golang.org/[email protected]/internal/transport/http2_client.go:470 +0xb8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 101
	google.golang.org/[email protected]/internal/transport/http2_client.go:468 +0x1b18

goroutine 65 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinWriter(0x4001e0a700, {0xaaaafa1df048, 0x4001e93e40}, 0x40024ed080)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:518 +0xd4
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func2()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:496 +0x64
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 102
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:494 +0x1bc

goroutine 114 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 103
	google.golang.org/[email protected]/stream.go:392 +0xb2c

goroutine 115 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x40032845a0)
	google.golang.org/[email protected]/internal/transport/transport.go:378 +0x6c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	google.golang.org/[email protected]/internal/transport/transport.go:393
google.golang.org/grpc.(*csAttempt).recvMsg(0x400333c680, {0xaaaaf9bc2fc0, 0x4003292200}, 0xaaaaf014981c?)
	google.golang.org/[email protected]/stream.go:1115 +0xc0
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x10?)
	google.golang.org/[email protected]/stream.go:969 +0x28
google.golang.org/grpc.(*clientStream).withRetry(0x4001a83200, 0x4003238ea0, 0x4003238e90)
	google.golang.org/[email protected]/stream.go:789 +0x188
google.golang.org/grpc.(*clientStream).RecvMsg(0x4001a83200, {0xaaaaf9bc2fc0?, 0x4003292200?})
	google.golang.org/[email protected]/stream.go:968 +0xf4
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentActionsClient).Recv(0x40020403f0)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/proto/elastic-agent-client_grpc.pb.go:140 +0x58
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsReader(0x4001e0a700, {0xaaaafa1deff0, 0x40020403f0}, 0x4001afc8a0)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:848 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func1()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:829 +0x88
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 103
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:826 +0x150

goroutine 116 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsWriter(0x4001e0a700, {0xaaaafa1deff0, 0x40020403f0}, 0x4001afc8a0, 0x40024ed200)
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:885 +0x140
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func2()
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:837 +0x94
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 103
	github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:834 +0x1e4

@cmacknz
Copy link
Member

cmacknz commented Feb 20, 2025

I suspect this is from the ES state store seeing it and the top of the stack trace and also what it is doing in that stack trace:

beats/libbeat/statestore/backend/es/store.go

Lines 64 to 85 in 2200a16

func openStore(ctx context.Context, log *logp.Logger, name string, notifier *Notifier) (*store, error) {
ctx, cn := context.WithCancel(ctx)
s := &store{
ctx: ctx,
cn: cn,
log: log.With("name", name).With("backend", "elasticsearch"),
name: name,
index: renderIndexName(name),
notifier: notifier,
chReady: make(chan struct{}),
}
chCfg := make(chan *conf.C)
unsubFn := s.notifier.Subscribe(func(c *conf.C) {
select {
case chCfg <- c:
case <-ctx.Done():
}
})
go s.loop(ctx, cn, unsubFn, chCfg)

CC @orestisfl

The agent policy here has multiple CEL inputs running the same Filebeat process, so perhaps related to that.

@orestisfl
Copy link
Contributor

Hmm, can't see anything in the stack trace that is supposed to be writing to the configuration value for it to cause a concurrent map read and map write. However, if not all go routines are captured at the exact moment when the exception happens, the most likely candidate I see is goroutine 105:

goroutine 105 [runnable]:
runtime.CallersFrames(...)
	runtime/symtab.go:77
go.uber.org/zap/internal/stacktrace.Capture(0x5, 0x0)
	go.uber.org/[email protected]/internal/stacktrace/stack.go:107 +0x158
go.uber.org/zap.(*Logger).check(0x4003330d00, 0x0, {0x40033366c0, 0x118})
	go.uber.org/[email protected]/logger.go:379 +0x540
go.uber.org/zap.(*Logger).Check(...)
	go.uber.org/[email protected]/logger.go:222
go.uber.org/zap.(*SugaredLogger).log(0x40020f8730, 0x0, {0xaaaaf723322b?, 0x24?}, {0x4003378cb0?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
	go.uber.org/[email protected]/sugar.go:354 +0xb8
go.uber.org/zap.(*SugaredLogger).Infof(...)
	go.uber.org/[email protected]/sugar.go:204
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
	github.com/elastic/[email protected]/logp/logger.go:156
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/elasticsearch.go:57 +0x204
github.com/elastic/beats/v7/libbeat/outputs.Load({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/output_reg.go:109 +0xd4
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createOutput(0x4001d8c008, {0xaaaafa201280, 0x4003330800}, {{0x40032bc410, 0xd}, 0x4003381560})
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1505 +0x198
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload.func1({0xaaaafa201280?, 0x4003330800?})
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:179 +0x68
github.com/elastic/beats/v7/libbeat/publisher/pipeline.loadOutput({0x400223ae00?, 0x40022f0100?, 0x4001e93690?, 0x40018e1ba0?}, 0x40033794a8?)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:139 +0x12c
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload(0x4001e8e480, 0x4003386100, 0x40020520a0)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:177 +0xbc
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater.(*Beat).makeOutputReloader.func2(0x4003386100)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1392 +0xd0
github.com/elastic/beats/v7/libbeat/common/reload.ReloadableFunc.Reload(0x40032ba510?, 0x40020399ff?)
	github.com/elastic/beats/v7/libbeat/common/reload/reload.go:226 +0x28
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reloadOutput(0x4002edd0e0, 0x40032a8540)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:784 +0x270
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload(0x4002edd0e0, 0x4003289e40)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:646 +0x7d0
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0x4002edd0e0)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:547 +0x6c4
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:295 +0x1f0

This line here is just 4 lines above the current line:

beats/libbeat/outputs/elasticsearch/elasticsearch.go

Line 53 in 34eb65a

overriddenFields, presetConfig, err := ApplyPreset(preset, cfg)

and ApplyPreset() calls Merge():

beats/libbeat/outputs/elasticsearch/config_presets.go

Line 114 in 34eb65a

err = userConfig.Merge(presetConfig)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Agent Filebeat Filebeat :integrations needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@qcorporation @cmacknz @orestisfl @efd6