Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DistributedTrace.generate_sampling timeouts. #385

Open
noizu opened this issue Jun 15, 2022 · 1 comment
Open

DistributedTrace.generate_sampling timeouts. #385

noizu opened this issue Jun 15, 2022 · 1 comment

Comments

@noizu
Copy link

noizu commented Jun 15, 2022

Describe the bug
Sampling timeouts occur under load on my IOT facing API nodes.

Need a way to increase timeouts or expose other configuration settings to avoid this exception.

image

Environment

  • Elixir & Erlang version (elixir -v):
    Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

  • Agent version (mix deps | grep new_relic_agent):
    new_relic_agent (Hex package)
    (mix) locked at 1.27.7 (new_relic_agent) a0aa7208

@noizu
Copy link
Author

noizu commented Dec 9, 2022

I was eventually able to resolve this by tweaking the following settings.

config :new_relic_agent,
analytic_event_per_minute: 250, # Reducing caps to avoid overwhelming api node telemetry
custom_event_per_minute: 250,
error_event_per_minute: 100,
span_event_per_minute: 100,
sampling_target: 5

Should we prepare a doc update to cover options for tweaking sampling rates and caps for high load environments to safe the next person some time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant