Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Efficiency Updates / Distribution & Aggregation / Timing calls/funs #4

Open
wants to merge 49 commits into
base: master
Choose a base branch
from

Conversation

soup-in-boots
Copy link

New Features:

  • Add support for timer:tc-like functions (uses timer:tc/1,timer:tc/3 to measure function duration then post results)
    • timing_fun, executes anonymous fun with timer:tc/1
    • timing_call, executes {M, F, A} style call with timer:tc/3
  • Add support for a "master node"; useful in distributed environments. Local estatsd instance tracks its requests, then sends its data on to the master node to be written to carbon/graphite

Major Changes:

  • Don't serialize ets updates through a process
    • For counters, use ets:update_counter and ets:insert_new
    • For timers and gauges, use duplicate bags to avoid serializing through the process
  • Double-buffer methodology for tables
    • Create two tables for stats, gauges, and timers, and one table to indicate which table is the active write table for each group
    • When flushing, swap the active table so external processes can immediately begin updating keys againswitch

Minor Changes:

  • Track the "last flush" time so that we can have more accurate averages (by using now_diff to determine the actual time between flushes)
  • Added a bunch of edoc style comments (still in progress)

Zachary Hueras and others added 30 commits July 20, 2012 16:17
Now aggregates data accurately
Apparently you can't have multiple processes
receiving packets from the same socket, so
scrap that effort.
* estatsd_server now a gen_leader
* stats aggregation is now simplified (no more break_aggregate)
* adds TCP aggregation methods (compressed and uncompressed)
* separates VM metrics gathering from carbon reporting, allowing
  VM metrics to be forwarded as well.
Add reference to cluster.config in etc
for peer/destination configuration
Needs testing in multi-node and multi-tiered configurations.
* Support integers for sample rate
* Add "other" clause to parse_value to prevent crashing
  when there's a badly encoded value.
Use sets and count the number of times each duration appears, as
opposed to using a duplicate bag.
Zachary Hueras and others added 17 commits March 26, 2013 11:28
Seperate segments for particularly large clusters.
Increases the number of TCP packets, but also
improves the performance of gen_leader in larger
clusters (by node count).

Hopefully.
Was doing a superfluous accumulate on the entries
from statsd_gauge_agg; values in their are already
in the accumulated format.
Fix for gauge accumulation
Updated ranch from 0.4.0 to 0.8.4
Adds third optional parameter to timing function
representing the number of readings for this value.
Defaults to one, indicating no sample.
Support for timing samples over UDP
Update jiffy to later version
@soup-in-boots soup-in-boots force-pushed the master branch 2 times, most recently from 908a5a3 to 1c9deb2 Compare January 19, 2015 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants