How different is this from artillery?

[danielo515]

Hello,
Thank you for such a nice benchmarking tool. I tried to use artillery before, but I found it a bit complex. Your approach seems simpler and easier to use. Do you know artillery? https://artillery.io/ How different is your project from it? Could you point some features your project has over it?

Maybe mentioning them on the docs could be a good idea.
Thanks in advance

[jeffbski]

Well I'm not that familiar with artillery, it wasn't around when I released the first version back in April 2014. We need an easy way to benchmark our system and record the metrics over time so we could watch for performance regressions. At the time there was nothing else like it out there, so I pulled together a few great Node.js modules and bench-rest was born.

It looks like the first release of artillery was in November 2015. They also use request for REST calls but they use stats-lite instead of measured. It looks like it supports testing web sockets as well.

bench-rest is 398 lines of code in a single file, while artillery is much larger. Whether that is a good thing or bad thing depends on your perspective and what features you want or need out of a benchmark runner.

Feature wise it looks like they have included the kitchen sink, but I'd have to leave it to you to determine how to use all those features. :-)

So I guess my answer is bench-rest is a simple command line app or Node.js library that can be used for creating multi-step REST flows and measuring statistics around them. I use a fairly simple DSL-like json structure to define the scenarios (flows) and it allows you to hook into the flow before and after the requests using a JS hook. You have access to the common metrics during and after a run, but it is up to you to determine where to store and how to chart.

[danielo515]

Thank you for your detailed answer.

As I said, I find artillery to be a it overcomplicated. I also prefer your approach, using it as a npm module, so it is more code over co figuration, while artillery is just a configuration file.

The only things that I miss from your implementation are :

• Ramp up and load increase. My server becomes unresponsive if I start with all the request directly, but with a bit of warm up works fine.
• Report generation. Artillery generates both Json and html reports. I have to admit that it is a very convenient feature.

Regards

[jeffbski]

I wonder if you could possibly just run some flows at a smaller concurrency for warm up and then run the real run. Basically having one or two warm up runs, then the real one might work. Just an idea.

Generated reports is a nice feature.

[danielo515]

Hello @jeffbski,

I'll try that way.

Another nice feature of artillery is the --insecure option. Does Bench-rest has something similar? If not, could you think any workaround?

This option is very useful for testing https on stage or even locally.

[jeffbski]

You can specify any of the request options in your actions, so add strictSSL:false

https://github.com/request/request#requestoptions-callback

[milahu]

I tried to use artillery before, but I found it a bit complex. Your approach seems simpler and easier to use.

for more simple/generic solutions see performance regression testing
aka "is the new version faster or slower?"

bench-rest and artillery focus on HTTP performance

[danielo515]

I tried to use artillery before, but I found it a bit complex. Your approach seems simpler and easier to use.

for more simple/generic solutions see performance regression testing aka "is the new version faster or slower?"

bench-rest and artillery focus on HTTP performance

Thanks, that’s a very good pointer, I’ll check it out. However their own author mentioned that this library form to ensure no performance regressions where introduced