On documentation

[alesch]

Hi Abigail,

Thanks for your efforts!
In the docs I did not find your email, so I started this issue instead.

I've just given starrynight a try and these are my notes on the docs.

starrynight.meteor.com

• Is this a github site?
  So you can get help spotting and fixing the docs.
• Version number at the top is 0.2.6.1
  Shouldn't i be 3.2.6.1?
• It is not clear where to start
  I suppose the examples in the doc site.
• Is people encouraged to create issues and help with pull requests?
  I guess so, but it wouldn't hurt to state it.
• How do you run nightwatch unittests?
• What database will the tests connect to?
  Can I specify a different one than the default? How?

github.com/awatson1978/starrynight

• The readme is different than the documentation site.
  Maybe keep a really simple readme pointing to the doc site?
• There is no reference to the documentation site.

General questions

I see in the generated nightwatch.json that tests can be run with phantomjs. How?

As starrynight did not install on OSX, I created a Meteor project in cloud9 (Ubuntu). But I did not manage to get Nightwatch connecting to selenium. It would be good with a checklist of things to check for troubleshooting.

At the end of the day, I still did not manage to get nightwatch tests running.

[alesch]

I managed to run nightwatch unittests by calling nightwatch directly.
I had to add ./tests/nightwatch/unittests folder to src_folders in nightwathc.json.

nightwatch --env unittests --config .meteor/nightwatch.json

How do you pass --env to nightwatch?

[alesch]

Looking at the code, I assume that the way to do it would be something like:

starrynight [run-tests --framework nightwatch || --type unit]

Unit is described in the help output, but the case for it is not defined.
Lines 113, 142 and 148 are confusing: the text does not match the launched framework.

[awatson1978]

Hi!
Great questions. Generally speaking, you're jumping into the deep-end of the pool by starting with the unit testing, since Nightwatch was originally written as an acceptance testing framework. But that's fine. :)

• Website for the StarryNight utility is located within the repository in the /website directory.
  https://github.com/awatson1978/starrynight/tree/develop/website
• Pull requests totally encouraged.
• We pulled some unneeded libraries this afternoon, and bumped the version number. Also pushed a fix for Fails to install in OSX #55.
• Lots of documentation on it's way, as listed in New Documentation Checklist #57. That should help people get started. In the meantime, follow the Examples walkthrough.
• For unit testing, you'll need to specify the filter field, rather than the src_folders field. See the Nightwatch.js documentation:
  http://nightwatchjs.org/guide#unit-testing
• To pass an --env to nightwatch, you'd want to do something like this:
  starrynight run-tests --framework nightwatch --env phantom,chrome

You'll need to edit your nightwatch.json config though. StarryNight can do that for you, if you starrynight generate-autoconfig. It will leave a ``.meteor/nightwatch.json` file for you that you can then edit.

[alesch]

Thanks for your answers.
I got unittests running like this:

1. starrynight generate-autoconfig
2. As my unittests folder is at the same level than ./tests/nightwatch/walkthroughs, I added it to src_folders.
3. starrynight run-tests --framework nightwatch --env unittests --autogenerated

For phantomjs:
starrynight run-tests --framework nightwatch --env phantomjs --autogenerated

[awatson1978]

Sweet!
To be perfectly honest, I've personally never tested it with that particular configuration before (phantom + unittests + autogenerated). I've certainly used each component in isolation, but not together. So that's great to hear.

If you got all that working, the rest of the Nightwatch/StarryNight features should all fall into place, like tagging, screenshots, logging, chai expect syntax, scaffolding, environment analysis, refactoring, etc.

Thanks for the feedback and have fun!

[alesch]

I think that claimed victory too soon.
How do I access the Meteor code that I want test, from my unittests?
Say I define a global variable X, how do I get access to it from a test?

[awatson1978]

Interesting question. There's a couple of different approaches, depending on whether you're wanting to test server, client, or package code.

The primary issue though, is whether or not you've written your code to be accessible in the first place. And that generally means naming functions, and cleanly attaching them to known objects (ie. Meteor, Templates, or on an isomorphic object exposed from a package). But lets assume you are.

Take a look at the following code, which uses unittests in the acceptance test runner, and which can hook into a running app and test code on the client and from packages. That's one way to access running code. (This is akin to running an engine diagnostics meter while your car is running.)

https://gist.github.com/awatson1978/e9fd118781e68c5e851e

To access static non-running code, the functions need to be attached to an object that can be found by the runner. This is similar to opening the hood of your car and looking at the engine while the engine if off. It's also where I'm a little less clear on how to use Nightwatch, as it's a new feature and being developed by others. But I'm more than happy to help.

If we look at the unittest example, we see the following line:

var Utils = require('lib/util/utils.js');

We need to extend this with something Meteorish...

var Foo = require('client/components/foo.js');

But that foo.js file isn't entirely guaranteed to have a module.export or otherwise export any objects. So, we may need to write a wrapper for it. If we're lucky it maybe has a reference to Template.foo.x and we can test on the Template object. But I wouldn't hold my breath. We may need something more like this:

var Template.foo = require('client/components/foo.js');

My guess is we're getting close to needing to stub the various default Meteor objects. Velocity has a project which is relevant; and I'm thinking we may want to grab the following file and figure out how it gets integrated.
https://github.com/meteor-velocity/meteor-stubs/blob/master/index.js

So, here's what I'd recommend trying:

• copy index.js to stubs.js
• add stubs.js to the src_folders path and/or filters path
• see if we can test something from the stubs file
• if not, try wrapping the stubs file in a export.module
• try adding Meteor client files to the src_folders path and/or filters path
• try adding Meteor server files to the src_folders path and/or filters path
• see if we can test something from the server / client files using unittests
• write an export.modules wrapper for Meteor files, as necessary

See if you can get anywhere with that. I'm writing documentation, and will be trying to get some of these answers written up and posted.

[alesch]

Maybe this is naive but...
would it be possible to ask the meteor tool to build the source for us? and then use that as our context for the test?

[awatson1978]

Oh, I don't think that's naive at all. Not 100% sure it will work. But it may be an excellent approach. For that, we'd need to reference files in the .meteor/local/build directory. We have a file scanning tool that's working great. We just need to figure out what else to maybe add to the scanning rules. If you can dig around in .meteor/local/build and make any progress putting files from there into the src_folders directory, please let me know.

[alesch]

I've been trying munit and package testing (both tintytest and velocity).
My conclusions so far are:

Integration tests and scope issues

Regular package testing seems to be much better supported than velocity integration tests.
So maybe the solution is to package my app into several packages, and then I get to choose which parts of the ecosystem should be available to my tests.
This is indeed a bit of extra work, but tests run quicker than with regular velocity.

Which runner to use?

I definitely do not want to rewrite my tests because I choose another runner. The ideal would be to write the tests in one framework, and then be able to choose runner freely. For example, as a Webstorm user I'd like to have a runner that is integrated with my IDE.

At the moment it seems that jasmine is the best supported runner in velocity, and velocity seems to be MDG's choice (right or wrong, that is the reality).
So I'd like to write my tests in jasmine (or mocha) and be able to choose a runner afterwards.

My guess is that there many other users in my situation, and if you'd like a wider adoption of starrynight, a path compatible with velocity could a wise strategy.

[awatson1978]

Hi Alex,
Thanks for the feedback. MDG might be choosing Velocity, but there's a portion of the Meteor community that doesn't agree with that decision, the Velocity architecture, or how the management of the Velocity project has been handled. Long story short, I've been funded specifically (among other things) to provide an FDA ready testing solution. And since some members of the Velocity project chose to team up and vote out our contributions to what was advertised as a community project, I'm basically now being funded to provide a community alternative to Velocity.

As you may have read about, my clients and our partner organizations are in the process of publishing a Clinical Meteor Track, where we'll be adding D3 back in, and replacing Velocity with Nightwatch. Because of the way that management of the Velocity project was handled, it's actually going to get murkier as the Clinical Meteor Track is published, since Velocity is not going to be the Clinical Meteor Track's official runner. (Similar to how some people use React or Angular instead of Blaze).

As for compatibility between StarryNight and Velocity, it would be fairly straight-forward to integrate the two. Simply need to add Velocity as a framework in the same way TinyTest was added. If somebody else wants to write that code, I can provide a pattern and files to follow, and would accept pull request for it. But I'm also not going to personally waste any more of my own time trying to integrate it.

As for getting additional people using StarryNight, there's already the healthcare working group and our partner organizations who are invested in using its features. It will grow simply because we've got a stable and growing base of clients, are currently interviewing and hiring developers for StarryNight/Nightwatch, it's going to be the official runner of the Clinical Meteor Track, and graduate students and interns are being assigned to use StarryNight instead of Velocity.

[alesch]

Thanks for the extra info. Great to know that you've got funding to get this working.
I love D3 and agree that Velocity is too complex.
I'll be eagerly watching your team's progress.