Nightwatch bridge and/or other processes do not stop on errors when starting the test runner #60
Comments
|
Hi Elliot, Once that's all created, you'll be able able to run the TinyTest-CI framework, and the tinytest results will be picked up the Nightwatch runner. Word of warning: this stuff is new, so you'll be reaching the edge of what the tool can do before long. But it's pretty open-ended what we can do at this point. For instance: it seems like we need to check whether the We're also working on an Atom package that's going to have a UI interface for running a bunch of StarryNight commands. That will give us an isomorphic environment and API between Client, Editor, and Test Harness. Lastly, I've got a ton of documentation in the works. Take a look at http://starrynight-beta.meteor.com for an early look at the upcoming documentation. |
|
Hi Abigail, Thanks so much for the quick response. Looks like that command has sorted me out with that original issue. Maybe there can be a 'getting started' section of the documentation with those two commands? Now I am hitting a new issue though: Running tiny tests on packages. Check http://localhost:3000 [10/404]
multiFramework::options undefined
Detecting release version...
Spawning Meteor instance on port 3300 for Package Verification testing using TinyTest...
I20150709-09:58:34.285(10)? test-in-console listening
/usr/local/lib/node_modules/starrynight/tool/frameworks/tinytest-on-server-console.js:44
options.tinytests = true;
^
TypeError: Cannot set property 'tinytests' of undefined
at Socket.startTesting (/usr/local/lib/node_modules/starrynight/tool/frameworks/tinytest-on-server-console.js:44:25)
at Socket.emit (events.js:129:20)
at readableAddChunk (_stream_readable.js:163:16)
at Socket.Readable.push (_stream_readable.js:126:10)
at Pipe.onread (net.js:538:20)Then, the second time I run the same command: $ starrynight run-tests --framework tinytest-ci
This command should be run in the root of an application.
Running tiny tests on packages. Check http://localhost:3000
multiFramework::options { _: [ 'node', '/usr/local/bin/starrynight', 'run-tests' ],
framework: 'tinytest-ci' }
Detecting release version...
Spawning Meteor instance on port 3300 for Package Verification testing using TinyTest...
Can't listen on port 3300. Perhaps another Meteor is running?
Running two copies of Meteor in the same application directory
will not work. If something else is using port 3300, you can
specify an alternative port with --port <port>.
Package Verification Testing exited with code 254
Detected a meteor instance on port 3300
Launching nightwatch bridge...
Starting selenium server...
started - PID: 5492
No tests defined! using source folder: /home/e/dev/bigspendr/tests/nightwatch/walkthroughs ; using tags: tinytests
Nightwatch exited with a code of 1I am defining my tiny-tests within my package right now using the Package.onTest() directive to specify dependencies and file load order etc (so that I can stub). When I run starrynight, is it loading my tests from individual packages? Or do these need to be moved somewhere else? Awesome tool - if I get some more hours up my sleeve I'll try get involved and make some PRs. Thanks for your help, |
|
Hi Elliot, Also, as I take the screenshots, I'm addressing more edge-cases, and flushing out bugs. So we're also going to bump versions within the next day or two, with lots of bug fixes. ps. I'm working on updating the scaffolds to have some default unittests and tinytests; and adding logic to detect that the necessary files are in place, and to prompt the user for action instead of throwing errors. |
|
You're a legend! Thanks so much!! |
|
Oi. I just finished making the screenshot/walkthrough of the Okay, so Also, I pushed online the screenshots of the new tutorials and examples. I still need to write text for them, but I think you may find the TinyTest Walkthrough to be useful as-is. Take a look at the preview site, if you wish: |
|
Hey Abigail, Great, thanks for that! I worked through the example you have under http://starrynight-beta.meteor.com/examples/tinytests. Have it now running the tinytest pickup, but not always displaying test results. Also I forced two of my tests to fail and the generated report is only reporting one assertion and reporting it as passing. Any idea what's happening? screenshot of tests in browser
command line outpute@m0ney:~/dev/project$ starrynight run-tests --framework tinytest-ci
This command should be run in the root of an application.
Running tiny tests on packages. Check http://localhost:3000
Detecting release version...
Spawning Meteor instance on port 3300 for Package Verification testing using TinyTest...
Can't listen on port 3300. Perhaps another Meteor is running?
Running two copies of Meteor in the same application directory
will not work. If something else is using port 3300, you can
specify an alternative port with --port <port>.
Package Verification Testing exited with code 254
Detected a meteor instance on port 3300
Launching nightwatch bridge...
Found .meteor/nightwatch.json
Starting selenium server...
started - PID: 9208
[Tiny Test Pickup] Test Suite
=============================
Running: Test Getting Logs
✔ Testing if element <body> is present.
http://localhost:3300/packages/test-in-console.js?59d9cde1f0563b9d9cc5b99297a0997b32d3f499 75:17 S: tinytest - getAwsRegions method throws to future when ec2 callout returns error
http://localhost:3300/packages/test-in-console.js?59d9cde1f0563b9d9cc5b99297a0997b32d3f499 75:17 S: tinytest - getAwsRegions inserts new regions where those returned do not exist
http://localhost:3300/packages/test-in-console.js?59d9cde1f0563b9d9cc5b99297a0997b32d3f499 75:17 S: tinytest - getAwsRegions deletes regions who no longer exist
http://localhost:3300/packages/test-in-console.js?59d9cde1f0563b9d9cc5b99297a0997b32d3f499 75:17 S: tinytest - getAwsRegions updates regions with new endpoints no longer exist
OK. 1 assertions passed. (2.604s)
Finished! Nightwatch ran all the tests!generated report:<?xml version="1.0" encoding="UTF-8" ?>
<testsuites errors="0"
failures="0"
tests="1">
<testsuite name="tinyTestPickup"
errors="0" failures="0" hostname="" id="" package="tinyTestPickup" skipped="0"
tests="1" time="2.604" timestamp="Mon, 13 Jul 2015 00:27:58 GMT">
<testcase name="Test Getting Logs" time="2.604" assertions="1">
</testcase>
</testsuite>
</testsuites> |
|
Hey! Yeah, we're about at the edge of development now. SpaceJam had a bunch of similar problems, which is exactly why I ported all of its functionality over to StarryNight/Nightwatch... to hopefully be able to hack on things and figure out how to make it more consistent. Obviously, part of the problem is that the console driver is writing to the console, and we have to do a pickup. TinyTest also writes a summary to a JSON object that sits in the browser console. It would be real nice to submit a pull request to MDG, and write all the results to the summary object. From a technical perspective, that would certainly be the easiest way to resolve the issue. But it also involves submitting a pull request (ug). If you want to try to put the pull request together, I'll certainly offer what assistance I can. Other possibilities include adjusting timeouts; making sure there are no stale process in place; moving synchronous code into async callbacks for more reliable timing; and clarifying the spawn/exit code process. If you're interested on hacking on things, the relevant files are below: https://github.com/awatson1978/starrynight/blob/develop/tool/frameworks/tinytest-on-server-console.js An 'npm install -g .' will compile and install a local version of starrynight. |
|
Hey, Yeah, I'd love to get involved. I'll get into it in my evenings here in Thanks for being super helpful - absolutely legendary response times and On 13 July 2015 at 11:05, Abigail Watson [email protected] wrote:
|
Hi Abigail,
Firstly, thanks for building out this tool! Velocity is rubbish and I'm hoping starrynight is the test framework for me. The doc look good.
Just having a little trouble getting started. In the process of trying to get everything started, it seems that if I have a fatal error when starting starrynight, not all processes that it spawns die and on the subsequent connection, the port is taken.
Then on the subsequent attempt
$ ps -ef | grep meteor e 7734 1091 0 12:27 pts/19 00:00:03 /home/e/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node /home/e/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/main.js test-packages --once --driver-package test-in-console -p 3300 e 7756 7734 0 12:27 pts/19 00:00:01 /home/e/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/mongodb/bin/mongod --bind_ip 127.0.0.1 --smallfiles --port 3301 --dbpath /tmp/meteor-test-runa8p3d5/.meteor/local/db --oplogSize 8 --replSet meteor e 7814 7734 0 12:27 pts/19 00:00:00 /home/e/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node /tmp/meteor-test-runa8p3d5/.meteor/local/build/main.js e 8357 3632 0 12:34 pts/19 00:00:00 grep meteorHopefully its just something crazy I have done wrong and there is no actual underlying bug.
Thanks,
Elliott
The text was updated successfully, but these errors were encountered: