testing guidlines

[anandkumarpatel]

Writing some notes down before committing so everyone can agree and provide input

[anandkumarpatel]

we should break down best practices and guilds for unit, functional, integration test.

unit test

stubbing

for unit test most external modules should be stubbed. the exceptions are util library like async, 101, ip, ...etc
we should use sinon for all stubbing. it has seemed most valuable from all the subbing modules we have tried.

usage

we should stub in beforeEach and restore in afterEach
example:

  describe('handleDied', function () {
    beforeEach(function (done) {
      sinon.stub(Events, '_isWeaveContainer');
      sinon.stub(Events, '_isThisHost');
      done();
    });

    afterEach(function (done) {
      Events._isWeaveContainer.restore();
      Events._isThisHost.restore();
      done();
    });
})

Asserts

for general value asserting code should be used

expect(value).to.be.true()

for funtions we should use sinon asserts
sinon.assert.calledOnce, sinon.assert.calledWith

functional test

functional test should run the entire application and use sinon to stub only modules which make external request. based on experience we should avoid using nock since it has lead us to some problems. functional test should just test the applications logic not external models (thats what integration test are for). functional test should test input and output of the application

integration test

these should be run without stubbing anything. runnable provides a great way to test with all our servers.
should be similar to functional test. testing input and outputs of the system.

[cflynn07]

I've at times failed to stub 100% of external method invocations of a function under test in a unit test. I'm going to make it a strong point to make sure all external method invocations to the method under test are stubbed whenever possible for unit tests.

Thanks to @Myztiq I've also seen the light with Sinon's assertion and stub APIs (ie, when to use .returns and .yields) so I'll be using those from now on.

[Myztiq]

I wholeheartedly agree with this sentiment. There are a bunch more things we should add here, like using sinon.assert.calledOnce, sinon.assert.calledWith. And perhaps? we should talk about where .returns and .yieldsAsync belongs. Personally, I think .returns and .yieldsAsync should be setup with the stub for the success case. On failure cases we should sinon.assert.notCalled those items.

[anandkumarpatel]

• 1 for sinon.assert.calledOnce, sinon.assert.calledWith will add that above. as for yields I think we need a bit more discussion.

will write something more once I have time :( prod issues

[bkendall]

I think I'm going to start putting some PRs towards this repo soon so we can start consolidating some of these ideas. I find error in your definitions of functional and integration tests, however. In the hierarchy of tests, I usually see unit, integration, and functional.

unit is pretty obvious (or should be). It tests exactly one unit of functionality. Most likely this should follow the boundaries of functions or at largest, modules.

integration tests make sure that one or more components function correctly together. This can be multiple functions within a module, multiple modules interacting with each other, or a module interacting with a database or external service.

functional tests should align with product ideas. If a user goes through a given flow, they should finish with some outcome. We've found these tests to be pretty difficult to work with as of late, and are trying to get rid of some of these.

An interesting read on how 'value' could be perceived in testing is one of the posts on Google's Testing Blog. It's not super long, so I encourage anyone to read. They very much encourage unit and integration tests over end-to-end (I read it as functional) tests as they help the developers much more and have the exact same value to the customer as any end-to-end test: a bug fix. If you can write a unit test for your bug rather than an end-to-end test, wouldn't you rather do that?