So I\'m trying to push for more automated testing in my company, which focus a l

Question

So I'm trying to push for more automated testing in my company, which focus a lot on prototype and proof of concept systems.

We currently use Google Test for unit testing. This tests specific test cases for correctness, but a lot of the stuff we do the metrics for how the code is performing follow more of a "does it work better or worse than before, and if it works worse, is it an acceptable loss in performance" task than a boolean "yes no". These tests I call "Characterization" for lack of a better term (I've got a EE background, so sue me): it describes how the code behaves. We NEED to have the metric for all builds, and we NEED to be able to compare it between builds in order to see what direction we are heading. We can often make a general cuttoff to make a high-level pass fail, but these tests can be extensive: they take a long time to run and often require number crunching and plotting stuff. This seems to be out of scope for a Unit Test, more for a Functional Test in my mind (but I'm no test expert). How do people/companies handle this type of test? Is there any framework for this type of test out there?

Explanation / Answer

You are talking about performance, which is essentially an Non Functional Requirement.

My view is that most of these are for the whole system.

A user is never going to say "I want the WidgetListCollection class to return a widget within 20 nanoseconds" -- he might say "I want a list of all products on my screen within 1.5 seconds".

The only way to really test these is to test the complete system. If the users screen is refreshed within .25 of a second exceeding the NFR by 600% than it does not really matter if the WdigetListCollection class is sub-optimal. Any effort spent optimizing it could be best spent elsewhere doing something that matters to the user.

Worse if you test this stuff at unit test level not only does it encourage you to waste time on micro-optimizations, your test may raise false alarms:

Say I may make a change to the class so it is now 5 times slower, however, it now also caches the widget usage history thus making the entire system run twice as fast. But I get nagged at the weekly meeting for poor performance anyway.

Unit testing is good. But it has its limits, in the end its the system as a whole that must work to spec. Be aware that the system can still fail miserably even though all the individual components are excellent.

Related Questions

Navigate

• Browse (All)
• Browse S
• Subjects

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.