On automated testing and protocols

Well, last week I continued to work on memory allocation. Things are shaping up pretty nicely, and I start to be pretty confident that at least the core functionality (allocating a chunk of memory to a process for exclusive access) will be ready for the first anniversary of the project. With a bit of luck, sharing will be completed too for that date, and with a lot of luck “auxiliary” features like removing all traces of a process in the memory management system and switching address spaces (as preliminary work towards full context switches) will be finalized in time. In short, everything is fine in the world of coding.

In meantime, I recently started to think about my goal of reaching very high reliability, and it appeared pretty quickly to me that I forgot something.

Apart from structural robustness, which I’ve gone in great lengths about, software reaches high reliability by being extensively tested. All parts must be tested in any way its creator may think of (and preferably others), and the more vital they are the more thoroughly tested they should be. In the context of this OS in particular, memory management is needed by the kernel and part of it, so it must be bullet-proof, and should preferably be able to survive industrial CO2 lasers and H bombs as well. Failure is not acceptable, memory leaks aren’t either.

But how do I check that my memory management code works ? Until now, I just did random manual testing : each time I had a new idea for stress-testing the thing, I tried it and saw if the chunk of code passed the test or not.

This way of doing things is probably wrong on many levels, but the biggest problem by far in my opinion is that it’s not reproducible : if tomorrow I change a tiny something in my code, I won’t remember all the tests I’ve applied in the past to see if it still works after the patch. And that is really annoying.

So I decided that I should keep a list of the test cases I’ve tried : anytime I think of a new testing method, I should check if it does not duplicate existing testing method in the list, and if not add it to the list. This way, my testing method gradually improves as time passes instead of stagnating on the average and varying in quality depending on my amount of sleep and spare time.

Since parsing checklists by hand is one of  the most boring tasks and time-consuming in the universe, I’ve decided to introduce a second innovation : automated testing. I write the checklist above in an implementable way, and anytime I get a stable release of something, I’ll make an automated testing system which parses the current checklist and checks all of its items, telling where the failure is if there’s one. This, combined with a quick human review of the code, will allow me to test minor patches for regressions fairly quickly, so I think it’s a good idea to do it.

So guys, I proudly present the memory management testing protocol, to be put on the official doc and implemented before freezing memory management and moving to something else.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s