A lot of data scientists will say that you cannot test a ML Model but in fact you can.
Iâve gotten on some projects where I used a Metamorphic testing technique to test a machine learning model. (also simple probability theory testing)
How you may ask? Because there is no expected result??
Very good question! But there isâŠ
Letâs say you are building a model to predict X. In a proper project this will not be your only acceptance criteria, what should also be acceptance criteria? The success rate. We are building a model to predict X with a 80% success rate. There you go, an expected outcome (just a bit different).
How do we approach this from QA?
Well a data scientist often gets training material (on which the model is trained) they split it up in multiple parts, 1 to train it on. 1 to verify it on for example.
This is what we want to do also. As a QA engineer, and this is not going to be simple, youâll have to make 10.000-100.000 cases to validate this.
Letâs say you are testing the value of a house. (in reality there are ~50+ parameters but in the following example Iâll use 1)
House A: build in 2020 is worth 500K
House B: build in 2015 is worth 350K
House C: build in 2010 is worth 250K
You wonât know the exact number itâs going to split out making validating it so much harder BUT you know the worth and can compare it.
Therefor House A > House B > House C
This is pretty easy. But what if you have 50 parameters and houses of the same year and maybe super similar to each other.
Youâll want the model to give you a list of values (estimated guesses of the worth of the house) and you as the QA Engineer will compare those values towards your test cases and if the order if correct for over 80% then youâve achieved your acceptance criteria. So basically 'Youâll create an order of house values in your test-set and compare it to the order, the model is giving. â
Do not underestimate the work of making testcases for this, again the above example is explained super simple, in reality there are 50+ parameters and making a test set takes some time and analysis.
DISCLAIMER: Every model is different, so for every model youâll have to use a different approach. Not every technique will work for each model, start thinking outside of the box
Now donât let data scientist say âyou canât test itâ
There is a way!