How to Test Facial Recognition Systems?

A recent app update on my phone got me thinking about different testing scenarios. The app update in question said something like

Altering the default login from facial recognition back to fingerprint authentication.

Now, this makes sense currently when a lot of the world are wearing face masks/protection and the default login of facial recognition shouldn’t work in that instance. The whole thing got me thinking about the technology behind facial recognition though.

Could you share ideas on how to test it? What are some edge cases? How to approach it?

Ignoring the ethics of if we should be using facial recognition in the first place (that’s a whole other discussion)…

Colors (is it as reliable for people of different skin tones?)
Colors (is it still reliable if a pasty-white me went out and spray tanned myself orange?)
Colors (How about if I put on make-up?)
Colors (How about if someone of a different skin tone put on make-up?)
Colors (If the light level in the room was darker / brighter, how would that affect the results?)
… And how similar would someone need to be to trigger a match? … Children might fall into the “close enough” category, and depending on the application, it might not be good enough. Identical twins would definitely be a match, and there are a lot of twins out there. Is there something to prevent a twin from imposing on the other twin’s privacy?
… And what if I looked at the camera from different angles. Would it still match me as me? or would I be someone else?
… And the people from the previous cases, would they still be identifiable?

If someone’s eyes are open, closed, red from lack of sleep?
Do they have a rash or a new injury?
Are they in the middle of talk-like-a-pirate day and have an eye patch?
Do they have a new collection of facial hair?

3 Likes

Unlike most facial recognition systems currently being deployed which are fixed installations in one location (and taking @brian_seg’s comments on board), using facial recognition as an authentication method on a phone will have to cope with a range of different lighting conditions and backgrounds. So can such a system cope with:

  • backlighting
  • highlighting from the left
  • highlighting from the right
  • underlighting (from below)
  • toplighting (from above, such as outdoors at noon)
  • low light
  • excessively bright light

And can the system discriminate between:

  • complex abstract backgrounds
  • backgrounds with geometric elements
  • backgrounds with other faces (i.e. advertising hoardings or posters, or in a portrait gallery)

and can it cope with those situations in different lighting levels (either high or low light might affect the way the system can discriminate between faces and backgrounds).

2 Likes

Ethically I don’t feel comfortable with these systems being used in our lives for various places(for the ‘good’)…so, I’m not going to test or think about testing them.

2 Likes

I’d like to add
Colors (Is the current light source cold (think bright white LEDs) or warm (think sunset))

2 Likes

I’m not sure if those are edge cases, but what about:

  • Alive detection (can be system be fooled by a printed picture? Moving a pen before the eyes on the picture to simulate blinking? A video?)
  • Performance / Overload (What happens if the system is faced with a crowd of people? How many faces can it process in a reasonable time frame?)
  • Body modifications/changes (glasses, contact lenses (even the fun Halloween ones), piercings, tattoos, band-aids, acne, scars)
  • The whole range of face and body types in the human population (not everyone has two eyes, not everyone has eyes with the same color, not everyone has a nose, …)

While this is a fun thought experiment I also agree with @ipstefan that I wouldn’t want to work on those systems since the potential for abuse is very high

1 Like

Looking at the things we came up with so far I wonder which kind of facial recognition you are interested in, @heather_reid?
Are you thinking about using the face as an authentication token, detecting known faces in a crowd, tracking faces, … ?
The range of application for this technology is wide and the test approaches and edge cases will vary from application to application.

With my edge cases above I was mostly thinking about the “face as an authentication token” application.

1 Like

I was kind of there too… since the initial post used changing of authentication systems as an example.

1 Like

Ah yes I could have made that clearer. So in the particular case I was referring to, it was for an authentication token.

It’s not something I’ve ever allowed this app to have access to, neither is the fingerprint recognition but the app update made me curious.

1 Like

I think we need to separate the technical from the ethics…and recognise that our sense of ethics might also differ (which makes it even more complex). For now we can just focus on the technical, and when confronted with an ethical issue at work, it is up to the individual to negotiate a consensus

Technical and ethics go hand in hand.
I wouldn’t have a problem if the ethics would be discussed first, laws, rules and world governing bodies to overview the use would be in place, before the technology being implemented/released.





If you can’t control it, why would you build it, sell it or give it away for free - then regret and beg authorities for some regulation?
Facial recognition is not very different from data tracking/usage/manipulation; e.g. Cambridge Analytica and the influence they’ve had over multiple countries, leadership and policies(see movie The Great Hack);

1 Like

For now we can just focus on the technical, and when confronted with an ethical issue at work, it is up to the individual to negotiate a consensus

That is a valid approach but in my opinion it’s a dangerous one.
So many products have been built where it’s clear that it was all about solving a technical challenge (thinking about FindFace for example) and once the product was out in the world it got abused in ways which should have been foreseen.
Thinking about ethics before solving the technical challenge may make you realize that this cool idea you have is actually dangerous or has negative impacts on peoples lives / the environment / …

To get back to an overused example: The emissions cheating scandal in the automotive industry was caused by solving the technical problem “Make the car emit the allowed emissions during a test cycle”. The technical solution is very interesting and a good example of the capabilities of technology and engineers.
However it is not ethical (at least according to the codes of ethics in the tech domain that I’m aware of) and ignoring the ethical aspects had many negative effects (including financial effects) for the involved companies.