Discussion: Security Testing

Tonight, @testerfromleic will be joined by three fantastically knowledgeable guests @saskia, @anneoikarinen and @coffeefueled to talk about security testing :lock:

We’ll use this thread to add questions we don’t get to during the session and any resources that the presenters mention.

If you’d like to continue the conversation from the session, this thread is an excellent place to do that :grin: Share resources and follow up success stories from your learnings here!

Questions we didn’t get to

  1. @dancaseley: Is security testing a “thing” in its own right, or is it an aspect of every other type of testing? Or is that just a lens thing?
  2. @dancaseley: Security testing suffers the same fate as a lot of other non-feature-driven types of testing. It happens at the end, like performance and accessibility testing. What kinds of security testing activity can I schedule for Sprint 0 & 1?
  3. @dancaseley: How could you estimate the amount of time required for security testing activities at the beginning of a project? This is sometimes a necessary evil of an agency model, and without it they might get skipped.
  4. @dancaseley: What can we do to ensure the security of testing itself? That tests or pipelines aren’t interfered with, that results can be depended upon, that vulnerabilities haven’t been suppressed, etc?
  5. @dancaseley: I’ve got a low severity CVE in a dependency. A new version was released today, which, among other things, fixes it. Do I always update? Aren’t there other inherent security risks? For instance, should I be trying to pentest all of my dependency updates?
  6. @viola: If you are given an app and you are given a chance to test just 1 idea, what would you selectwith number one priority?
  7. @dancaseley: If I’m not a bank or a civil servant, are adversaries at the sophistication level of nation states something I should worry about, or include in any threat modelling?
  8. @andy_hird: If not certification, what do hiring managers look for in beginner security testers?
  9. @bjpalmz: How do you promote good security practices as the only tester in a team?
  10. @dancaseley: To what extent do NCSC (or country-local equivalent) represent “best in class” advice for cyber security?

Resources mentioned

Saskia’s Manchester talk

Anne’s TestBash Home talk

Elevation of Privilege (EoP) Threat Modeling Card Game

Cyber Bogies


James recommended this book to learn about thread modelling

Club threads on getting started in certain specialities, security included Club Posts To Help You Get Started With Software Testing And QA

OffSec do a couple of free courses: https://kali.training/ and Metasploit Unleashed - Free Online Ethical Hacking Course




To throw in my own answers (which are not authoritative, just mine):

Perfect world answer, security is simply an aspect of everything else, not a separate area at all. That applies to testing as much as anything else. This only works when every area of a business and every person is able to (as in has the expertise, resources, and autonomy) take ownership of their own area of security.

Threat modelling at sprint 0, which can inform design principles and functional testing for each subsequent sprint. In sprint 1, update the threat model (also at every subsequent sprint - unless you can fully test everything you need something to guide your security testing, and that’s the purpose of a threat model).

Flippant answer, make an estimate and double it. It’s a hard one to answer without more detail. If you take security as a functional requirement (looking at misuse cases throughout the process as an example) then you can use the same methods as for any other testing. For the final penetration testing step, it’s more a case of deciding what your risk appetite is, and a conversation with your pen testers will be able to help with that.

That’s looking into integrity testing, so version control, good change management, solid audit trails, will all help. Particularly around findings from security testing, having strong risk management and ensuring that ownership of findings is recorded and attributable to an identifiable accountable person is key.

Risk appetite is key here. There is no absolute answer. What’s the impact of the CVE, is it viable for an actual attack? Does the update break anything or raise any other risks? Pen testing all dependency updates isn’t realistically going to happen, and would slow the update cycle to a crawl in any case. This needs a dialogue with the wider business to understand the priorities and risk appetite, and with the security team to make sure that risk appetite translates into appropriate action.

Thoroughly fuzzing every input.

When teaching threat modelling I often use the example of an alien invasion. Making sure the threats you are working with are realistic is key, as it’s perfectly possible to dive down a rabbit hole, concerning yourselves with threats that take more resources than are ever going to be provided to mitigate, and/or which are never going to concern themselves with your systems.

One way I’ve seen successfully used a lot is speaking at rookie conferences, writing about what you’re learning, and generally talking about things. Hiring managers are actively looking for people, and putting out content on topics that interest you seems to be successful at drawing attention of hiring managers, particularly where you’re talking about a niche topic.

I’ve not been in this situation, but I have found that generally people actively want to do security right when they understand it and feel that they can do something about it.

No one-size-fits-all advice will ever be best in class, but NCSC and similar do put out good basic advice. It will never compare to a proper assessment that puts work into understanding individual organisation context, but is worth following if you don’t have that option.


The recording is now live :tada: