Anyone want to give me feedback on a security testing idea?


(Joe) #1

Hi! I am a product guy. I am working on some new ideas (no product yet) around security testing inside Devops pipelines. I envision a world where security testing can be easily added to any QA process - no waiting for results and no ongoing tuning or configuration – just fast, high quality findings. Of course, all of this integrated with your favorite ticket systems or CI/CD platforms :wink:

What challenges do you have with security testing today

I don’t have a product but I am looking for feedback, so if you have other ideas or want to provide some input, just reply here!


(Ady) #2

Hi Joe, our pipeline uses Team City and we have Veracode builds set up to run weekly. I also use Bug Magnet as part of day to day testing which has SQL injections.

Is that the sort of thing you are after?


(Joe) #3

Hi Ady, yes, that is exactly what I am after. Does the above solution work well for you? Or are there things you wish you could change or improve?


(Rosie) #4

Pinging @danielbilling @santhosh.tuppad


(Daniel) #5

Hi there,

Security is one of the most complex aspects of software development and quality. There really isn’t a one size fits all approach. Your approach needs to be organisation wide, and agreed with many stakeholders because of its wide ranging impact. A lot of it will be informed by the tech stack you sit within.

Have you done any threat modelling exercises? This is a must to understand where you need to target your testing. Combinations of defence in depth and breadth will need to be considered:

Static analysis approaches are essential, looking at errors in the code before it’s compiled or deployed. Dynamic scanning of the application under usage is also important, where tools like ZAP or BurpSuite can be useful. It’ll take a while to understand how these tools will be useful to you, by tuning them, monitoring and filtering results to avoid false positives, but bubbling up real bugs and false negatives.

If you are using exploratory approaches to your day to day testing, then the DAST tools can help you target where your testing needs to be most effective. Bug Magnet can be really useful here. I’ve added some extensions to Bug Magnet to my github, (github.com/danielbilling/security-test-learning) so you can add these if you like. One for XSS and one for SQLi.

These tools are powerful, and potentially dangerous if not used appropriately. Be wary. Let me know if you want some help or guidance.

Another aspect to consider is the monitoring of applications in production. What errors and usage patterns are kicked up when real users interact with the application. Does it surface nasties such as information disclosure, sql statements or sensitive data in the logs. Does this stuff get surfaced to the user in a way a hacker could exploit? Look for using tools like Papertrail, SumoLogic or New Relic to monitor your app in production.

Maybe your ops guys have advice here, especially around things like intrusion detection and monitoring, to identify weak points and areas of interest.

Drop me a line if you want a chat.


(Santhosh ) #6

Hello, Joe!

I would personally tamper with your question and rephrase as - “I envision a world where security checks can be added to the development cycle or process or activity”. (Testing Versus Checking are two different things in my opinion and I strongly believe in what I believe). Anyways, understanding the depth of your question here are my thoughts.

Challenges that we face today?

  • We have a massive skill shortage compared to the black-hat world of hackers.
  • Industry-based white-hat hackers are trained to do “X” number of attacks and on the other hand, black-hat hackers are beyond white-hat hackers skill-set (usually).
  • Customers / People failing to understand that, security testing is a continuous process and not just one-time activity. Educating people (customers) is a key here. How to do it? Well, it’s time-consuming and I am trying to crack this nut in a better way.
  • We have misconceptions about automation or DevOps or Tools helping us to create fool-proof systems. This is one of the biggest challenges where we need to educate professionals about how these ideas by themselves don’t do the job “well done”. Black-hat hackers who are the evil customers of your product are not using DevOps or Automation to hack into your product. They are something else and are equipped with a criminal mindset, script kiddie attitude, skills like investigation/analysis/critical thinking/general systems thinking, choosing the right tools in order to implement an idea or attack generated in their brain through thinking skills. In the current world, it looks like we are trying to fight the monster by running away from it. I don’t call it as “fighting” the “monster” (I am referring to usage of ONLY automated checkers, DevOps, and other fancy ideas. Well, this helps; but not really helpful when we don’t mix these with real skills of hacking. I insist the world have professionals who are hackers and have good intention to help the software industry).
  • Disable the IT policies whenever hackers are on-board. Most of the bigger companies have an environment where it’s hard for security test professionals to install their arsenals.
  • Every-day black-hat hackers are working on inventing new attacks and that’s why OWASP community keeps adding or updating new attacks every year. We need white-hat hackers working inside a company full-time instead of using security services only after all other tests are done (functional, usability, performance). And I also I strongly hate the world when they call it as “Non-Functional”. Nuts! We need to work on hiring kickass real-world hackers inside the company working on “secure coding”, “providing consistent feedback on day to day implementation”, “working with developers, business analysts, product owner, sales, marketing and what not to bring in the value”. Alas! Currently, we don’t have such agility and powerful teams across the world. We need change!
  • Last, but not least. Passion is an ingredient that’s tricky and it’s missing heavily. I am not complaining, but being a realist here (Probably).

Now, let’s speak about how I can try to help you with security checks integration in your development process or activity. Also, I assume that your application could be web platform (If it’s not, I request you to provide me with the details or an example).

  • “Design is critical, so are security tests”: Upon deciding on the technology stack that you would like to use for your product, identify the “secure coding guidelines”. However, before that, it’s good to test the design or architecture of your product. Hire the hackers who have skills and mental modeling of finding smaller to critical vulnerabilities in the design phase. Well, in “Security” even a “smaller” vulnerability is “bigger” based on the intelligence of hacker in terms of exploiting the vulnerability. Careful, please!
  • Identify the tools which can find security misconfiguration errors in the coding or configuration of infrastructure like servers, load balancers, headers etcetera. Please make sure that you analyze the results and get rid of false-positives. In my experience, I have come across false-positives which were fixed and then I started seeing functional bugs. You need to know what you are fixing.
  • Identify the scripts on Github which do “intelligence gathering” or “passive reconnaissance” or probably use “Maltego”. There are various editions where you can pay for extra features or just use community edition that is free. It will generate the report for you. I also use “Passive Recon”, Netcraft Site Report, MailTester.com and more tools. You can also take a look at http://osintframework.com/ to get greater ideas and also a list of tools. However, once the intel is gathered, someone with skills of analysis this data has to go through it and identify risks or leaking this intelligence. The counter-measure can be blinding or hiding this intel from eyes of prey.
  • http://packetstormsecurity.com/ You or someone from your team can explore all various “payloads” from this website where there are plenty of files for attacking. They include OWASP Top 10 Payloads (wherever applicable) and also, customize the payloads to suit your product needs in terms of attacking successfully. Running them blindly can be waste of time and resources which could be expensive spend.
  • Identify all the web browser plug-ins and create automation script so that these plug-ins or add-ons can run the checks overnight or weekly once or whatever timeline you think suits you. To name a few, I use “Tamper Data”, “HTTP Headers”, “XSS Scanner”, “KnowXSS (Paid)”, WebSecurify.
  • Identify the scanner frameworks and configure them to run whenever there is a new build. For example http://www.arachni-scanner.com/ (Trust me, there is a huge number of scanners. Never mind in combining the power of all of them or more scanners. Every scanner has got a way to help you).
  • Run Active and Passive Scanners using the appropriate tool (free/commercial). I use BurpSuite extensively.
  • Also, explore the addons from BAppStore to integrate them with BurpSuite.
  • Bash Scripting, Python Scripting combined with Kali Linux can be a powerful skill. Using Python, one can use the extensive libraries and scripts from Python in order to integrate the security checks or any security-related activities to run overnight and smoothly. I love Python. Not the reptile J
  • I think Dan Billing has also helped with other ideas like log files and analyzing the log files, Bug Magnet etc. which are equally powerful.

Disclaimer: The tools and scanners I have listed here are with the assumption of product being “Web”. Well, the concept can be used wherever applicable (different platforms). If you can help me with more details or imaginary product that you have in mind, we can dig deeper.

Also, I personally cannot answer deeply if the question is not too specific when it comes to “Security” topic. To me, Security is a vast area in an unbelievable way. I will stop here to see if you can help me with more details or specific question. I am here to help!

Remember, “Checks aren’t sufficient and Tests aren’t sufficient if the skills are missing”. “Hire the best” or "Hire someone who can learn better with self-learning attitude or under someone’s coaching.

Cheers!


(Joe) #7

Daniel,
Thank you for the insight! This is very valuable. I’m very familiar with ZAP and Burp (am a user of each). In fact, I am hoping to extend these tools and make them even more ‘friendly’ to devs.

I’m actually starting to pursue something entrepreneurial in this space… would you be open to a 30 minute Zoom meeting so I can get your feedback?

Thanks!


(Joe) #8

Thank you for sharing these tools, Santosh!

I am familiar with some, but others look like interesting scripts. What would you say are the biggest problems in this space?

Would you be open to a 30 minute discussion so I can share with you what I am thinking?


(Daniel) #9

Yeah, I’m happy to chat. Hit me up here, on Slack or on Twitter to arrange. @thetestdoctor


(Joe) #10

HI Daniel,
Great! Thanks for taking the time to meet. I am a product manager at CA Technologies focusing on building a ‘startup inside the company’. We are conducting problem and solution interviews, so your feedback will be very valuable!

Do you want to grab 30 minutes on my calendar? You can find a timeslot here: https://calendly.com/jp-product/solutioninterview

Looking forward to meeting.

-Joe


(Ady) #11

Hi Joe, it works really well to be fair. There’s always thing we wish we could improve, environment stability etc. but we pair Team City with Octopus Deploy so there’s good turn around between the devs and myself. They can give me a shout when a change is ready to deploy and I can be testing within a few minutes. The company I work for also has a dedicated security team which we can call on and we also have annual external and independant reviews.

All that said, Veracode popped up an issue last week which has been present since go live over a year ago so it’s only as good as the checks that are done and you have to accept you are in a changing environment and new threats can appear at any time.

Good luck with your work fella.