How to test a flight simulator?

šŸ—„ļø Archive
1

I spoke to @neillb about this when I came back from a recent trip to Australia. I didnā€™t sleep for over 30 hours on the trip over. Sounds crazy right? What kept me awake on the trip over? Well when I told Neill it got him thinking too, letā€™s share the sleep deprivation :laughing:

I had read a lot of testing blogs before we boarded the plane. I also had a lengthy discussion with my husband about how pilots are trained. We wondered how a flight simulator could prepare them for all eventualities. He eventually drifted off to sleep but the seed had been set in my head. I started to think (and overthink) about how I would approach testing a flight simulator.

  • Where would I start?
  • What list of things would I test that a plane (via the simulator) should do?
  • What would I check that the simulator shouldnā€™t do?
  • Would it actually be me testing it? Would it be a pilot? Would it be both of us?

As I got more sleep deprived, I started to think about the requirements. How would you check the requirements for such a device/machine? Could you ever truly achieve an exhaustive list? (I didnā€™t think so).

I think if I was testing such a machine Iā€™d look to see how many safe landings in challenging conditions were covered. (Iā€™m thinking the fog in Gatwick and the coastal winds that hit Donegal airport as general conditions).

Would you cover crash landing scenarios? Iā€™m still undecided on that one. I imagine youā€™d have to cover it in some capacity but Iā€™m not sure how much.

How would you test a flight simulator? Was my sleep deprived mind raving mad? :see_no_evil:

1 Like
2

Hi,

Hereā€™s some things about the flight simulators and pilots:
Based on the locations they have to fly to, on the number of times they flew there, on the airport classifications and aircraft types, on the type of position the pilot had in the cockpit for the previous flights(copilot/second/trainee or captain), and other factors - in the last 12 months(usually)ā€¦they have to perform simulator flights if they want to get the license to fly that plane to that airport.

A flight simulator should simulate the real flight conditions. Iā€™d assume these components being involved:
Hardware of the airplane - which is a ā€˜simulationā€™ of the real airplane.
Hardware from inside of the airplane(cockpit) - probably the real one?
Internal Software of the airplane - which should probably be almost the real software - but for multiple types of airplanes.
External environment Software - simulated part of the environments(weather, airplane movement, contact with objects, and other variables)

Which hardware or software youā€™d want/have to test and in which combination?

Other information: most of the software in the simulators and aircraft is based on model driven development, each line of code has to be traced back to requirements. Other information about how software is built:


Then there are the companies that provide various parts of the hardware or software. Do you know for which one youā€™d be working for in your scenario? I just found an example of one of them.

On the aircraft simulators:


As Iā€™ve noticed and was expecting already you have to comply with lots of standardsā€¦dozens.

Then thereā€™s the usage of the simulator: personal use, commercial or private use, military, etc.

And so onā€¦A lot of things to coverā€¦
Soā€¦Unless you start to learn a lot about a lot of things youā€™d have very little ideas what would be useful to test and how.

3

I donā€™t see this as a bad thing. As testers weā€™re always doing this I think.

5

Given that you commented in the MoT Slack that you were thinking about both simulators for commercial pilots, and sims for ā€œJoe Publicā€ to try out the feel of flying an airliner, Iā€™ll add my own experience, which is very slight, and definitely towards the latter side. I have never even sat in a real airliner cockpit, or a simulator for commercial pilots, but I have a little recreational flying experience and have tried two different recreational flight simulators. (Looking forward to reading Alexandreā€™s update in a couple of days time to hear about the commercial side).

I used to be a pretty keen glider pilot up until I ran out of the fairly significant time required to stay current about 6 years ago (stupidly, I had completed all of my bronze cert reqs except for a written test, which I then let lapse w/o doing the test). I canā€™t find my logbook, but I had something like 200+ hours of flying logged which involved hundreds of landings and takeoffs in varied airfields. So, beyond complete beginner level, but quite some way off expert. Experienced enough that a fair few normal flying tasks had become unconscious for me.

Despite this, when I tried out a home-built simulator (built by a member of my club) with controls, seat, cockpit etc from a real glider, I was completely unable to even get the virtual plane off the virtual ground with it - I crashed on takeoff repeatedly. The physical feedback on the controls was missing, the visual input was limited to a couple of screens directly ahead rather than wrap-around, and I just could not get past that. The cognitive dissonance was too strong so I was unable to control the aircraft sufficiently through one sense alone.

Flying a small glider, of course, is going to be pretty different from flying a huge airliner - in a glider, Iā€™m used to receiving a lot of physical input (when people talk about ā€œflying by the seat of your pantsā€, this is a real thing in gliding - you are looking for rising air, and you literally do feel a bump through your seat when you fly through a thermal, the sound you are hearing is a very important cue to your flying speed etc) plus the movements you make are way more extreme than in an airliner (much sharper angle of bank, for one, and I know glider pilots transitioning to power tend to overuse the rudder). But I would presume that movement and pressure from your controls is still important. Iā€™ve tried a ā€œcommercial-for-Joe-Publicā€ airliner simulator (the software they used was Microsoft Flight Simulator, but built into a mocked up cockpit - not full motion, but force feedback on the controls etc) and I was able to actually fly that, because I had enough feedback to trigger my reactions appropriately.

The BGA apparently has a gliding simulator that they loan out to clubs, but Iā€™ve never had an opportunity to try it. Itā€™d be interesting to contrast it with the other two. I would note that I am not a gamer - I wonder if I would have found the home-built sim easier if I was, which makes me think about selection of user groups and how you might not be testing for what you think you are testing for.

1 Like
6

This is really interesting!
I have worked with a developer who used to make software for the military especially for things such as helicopters and planes. They used simulators to test with but also (unless he was pulling my leg) he said they had test for how the actual simulator reacted to certain conditions and you could see those being run, like selenium tests apparently.

At home my other half has had a home built simulator for a cessna for a few years and just took up learning to fly in a real plane. He said similar to Anna above that the main thing that he noticed that was different is the lack of feedback from the controls.
Being a developer himself he is looking into hooking up a raspberri pi to make it more realistic.

I thing in this industry you need real domain knowledge to be really useful or pairing with pilots could be like user acceptance testing.

I know people who build the simulator software will use real pilots of certain aircrafts and collaborate to simulate the experience of that aircraft. It is all super interesting and an exciting industry to work in I think.

2 Likes
7

Hello @heather_reid
I will talk here about flight simulators that are used for training airline pilots https://youtu.be/Wqq4-XC5f8U
and to put you in context this is what CAE is building when we talk about simulation. As a new user Iā€™m limited to the number of links and pictures I can put in there. I will figure out something!
As a starter I would say that my education path made me very comfortable in the flight simulation test and certification. Iā€™ve learned how to draw mechanical parts, mill and assemble them. Learned how to design electronic boards, create some firmware or middle-ware and program and test them. And my initial carrer as software developer and integrator was in aerospace (radar, inertial navigation, cockpit instrumentsā€¦) and one day I decided to be an airline pilot, flying the big jets.
In CAE my job was to test flight those devices (Boeing 777) and bring them to certification. But Iā€™ve worked also with design, manufacturing and software engineering in order to find the most efficient ways of testing. Being included in design allowed us to be very innovative on how to test cockpit hardware and on software side we added automatic testing in order to be able to test most of the systems before going onto the final hardware platform. In general I will answer your question from my former job position (test pilot and certification).

Now your questions.

Where would I start?
The way I understood your question is ā€œWhere would I start if I want to test a flight simulatorā€.

  • know the test requirements (earlier post) per country. To make it easy just pick the FAA Part 60.
  • know the airplane, not only how to operate it but you will have to know the airplane systems (fuel, pneumatic, engines, APU, hydraulics, electric ā€¦) and architecture of the whole thing
  • know aerodynamics and physics (sometimes we are talking about tyre friction, torque, aquaplaning, macro abrasion, water temperature on runway, wind direction and ground frictionā€¦)
  • know the architecture of the flight simulator and various components.
  • know weather (relation between temperature, pressure, wind gradient, cloud, fog, ice formation and accretionā€¦)
    If you do no have the knowledge on those topics, then join a system team (fuel team for example) and start testing/integration in there. move to next team, take some flying lessons.
    There are some flight test pilot schools but they tend to focus on plane itself and not all the systems around simulation. On the other hand you have schools for being a flight simulator inspector that focus more on requirements and not so much on the plane itself.
    One of the good desktop simulation is the Xplane because it includes the ā€œIOSā€ (instructor operating station). That piece of software allow the instructor who is in the cockpit with us to control all the simulation (weather, weight, temperature, airport we are flying to and most important, failures we are going to encounter the most classical one would be engine failure during take off close to V1 on a slippery runway).
    You might offer to test some ā€œopen sourceā€ flight simulator (Digital combat simulator might be one) but do not expect to get paid for that.

What list of things would I test that a plane (via the simulator) should do?
You should be able to do more in a simulator than in the real plane. Reason being is that rules for operating an airplane are bound to limits (max 80 kts crosswind or -48c for engine start or 670000 lbs for max landing mass) but during training we might exceed those limits. Example: you have fire onboard with smoke in the cockpit but you are too heavy to land and the crosswind is too high (according to the limits). Would you wait to explode in the air or make an attempt to land on the rwy? Another thing is that you should be able to ā€œtransportā€ the simulated airplane within a couple of seconds in a totally different environment and none of the systems should complain (inertial system, EGPWS, cabin pressurization system). For example we start training on ground at night cold weather in Fairbanks. Then second part of training is in Dubai at FL310. All the systems should be in a state that represents ā€œnormal operationā€ (fuel temperature adjusted, brake temperature, ice quantity on airframeā€¦). After the Air France Rio Paris crash the aviation community decided to open up the flight enveloppe. Before it was "no, those data are not valid because your are not supposed to operate there (high altitude, high bank or pitch) and we do not have objective data. Aircraft manufacturers were now forced to provide data for those part of the flight enveloppe.
In general you should be able to perform everything that is present in the ā€œFCOMā€

What would I check that the simulator shouldnā€™t do?
In short the FAA part 60 is a law. Meaning you can be put in court if you messed up something. So the most simple answer is called ā€œnegative trainingā€. A simulator should never ever give false cues to trained pilots.

Would it actually be me testing it? Would it be a pilot? Would it be both of us?
Being a pilot helps. Being a pilot on complex airplane (variable pitch, multi engine, retractable gear) helps more and being an airline pilot helps even more (the knowledge gap is smaller). Knowledge in electronics, use of lab equipment an software engineering helped me a bunch.
I was ā€œboredā€ running manual testing in the cockpit months after months. In the end Iā€™ve implemented test written in natural language for most of the systems. It was my pilot knowledge transformed into code that Iā€™ve pushed back to engineering so their could run automatic tests into more realistic conditions. You cannot expect the brake team to know how to land the ship as real pilot do (speed, flap setting, weight, flare height and technique) when you want to test the autobrake system. With the test framework designed it was a one function call for them to have the ā€œsimulationā€ ready for landing starting 100 feet above the runway.

I had several failures during my airline pilot time and for sure, training in the simulator helped us to remain in control.

Hope that I have answered a couple of your questions, feel free to ask for more.
Good luck.

Be aware that the world might have just a couple of hundred simulator test pilots. This is a very unique job.

4 Likes
8

Good happy new year,

As a new user I was restricted about what I could post here. The link below will allow you to download my 11 pages response to the questions from @heather_reid.

https://drive.google.com/open?id=1xKZN5-jxIU-DtrLPGD90E5aZEnuhbB2M

Enjoy reading and drop me an email if you have other questions about flight simulators.

Alex.

image
image.jpg1222Ɨ851 231 KB

4 Likes
10

We do simulate landing gear failure if the landing is too hard as well as scraping wing on the ground. We also simulate ā€œlandingā€ at sea but this is very theoric because also every time the airplane collapse. The Hudson case was special because water was calm. How you test them? This is a very long story but you can search me on the web and you will find some articles and video about it.