In December, Harvard University joined forces with several other universities to scrap ACT and SAT requirements — the standardized tests these institutions recognize won’t tell the full story about applicants. Is it time to track down the airline industry?
The tendency of colleges to abandon SAT admission protocols — while mostly for social reasons to encourage equality among applicants — shows that sometimes when we look at things with fresh eyes, we realize that there are multiple ways to solve the problem.
The FAA Knowledge Exam – generally a requirement to pass before applying to take the round of the check – is in my opinion a perennial and wrong system recognized by all stakeholders from pilots to examiners and even policy makers with repair needs.
Or, in my estimation, removal.
Tradition says it excludes some candidates, but I would argue that this system provides very little evidence of actual knowledge. Why do we cling to it?
Just like college test scores, proponents who advocate retaining knowledge exams might say that examiners need a scale to compare applicants to a certain standard. Examiners are tasked with questioning applicants about any weaknesses revealed through the knowledge test missed questions. With the recent move to pilot certification standards, this connection is made more clear by the mission codes alignment.
Getting rid of the tests could mean revising the entire flight screening mechanism — and rethinking an entire methodology. It would certainly be an expensive endeavor, but I refuse that keeping the tests negatively impacts the entire system.
knowledge testing industry
Consider what students now have to pass in order to take the knowledge test. In 2018, the FAA awarded a 10-year contract to PSI Services LLC (PSI), a provider of testing services across North America.
PSI (formerly LaserGrade) offers computer-based exam management to monitor professional and professional licensing and certification exams, and FAA certification using its secure network and software. PSI is one of only two computer-based test companies authorized by the federal government to administer all FAA pilot knowledge exams. PSI offers more than 60 in-demand tests for commercial and private pilots, as well as airframe and powertrain technicians.
On the surface, this move to standardize things seems to mean more efficiency. All you have to do is register on the FAA’s website for the specific knowledge test you wish to take, and visit your local PSI Test Center.
Except, pilots know that the entire process of taking the test—from prep to checking—isn’t simple.
Because knowledge tests play an important role as a precursor to most screening rounds – and then as a component of the screening journey itself – applicants know they have to make the most of it. Like the SAT and ACT, these knowledge tests have led to their own market, with an entire industry of test preparation companies now training students through the tests, with exorbitant fees. In fact, a March 2019 report by IBISWorld estimated the teaching and test preparation industry to be $1.1 billion, with test preparation services accounting for 25 percent of the industry.
To be clear, this pertains to the entire US test preparation market for all levels of education, including college admissions and other industries outside of aviation. The aviation industry has its own commercial players such as King School, ASA, Sporty’s Jeppesen, Boldmethod and many more.
While it is difficult to gauge the number of pilots using these commercial services to prepare, a 2020 webinar by the Federal Aviation Administration (FAA) gives an industry-wide look, showing that it runs more than 150,000 tests annually. In fact, the presentation describes it as one of the “world’s largest and most complex testing programs of any industry.”
Pilots may familiarize themselves with the process: It’s almost time to take the flight check, and you need to pass a knowledge test before applying to take the check round with at least 70 percent. Your well-meaning teacher may encourage you to pay a subscription fee to a reputable test preparation company that advertises high success rates and gives advice on how to study the question bank. Additionally, when you get to the actual check round and the examiner has to review your knowledge test to test for any deficiencies, the pilot is incentivized to score as high as possible on the knowledge test. This makes the whole process more rigorous for pilots, who spend a lot of time memorizing answers rather than fundamentally understanding questions problems – and paying more to re-test if they don’t pass the first time.
Furthermore, as an example to demonstrate that the process has become a proxy for the actual outcome – pilot knowledge – the FAA’s Pilot Knowledge Test Matrix, which the administration released in May 2021 in an effort to clarify parts of the process, adds to the complexity. In other words, in their quest to make the process as efficient as possible, the system ignores the fact that pilots still view the process as pass-through.
The cost of already cash-strapped pilots
What then? The costs of testing applicants are prohibitive. The fee for all FAA pilot knowledge tests is now $175. Given that pilots’ journey from private pilot to flight instructor involves taking at least six knowledge tests, in the end, they’ve spent a lot of capital.
It can be shown that this model is discouraging potential pilots. Sure, compared to the direct cost of flight training, it’s a lot less, but it’s a thing for a lot of people.
Do the numbers prove anything?
Is there any correlation between knowledge tests and initial screening pass rates? Do pilots who are good at knowledge tests pass the screening rounds on initial attempts?
Here is the 2020 data for initial ride pass/fail rates in 2020. Across the board—at the private, commercial, and even flight instructor level—the average initial ride pass rate is 77.5 percent. Can we say this is enough?
Meanwhile, here are the statistics for the pilot knowledge tests for 2020. The interesting thing to note is the higher pass rate and average score for the knowledge tests compared to the pass/fail statistics for the initial screening. Is it enough to continue to prove that knowledge test scores are a good measure of whether or not someone will do well?
Test questions lack the nuances of the Global Aviation Regiment
The Federal Aviation Administration (FAA) is seeking more comprehensive testing, with more emphasis on a pilot’s level of competency. What does the bank of limited test questions really prove that a candidate should not be able to find out about a check pilot?
Another thing to keep in mind is that more and more foreign students, with English as a secondary language, are taking part in training globally. While English is the industry standard language, this language’s strict multiple-choice structure provides no room for maneuver for students who might want to clarify something. Permanently, this enhances the test preparation cycle as more memorization occurs. Part of the system’s renewal is creating space for a broader base of multicultural contributors at the table to encourage better inclusion.
It’s time to rethink our approach to standardizing knowledge tests.
Is it time for the airline industry to take cues from what’s happening in the broader college testing industry? As more and more universities look for ways to consider applicants who do not have SAT and ACT scores, this proves that the options exist. Likewise, if our industry recognizes the limits of knowledge tests and the lack of perceived value, we may realize that options exist. While the FAA is making efforts – in its favor – more could be done, in my opinion.