The Climb to make AR-Craft Academy

June 2018 – August 2018

Booz Allen Hamilton, Summer Games Internship

Teammates: James Baber, Jack Park, David Cross, Daniel Sanchez

Challenge Leaders: Thomas Canon, Tom Bigelow, and Eric Dziurzynski

Background and Summary: This summer at my internship for Booz Allen Hamilton, I worked on a team of five interns to create an Augmented Reality phone application to teach pilots how to fly a T-6 Texan II. I participated on the team as the designated User Experience Researcher, and I guided the team through a Rapid Prototyping process where everybody had input on the application design.

We were an intern team within the San Antonio location’s Interactive Media and Gaming Department (IMAG). Everybody else around us had projects that taught people Militarily related tasks through Virtual Reality. There was a UX person that I talked to a bunch who did UX Design for Call of Duty, and I had a manager who helped create Killing Floor back in the day. Every day was an adventure learning and talking to people, and I wouldn’t trade this experience for the world.

The User’s Story

The Problem Summary:

1. Dome Simulators and Planes are expensive to operate.
2. There is a shortage of pilots in the military.
3. Students drop out of the training programs.
4. Students must study a lot at night, mainly with an extensive manual, a poster board, and a plunger.

The Problem’s Story: Training people to fly a plane is expensive. The Air Force has been trying to become more efficient and effective at producing their pilots for several years. The first push of Virtual Reality Training came in the form of Dome Simulators. The ones that are just stationary cockpits, without hydraulics to move them, cost around $400,000 each, depending on the military contractor that puts them together. A Boeing 777 Dome Simulator to train commercial pilots can cost upwards of $4 Million, depending on how many bells and whistles you want in your simulator.

Another layer of complexity is that a limited number of students can learn on a Dome Simulator, and a limited number of instructors can teach. The Air Force’s original fix to this was Super Sorties, where there would be two students and an instructor. One student would fly the dome simulator plane, the instructor would stand off to the side, running the student through checklists and scenarios, and another student would be in the room waiting to trade off with the first student.

When students go home, they are expected to study for around three hours a night with an extensive 300-page manual for RPA Pilots and a 600-page manual for Pilots. Students run through checklists with a poster board of the cockpit and a plunger for the throttle. School can last from 5 AM to 7 PM, and students are expected to also care for themselves, such as eating, sleeping, and cleaning their living arrangements while not at school. Pilots have the extra stress of daily quizzes, checklists, and emergency procedures that must be recited perfectly, or they are grounded for the day, which prevents them from flying the physical plane.

Bonus Problems: Furthermore, there is a shortage of Air Force Pilots, so much so that commercial airlines had a hands-off policy until earlier this summer on actively recruiting those Pilots in the military. The commercial pilot positions usually pay a lot more with some Air Force flying experience, and the original policy was that until the commercial pilot industry was experiencing a shortage, they wouldn’t try to actively recruit pilots from the military. Besides that, a second career for military pilots is often transferring to the commercial industry.

At the time, Recruiters were also experiencing trouble recruiting kids out of high school who would qualify to become pilots. Recruiters often get kids in high school excited about their potential military career by helping them figure out what they want to do in the military. This also allows kids to determine if they wish to apply to college because there are several jobs in the military that require a college degree, such as being a pilot.

The Climb to the Final Product

Our team set up weekly sprints with User Testing and deliverable presentations on Friday. Every week, I would schedule users for testing, research whatever I could find about users, and then make a presentation summarizing everything everybody had done that week. You can, of course, read more about the road bumps in my other article on my portfolio about my trials and errors on this project. There are so many that I had to make another article about it because it didn’t make sense to make my entire portfolio piece. Otherwise, I have a short list of all the things I accomplished:

A Competitive Evaluation

• Researching what other Augmented Reality Instructional Apps exist on the market,
• The differences between Virtual Reality and Augmented Reality for processing power on the phone and also instructional differences with the effects on the user,
• What did other Air Force Bases do to train their pilots, and how did that differed from what we had in San Antonio
  

Foundational Research

This part of the research goes back to my Technical Communication background. I first took the time to understand who I was working with on my team and how we could best utilize their skills. For example, Jack automatically loved Project Management tasks, so we let him care for Jira. David had a little more experience with AI than Unity, so we let him take the front of designing an AI portion of the app while Jack and Daniel coded the app’s visual elements. James has a minor in Computer Science, so he got to add to the code when he had a really cool idea besides just doing all the 3D visualization for the project.

I naturally also wanted to conduct stakeholder and user interviews to understand the need we were filling with the potential product. Every interview involved the whole team and a list of questions to ensure we didn’t miss anything. With a lot of stakeholders, I made sure to schedule follow-up interviews to get their perspectives and vision for the project.

Discount Usability Testing with an Iterative Design Process

I convinced the team to give me a build of the app every week on Friday so I could dedicate the day to doing Discount Usability Testing with one-on-one user Interviews. I tested with around 45 users, and my results helped inform the teams of potential redesigns. This was an extensive learning process for me in communicating those results, sometimes beautifully and sometimes quickly.

Kleenex Usability Testing is a game design term that I learned about for inexperienced Usability testers like those on my team, and it was the perfect method to get the team involved. There was a period when I was having trouble getting the team to believe me when I said the users were running into bugs and unable to complete any of the tasks in the Usability Test. So I got them to commit to a few days of Usability Testing without intervening in the testing process and they got the message that there were bugs that needed to be fixed.

A Final Business Plan to sell this product to the Military

This part was complicated because I was trying to find keywords in an area I didn’t know much about. Thankfully, the Air Force has one of the most documented budgets for readily available online content. A lot of these numbers, in the end, came from research papers projecting future costs with algorithms, The Air Force Times, Time Magazine, and several Wikipedia pages.

My research colleagues taught me that no one is trying to mislead you. For example, I hesitated to use The Air Force Times because all their sources were based on interviews and word of mouth. Those who document the Wikipedia pages also dive deep into historical documents and the Congress-approved military budget. Ultimately, this all came down to a simple formula for how much we could save the military. The first part was to understand how much money the Air Force spent on the T-6 Texan II part of the training pipeline because that was the cockpit our app modeled after.

1. Flight Hours Required for RPA Pilots * Operating Cost per Hour of Running the Dome Simulator * How many RPA Pilots per year minimum go through the program per year = How Much Money the Military spends on Simulator Training for an entire year of RPA Pilot Students
2. Live Flight Hours Required for Pilot Students * Operating Cost per Hour of T-6 Texan II * How many Pilot Students per year minimum go through the program = How Much Money the Military spends on Live Flight Training for an entire year of Pilot Students
3. Dome Simulator Hours Required for Pilot Students * Operating Cost per Hour of Dome Simulator * How many Pilot Students per year minimum go through the program = How Much Money the Military spends on Dome Simulator Training per year for Pilot Students

Remember that the total cost might be plus or minus a few dollars due to retake flights when students just need more practice. The numbers I got here were based on hour requirements for graduation from the RPA Pilot Program and the Pilot Program. These are also hours just for the T-6 Texan II training portion and not the hours required of those students when they move onto the next phase, where they really fly a drone or the C-130 they got placed with.

The next step was to understand that there is still that Operating Cost per hour charge, so if our application was to save the military 1 hour of Dome Simulator time or Live Flight Time, then we needed to multiply that 1 hour saved by the number of the projected amount of students in the program.

Then, you could draw the picture a lot bigger depending on your audience. We decided to count every actively used military vehicle and treat them like a T-6 Texan II for a minimum operating cost because some of those Operating Costs were harder to find than expected. For example, I could easily see the overall government spending for a new air carrier for the Navy. Still, I couldn’t find the per-hour cost of operation for that same air carrier with roughly 300 people working on it. Operating Cost per hour is a much more popular quantification for airplanes in the Air Force than other military vehicles, so a little wiggle room on the numbers was to be expected.

Thus, with all that in mind, we came up with a conservative estimate of $2.4 Billion that Augmented Reality Training could save the military 1 hour across every training pipeline.

Our Final Prototype

Our solution to this multifaceted problem was to create an Augmented Reality training phone app to teach pilots how to fly the T-6 Texan II. The team designed the code to be quickly built upon for adding other planes and vehicles. That way, our department at Booz Allen could easily make other training apps. You can see the video that we presented at our big presentation in DC below:

This app was supposed to have the primary purpose of backing up the manual through Gamification. We had the user go through a simple memory game in our prototype application, and this app is far from being done. Our Interactive Media and Gaming Department at Booz Allen will probably add more to our work later, but this is what we could accomplish in ten weeks.

What I would like to Research for the future

There were several research questions that I was left with for the future of Booz Allen Hamilton’s Augmented Reality Contract with the military:

1. How long do students in the Air Force and other branches actually study? In my user interviews, I found that I was getting conflicting results when I asked students how long they studied each night. Some people who wanted to become fighter pilots said an hour to three hours, while some other students said that they don’t study at night. The military needs to look candidly at this with some outside researchers to get accurate data for future contracts and projects relating to training.
2. How does Gamification actually help people learn in the Military? I found a lot of research saying that Gamification and real-world virtual reality training gave students better reactions to real-world situations. There is a lot of peripheral research on how Gamification and learning affect children and even adults learning a different language, but not a ton of participants in the military.
3. Can I get my hands on those tests that tell students whether they will be Fighter Pilots or C-130 Pilots? This test and how well students think they will do might predict how much they study and how they study. Some tests tell High Schoolers what position they would be suited for in the military, and the same goes for College Students about to graduate from their ROTC program as a 2nd Lieutenant. This information all goes towards further defining the user story.
4. Wild Goal: Observe a day of pilot training for the Air Force and Army, basic training for a few other branches, and a day of students taking various tests using the current training pipeline. Observation Studies aren’t out of the normal for User Experience Researchers, but there is a ton of red tape surrounding observing anything to do with the military.

Conclusion

My summer was packed with a ton of learning. There is a ton of information I uncovered that is surrounded by a bit of red tape. Therefore, please contact me to see my deliverables, such as the application. If you are impressed by my resume, you will be blown away by my team’s final results. You should email me at elizabeth.newman.tate@gmail.com to talk to me more!