RQ-21A Blackjack Small Tactical Unmanned Aircraft System (STUAS)

Transcription

1 Director, Operational Test and Evaluation RQ-21A Blackjack Small Tactical Unmanned Aircraft System (STUAS) Initial Operational Test and Evaluation Report June 2015 This report on the RQ-21A Blackjack Small Tactical Unmanned Aircraft System fulfills the provisions of Title 10, United States Code, Section It assesses the adequacy of testing and the operational effectiveness and operational suitability of the RQ-21A Small Tactical Unmanned Aircraft System. J. Michael Gilmore Director

2 Report Documentation Page Form Approved OMB No Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 29 JUN REPORT TYPE N/A 3. DATES COVERED 4. TITLE AND SUBTITLE RQ-21A Blackjack Small Tactical Unmanned Aircraft System (STUAS) Initial Operational Test and Evaluation (IOT&E) Report 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Director, Operational Test and Evaluation (DOT&E) 8. PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited. 13. SUPPLEMENTARY NOTES The original document contains color images. 14. ABSTRACT 15. SUBJECT TERMS 11. SPONSOR/MONITOR S REPORT NUMBER(S) 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT SAR a. REPORT unclassified b. ABSTRACT unclassified c. THIS PAGE unclassified 18. NUMBER OF PAGES 53 19a. NAME OF RESPONSIBLE PERSON Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

3 RQ-21A Blackjack Small Tactical Unmanned Aircraft System

4 Executive Summary This document reports on the evaluation of test adequacy, operational effectiveness, and operational suitability of the RQ-21A Blackjack Small Tactical Unmanned Aircraft System (STUAS) during Initial Operational Test and Evaluation (IOT&E). The Navy s Commander, Operational Test and Evaluation Force (COMOPTEVFOR), with assistance from the Marine Corps Operational Test and Evaluation Activity (MCOTEA), conducted the RQ-21A IOT&E between January and December The Navy and Marine Corps plan to use the results of IOT&E to inform the Services Full-Rate Production (FRP) decision, which was originally planned for October 2015, and is now scheduled to occur in September Additional testing is planned to occur before the September 2016 FRP. DOT&E intends to provide an updated operational assessment prior to the FRP to further support that decision. The RQ-21A is an Acquisition Category II program. The RQ-21A is not operationally effective. The detachment equipped with RQ-21A is not effective in supporting the ground commander s mission because of an inability to have an unmanned aircraft arrive on station at the designated time and remain on station for the duration of the tasked period. The electro-optical/infrared sensor provides accurate target locations. Operators using the sensors were not able to correctly classify 1-meter targets, which is necessary for consistent classification of hostile/non-hostile intent on the part of individuals by classifying such items as rifles, rocket-propelled grenade launchers, and shovels. The RQ-21A is not operationally suitable. The RQ-21A demonstrated a Mean Flight Hour between Abort for the System (MFHBA SYS ) of 15.2 hours versus the 50-hour requirement. Because of air vehicle reliability, overall system availability did not meet the 80 percent key performance parameter threshold (demonstrated value = 66.9 percent). The average service life of the propulsion modules was 48.9 hours, which does not meet the manufacturer s stated 100-hour capability. Production quality control issues contributed to the system s poor reliability and availability. Cybersecurity testing demonstrated that the system has exploitable vulnerabilities. The classified appendix contains additional detail regarding cybersecurity testing. The RQ-21A testing was adequate and executed in accordance with the DOT&Eapproved test plan. System Description and Mission The RQ-21A system includes five air vehicles (each air vehicle is equipped with an electro-optical/infrared payload and a communications relay payload), two ground control stations (each ground control station has two operator workstations), four ground data terminals, a launcher, a recovery system, and government-furnished vehicles and support systems. Marine Corps commanders intend to use the RQ-21A to provide units ashore with an organic battlefield intelligence, surveillance, and reconnaissance capability to reduce their dependence on higher headquarters for such assets. In addition to operating from land bases, i

5 detachments from Marine Corps unmanned aerial vehicle squadrons will embark personnel and equipment aboard L-class ships to support operations in the maritime domain. Test Adequacy Operational testing of the RQ-21A was adequate to support an evaluation of the operational effectiveness and operational suitability of the system. The Navy and Marine Corps conducted testing from January to December 2014 in three phases from three different locations. The test locations in sequence were Marine Corps Air-Ground Combat Center Twentynine Palms, California; Marine Corps Base Camp Lejeune, North Carolina; and aboard USS Anchorage (LPD 23). Total flight time accrued from all three test sites was hours during 38 flights. In May 2014, the Marine Corps deployed an Early Operational Capability RQ-21A system to support overseas combat operations. This deployed RQ-21A detachment consisted of up to eight air vehicles, two launchers, and two recovery systems. Between May and September, the system flew approximately 121 flights for 995 flight hours. Each day the detachment sent a daily summary containing the status of each major piece of equipment and a narrative summary of the day s flight activities. Data collected by the Early Operational Capability detachment while deployed overseas provided additional insights into system reliability. Previous Operational Testing Results Revisited in IOT&E In support of the RQ-21A Milestone C, the Navy conducted Operational Test B-2 (OT-B2) from September to November 2012, from which DOT&E provided an assessment of demonstrated performance. Subsequent to that test, the Navy addressed four of the seven recommendations contained in the DOT&E April 2013 memorandum (updated the reliability growth curve, verified fuselage structural integrity, identified and corrected the root causes for the excessive number of payload resets, quantified and compared the power requirements for the system s ground components to the generating capabilities of standard Marine Corps generators). The Navy did not fully address two recommendations from this memo, and both issues (strengthening quality control processes and communications relay payload frequency changes) adversely affected system performance during IOT&E. DOT&E s final recommendation to provide a reliability rationale in the Capability Production Document was not addressed and would have provided operational insight into the poor system reliability demonstrated during IOT&E. Based upon data collected during OT-B2 in 2012, DOT&E determined that the electrooptical/ infrared payload demonstrated the potential to provide tactical commanders with accurate target acquisition intelligence. In this report, DOT&E determined that, while the electro-optical/ infrared sensor provides accurate target locations, the low probability of correct classification for 1-meter targets, such as rifles, shovels, or rocket-propelled grenade launchers, does not allow for consistent determination of hostile/non-hostile intent on the part of individuals. The difference between the DOT&E OT-B2 and IOT&E assessment is attributable to changes between the Capability Development Document and the Capability Production ii

6 Document (CPD). The more stringent requirements in the CPD and the operating environment of IOT&E resulted in poor payload performance. DOT&E first identified the communications relay payload lack of in-flight frequency change capability as a potential deficiency during OT-B1 (January 2010) and highlighted this again in the DOT&E 2013 OT-B2 memorandum. In this report, DOT&E determines that the communications relay payload limits the commanders tactical flexibility. Just prior to OT-B2, developmental testing experienced a fuselage failure that resulted in a cracked airframe, possibly attributable to production quality control issues. DOT&E recommended that the Navy strengthen the quality control process in those areas where manufacturing shortcomings could result in air vehicle losses in order to minimize the potential for future mishaps. During IOT&E, production quality control issues contributed to the system s poor reliability and availability. During OT-B2, the system demonstrated an MFHBA SYS of 10.6 hours, below the threshold requirement of 50 flight hours. The demonstrated MFHBA SYS of 15.2 hours during IOT&E continues to be well below the threshold requirement. This contributed towards DOT&E finding the RQ-21A not suitable. Operational Effectiveness The RQ-21A is not operationally effective. Mission accomplishment for the detachment equipped with the RQ-21A begins with the unmanned air vehicle arriving at the desired location, at the prescribed time, and establishing contact with the supported unit. The unmanned aircraft should then be capable of performing assigned tasks for the duration of the required on-station time. Once on station, the air vehicles sensors should support the commander s intelligence requirements by providing accurate locations and reports of enemy actions. The unmanned aircraft should support additional missions for which it is equipped, such as communications relay, and the unmanned aircraft should have the capability of accepting dynamic retasking to support the commander s emergent needs. The mission concludes with the safe recovery of the air vehicle without encountering a significant amount of damage. The Mission Coverage metric measures the percentage of time an air vehicle is on station when tasked to do so. This metric provides an indication of the level of support a ground commander can expect from the RQ-21A. During the IOT&E, the RQ-21A-equipped unit provided coverage during 68 percent of the tasked on-station hours (83.8 of hours). In essence, RQ-21A was absent or not capable of providing requisite support one-third of the time the commander expected to have support. The RQ-21A demonstrated a low probability of successfully completing realistic missions during IOT&E. Nineteen of 45 attempted flights (42 percent) launched on time, supported assigned tasking, and remained on station for the duration of the assigned period. Of the 45 attempted launches, 35 (78 percent) provided some level of utility to ground commanders (were able to launch and provide some on-station time). iii

7 The RQ-21A sensor does not meet one of the two target classification key performance parameters established in the CPD. The electro-optical sensor is required to provide a 50 percent probability of correct classification for 1-meter linear targets (weapons or tools) while the infrared sensor is required to support 50 percent correct classification for 3-meter sized linear targets (vehicle chassis type). The infrared sensor demonstrated 100 percent correct classification of 3-meter objects, exceeding the key performance parameter threshold. For the electro-optical sensor, regression models based upon IOT&E data indicate a probability of correct classification of 41 percent in a desert environment and 43 percent in the maritime environment. The communications relay payload limits the commanders tactical flexibility and mission accomplishment. The communications relay payload is constrained to a single frequency in each of the two radios. Before launch, maintainers set the frequency in each radio. Once airborne, the only interface between the air vehicle and the communications relay payload is a power connector. Operators can power the communications relay payload on and off, but cannot change frequencies. This limits the communications relay payload to supporting preplanned relay missions on a single pair of pre-set frequencies. On four separate occasions during IOT&E, the RQ-21A-equipped unit was not able to provide support to ground units because the communications relay payload was set to frequencies different from those in use by the ground units. When able to establish compatible frequencies, the payload is capable of relaying clear and secure voice communications and secure data as required. The payload is capable of relaying voice communications between two ground stations located 50 nautical miles apart as required. The RQ-21A provides tactically useful target locations. While the CPD does not specify a threshold value for sensor point of interest accuracy, Marine Corps guidance indicates that 100-meter accuracy is sufficient to support tactical operations. RQ-21A provides a 90 percent circular error probable target location error of 43.8 meters. The RQ-21A target locations provide Category IV accuracy as defined in Joint Publication , Close Air Support (31 to 90 meters). Such accuracy is sufficient to support targeting in a conventional linear battlefield, where less target location accuracy is required for successful target engagement. This accuracy does not support targeting in a dense urban environment that requires more accurate target locations. Air vehicle endurance meets the 10-hour key performance parameter. Nine flights during IOT&E exceeded 10 hours (COMOPTEVFOR did not schedule every flight for 10-plus hours). The longest IOT&E flight lasted 12.3 hours. The longest flight during the Early Operational Capability deployment lasted 12.6 hours, with 48 of 121 flights lasting longer than 10 hours. The modular air vehicle design allows for payload changes well within the 60-minute Modular/Interchangeable Payload key performance parameter threshold. The air vehicle carries payloads in one of two modules: the nose module or the Center of Gravity (CG) Bay. The electro-optical/infrared sensor mounts to the nose module. The CG Bay carries the communications relay payload and the automatic identification system. Build-up of an entire air vehicle takes only 20 minutes. iv

8 The recessed, nose-mounted electro-optical/infrared payload requires circular orbits over top of the target to maintain continuous coverage and positive target identification. The use of offset orbits results in the fuselage blocking the payload field of view for significant periods of time. The actual duration of the payload obstruction ranged from 5 to 30 seconds, depending upon air vehicle altitude, bank angle, and distance from the target. During IOT&E, ground units directed RQ-21A operators to use offset orbits for 28 of 32 missions. These offset orbits resulted in auto-track break locks and loss of positive identification of high-value targets. There are orbit shapes that would allow RQ-21A operators to maintain continuous coverage of a target, but the current RQ-21A operating system limits operators to circular orbits. The medium-wave infrared sensor does not support search operations in a maritime environment. Looking at a large, relatively uniform temperature body such as the ocean hinders the sensor s auto-focus capability, resulting in image blurring and haloing. The RQ-21A program office addressed the cybersecurity vulnerabilities of the RQ-21A electronics architecture identified during the first cybersecurity adversarial assessment at Twentynine Palms before the second adversarial assessment on the USS Anchorage. During the second assessment on-board the USS Anchorage, new vulnerabilities were discovered. The classified appendix to this report contains findings related to cybersecurity testing and the accompanying recommendations. Operational Suitability The RQ-21A is not operationally suitable. The RQ-21A demonstrated an MFHBA SYS, of 15.2 hours versus the 50-hour requirement (80 percent confidence interval: ). Because of air vehicle reliability, overall system availability did not meet the 80 percent key performance parameter threshold (demonstrated value = 66.9 percent). The average service life of the propulsion modules was 48.9 hours, which does not meet the manufacturer s stated 100-hour capability. Given that the CPD does not provide a rationale for the 50-hour MFHBA SYS threshold requirement discussed above (which includes all aborts Pre-flight and In-flight aborts), DOT&E chose to assess reliability against the probability of completing a 12-hour continuous coverage mission without experiencing an in-flight aborting failure that causes the air vehicle to return to base prematurely. Once launched, the RQ-21A demonstrated a mean time between in-flight aborting failures resulting in a premature return to base of 46.7 hours. This translates to a 75 percent probability of completing a 12-hour on-station period without an in-flight aborting failure (80 percent confidence interval: 59 87). Although there are no Navy reliability requirements for failures rates, evaluation of the failures provides insight into the future maintenance and logistics burden to the Navy for the system. RQ-21A system demonstrated a Mean Flight Hours between Failure for the system (MFHBF System ) of 4.6 hours. The Mean Time between Failures for the surface components (MTBF Surface Components ), which includes the ground control station, launcher, and recovery system, is 31.3 hours in the land configuration and 14.6 hours in the sea configuration. The air vehicle demonstrated MFHBF is 5.9 hours. v

9 The propulsion module unit has a manufacturer-stated service life of 100 operating hours. Upon reaching 100 hours, maintainers replace the propulsion module unit and return the high-time unit to the manufacturer for refurbishment. Over the course of 995 flight hours, the Early Operational Capability detachment replaced 26 propulsion modules for a demonstrated average life span of 48.9 hours (median 42.2 hours). The RQ-21A demonstrated an availability rate of 66.9 percent (148.5 down hours during 449 test hours) during operations at Twentynine Palms. The demonstrated availability does not meet the 80 percent Operational Availability key performance parameter. Air vehicle reliability caused 148 of the total system down hours. During those 148 hours, the system did not have the minimum of two air vehicles in a mission-capable status. The STUAS Recovery System accounted for the remaining 0.5 hours of system down time. The IOT&E test unit required an influx of spare parts on February 6, 2014, to complete that phase of testing. Without the influx of spare parts, overall system availability would have been less than the demonstrated 66.9 percent. The RQ-21A Naval Air Training and Operations Standardization manual are missing important information regarding mission computer logic. This lack of information is especially critical during emergencies where operators are unaware of which conditions enable/disable various aspects of air vehicle functionality. This lack of system operations information contributed to the loss of Air Vehicle 1010 during the first IOT&E flight. To support shipboard operations, the Navy permanently installed some RQ-21A ground components (antennae interface modules, data link antennae) on selected ships. The ships personnel, and not the RQ-21A detachment, own and maintain these components. During IOT&E, none of the ships personnel received training on maintaining these installed components. The ship did not receive spare parts, maintenance manuals, or wiring diagrams with which to facilitate repairs. Field service representatives are required to conduct maintenance in accordance with the Naval Aviation Maintenance Program (Commander Naval Air Forces Instruction A). Instead of using the established Naval Aviation Logistics Command Management Information System to track maintenance and flight information, the field service representatives use a program called Sapphire. At the completion of testing, operational testers conducted a comparison of paper maintenance action forms completed by Marines to the Sapphire database. Of the 136 unscheduled maintenance actions completed over the course of IOT&E, Sapphire accurately archived 15. Because of these inaccuracies, the Marine Corps will not be able to determine the true maintenance burden associated with RQ-21A operations. Extended logistics delay times and production quality control issues contributed to the system s poor reliability and availability. In six instances, air vehicles spent time in a nonmission capable status while awaiting spare parts. Incorrectly assembled/configured components increase maintenance time to repair or replace components, resulting in reduced mission availability. vi

10 RQ-21A support equipment does not facilitate expeditionary operations. As fielded, each RQ-21A system includes two tents used as shelter for the ground control stations. Three module containers for each air vehicle are supposed to provide weather protection between flights. The system does not provide for inclement weather protection for support equipment and spare parts. During eleven days of operations at Camp Lejeune, maintainers discovered six instances where storage containers failed: On two occasions, foam cushion inserts separated from the wing container. This allowed two wings to come in contact and break winglet bolts. The design of the fuselage container allows stacking two high. On one occasion, stacking containers caused a 6-inch hole in the bottom container. After overnight rain, maintainers discovered approximately 16 ounces of water on the bottom of two wing containers and one fuselage container. All RQ-21A associated vehicles and trailers received certification for transport by CH-53E helicopters, as required by the air transportability key performance parameter. All trailers meet the weight and structural limits to allow for towing behind High Mobility Multipurpose Wheeled Vehicles (HMMWVs), as required by the ground transportability key performance parameter. The Marine Corps intends that the planned ground transport configuration (four HMMWVs, two cargo trailers, STUAS Recovery System trailer, and launcher trailer) would provide a fully functional RQ-21A detachment to support expeditionary operations. As currently configured, the expeditionary configuration relies upon significant help from supported units to conduct flight operations. The expeditionary detachment does not include a Marine communications specialist or dedicated generators to power the ground control stations. Recommendations The Navy and Marine Corps should consider the following recommendations in order to improve operational effectiveness and operational suitability before entering full-rate production and should verify the corrections to deficiencies during follow-on test and evaluation. The main body of this report contains additional recommendations not presented below. Operational Effectiveness Improve sensor resolution to increase probability of correct classification for 1-meter sized targets. Increase the number of programmed air vehicle loiter patterns to increase tactical flexibility and reduce fuselage obstruction of the payload. Fully integrate the communications relay payload into the air vehicle architecture to allow for in-flight frequency changes and altering of other radio settings to increase tactical utility. Implement the cybersecurity recommendations in the classified annex to improve system security. vii

11 Operational Suitability Expand the systems description and flight characteristics section of the RQ-21A Naval Air Training and Standardization Procedures manual to allow operators to safely react to system emergencies. Increase propulsion module service life and reliability to reduce maintainer workload, the number of spares required, and operating costs. Increase production quality control and implement thorough acceptance procedures for delivered systems and spare parts to reduce the number of faulty items received by Fleet operators. Fully train and provide ship personnel with technical manuals, wiring diagrams, and spare parts related to RQ-21A shipboard components to increase RQ-2A1 full mission capability. Require field service representatives to utilize the Naval Aviation Logistics Command/Management Information System to track system maintenance to provide the Marine Corps with better maintenance burden data. Increase expenditures on spare parts to reduce administrative logistics delay times. Increase the number of spare parts in the pack-up kit to increase system availability. J. Michael Gilmore Director viii

12 Contents System Overview... 1 Test Adequacy... 7 Operational Effectiveness Operational Suitability Recommendations ix

13 This page intentionally left blank. x

14 Section One System Overview This document reports on the evaluation of test adequacy, operational effectiveness, and operational suitability of the RQ-21A Small Tactical Unmanned Aircraft System (STUAS) during Initial Operational Test and Evaluation (IOT&E). The Navy s Commander, Operational Test and Evaluation Force (COMOPTEVFOR), with assistance from the Marine Corps Operational Test and Evaluation Activity (MCOTEA), conducted the RQ-21A IOT&E between January and December 2014 in three phases from three different locations. The test locations in sequence were Marine Corps Air-Ground Combat Center Twentynine Palms, California; Marine Corps Base Camp Lejeune, North Carolina; and aboard USS Anchorage. The Navy and Marine Corps plan to use the results of IOT&E to inform the Services Full-Rate Production decision, originally planned for October 2015, now scheduled to occur in September The RQ-21A is an Acquisition Category II program. Mission Description and Concept of Employment The Services intend the RQ-21A-equipped detachment to provide an expeditionary, organic Intelligence, Surveillance, and Reconnaissance (ISR) asset at the tactical level to support both maritime and shore-based operations within the respective tactical commander s area of responsibility. Mission support areas include: target development (to include locating and tracking), maritime and shore target identification, near-shore and beach landing site and helicopter landing zone reconnaissance/surveillance, area/point/route reconnaissance, vehicle convoy over-watch, riverine and small craft transit over-watch, maritime/land direct action over-watch. The Marines intend task-organized detachments from Marine Unmanned Aerial Vehicle Squadrons (VMU) to operate and maintain RQ-21A in support of any sized Marine Air-Ground Task Force. Each VMU will possess equipment and manpower sufficient to support nine RQ-21A detachments. Detachments deploy with Marine Expeditionary Units (MEUs) and regimental combat teams. To support an MEU, the VMU detachment will integrate with the MEU s aviation combat element during the pre-deployment training cycle, embark the requisite personnel and equipment aboard an L-class ship, and conduct flight operations in the maritime domain. If required to go from ship to shore, the VMU detachment will disembark the requisite personnel and equipment on CH-53E helicopters and surface landing craft, transition ashore, deploy to and establish a suitable expeditionary land-based site (usually co-located with the 1

15 supported unit headquarters), to commence flight operations in support of the ground commander s objectives. The VMU detachment may conduct hub-and-spoke operations during which a launch, recovery, and maintenance hub capability remains afloat while a mission control spoke capability goes ashore in order to extend the range at which the RQ-21A can support MEU objectives inland with a minimal footprint ashore. During land-based operations, the VMU detachment must be able to transport requisite personnel and equipment via High Mobility Multipurpose Wheeled Vehicle (HMMWV) to maneuver as required in support of ground units. System Description As depicted in Figure 1-1, the RQ-21A system includes five air vehicles; each air vehicle is equipped with an electro-optical/infrared payload and a communications relay payload, two ground control stations (each ground control station has two operator workstations), four ground data terminals, a launcher, a recovery system, and government-furnished vehicles and support systems. The Pack-Up Kit provides spare components and a tool kit. Figure 1-1. RQ-21A Small Tactical Unmanned Aircraft System The air vehicle is a modular system, composed of small easily assembled line replaceable units. It has a multi-function ball turret that houses mission sensors, and various connection points and bays for custom payloads (see Figure 1-2). The air vehicle is capable of taking off and landing without a runway via the launch and recovery systems. The air vehicle is powered by a two-cylinder, two-stroke, 100 cc electronically fuel-injected engine that operates on JP-5 (military aviation jet fuel) and equivalent fuel types (with the addition of a special synthetic lubricating oil additive). 2

16 Figure 1-2. RQ-21A Air Vehicle The ground control station (GCS) is used to plan missions, control and monitor the air vehicle, and manage data received from the air vehicle. Figure 1-3 shows the tabletop ground control station configuration used during operations at Marine Corps Air-Ground Combat Center Twentynine Palms. The GCS includes two operator workstations, two ground data terminals for air vehicle command and control and video downlink, and a Global Positioning System (GPS) Electronics Module. Air vehicle operators control the air vehicle and multi-function ball turret through the GCS. Operator Workstation #1 Operator Workstation #2 Figure 1-3. Tabletop Ground Control Station Configuration 3

17 Each operator workstation consists of four modules: the electronics power module, network computer module, user interface module, and the data storage module (see Figure 1-4). Figure 1-4. Operator Workstation The RQ-21A launcher is a pneumatically-controlled device that accelerates the air vehicle to flight speed at takeoff and enables expeditionary employment of the RQ-21A in austere conditions without runways (see Figure 1-5). The launcher system is self-powered and designed for transport by air (via CH-53 helicopter, C-130 aircraft, and C-17 aircraft) or by land (towed by HMMWV). Figure 1-5. The RQ-21A Launcher 4

18 The RQ-21A recovery system employs a telescoping hydraulic mast and boom system that suspends a rope to capture the air vehicle in mid-flight (see Figure 1-6). After the rope captures the air vehicle, the air vehicle spins about the rope down to the ground. The recovery system mounts on a modified tactical Military Standard (MIL-STD) trailer chassis with an engine-driven hydraulic power unit. External electrical power and fiber optic signal cables are required to support the GPS electronics module onboard the RQ-21A recovery system. Figure 1-6. RQ-21A Recovery System Each RQ-21A system has two Tactical ISR Processing, Exploitation, and Dissemination Systems (TIPS) that serve as the cross-domain solution from the unclassified GCS to any classified network. The TIPS allows intelligence analysts to monitor near real-time RQ-21A imagery and create intelligence products for use by supported units. The TIPS provides the conduit for streaming live RQ-21A imagery across classified networks for viewing by any user with access to that particular network. The Marine Corps expects each detachment to operate and maintain the RQ-21A for up to 12 hours a day with a surge rate of 24 hours a day for a 10-day period. Additional Marines may be required to sustain surge operations and normal operations for extended periods of time. The VMU commander will tailor actual detachment sizes based on the expected mission and 5

19 duration of operations. This arrangement is typical for all Marine Air-Ground Task Force operations. In accordance with Marine Corps planning, the Navy s program manager for Navy and Marine Corps Small Tactical Unmanned Air Systems (PMA-263) has funded for contractor maintenance support to the VMUs and all detachments until two years after the Marine Corps declares RQ-21A Initial Operating Capability, which is currently expected in mid FY16. Four field service representatives will deploy with each RQ-21A-equipped Marine Expeditionary Unit. Organizational level maintenance is limited to the removal and replacement of individual modules for the air vehicles and operator workstations. Units return removed modules to the manufacturer for repair. 6

20 Section Two Test Adequacy Operational testing of the RQ-21A was adequate to support an evaluation of operational effectiveness and operational suitability. The Navy s Commander, Operational Test and Evaluation Force (COMOPTEVFOR) conducted the testing in accordance with a DOT&Eapproved test plan. The Navy and Marine Corps will use the results of Initial Operational Test and Evaluation (IOT&E) to support a Full-Rate Production decision, originally planned for October 2015, now scheduled to occur in September Operational Testing The operational test unit consisted of Sailors and Marines from three units. Marine Unmanned Aerial Vehicle Squadron 2 (VMU-2) provided air vehicle operators and maintainers, Marine Operational Test and Evaluation Squadron 22 (VMX-22) provided air vehicle operators, and Air Test and Evaluation Squadron ONE (VX-1) provided a maintainer and an air vehicle operator. Four Insitu (the RQ-21A prime contractor) field service representatives provided technical and maintenance support in a manner consistent with the Navy fielding plan. VX-1 Operational Test Directors provided day-to-day control of test events. The Marine Corps Operational Test and Evaluation Activity assisted with IOT&E planning, execution, and reporting. COMOPTEVFOR conducted testing in three phases between January and December 2014, as shown in Table 2-1. Test Phase Table 2-1. Phases of Initial Operational Test and Evaluation Dates Location Flights Flight Hours (Flown/Planned) Phase 1 January 15 February 13 Marine Corps Air-Ground Combat Center /324 Phase 1a June 4-14 Phase 2 December 1-10 Marine Corps Base Camp Lejeune USS Anchorage (LPD 23) /NA /24 Total /348 During Phase 1, the RQ-21A-equipped unit supported a Marine Corps Integrated Training Exercise. Missions flown during Phase 1 included communications relay, convoy over-watch, tactical recovery of aircraft and personnel, and various reconnaissance and surveillance missions. An air vehicle mission computer failure during the first flight of Phase 1 resulted in the destruction of an air vehicle. Post-mishap investigations suspended operational test flights for 10 days. The first-flight mishap and reliability issues throughout Phase 1 limited the RQ-21A detachment to flying of the 324 planned flight hours. 7

21 Because of the reduced number of flight hours, overall poor system performance during Phase 1, and a change to air vehicle software, COMOPTEVFOR added Phase 1a, a second land-based phase of test not contained in the DOT&E-approved test plan. RQ-21A operations during this period supported a forward air controller training exercise and repeated collection of target location data points because of the software changes made to the system post-phase 1 testing. Operational testing concluded with the ship-based Phase 2. During this phase of test, the RQ-21A-equipped unit assessed shipboard compatibility for maintainability and flight operations. Missions included support for visit, board, search, and seizure operations, and provided full-motion video of surface contacts to the ship s Combat Information Center. Cybersecurity The two RQ-21A cybersecurity test events executed during IOT&E were adequate to assess cybersecurity vulnerabilities. The Marine Corps Information Assurance Red Team conducted testing on the land-based system during operations at Twentynine Palms. This period assessed operator workstations and the Tactical Intelligence, Surveillance, and Reconnaissance (ISR) Processing, Exploitation, and Dissemination System (TIPS) against the insider threat. COMOPTEVFOR, working with the Navy Information Operational Command-Norfolk, conducted cybersecurity testing of the sea-based system aboard USS Anchorage (LPD 23). This test period assessed the operator workstations and TIPS, in addition to the antenna interface module, the maintenance laptop, and the air vehicle. Testers attempted to penetrate the RQ-21A system externally through the Fleet Network Operations Center, Automated Digital Network System, and Shipboard Wide Area Network. The air vehicle remained grounded during cybersecurity testing for flight safety reasons. Figure 2-1 shows the RQ-21A cybersecurity architecture. Figure 2-1. RQ-21A Architecture 8

22 Additional Data In May 2014, the Marine Corps deployed an Early Operational Capability RQ-21A system to support overseas combat operations. This deployed RQ-21A detachment consisted of up to eight air vehicles, two launchers, and two recovery systems. Between May and September, the system flew approximately 121 flights for 995 flight hours. Each day the detachment sent a daily summary containing the status of each major piece of equipment and a narrative summary of the day s flight activities. Data collected by the Early Operational Capability detachment while deployed overseas provided additional insights into system reliability and availability. After experiencing communications relay payload difficulties during the Camp Lejeune phase of IOT&E, COMPTEVFOR and the RQ-21A program office sent personnel to observe two periods of VMU-2 flight operations in the vicinity of Marine Corps Air Station Cherry Point, North Carolina. Over two separate weeks, observers identified the cause of the communications relay problems and updated technical publications to ensure future repeatability in the payload programming and employment process. Test Limitations There were no test limitations affecting the scope of DOT&E s assessment. 9

23 This page intentionally left blank. 10

24 Section Three Operational Effectiveness The RQ-21A is not operationally effective. The detachment equipped with RQ-21A is not effective in supporting the ground commander s mission because of an inability to have an unmanned aircraft arrive on station at the designated time and remain on station for the duration of the tasked period. The electro-optical/infrared sensor provides accurate target locations. Operators using the sensors were not able to correctly classify 1-meter targets, which is necessary for consistent classification of hostile/non-hostile intent on the part of individuals by classifying such items as rifles, rocket-propelled grenade launchers, and shovels. Mission Accomplishment Mission accomplishment begins with the unmanned air vehicle arriving at the desired on-station location, at the prescribed time, and establishing contact with the supported unit. The unmanned aircraft should then be capable of performing assigned tasks for the duration of the required on-station time. Air vehicles that arrive late or depart early may imperil ground forces and adversely affect mission accomplishment. Once on station, the air vehicle s sensors should support the commander s intelligence requirements by providing accurate locations and reports of enemy actions. The unmanned aircraft should support additional missions for which it is equipped, such as communications relay, and the unmanned aircraft should have the capability of accepting dynamic retasking to support the commander s emergent needs. The mission concludes with the safe recovery of the air vehicle without encountering a significant amount of damage. Mission Coverage The RQ-21A does not provide commanders with a timely and dependable intelligence asset. This metric provides an indication of the level of support a ground commander can expect from the RQ-21A system. The Mission Coverage metric measures the percentage of time an air vehicle is on station when tasked to do so. During the Initial Operational Test and Evaluation (IOT&E), the RQ-21A equipped unit provided coverage during 68 percent of the tasked onstation hours (83.8 of hours, see left side of Figure 3-1). This means RQ-21A was absent or not capable of providing requisite support one-third of the time the commander expected to have support. Losses in on-station time resulted from the following: RQ-21A failed to support 20.1 hours (16.4 percent) of the on-station hours because of in-flight aborting failures that caused an early return to base. Pre-flight aborting failures that delayed or cancelled launches resulted in the loss of 10.7 hours (8.7 percent) of the on-station flight hours. In-flight troubleshooting of sensor anomalies caused the loss of the final 8.1 hours (6.6 percent) of on-station hours. RQ-21A flew 77 percent of scheduled flights hours (233.6 of 301.7, see right side of Figure 3-1). Unlike the Mission Coverage metric, this number considers all flight hours 11

25 scheduled across the entirety of the IOT&E. This includes the tasked on-station times, transit times to and from the on-station location, training flights, and flights to collect specific operational test data points. In this case, pre-flight aborts caused the greatest loss of 33.9 flight hours (11.2 percent), followed by in-flight aborting failures, which caused a loss of 23.4 hours (7.7 percent), and in-flight troubleshooting, which caused a loss 10.7 hours (3.6 percent). Mission Completion Rate Figure 3-1. Mission Coverage The RQ-21A demonstrated a low probability of successfully completing realistic missions during IOT&E. As shown in Figure 3-2, 19 of 45 attempted flights (42 percent) launched on time, supported assigned tasking, and remained on station for the duration of the assigned period. Of the 45 attempted launches, 35 (78 percent) provided some level of utility to ground commanders (launched at some point and provided some on-station time). 12

26 Target Classification Figure 3-2. Mission Completion Rate The RQ-21A sensor does not meet one of the two classification key performance parameters (KPPs). The Capability Production Document (CPD) establishes a classification KPP for the electro-optical sensor and one for the infrared sensor. Each classification KPP specifies a 50 percent probability of classifying an object at an air vehicle-to-target slant range of 4,242 feet (3,000 feet above ground level with a sensor depression angle of 45 degrees). The electro-optical sensor is required to support classification of 1-meter sized linear objects (weapons or tools) while the infrared sensor is required to support classification of 3-meter sized linear objects (vehicle chassis type). Using design of experiments, the Navy s Commander, Operational Test and Evaluation Force (COMOPTEVFOR) developed a data collection plan to characterize target classification capabilities for each sensor across a range of air vehicle-to-target distances and sensor depression angles. As shown in Table 3-1, test results indicate that the electro-optical sensor does not meet the 50 percent probability threshold for correct classification of a 1-meter object in desert (41 percent [80 percent confidence interval: 24-60]) or maritime (43 percent [80 percent confidence interval: 24-64]) environments at the specified slant range. The infrared sensor does meet the 50 percent threshold probability for correctly classifying 3-meter objects by demonstrating 100 percent correct classification. 13

27 Table 3-1. Percent Correct Classification at Slant Range = 4,242 feet (With 80 percent Confidence Intervals) Sensor Environment 1-meter Object 3-meter Object Electro-optical Infrared Desert Maritime Desert Maritime 41% (24, 60) 43% (24, 64) 48% (34, 63) 11% (4, 25) 3-meter objects had 100 percent classification. Infrared sensor performance in a maritime environment discussed in the Sensor Performance section below could explain the poor probability of correct classification results in Table 3-1. Communications Relay The communications relay payload limits the commander s tactical flexibility and mission accomplishment. The payload is capable of relaying clear and secure voice communications and secure data as required. The payload is capable of relaying voice communications between two ground stations located 50 nautical miles apart, as required. The communications relay payload supported plain voice relay operations at Twentynine Palms between two aircraft, and between VMU-1 Shadow operators and Joint Terminal Attack Controllers conducting close air support missions. Operational testers demonstrated secure voice communications relay during the two troubleshooting periods in North Carolina and demonstrated secure data relay aboard USS Anchorage. The communications relay payload is constrained to a single frequency in each of the two radios, which limits the commanders tactical flexibility. Before launch, maintainers pre-set the frequency in each radio. Once airborne, the only interface between the air vehicle and the communications relay payload is a power connector. Operators can power the communications relay payload on and off, but cannot change frequencies. This means that the communications relay payload supports pre-planned communications relay missions on a single pre-set frequency pair. In practice, when an air vehicle launches on a mission, it may or may not be tasked to support a pre-planned communications relay mission. Over the course of a 10-hour mission, the air vehicle may receive several dynamic retasking missions, one of which might be communications relay. If the units requesting communications relay support are not on the single pair of pre-programmed frequencies on the air vehicle, the supported units must change their communications plan to accommodate the air vehicle s frequencies. Due to the time and 14

28 coordination necessary for such an undertaking, this is not a practical solution for the supported ground unit. Four examples from IOT&E demonstrate the mission failures inherent to the single pair of frequencies: During a tactical recovery of aircraft and personnel mission, the communications relay payload was not set to the frequency used by the survivors on the ground. The RQ-21A operators were not able to monitor the survivors communications for information on their location. During one mission, RQ-21A operators received a tasking to support a ground unit in the search for missing personnel. The RQ-21A operators were not able to switch the communications relay payload to the search frequency, so they could not contribute to the search. During one pre-planned communications relay mission, maintainers loaded the wrong combination of frequencies into two separate air vehicles. One air vehicle was to support the pre-planned communications relay mission while the other was to support IOT&E data collection. Because operators could not change to the correct frequencies, both air vehicles were required to support the pre-planned communications relay mission. Communications went from Ground Unit A, to Air Vehicle A, to Air Vehicle B, to Ground Unit B and back again. During a forward air controller training exercise, controllers planned for the communications relay payload to switch frequencies to support different teams of forward controllers throughout each flight. In this case, the communications relay payload was inoperable. Had the communications relay payload functioned properly; all teams would have been required to operate on a single set of frequencies that would have delayed the exercise. The CPD does not call for the ability to change frequencies while airborne. COMOPTEVFOR determined such a capability to be a derived requirement and sought concurrence from the Marine Corps Capabilities Development Directorate. The Director of the Capabilities Development Directorate did not concur with COMOPTEVFOR s derived requirement because they believe the capability to change frequencies in-flight is not technologically mature and not a present day operational requirement. The Capabilities Development Directorate did not incorporate the ability to change frequencies during flight capability requirement into the CPD. The AN/PRC-152 radios used in the RQ-21A communications relay payload support frequency changes. Currently, connections between the communications relay payload and the RQ-21A air vehicle consist of a power connector and two antennae connections. In-flight frequency changes would require the addition of a data cable to the AN/PRC-152 radios and a user interface in the Operator Workstation software. Such technology is available and mature; the RQ-7 Shadow communications relay package, which the Marine Corps employs, has had an in-flight frequency changing capability for several years. 15

29 The communications relay payload does not support operations between ground units employing Network Identifiers (Net IDs) that provide frequency hopping to protect against communications jamming. The communications relay payload cannot load the required Transmission Security Key nor can it maintain the required GPS Time Synchronization. As a result, the communications relay payload cannot support convoy over-watch missions where the supported ground unit employs the frequency hopping capability in their normal combat operations. Programming communications relay payload radios is a burdensome process consisting of 72 steps (36 for each radio). This process proved cumbersome to maintainers and is exacerbated by the lack of formal training and poorly written manuals. Twenty-four of these steps contain multiple options. Five of the steps contain multiple options that must be set to one specific option. One step has an option that must be the same as the supported ground unit radios using the communications relay payload to communicate, requiring additional coordination with the supported unit well prior to launching the air vehicle. Choosing the wrong option during the programming process could degrade or prevent communications relay after the air vehicle launches. Target Attack The RQ-21A provides tactically useful target locations. While the CPD does not specify a threshold value for sensor point of interest accuracy, Marine Corps guidance indicates that 100-meter accuracy is sufficient to support tactical operations. RQ-21A provides a 90 percent circular error probable target location error of 43.8 meters at 3,000 feet above ground level with a 45-degree sensor depression angle. RQ-21A target locations provide Category IV accuracy as defined in Joint Publication , Close Air Support (31 90 meters). Such accuracy is sufficient to support targeting in a conventional, linear battlefield, where less target location accuracy is required for successful target engagement. This accuracy does not support targeting in a dense urban environment that requires more accurate target locations. Target locations reported by RQ-21A operators showed a systematic error, consistently reporting target locations as further east (on average, 19 meters) and further north (on average, 17 meters) than actual locations (see Figure 3-3). 16

30 Figure 3-3. Sensor Point of Accuracy Data The RQ-21A does provide a laser rangefinder to measure the air vehicle-to-target distance. Other unmanned aircraft systems use this information in their target location calculations to reduce location error. The RQ-21A does not use laser rangefinder information to assist in targeting. The infrared marker provides utility when used to identify specific targets for ground observers. Successful employment of the laser marker required RQ-21A operators to start by putting the marker within 100 meters of the ground observer s position. The RQ-21A operator would then walk the marker out to the target. Ground observers could follow the marker out to 500 meters, at which point they lost sight of the marker. Low infrared marker energy and micro-terrain (convex and concave terrain features) made it difficult to pick out the infrared marker spot from background clutter. System Performance Air Vehicle Endurance Air vehicle endurance meets the 10-hour KPP. An unfueled air vehicle (with payloads) weighs 110 pounds, with a maximum gross weight of 135 pounds. With 25 pounds of fuel and an average 2-pound per hour burn rate, the calculated endurance is approximately 12 hours. Actual endurance depends upon the amount of climbing and descending and high-speed flight required to support mission tasking. Frequent climbs and higher airspeeds increase the fuel burn rate, thus decreasing endurance. COMOPTEVFOR did not schedule every IOT&E flight to last ten or more hours. Mission tasking, training requirements, and data collection drove actual flight durations. Of the 38 IOT&E flights, nine exceeded 10 hours. The longest IOT&E flight lasted 12.3 hours. The 17

31 longest Early Operational Capability deployment flight lasted 12.6 hours, with 48 flights lasting longer than 10 hours. Payload Modularity The modular air vehicle design allows for payload changes within the 60-minute Modular/Interchangeable Payload KPP threshold. Build-up of an entire air vehicle takes 20 minutes. As shown in Figure 3-4, the air vehicle carries payloads in one of two modules: the nose module or the Center of Gravity (CG) Bay. Figure 3-4. Air Vehicle Payload Modules The electro-optical/infrared sensor mounts to the nose module. The CG Bay carries the communications relay payload and the automatic identification system. The CG Bay will carry additional payloads in the future provided they have the appropriate interface connections. Maintainers are able to remove and replace the nose module or the CG Bay well within the required 60-minute threshold. The nose module connects to the air vehicle fuselage with eight bolts and has a pneumatic tubing connection for the pitot systems and a serial port type connection to the avionics module. The CG bay connects to the air vehicle fuselage with eight bolts and a single power connection to the avionics module. Sensor Performance The recessed nose-mounted electro-optical/infrared payload requires circular orbits over top of the target to maintain continuous coverage and positive target identification. The use of offset orbits results in the fuselage blocking the payload field of view for significant periods of time (see Figure 3-5). The actual duration of the payload obstruction ranged from 5 to 30 seconds depending upon air vehicle altitude, bank angle, and distance from the target. During IOT&E, these periods of payload obstruction resulted in auto-track break locks and loss of positive identification of high-value targets. Battalion Air Officers and Joint Tactical Air Controllers directed RQ-21A operators to place the air vehicle in an offset loiter position in 28 of 32 missions in accordance with Joint Publication 3-52 Joint Airspace Control and airspace control plans used in overseas operations. Offset orbits are typically preferred to orbits over top of the target because they make the air vehicle less visually and aurally detectable to the target. In cases where the target may be fired 18

32 upon, an offset orbit may be necessary to ensure that the air vehicle will not obstruct the projectile s approach to the target. Figure 3-5. Sensor Payload Fuselage Obstruction As shown in Figure 3-6, there are orbit shapes that allow RQ-21A operators to maintain continuous coverage of a target from an offset orbit. The so-called bow-tie orbit solves the fuselage blocking the payload issue while maintaining continuous target coverage. As currently designed, the RQ-21A operating system limits operators to circular orbits. An experienced air vehicle operator was able to execute a bow-tie orbit by manually moving a circular orbit back and forth as the payload approached the fuselage blocking point. This manual bow-tie pattern requires an increased workload that prevents the operator from performing any other task during its conduct. Figure 3-6. Air Vehicle Orbits 19

33 The medium-wave infrared sensor does not support search operations in a maritime environment. Looking at a large, relatively uniform temperature body such as the ocean hinders the sensor s auto-focus capability, resulting in blurring and haloing of the image, as shown in Figure 3-7. Ocean Water With No Visible Contrast 20 Whitecaps Provide Some Contrast Figure 3-7. Infrared Sensor in Maritime Environment The left image in Figure 3-7 shows the ocean without contrast. Without contrast, the sensor s auto-focus feature is not able to provide a clear image. The right image in Figure 3-7 shows some contrast in the form of whitecaps that allows for auto-focus engagement. Both images show the haloing effect resulting from the infrared sensor looking at a body of uniform temperature. The combination of image haloing and lack of auto-focus precluded operators from locating surface vessels when cued by the ship s Combat Information Center. Once a target is located and auto-focus is able to engage, the sensor provides operators with a clear image. Operational testers encountered four other payload anomalies that hindered their ability to maintain continuous target coverage: When operating in Location Mode, the payload would wander or drift over time, losing sight of the target. While panning, the sensor hits a stop or limit obstructing the panning movement, causing operators to lose sight of the target. The sensor would randomly slew off target unexpectedly without operator input. Operators would encounter a split screen condition where the top of the sensor image appears at the bottom of the image on the operators display. Previous Operational Testing Results Revisited in IOT&E In support of the RQ-21A Milestone C, the Navy conducted Operational Test B-2 (OT-B2) from September to November Subsequent to that test, the Navy addressed four

34 of the seven recommendations contained in the April 2013 DOT&E Milestone C Operational Assessment (updated the reliability growth curve, verified fuselage structural integrity, identified and corrected the root causes for the excessive number of payload resets, quantified and compared the power requirements for the system s ground components to the generating capabilities of standard Marine Corps generators). The Navy did not fully address two recommendations (strengthening quality control processes and communications relay payload frequency changes) that adversely affected system performance during IOT&E. DOT&E s final recommendation to provide a reliability rationale in the CPD was not addressed and would have provided operational insight into the poor system reliability demonstrated during IOT&E. Based upon data collected during OT-B2 in 2012, DOT&E determined that the electrooptical/infrared payload demonstrated the potential to provide tactical commanders with accurate target acquisition intelligence. In this report, DOT&E determined that, while the electrooptical/infrared sensor provides accurate target locations, operators using the sensors were unable to correctly classify 1 meter targets as required by the CPD, which would allow for consistent determination of hostile/non-hostile intent on the part of individuals by identifying such items as rifles, rocket-propelled grenade launchers, and shovels. The difference between DOT&E s OT-B2 and IOT&E assessment is attributable to two differences between the tests: The Capability Development Document specified the requirement for air vehicle flight at an altitude of 3,000 feet above ground level, while the CPD specified flight at an altitude of 3,000 feet above ground level AND a sensor depression angle of 45 degrees, resulting in an assessment at a slant range of 4,242 feet. Test results indicate that probability of correct classification exceeded 60 percent at 3,000-foot slant ranges versus the 41 percent at 4,242 foot slant ranges. IOT&E included testing against maritime targets while OT-B2 did not include overwater activities with maritime targets. During IOT&E, RQ-21A demonstrated poor performance against maritime targets. While the payload demonstrated potential during OT-B2, the more stringent requirements and operating environment of IOT&E described above changed the nature of the probability of correct classification assessment between OT-B2 and IOT&E resulting in a less favorable assessment. DOT&E first identified the communications relay payload lack of in-flight frequency changes as a potential deficiency during OT-B1 (January 2010) and highlighted this again in the 2013 OT-B2 memorandum. In this report, DOT&E determines that the communications relay payload limits the commanders tactical flexibility. System Performance Cybersecurity The adversarial cybersecurity assessment during the second cybersecurity test event on USS Anchorage found that the Navy addressed vulnerabilities of the RQ-21A electronics architecture discovered during the first cybersecurity test event at Twentynine Palms. 21

35 Cybersecurity testing demonstrated that the system has exploitable vulnerabilities. The classified appendix contains a complete description of cybersecurity test events and findings. 22

36 Section Four Operational Suitability The RQ-21A is not operationally suitable. The RQ-21A demonstrated a Mean Flight Hour between Abort for the System (MFHBA SYS ) of 15.2 hours versus the 50-hour requirement. Because of air vehicle reliability, overall system availability did not meet the 80 percent key performance parameter (KPP) threshold (demonstrated value = 66.9 percent). The average service life of the propulsion modules was 48.9 hours, which does not meet the manufacturer s stated 100-hour capability. Table 4-1 depicts system suitability parameters, threshold requirements, and performance demonstrated during test. Table 4-1. Suitability Parameters Parameter 23 Threshold Performance [80% confidence interval] Mean Flight Hours between Abort - System (hours) [ ] Mean Time between Aborts Surface Components (hours) Land [ ] Mean Time between Aborts Surface Components (hours) Ship [ ] Mean Flight Hours between Abort Air Vehicle (hours) [ ] Mean Time to Repair (hours) N/A 1.66 [ ] Operational Availability 80% 66.9% [ ] Reliability Scoring The Navy defines Operational Mission Failures (OMFs) as a hardware failure, software fault, or government-furnished equipment integration failure that prevented the system from performing one or more mission-essential functions. Essential functions for the RQ-21A include the following: Prepare/configure system for use Collect, process, exploit, and disseminate information Perform post mission tasks Exchange information An abort is defined as a failure that precludes the system from commencing or completing a mission and puts the system in a not mission-capable status. The Navy s Commander, Operational Test and Evaluation Force (COMOPTEVFOR) collected failure data during each phase of the Initial Operational Test and Evaluation (IOT&E). At the conclusion of each event, scoring conferences determined the severity of each failure incident using the above definitions. There were scoring differences between the Navy and DOT&E. Air vehicle failures in which maintainers replaced one air vehicle with another before launch, thus delaying the start of a mission for periods of up to 100 minutes, were scored OMFs

37 by COMOPTEVFOR and not as an abort even though the system could not commence a mission as was prescribed in the definition of abort. DOT&E scored this as a system abort. In general, OMFs require situational scoring, in which scoring is based on the circumstances that exist at the time of the failure. DOT&E did not score incidents as OMFs. Scoring the incidents based solely on the loss of functionality and the duration of that loss provides a non-situational method of scoring. This method is independent of the circumstances that exist during the period functionality is lost. Scoring incidents using the non-situational method results in a more conservative (lower) reliability estimate and, in some respects, does penalize the RQ-21A system by ignoring redundancies inherent in the system. This report employs a non-situational scoring methodology. Incidents were scored as failures if they could not be remedied quickly, required parts, or could not be deferred. Some failures were further scored as aborts if they constituted a major subsystem reliability failure during preflight, caused a crash or return to base, caused a complete loss of video or communications with the supported ground unit, or if the air vehicle was not recovered in a mission-capable status because of a reliability failure during flight or recovery. For the purposes of calculating reliability metrics, time is determined in terms of flight hours and operating hours. For the system and air vehicle, reliability requirements specify the use of flight hours between aborts. For surface components, requirements specify the use of operating hours between aborts. For this case, calculations use ground control station operating hours. Mean Flight Hours between Aborts Table 4-2 presents reliability calculations for all phases of IOT&E along with the 80 percent confidence intervals. The gray cells in Table 4-2 indicate the combinations of data used to compute each metric. In some aspects, the Camp Lejeune results vary significantly from the Twentynine Palms and USS Anchorage results. DOT&E omitted Camp Lejeune data from some metrics. 1 Because of differences in the surface component configuration between land-based and sea-based systems, DOT&E chose to report Mean Time between Aborts for surface components (MTBA Surface Component (SC) ) for both configurations individually instead of combining that data. The RQ-21A demonstrated a Mean Flight Hours between Aborts (MFHBA System ) of 15.2 hours versus a requirement of 50 hours. For the land-based configuration, the demonstrated MTBA Surface Component is 58.8 hours versus a requirement of 240 hours. The sea-based configuration demonstrated an MTBA Surface Components of 36.5 hours versus the same 240-hour threshold. The RQ-21A demonstrated an MFHBA Air Vehicle of 35.5 hours versus the requirement of 50 hours. 1 Gray-Lewis test between Lejeune and Twentynine Palms data demonstrates a p-value of 0.02 for MFHBA System and for MFHBA Air Vehicle, indicating that failure rates differed significantly across the two periods, and that analysts should not combine the data. The location of the RQ-21A launch and recovery site at Camp Lejeune was more austere than the location at Twentynine Palms, which may have contributed to the substantially lower MFHBA Air Vehicle observed at Camp Lejeune. 24

38 Metric (Aborts) MFHBA SYS (hours) MTBA SC (hours) Land MTBA SC (hours) Ship MFHBA AV (hours) Table 4-2. Summary of Reliability Results for Aborts Test Event Hours Aborts Performance [80% confidence interval] Twentynine Palms [ ] Camp Lejeune [ ] USS Anchorage [ ] Twentynine Palms and USS Anchorage Combined [ ] Twentynine Palms [ ] Camp Lejeune [ ] Twentynine Palms and Lejeune Combined [ ] USS Anchorage [ ] Twentynine Palms [ ] Camp Lejeune [ ] USS Anchorage (80% lower confidence bound) Threshold Twentynine Palms and USS Anchorage Combined [ ] The Capabilities Production Document (CPD) does not provide a rationale for the 50-hour threshold requirement. DOT&E chose to assess reliability against the probability of completing a 12-hour continuous coverage mission without experiencing an in-flight aborting failure that causes the air vehicle to return to base prematurely. Once launched, the RQ-21A demonstrated a mean time between in-flight mission aborting failures of 46.7 hours (5 in-flight aborting failures over flight hours). This translates to a 75 percent probability of completing a 12-hour on-station period without an in-flight aborting failure (80 percent confidence interval: 59 87). 2 Two additional failures scored as aborts did not result in mission aborts. During three missions at Camp Lejeune, the communications relay package was not operational. The RQ-21A operators could not speak to the supported ground unit. Although the communications relay package failure occurred over multiple flights, DOT&E scored it as a single failure since the payload never functioned properly. The second failure occurred on the USS Anchorage. Midway through the last flight, the portside Antenna Interface Module ceased functioning, resulting in loss of video and air vehicle command and control on that side of the ship. 2 Assumes two air vehicle flights, each with six hours on-station time and 0.62 hours of transit time for a total of 6.62-hour flights for each air vehicle. 25

39 Figure 4-1 shows the breakdown of aborting failures. Eight of the 19 failures occurred with the air vehicle in flight, resulting in mission aborts as follows: Because of a mission computer reset during launch, one air vehicle was destroyed upon recovery. Four failures required the air vehicle to return to base and abort the mission. These failures included a tachometer generator failure alarm, propulsion module unit cylinder head temperature discrepancy, and two cases of lost link. An air vehicle modem and an operator workstation networking issue caused the two lost link events. While the air vehicle was attempting to land, the Differential Global Positioning System failed, delaying recovery by 97 minutes. Had this failure occurred during a longer flight, when the air vehicle was returning with less fuel, this could have led to a crash. Figure 4-1. Aborting Failure Breakdown Ten preflight failures resulted in aborts. Five of these failures caused the flight aborts. The remaining five were failures of a major subcomponent and caused the sortie to be delayed. The major failures for these events included: Air vehicle modem Propulsion module unit GPS Electronic Module software and hardware Air vehicle engine preheater cable 26

40 Antenna interface module Air vehicle avionics module. In addition, two quality control problems caused aborting failures: Ground control station video settings incorrectly configured at the factory A CAT-5 cable on the ground control station stack plugged into the wrong port at the factory. The final abort was due to the lack of Digital Terrain Elevation Data maps on the operator workstation to operate in a maritime environment. The manuals did not provide procedures to input elevation of sea level that is necessary for maritime operations. Mean Time between Failures Although there were no Navy reliability requirements for failures, evaluation of the failures provides insight into the future maintenance and logistics burden to the Navy for the system. Table 4-3 provides a summary of the failure metrics for each phase of IOT&E. The gray cells in Table 4-3 indicate the combinations of data used to compute each metric. The RQ-21A system demonstrated a Mean Flight Hours between Failure (MFHBF SYS ) of 4.6 hours. The Mean Time between Failures (MTBF) for the surface components in the land configuration is 31.3 hours and 14.6 hours in the ship configuration. The air vehicle demonstrated a MTBF of 5.9 hours. Table 4-3. Summary of Reliability Results for Failures Metric Test Event Hours Failures Performance (80% confidence interval) Twentynine Palms [ ] Lejeune [ ] MFHBF System USS Anchorage [ ] Twentynine Palms and USS Anchorage Combined [ ] Twentynine Palms [ ] MTBF Surface Components Land Lejeune [ ] Twentynine Palms and Lejeune Combined [ ] MTBF Surface Components Ship USS Anchorage [ ] Twentynine Palms [ ] Lejeune [ ] MFHBF Air Vehicle USS Anchorage [ ] Twentynine Palms and USS Anchorage [ ] 27

41 Figure 4-2 presents a breakdown of failure chargeability. There were 68 failures during the IOT&E (this includes Camp Lejeune failures). Air vehicle components accounted for 68 percent (46 of 68) of the failures. The surface components contributed 29 percent (20 of 68) of the failures. Inaccurate/incomplete technical publications caused the remaining 3 percent (2 of 68) of the failures. This figure shows the severity of the failures; in the figure key, the dark blue, green, and red colors represent the aborts and the light blue, green and red colors represent the failures. It breaks down the failure mode by cause: hardware, software, or other (technical publications or built-in test). The four most common failure modes accounted for 54 percent (37 of 68) of all failures. Structural damage to the air vehicle accounted for 23 percent (16 of 68). The forces experienced by the air vehicle during capture with the recovery system cause parts to break. This damage ranged from hinge clips to replacement of entire wing and empennage modules. Payload software accounted for 15 percent (10 of 68) of the failures. Operators remedied these failures, such as pixilated imagery, loss of payload control, and the payload slewing away automatically, by resetting payload software. Nine percent (6 of 68) of failures were operator workstation software. Propulsion module units contributed 7 percent (5 of 68) of all failures. Figure 4-2. Failure chargeability 28

42 Propulsion Module Units Propulsion module units have a manufacturer-stated service life of 100 operating hours. Upon reaching 100 hours, maintainers replace the propulsion module unit and return the high-time unit to the manufacturer for refurbishment. Over the course of 995 flight hours, the Early Operational Capability detachment replaced 26 propulsion modules. Figure 4-3 presents the reasons for propulsion module replacements and the distributions of module service hours at the time of replacement for each of the 26 modules. This resulted in a median life span for the propulsion modules of 42.2 hours (average life span 48.9 hours). Replacement Reason Count Fuel Leak 9 Low RPMs 6 Pulley Cracks 4 Reached End of Life 1 3 Cylinder Head Temp 3 Fuel Pump 1 Note: Two propulsion module units reached 100-hour life limit, one reached 80-hour adjusted life limit. Figure 4-3. Reasons for Propulsion Module Replacement and Distribution of Lifespans On Day 103 of the Early Operational Capability deployment, after two previous in-flight engine failures, and the discovery of cracks on the propulsion module pulley cranks on other air vehicles, the detachment lowered service life to 80 hours. On day 113 of the deployment, the detachment further reduced expected propulsion module service life to 50 hours to be more aligned with the demonstrated average life span of the propulsion modules. Availability The RQ-21A demonstrated an availability rate of 66.9 percent (148.5 down hours during 449 test hours) during operations at Twentynine Palms. The demonstrated availability does not meet the 80 percent Operational Availability KPP. The CPD describes an available system as follows: 2 of 5 air vehicles mission-capable 1 of 2 Ground Control stations mission-capable (each ground control station consists of two operator workstations) A mission-capable launcher A mission-capable Small Tactical Unmanned Aircraft System (STUAS) Recovery System. 29

43 Figure 4-4 presents the status of each RQ-21A component during operations at Twentynine Palms. The period between the January 15 air vehicle mishap until the morning of January 26, 2014 is not counted in the availability calculation. During this period, the Marine Corps suspended flight operations pending the outcome of the mishap investigation. Figure 4-4. RQ-21A System Availability at Twentynine Palms A lack of air vehicles caused 148 of the total system down hours. During those 148 hours, the system did not have the minimum two mission-capable air vehicles. The STUAS Recovery System accounted for the remaining 0.5 hours of system down time. Operator workstations and the launcher did not account for any system down time. The loss of Air Vehicle 1010 on the first IOT&E flight without replacement adversely affected system availability. Instead of five air vehicles being available to make the two mission-capable air vehicles system requirement, the mishap left the system with four air vehicles to make two mission-capable air vehicles system requirement, thereby reducing built-in system redundancy. To assess the impact of the loss of Air Vehicle 1010 without replacement, DOT&E notionally included a fifth aircraft as if it were available for operations and conducted a bootstrap analysis to compare system availability with four and five air vehicles to account for the loss of air vehicle 1010 for the duration of operations at Twentynine Palms. Bootstrapping is a resampling method that relies on random sampling with replacement. Bootstrapping allows assigning measures of accuracy (in this case confidence intervals) to sample estimates. 10,001 bootstrap data sets were generated by re-sampling the original data set of system up and down 30

44 times with replacement, generating a bootstrap distribution for System Availability. DOT&E derived confidence intervals based on the 10th and 90th percentiles of this availability bootstrap distribution. Figure 4-5 presents the results of this bootstrap analysis. There is more variability in the results with four air vehicles as shown by the dispersion of the distribution (length of vertical lines). Conversely, the bootstrapping results for five air vehicles demonstrates that there was a higher availability and lower variability (dispersion) as seen in the shorter vertical line length. The box plot for 5-Air Vehicles does include some outliers, symbolized as dots at the bottom of the figure, but the number is small and falls outside the line. 3 Figure 4-5. Distribution of Bootstrap Availability Results by Number of Air Vehicles Using the bootstrap methodology, system availability exceeded 80 percent for 39.6 percent of the four-air vehicle system iterations. For the five-air vehicle system, 61 percent of the bootstrap iterations exceeded the 80 percent availability threshold. While the presence of a fifth air vehicle might have allowed the system to demonstrate meeting the 80 percent availability threshold, it would not have done so with statistical confidence. Air Vehicle Availability As described above, air vehicles became the driving factor for system availability. Figure 4-6 presents the number of mission-capable air vehicles over the course of testing at Twentynine Palms. The red region in the left side of Figure 4-6 (depicting either one or no air vehicles) represents periods when the overall system is in a down status because of air vehicle availability. As shown in right side of Figure 4-6, individual air vehicle availability averaged 51.4 percent. The IOT&E test unit required an influx of spare parts on February 6, 2014 to complete the Twentynine Palms phase of testing. Without the influx of spare parts, overall system availability would have been significantly less than the demonstrated 66.9 percent. 3 The length of the line is equal to 1.5 times the Interquartile Range. 31

45 Air Vehicle Availability % % % % 1010 Mishap 1st Flight Overall 51.4% Figure 4-6. Number of Mission-Capable Air Vehicles over Time and Air Vehicle Availability Air vehicle availability was poor for the Early Operational Capability detachment. This detachment started operations with seven air vehicles, with an eighth air vehicle delivered on Day 57 of the deployment. As shown in Figure 4-7, after 119 days, the Early Operational Capability detachment had two mission-capable air vehicles (25 percent). Four of eight air vehicles were beyond repair (one cracked fuselage, two in-flight engine failures, and one damaged fuselage power distribution board) and two air vehicles required propulsion modules. Of the two operable air vehicles, one had 8 hours remaining on its propulsion module unit and would then require a replacement, which the detachment did not possess. Maintainability Figure 4-7. Early Operational Capability Air Vehicle Availability Operators, maintainers, and field service representatives performed three types of maintenance: pre- and post-flight, scheduled, and unscheduled maintenance. Pre- and post-flight maintenance includes aircraft preparation and recovery. Scheduled maintenance includes events delineated in maintenance manuals to be performed at a specific time, e.g., 25-hour engine inspection. Failures that were quickly remedied, did not require parts, or could be deferred are referred to as other maintenance actions. These are also included in the maintenance burden calculations. 32

46 During the testing at Twentynine Palms, maintainers expended 65.1 hours repairing 44 failures. 4, 5 The distribution of repair times is shown in Figure 4-8. Including the other maintenance actions, the total unscheduled maintenance time was 77.7 hours. Maintainers and field service representatives spent an additional hours conducting scheduled and pre- and post-flight maintenance. The total maintenance time expended was hours. The maintenance man hour per flight hour is 1.7 hours (324.6 hours/188.3 hours). The unscheduled maintenance man hour per flight hour is 0.41 (77.7/188.3). Assuming a lognormal distribution, the mean time to repair is 1.66 hours [ ]. The mean time to repair aborts is 3.92 hours [ ]. Logistics Supportability Technical Publications Figure 4-8. Distribution Twentynine Palms Repair Times Unlike manned aircraft, the RQ-21A does not have a pocket checklist for operator use during start-up and recovery, and when reacting to emergencies. Instead, operators must balance a 4-inch, 914-page binder on their laps while conducting normal and emergency procedures. This cumbersome and time-consuming process diverts operator attention from the problem at hand. Organizational level maintenance procedures stop at the module level. During shipboard operations, Air Vehicle 1002 reported a bad propulsion module generator during start-up. Maintainers installed a new propulsion module on Air Vehicle 1002, which indicated a bad generator on the new propulsion module. At that point, maintainers suspected that Air Vehicle 1002 had an electrical problem and that both propulsion module generators were in fact operating normally. Because maintenance manuals did not contain troubleshooting procedures for the propulsion modules, the option available to the Marines was to pack up both propulsion 4 5 Maintainability data from Twentynine Palms were the only data adequate for analysis. There are two fewer failures here than were accounted for in Table 4-3. These failures were not collected by COMOPTEVFOR, but were observed by DOT&E during the test. 33

47 modules and return them to the manufacturer. The incorporation of module troubleshooting procedures would reduce administrative logistics delay time and the number of spare parts in the pack-up kit. To support shipboard operations, the Navy permanently installed some RQ-21A ground components (antennae interface modules, data link antennae) on selected ships. The ships personnel, and not the RQ-21A detachment, own and maintain these components. None of the ships personnel received training on maintaining these installed components. The ship did not receive spare parts, maintenance manuals, or wiring diagrams with which to facilitate repairs. During the last IOT&E flight, the port-side antennae interface module failed. As a result, the operators could not fly the air vehicle on the port side of the ship. Without trained personnel and spare parts, this failure could have precluded further tactical flight operations. After the Marine Corps declares the fielding of an RQ-21A Initial Operational Capability, they plan to rely upon field service representatives to perform organizational level maintenance for two years. After the initial two years, the Marine Corps will decide whether to train Marine maintainers or continue to rely upon field service representatives. During this two-year period, field service representatives are required to conduct maintenance in accordance with the Naval Aviation Maintenance Program (Commander Naval Air Forces Instruction A). Instead of using the established Naval Aviation Logistics Command Management Information System to track maintenance and flight information, the field service representatives use an Insitu program called Sapphire. At the completion of testing, operational testers conducted a comparison of paper maintenance action forms completed by Marines to the Sapphire database. Of the 136 unscheduled maintenance actions completed over the course of IOT&E, Sapphire accurately archived 15. Because Sapphire was not available during IOT&E operations at Twentynine Palms and Camp Lejeune, field service representatives attempted to enter data after the completion of each test period. Instead of allowing maintainers to retroactively enter data using actual times and dates for each maintenance action, Sapphire automatically enters the current date and time as the failure time. Given the delays between the test periods and when data were entered into Sapphire, this created a 30-day lag in maintenance tracking data. Supply Support The lack of spare parts resulted in an administrative logistics delay time of 27 days. Administrative logistics delays fell into two categories: items not received at the start of the IOT&E and lack of air vehicle spare parts. Twelve items that provide system redundancy did not arrive with the system at the start of IOT&E, including a single refueler when a unit should have two, and a single pitot system test set when the unit should have two. The test unit did not receive these 12 items for 34 days. The lack of the second refueler caused mission aborts and required the test unit to borrow equipment from the Early Operational Capability detachment (which had yet to deploy) to commence flight operations. 34

48 Lack of air vehicle spare parts caused administrative logistic delays. In six instances, air vehicles spent a significant amount of time in a non-mission capable status while awaiting spare parts (7, 9, 13, 15, 15, and 19 days). Propulsion modules caused four of the extended delays, an avionics module caused one, while an empennage module caused the last. Extended logistics delays increased system down time, and without the use of unacceptable workarounds, would have halted flights during operational testing. Production quality control issues contributed to the system s poor reliability and availability. In seven instances, operators and maintainers found incorrectly assembled/configured components during initial system set-up, maintenance inspections, and pre-flight (see Table 4-4). Incorrectly assembled/configured components caused increased maintenance time to replace components, resulting in reduced availability. Table 4-4. Production Quality Issues Component Discrepancy Air Vehicle Fueler Air vehicle fueler would not prime Air Vehicle Support Equipment Operator Workstation New Modems were not properly configured Broken servo connector and retention clip Avionics module software not loaded Lost link Network communications issue Operator Workstation not displaying video RQ-21A support equipment does not facilitate expeditionary operations. As fielded, each RQ-21A system includes two tents used as shelter for the ground control stations. The system does not provide for inclement weather protection for air vehicles, support equipment, and spare parts. As shown in Figure 4-9, air vehicle module packing cases and spare parts would be stored under a tarpaulin strung between two tents or in the back of High Mobility Multipurpose Wheeled Vehicles (HMMWVs). Figure 4-9. RQ-21A Expeditionary Configuration 35