MALINA

Automated Performance Assessment for Training Satellite Planners Based on Learned Metrics

When planners at Satellite Operations Centers (SOCs) develop schedules for requested satellite contacts, their decision-making must account for a range of planning constraints, specialized for different satellites. In order to train SOC planners in simulation-based scenarios, performance assessment measures are needed to evaluate their planning decisions in context and provide feedback. These assessment measures can be complex and time-intensive to develop, especially when there are many different satellites and SOCs, with different sets of mission-related constraints.

Stottler Henke developed a set of authoring tools called Modular Annotated Learning for Instructional Authoring (MALINA) that provides a novel collaborative approach to constructing assessment knowledge, by combining machine learning with instructor-provided annotations. Input data include sample solutions collected from real-world operations, logged training events, or instructor-created examples. MALINA provides an annotation utility to supplement learned assessment measures with additional information from a human instructor or expert. This is particularly important for cases where there are soft rules that a human expert may follow, and that the system must learn. The system then induces performance assessment knowledge from the annotated sample data. MALINA is conceived as a domain-independent approach to building automated assessment for simulation-based training. However, the domain of SOC planning is the initial development application for the MALINA approach with an exemplar training prototype. The MALINA authoring methodology was applied in a training tool for a specific SOC, with planning and deconfliction tasks involving 3 satellites and 60 constraints. All constraints were derived directly from historical sample data, and then applied in training exercises both for direct performance assessment and comparison against generated sample solutions. A second training application with a different SOC will present an opportunity to explore generalizability to the planning decisions involved with different satellites and different constraints.

The MALINA learning mechanism also presents an additional potential benefit beyond the training context. By learning the constraints used in decision-making for a given domain such as SOC satellite planning, these constraints can be employed not only for automated performance assessment in a training setting, but also for suggesting possible solutions to challenging cases in an operational setting.