Technological advances are increasing the complexity of, interactivity with, and cognitive demands on, human agents, particularly when AI is introduced. If we are to successfully reduce cognitive workload (CWL), we must adjust how we conceptualize cognition in manned-unmanned teaming environments. Towards this end, we draw from cognitive science theory, and the idea that the processes and products of cognition are extended across natural and artificial cognitive systems and embedded within a socio-technical environment. This necessitates a consideration of the system and environment as the same cognitive unit as the operator. We take a cognitive systems perspective and focus on the role of interaction and interdependence of human-machine teaming to provide insights for mitigating workload by extending cognition via scaffolding and offloading. Through this application of extended cognition theory, we offer a multi-disciplinarily informed approach, allowing us to provide a set of theoretically grounded research questions for understanding and improving workload.