Reproducibility in TMS–EEG studies: A call for data sharing, standard procedures and effective experimental control

A recent study by Conde, Tomasevic et al. (2019) [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] puts a spotlight on the subtleties of experimental design and analysis of studies involving TMS-evoked EEG potentials (TEPs), specifically focusing on the challenge of disentangling genuine cortical responses to TMS from those resulting from concomitant sensory activation. This is a relevant topic that the TMS–EEG community has previously identified [[2]Ilmoniemi R.J. Kičić D. Methodology for combined TMS and EEG.Brain Topogr. 2010; 22: 233-248https://doi.org/10.1007/s10548-009-0123-4Crossref PubMed Scopus (256) Google Scholar] and addressed with different strategies [3Gordon P.C. Desideri D. Belardinelli P. Zrenner C. Ziemann U. Comparison of cortical EEG responses to realistic sham versus real TMS of human motor cortex.Brain Stimul. 2018; 11: 1322-1330https://doi.org/10.1016/j.brs.2018.08.003Abstract Full Text Full Text PDF PubMed Scopus (52) Google Scholar, 4Gosseries O. Sarasso S. Casarotto S. Boly M. Schnakers C. Napolitani M. et al.On the cerebral origin of EEG responses to TMS: insights from severe cortical lesions.Brain Stimul. 2015; 8: 142-149https://doi.org/10.1016/j.brs.2014.10.008Abstract Full Text Full Text PDF PubMed Scopus (57) Google Scholar, 5Herring J.D. Thut G. Jensen O. Bergmann T.O. Attention modulates TMS-locked alpha oscillations in the visual cortex.J Neurosci. 2015; 35: 14435-14447https://doi.org/10.1523/JNEUROSCI.1833-15.2015Crossref PubMed Scopus (116) Google Scholar, 6Nikouline V. Ruohonen J. Ilmoniemi R.J. The role of the coil click in TMS assessed with simultaneous EEG.Clin Neurophysiol. 1999; 110: 1325-1328https://doi.org/10.1016/S1388-2457(99)00070-XCrossref PubMed Scopus (195) Google Scholar]. Based on the similarity of the evoked EEG responses they obtained in real TMS at different sites and in sham conditions (auditory and somatosensory scalp stimulation), the authors of [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] inferred that TEPs can be significantly contaminated by the effects of concurrent, non-transcranial stimulation. We acknowledge this is a valuable reminder to the TMS-EEG community; however, we contend that another fundamental implication of the work by Conde, Tomasevic and colleagues [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] - only incidentally mentioned at the end of their discussion – is that the evoked responses they obtain from both real TMS and sham conditions are substantially different from the TEPs reported in many of the previous studies (see, for example [7Bonato C. Miniussi C. Rossini P.M. Transcranial magnetic stimulation and cortical evoked potentials: a TMS/EEG co-registration study.Clin Neurophysiol. 2006; 117: 1699-1707https://doi.org/10.1016/j.clinph.2006.05.006Crossref PubMed Scopus (228) Google Scholar, 8Casarotto S. Romero Lauro L.J. Bellina V. Casali A.G. Rosanova M. Pigorini A. et al.EEG responses to TMS are sensitive to changes in the perturbation parameters and repeatable over time.PLoS One. 2010; 5e10281https://doi.org/10.1371/journal.pone.0010281Crossref PubMed Scopus (146) Google Scholar, 9Mäki H. Ilmoniemi R.J. The relationship between peripheral and early cortical activation induced by transcranial magnetic stimulation.Neurosci Lett. 2010; 478: 24-28https://doi.org/10.1016/j.neulet.2010.04.059Crossref PubMed Scopus (72) Google Scholar, 10Paus T. Sipila P.K. Strafella A.P. Synchronization of neuronal activity in the human primary motor cortex by transcranial magnetic stimulation: an EEG study.J Neurophysiol. 2001; 86: 1983-1990https://doi.org/10.1152/jn.2001.86.4.1983Crossref PubMed Scopus (289) Google Scholar, 11Rosanova M. Casali A. Bellina V. Resta F. Mariotti M. Massimini M. Natural frequencies of human corticothalamic circuits.J Neurosci. 2009; 29: 7679-7685https://doi.org/10.1523/JNEUROSCI.0445-09.2009Crossref PubMed Scopus (441) Google Scholar]). This discrepancy offers a timely opportunity to focus on the issue of the reproducibility of TEPs across laboratories and, most important, can encourage a constructive debate within the whole TMS–EEG community towards the optimization of shared procedures to obtain genuine responses to TMS. In this vein, Fig. 1 directly compares the TEPs reported in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] with others previously published in different studies taken as a reference by Conde, Tomasevic and colleagues [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar]. The inspection of Fig. 1 clearly shows that it is possible to effectively trigger high-amplitude, sharply rising early (<50 ms) components and overall TEP wave-shapes that are specific for the angle and site of stimulation and that are very different from those obtained in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar]. This simple comparison highlights a general problem of reproducibility and offers an excellent opportunity to discuss two critical steps in TMS–EEG data acquisition: (i) maximising the impact of TMS on the cortex, and (ii) minimizing EEG confounding factors due to sensory co-stimulation. Regarding the impact of TMS on the cortex, it is very likely that the authors of [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] were not as effective as other investigators for the following reasons. First, they applied TMS with a maximum electric field (E-field) intensity between 70 and 90 V/m according to their estimation, assuming a priori that this would have warranted effective cortical activation based on a previous work [[12]Casali A.G. Casarotto S. Rosanova M. Mariotti M. Massimini M. General indices to characterize the electrical response of the cerebral cortex to TMS.Neuroimage. 2010; 49: 1459-1468https://doi.org/10.1016/j.neuroimage.2009.09.026Crossref PubMed Scopus (106) Google Scholar]. However, in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] the authors adopted a small coil (outer winding diameter: 45 mm) which, compared to the larger ones (outer winding diameter: 70–90 mm) employed in previous works [[3]Gordon P.C. Desideri D. Belardinelli P. Zrenner C. Ziemann U. Comparison of cortical EEG responses to realistic sham versus real TMS of human motor cortex.Brain Stimul. 2018; 11: 1322-1330https://doi.org/10.1016/j.brs.2018.08.003Abstract Full Text Full Text PDF PubMed Scopus (52) Google Scholar,[7]Bonato C. Miniussi C. Rossini P.M. Transcranial magnetic stimulation and cortical evoked potentials: a TMS/EEG co-registration study.Clin Neurophysiol. 2006; 117: 1699-1707https://doi.org/10.1016/j.clinph.2006.05.006Crossref PubMed Scopus (228) Google Scholar,[9]Mäki H. Ilmoniemi R.J. The relationship between peripheral and early cortical activation induced by transcranial magnetic stimulation.Neurosci Lett. 2010; 478: 24-28https://doi.org/10.1016/j.neulet.2010.04.059Crossref PubMed Scopus (72) Google Scholar,[10]Paus T. Sipila P.K. Strafella A.P. Synchronization of neuronal activity in the human primary motor cortex by transcranial magnetic stimulation: an EEG study.J Neurophysiol. 2001; 86: 1983-1990https://doi.org/10.1152/jn.2001.86.4.1983Crossref PubMed Scopus (289) Google Scholar,[12]Casali A.G. Casarotto S. Rosanova M. Mariotti M. Massimini M. General indices to characterize the electrical response of the cerebral cortex to TMS.Neuroimage. 2010; 49: 1459-1468https://doi.org/10.1016/j.neuroimage.2009.09.026Crossref PubMed Scopus (106) Google Scholar,[13]Harquel S. Bacle T. Beynel L. Marendaz C. Chauvin A. David O. Mapping dynamical properties of cortical microcircuits using robotized TMS and EEG: towards functional cytoarchitectonics.Neuroimage. 2016; 135: 115-124https://doi.org/10.1016/j.neuroimage.2016.05.009Crossref PubMed Scopus (27) Google Scholar] activates a smaller cortical volume, maximum E-fields being comparable. Second, the authors of [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] reported having controlled online for the amplitude of the early TEP to obtain a minimum peak-to-peak amplitude of 6 μV. This real-time check is indeed a crucial step to assess the actual impact of TMS on the cortex, given the complexity of the interactions among coil dimensions, the different methods for E-field estimation (computed in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] at a gyrus of the cortex, thus possibly overestimated [[14]Thielscher A. Opitz A. Windhoff M. Impact of the gyral geometry on the electric field induced by transcranial magnetic stimulation.Neuroimage. 2011; 54: 234-243https://doi.org/10.1016/j.neuroimage.2010.07.061Crossref PubMed Scopus (264) Google Scholar]), and the inter-individual variability in cortical morphology and excitability. In this regard, it is key to note that the authors of [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] performed this control by inspecting on-line average TEPs using a cephalic common reference, an EEG montage where the observed voltage largely depends on the reference position with respect to the electrode of interest. Conversely, previous studies employed the average-referenced signal (see, for example [[15]Casarotto S. Comanducci A. Rosanova M. Sarasso S. Fecchio M. Napolitani M. et al.Stratification of unresponsive patients by an independently validated index of brain complexity.Ann Neurol. 2016; 80: 718-729https://doi.org/10.1002/ana.24779Crossref PubMed Scopus (185) Google Scholar]), which provides a reference-independent presentation of the potentials generated by cortical sources [[16]Dien J. Issues in the application of the average reference: review, critiques, and recommendations.Behav Res Methods Instrum Comput. 1998; 30: 34-43https://doi.org/10.3758/BF03209414Crossref Scopus (319) Google Scholar]. As a result, when the authors of [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] analysed their responses off-line using the average reference, they found much smaller early TEP components than what they had originally aimed at (as reflected in the grand averages reported in Fig. 1). All the above highlights the need for a careful selection of stimulation parameters and for an adequate real-time monitoring of average TEPs in order to warrant an effective cortical stimulation, a necessary requirement for TEPs reproducibility. On the other hand, regarding sensory co-stimulation, the fact that the responses observed in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] were stereotypical and did not vary depending on the stimulation site, may be explained by a deficient control over confounding factors. In fact, despite the use of noise masking, the participants in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] could still perceive the click associated with TMS as indicated by the average VAS rating ≥3 (individually rated up to 8 out of 10). This likely contributed to the large EEG components (50–200 ms) reported for both the real TMS and sham conditions and suggests a suboptimal masking of the TMS click (possibly related to the specific coil used by the authors). Although it is difficult to systematically rule out the contribution of sensory co-stimulation in each and every measurement, previous studies showed that ad-hoc auditory masking procedures associated to the use of other coil types may be effective in minimizing sensory event-related potentials (ERPs) [[4]Gosseries O. Sarasso S. Casarotto S. Boly M. Schnakers C. Napolitani M. et al.On the cerebral origin of EEG responses to TMS: insights from severe cortical lesions.Brain Stimul. 2015; 8: 142-149https://doi.org/10.1016/j.brs.2014.10.008Abstract Full Text Full Text PDF PubMed Scopus (57) Google Scholar,[17]Massimini M. Ferrarelli F. Huber R. Esser S.K. Singh H. Tononi G. Breakdown of cortical effective connectivity during sleep.Science. 2005; 309: 2228-2232https://doi.org/10.1126/science.1117256Crossref PubMed Scopus (1033) Google Scholar]. Thus, the results reported in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] suggest that the combination of insufficient cortical stimulation and insufficient masking of auditory inputs may lead to an EEG response dominated by sensory ERPs even in the real TMS conditions, similar to a sham condition. This is very important, because this unwanted outcome may have affected, to various degrees, a number of other TMS–EEG studies (including some by the authors of this letter), where small (or non-detectable) early components are followed by prominent N100 P200 sequences with a topography that is invariant depending on stimulation site/angle. However, it is equally important to recognize that this does not generalise to all TMS–EEG measurements. Indeed, previous works, employing different coils and stimulation parameters and applying real-time inspection of average EEG responses, have shown qualitatively different TEPs that retain spectral features and overall spatiotemporal dynamics that are specific for the site/angle of stimulation [[7]Bonato C. Miniussi C. Rossini P.M. Transcranial magnetic stimulation and cortical evoked potentials: a TMS/EEG co-registration study.Clin Neurophysiol. 2006; 117: 1699-1707https://doi.org/10.1016/j.clinph.2006.05.006Crossref PubMed Scopus (228) Google Scholar,[8]Casarotto S. Romero Lauro L.J. Bellina V. Casali A.G. Rosanova M. Pigorini A. et al.EEG responses to TMS are sensitive to changes in the perturbation parameters and repeatable over time.PLoS One. 2010; 5e10281https://doi.org/10.1371/journal.pone.0010281Crossref PubMed Scopus (146) Google Scholar,[11]Rosanova M. Casali A. Bellina V. Resta F. Mariotti M. Massimini M. Natural frequencies of human corticothalamic circuits.J Neurosci. 2009; 29: 7679-7685https://doi.org/10.1523/JNEUROSCI.0445-09.2009Crossref PubMed Scopus (441) Google Scholar,[13]Harquel S. Bacle T. Beynel L. Marendaz C. Chauvin A. David O. Mapping dynamical properties of cortical microcircuits using robotized TMS and EEG: towards functional cytoarchitectonics.Neuroimage. 2016; 135: 115-124https://doi.org/10.1016/j.neuroimage.2016.05.009Crossref PubMed Scopus (27) Google Scholar,[18]Ferrarelli F. Sarasso S. Guller Y. Riedner B.A. Peterson M.J. Bellesi M. et al.Reduced natural oscillatory frequency of frontal thalamocortical circuits in schizophrenia.Arch Gen Psychiatr. 2012; 69https://doi.org/10.1001/archgenpsychiatry.2012.147Crossref Scopus (93) Google Scholar] (see Supplementary Material: Supplementary Fig. 1 and animation file downloadable from https://data.mendeley.com/datasets/y747bdb42h/1). Crucially, these responses are also characterized by state-dependent changes in time–frequency features [[19]Rosanova M. Fecchio M. Casarotto S. Sarasso S. Casali A.G. Pigorini A. et al.Sleep-like cortical OFF-periods disrupt causality and complexity in the brain of unresponsive wakefulness syndrome patients.Nat Commun. 2018; 9: 4427https://doi.org/10.1038/s41467-018-06871-1Crossref PubMed Scopus (59) Google Scholar] and overall spatiotemporal complexity both at early and late latencies [[17]Massimini M. Ferrarelli F. Huber R. Esser S.K. Singh H. Tononi G. Breakdown of cortical effective connectivity during sleep.Science. 2005; 309: 2228-2232https://doi.org/10.1126/science.1117256Crossref PubMed Scopus (1033) Google Scholar] that can be fully replicated by recording the cortical responses to single-pulse intracranial electrical stimulation [[20]Comolatti R. Pigorini A. Casarotto S. Fecchio M. Faria G. Sarasso S. et al.A fast and general method to empirically estimate the complexity of brain responses to transcranial and intracranial stimulations.BioRxiv. 2019; : 445882https://doi.org/10.1101/445882Crossref Google Scholar,[21]Pigorini A. Sarasso S. Proserpio P. Szymanski C. Arnulfo G. Casarotto S. et al.Bistability breaks-off deterministic responses to intracortical stimulation during non-REM sleep.Neuroimage. 2015; 112: 105-113https://doi.org/10.1016/j.neuroimage.2015.02.056Crossref PubMed Scopus (104) Google Scholar], which is not associated with any auditory or somatosensory response. Having recognized the heterogeneity of TEPs' quality reported in the current literature, future methodological studies should consider open data sharing and direct comparisons across laboratories in order to appraise the current state of the field and propose constructive exchanges to jointly define standard procedures. In this vein, we share a dataset (see link in the supplementary material) from the original study reported in Fig. 1A [[11]Rosanova M. Casali A. Bellina V. Resta F. Mariotti M. Massimini M. Natural frequencies of human corticothalamic circuits.J Neurosci. 2009; 29: 7679-7685https://doi.org/10.1523/JNEUROSCI.0445-09.2009Crossref PubMed Scopus (441) Google Scholar], as a first opening toward the implementation of a TMS-EEG data repository. Future methodological work should certainly also include the use of realistic sham conditions (scalp and auditory stimulation) to reassess the contribution of sensory ERPs on TEPs absolute amplitude and topography in cases where the effectiveness of cortical stimulation has been first documented. Yet, it is important to note that designing realistic sham conditions may be rather challenging even in healthy controls, as demonstrated in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar], and utterly unfeasible in patients in whom a reliable psychophysics assessment cannot be performed. In conclusion, the example provided in Ref. [[1]Conde V. Tomasevic L. Akopian I. Stanek K. Saturnino G.B. Thielscher A. et al.The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.Neuroimage. 2019; 185: 300-312https://doi.org/10.1016/j.neuroimage.2018.10.052Crossref PubMed Scopus (139) Google Scholar] emphasizes an intrinsic challenge about the nature of TMS–EEG that we all in the TMS–EEG community need to reappraise. The genuine impact of TMS on the cortex depends on several factors that a priori are hard to control for, even when following the best guidelines and using state-of-the-art neuronavigation. These factors encompass coil design, E-field orientation with respect to axons, the 3D morphology and cytoarchitectonic of the underlying cortical tissue in individual brains. In this respect, TMS–EEG is not a magic wand but more akin to an ultrasound probe that requires skilful handling in order to recover a strong signal of interest amidst layers of noise. The first duty of an ultrasound operator is to aim and orient the probe so as to recover on her/his monitor a robust echo from the target structure; when basic signal-to-noise criteria are met, only then can the actual measurement start. We all trust ultrasound for key decisions in life-threatening cases, even though we know that the same probe would only recover noise if placed on our skin by a blinded operator. Just like the ultrasound community, the TMS–EEG community is now ready to recognize this fact, to share and compare datasets, to develop standard criteria, and to focus on the development of shared real-time TEPs visualization tools and artifact-rejection algorithms [e.g. Ref. [[22]Wu W. Keller C.J. Rogasch N.C. Longwell P. Shpigel E. Rolle C.E. Etkin A. ARTIST: a fully automated artifact rejection algorithm for single-pulse TMS-EEG data.Hum Brain Mapp. 2018; 39: 1607-1625https://doi.org/10.1002/hbm.23938Crossref PubMed Scopus (37) Google Scholar]] aimed at maximising the effects of direct cortical impact against noise, so that different laboratories and operators can reproducibly elicit genuine echoes from the brain. AE declares equity in Akili Interactive and Mindstrong Health. RI is founder, minority shareholder and advisor of Nexstim Plc and has acted in an advisory role for Brainsway Ltd. PJ declares consulting and shared patent pending with Nexstim Plc. All the other authors declare no competing financial interests. The following is the Supplementary data to this article: Download .docx (.48 MB) Help with docx files Multimedia component 1