Conditional Granger causality, based on functional magnetic resonance imaging (fMRI) time series signals, is the quantification of how strongly brain activity in a certain source brain region contributes to brain activity in a target brain region, independent of the contributions of other source regions. Current methods to solve this problem are either unable to model nonlinear relationships between source and target signals, unable to efficiently quantify time lags in source-target relationships, or require ad hoc parameter settings and post hoc calculations to assess conditional Granger causality. This paper proposes the use of deep stacking networks, with dilated convolutional neural networks (CNNs) as component parts, to address these challenges. The dilated CNNs nonlinearly model the target signal as a function of source signals. Conditional Granger causality is assessed in terms of how much modeling fidelity increases when additional dilated CNNs are added to the model. Time lags between source and target signals are estimated by analyzing estimated dilated CNN parameters. Our technique successfully estimated conditional Granger causality, did not spuriously identify false causal relationships, and correctly estimated time lags when applied to synthetic datasets and data generated by the STANCE fMRI simulator. When applied to real-world task fMRI data from an epidemiological cohort, the method identified biologically plausible causal relationships among regions known to be task-engaged and provided new information about causal structure among sources and targets that traditional single-source causal modeling could not provide. The proposed method is promising for modeling complex Granger causal relationships within brain networks.