Deep neural networks (DNNs) have surpassed human-level accuracy in various fields, holding the promise of emerging technologies, such as self-driving cars and autonomous unmanned aircraft systems, smart cities infrastructure, personalized treatment in medicine, and cybersecurity. However, unlike Humans who have a natural cognitive intuition for probabilities, DNN systems – being inherently deterministic – are unable to evaluate their confidence in the decisions. To truly deserve its name, an artificial intelligence system must be aware of its limitations and have the capacity for insightful introspection.

This talk will advance Bayesian deep learning methods that are able to quantify their uncertainty in the decision and self-assess their performance, are robust to adversarial attacks, and can even expose an attack from ambient noise. This talk will establish the theoretical and algorithmic foundations of uncertainty or belief propagation through complex deep learning models by adopting powerful frameworks from optimal estimation problems in non-linear and non-Gaussian dynamical systems.

The challenge in DNNs is the multi-layer stages of non-linearities in deep learning models, which makes propagation of high-dimensional distributions mathematically intractable. Drawing upon powerful statistical frameworks for density propagation in non-linear and non-Gaussian dynamical systems, we introduce Tensor Normal distributions as priors over the network parameters and derive a first-order Taylor series mean-covariance propagation framework. We subsequently extend this first-order approximation to an unscented framework that propagates sigma points through the model layers. The unscented framework is shown to be accurate to at least the second-order approximation of the posterior distribution. We finally learn the entire predictive distribution using Particle Filtering, a powerful class of numerical methods for the solution of optimal estimation problems in non-linear, non-Gaussian systems.

The uncertainty in the output decision is given by the propagated covariance of the predictive distribution. We show that the proposed framework performs an automatic logit squeezing, which leads to significantly enhanced robustness against noise and adversarial attacks. Experimental results on benchmark datasets, including MNIST, CIFAR-10, real-world synthetic aperture radar (SAR), and Brain tumor segmentation (BraTS 2015), demonstrate: 1) superior robustness against random noise and adversarial attacks; 2) self-assessment through predictive confidence that monotonically decreases with increasing levels of ambient noise or attack; and 3) an ability to detect a targeted attack from ambient noise.