Empirical Bayesian Neural Network

Bayesian Neural Networks

For BNNs, they follow windowed input and a feed-forward two-layer neural network (FNNs) architecture. The difference is that all the weights in BNNs are represented by probability distributions over their possible values, while all the weights have fixed values to be learned in FNNs [1]. Define \(w_{ij}\) as a single weight in the model parameters \(\mathbf{W}, \mathbf{U}, \mathbf{b}_1, \mathbf{b}_2\), and \(\mu_{ij}, \sigma_{ij}\) as the mean and scale for \(w_{ij}\), then the window-based BNNs can be expressed as

\[ \begin{aligned} \mathbf{e}_t &= [\mathbf{x}_{t-w}\mathbf{E}, \dots, \mathbf{x}_{t}\mathbf{E}, \dots, \mathbf{x}_{t+w}\mathbf{E}]\\ \mathbf{h_t} &= \phi(\mathbf{e}_t\mathbf{W}+\mathbf{b}_2)\\ \hat{\mathbf{y}_t} &=softmax(\mathbf{h}_t\mathbf{U}+\mathbf{b}_2)\\ &\mbox{where } w_{ij} \sim N(\mu_{ij}, \sigma_{ij}) \quad (1)\\ \end{aligned} \] The weights in BNNs are assumed to be independent, meaning that each weight has its own mean and scale.

Variational Bayesian Neural Networks

Let \(\mathbf{D} = \left\{X, y\right\}, X\in \mathbb{R}^{N\times D}, y\in \mathbb{R}^{N\times 1}\) be the dataset containing windowed input matrix and the associate target vector in the named entity recognition, let denote the weights and biases in a BNN, it is assumed that \(p(Y|\mathbf{w}, X)\) is a categorical distribution conditioned on , the goal of model training is to find out \(p(y|X) = \int_\mathbf{w}p(y|\mathbf{w}, X)p(\mathbf{w|\mathbf{D}})d\mathbf{w}\) , which is the probability of given by integrating our all possible \(\mathbf{w}\).

To achieve this goal, one needs to find out the posterior distribution of \(\mathbf{w}\), i.e., \(p(\mathbf{w}| \mathbf{D})\). Since the likelihood distribution \(p(y|\mathbf{w}, X)\) is not Gaussian in classification problem, the analytic form of the posterior \(p(\mathbf{w}| \mathbf{D})\) is not tractable. To address this problem, a variational approach [8][9][10] is developed by introducing a variational distribution \(q(\mathbf{w}| \mathbf{\theta})\) within a known distribution family with parameters \(\mathbf{\theta} = (\mathbf{\mu}_q, \Sigma_{q})\) to approximate the true posterior \(p(\mathbf{w}| \mathbf{D})\). The approximation is learned by minimizing the Kullback-Leibler divergence (KL) between \(q(\mathbf{w}| \mathbf{\theta})\) and \(p(\mathbf{w}| \mathbf{D})\) with respect to . Therefore, in BNNs, the objective function to minimize is

\[ \begin{aligned} KL(q(\mathbf{w}| \mathbf{\theta}) || p(\mathbf{w}|\mathbf{D})) &= \int_{w} q(\mathbf{w}| \mathbf{\theta})log\frac{q(\mathbf{w}| \mathbf{\theta})}{p(\mathbf{w}|\mathbf{D}))}dw\\ & = \int_{w} q(\mathbf{w}| \mathbf{\theta})log\left(\frac{q(\mathbf{w}| \mathbf{\theta})}{p(y|\mathbf{w}, X))p(\mathbf{w})}p(y|X)\right)dw\\ & = KL[q(\mathbf{w}| \mathbf{\theta}) || p(\mathbf{w})]-\mathbb{E}_{q(\mathbf{w}|\theta)}[logp(y|\mathbf{w}, X)] \quad (2)\\ & \qquad\qquad\qquad\qquad\qquad\qquad+log[p(y|X)]\\ \end{aligned} \] Since \(log[p(y|X)]\) doesn’t depend on \(\mathbf{w}\), minimizing \(KL(q(\mathbf{w}|\mathbf{\theta}) || p(\mathbf{w}|\mathbf{D}))\) s equivalent to minimize the first two terms on the right hand side, which is called the variational free energy \(\mathcal{F}(\mathbf{D}, \mathbf{\theta})\),

\[ \begin{aligned} \mathcal{F}(\mathbf{D}, \mathbf{\theta}) = KL[q(\mathbf{w}| \mathbf{\theta}) || p(\mathbf{w})]-\mathbb{E}_{q(\mathbf{w}|\theta)}[logp(y|\mathbf{w}, X)] \qquad (3) \end{aligned} \]

People often maximize the negative of \(\mathcal{F}(\mathbf{D}, \mathbf{\theta})\), which is called the evidence lower bound \(\mathcal{L}(\mathbf{D}, \mathbf{\theta} )\),

\[ \begin{aligned} \mathcal{L}(\mathbf{D}, \mathbf{\theta})& = \mathbb{E}_{q(\mathbf{w}| \mathbf{\theta})}[logp(y|\mathbf{w}, X)]-KL[q(\mathbf{w}| \mathbf{\theta}) || p(\mathbf{w})]\\ &=\mathbb{E}_{q(\mathbf{w}| \mathbf{\theta})}[logp(y|\mathbf{w}, X)] - \mathbb{E}_{q(\mathbf{w}| \mathbf{\theta})}logq(\mathbf{w}|\theta ) + \mathbb{E}_{q(\mathbf{w}| \mathbf{\theta})}log p(\mathbf{w}) \quad (4) \end{aligned} \]

Since BNNs involves nonlinear transformation \(\phi\)n each layer, each \(\mathbb{E}\) term in Equation (4) is intractable, monte carlo (MC) can be used to estimate \(\mathbb{E}\) by sampling \(\mathbf{w}\) from \(q(\mathbf{w}|\theta)\) each epoch to maximize \(\mathcal{L}(\mathbf{D}, \mathbf{\theta} )\),

\[ \mathcal{L}(\mathbf{D}, \theta) \approx \frac{1}{S}\sum_{s=1}^{S}log p\left(y|\mathbf{w}^{(s)},\mathbf{x}\right) - \frac{1}{S}\sum_{s=1}^{S}log q(\mathbf{w}^{(s)}|\theta)+\frac{1}{S}\sum_{s=1}^{S} p(\mathbf{w}^{(s)}) \quad (5) \] Maximizing \(\mathcal{L}(\mathbf{D}, \mathbf{\theta} )\) encourages the model to find \(\theta\) such that \(q(\mathbf{w}|\mathbf{\theta})\) is closest to \(p(\mathbf{w}| \mathbf{D})\) within a distribution family, then it can be assumed that \(w_{ij} \sim q(\mathbf{w}|\mathbf{\theta})\) independently.

Empirical Variational Bayesian Neural Networks

      m = tf.cast(mu.shape[0], dtype='float32')
      S = tf.reduce_sum(sigma + mu * mu, axis=0)
      m_plus_2alpha_plus_2 = m + 2.0 * self.prior_alpha + 2.0
      S_plus_2beta = S + 2.0 * self.prior_beta
      s_hat = S_plus_2beta / m_plus_2alpha_plus_2  

   def output(self, sess, inputs_raw, inputs):

        preds = []
        std_array_all = None
        prog = Progbar(target=1 + int(len(inputs) /self.batch_size))
        def counts(x):
            return np.argmax(np.bincount(x))
            
        for i, batch in enumerate(minibatches(inputs, self.batch_size, shuffle=False)):
            # Ignore predict
            batch = batch[:1] + batch[2:]
            y_pred_list = []
            ps_list= []
            for j in range(50):
                preds_, ps_ = self.predict_on_batch(sess, *batch)
                y_pred_list.append(preds_)
                ps_list.append(ps_)
            
            stds = np.transpose(np.std(np.transpose(np.array(ps_list)), -1)) + np.mean(np.transpose(np.array(ps_list), (1,0,2))-np.transpose(np.array(ps_list), (1,0,2))**2, 1)
            #std[(5, bs, 50), -1]->t(5, bs)->(bs, 5) + mean(bs, 50,5)->(bs,5)
            if std_array_all is None:
                std_array_all = stds
            else:
                std_array_all= np.concatenate((std_array_all, stds), axis=0)
                
            preds_ = np.apply_along_axis(counts, 0, np.array(y_pred_list))
            preds += list(preds_)
            prog.update(i + 1, [])
            
        #np.save('./results/stds.npy', std_array_all)
        return self.consolidate_predictions(inputs_raw, inputs, preds)

Experimental Results

Conclusions

In this study, we apply EBNN to classification problem in NLP area, which is an under-explored area. Results show that, for the named entity recognition task, EBNN converges faster and is comparable with other four deep learning models, in addition, EBNN provides the measure for inference uncertainty, which is informative especially for the wrong predictions.

[1] Blundell, Charles, et al. “Weight uncertainty in neural networks.” arXiv preprint arXiv:1505.05424 (2015).

[2] Neal, Radford M. “Bayesian learning via stochastic dynamics.” Advances in neural information processing systems. 1993.

[3] Wu, Anqi, et al. “Deterministic variational inference for robust bayesian neural networks.” arXiv preprint arXiv:1810.03958 (2018).

[4] Nickisch, Hannes, and Carl Edward Rasmussen. “Approximations for binary Gaussian process classification.” Journal of Machine Learning Research 9.Oct (2008): 2035-2078.

[5] Rasmussen, Carl Edward. “Gaussian processes in machine learning.” Summer School on Machine Learning. Springer, Berlin, Heidelberg, 2003.

[6] Lee, Jaehoon, et al. “Deep neural networks as gaussian processes.” arXiv preprint arXiv:1711.00165 (2017).

[7] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).

[8] Jaakkola, Tommi S., and Michael I. Jordan. "Bayesian parameter estimation via variational methods.

[9] Graves, Alex. “Practical variational inference for neural networks.” Advances in neural information processing systems. 2011.

[10] Krishnan, Rahul G., Uri Shalit, and David Sontag. “Deep kalman filters.” arXiv preprint arXiv:1511.05121 (2015)." Statistics and Computing 10.1 (2000): 25-37.

[11] Hinton, Geoffrey E., and Drew Van Camp. “Keeping the neural networks simple by minimizing the description length of the weights.” Proceedings of the sixth annual conference on Computational learning theory. 1993.

[12] Kwon, Yongchan, et al. “Uncertainty quantification using bayesian neural networks in classification: Application to ischemic stroke lesion segmentation.” (2018).

[13] Kwon, Yongchan, et al. “Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation.” Computational Statistics & Data Analysis 142 (2020): 106816.

[14] Shridhar, Kumar, Felix Laumann, and Marcus Liwicki. “Uncertainty estimations by softplus normalization in bayesian convolutional neural networks with variational inference.” arXiv preprint arXiv:1806.05978 (2018). [15] Sang, Erik F., and Fien De Meulder. “Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.” arXiv preprint cs/0306050 (2003).

[16] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).

[17] Goller, Christoph, and Andreas Kuchler. “Learning task-dependent distributed representations by backpropagation through structure.” Proceedings of International Conference on Neural Networks (ICNN’96). Vol. 1. IEEE, 1996.