Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data

Homomorphic encryption (HE) is one of the representative solutions to privacy-preserving machine learning (PPML) classification enabling the server to classify private data of clients while guaranteeing privacy. This work focuses on PPML using word-wise fully homomorphic encryption (FHE). In order to implement deep learning on word-wise HE, the ReLU and max-pooling functions should be approximated by polynomials for homomorphic operations. Most of the previous studies focus on HE-friendly networks, which approximate the ReLU and max-pooling functions using low-degree polynomials. However, this approximation cannot support deeper neural networks due to large approximation errors in general and can classify only relatively small datasets. Thus, we propose a precise polynomial approximation technique, a composition of minimax approximate polynomials of low degrees for the ReLU and max-pooling functions. If we replace the ReLU and max-pooling functions with the proposed approximate polynomials, standard deep learning models such as ResNet and VGGNet can still be used without further modification for PPML on FHE. Even pre-trained parameters can be used without retraining, which makes the proposed method more practical. We approximate the ReLU and max-pooling functions in the ResNet-152 using the composition of minimax approximate polynomials of degrees 15, 27, and 29. Then, we succeed in classifying the plaintext ImageNet dataset with 77.52% accuracy, which is very close to the original model accuracy of 78.31%. Also, we obtain an accuracy of 87.90% for classifying the encrypted CIFAR-10 dataset in the ResNet-20 without any additional training.