Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images

Fang Nan; Wei Jing; Feng Tian; Jizhong Zhang; Zhenxin Hong; Qinghua Zheng

doi:10.1016/j.knosys.2021.107678

Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images

Fang Nan, Wei Jing, Feng Tian, Jizhong Zhang, Zhenxin Hong, Qinghua Zheng

Centre for Research in Arts and Creative Exchange

Research output: Contribution to journal › Article › peer-review

Abstract

Facial Expression Recognition (FER) for various low-resolution images is an important task and need in applications of analyzing crowd scenes (station, classroom, etc.). Due to the discriminative feature loss caused by reduced resolution, classifying various low-resolution facial images into the right category is still a challenging task. In this work, we proposed a novel generative adversarial network-based feature level super-resolution method for robust facial expression recognition (FSR-FER), which can reduce the chance of privacy leaking without restoring high-resolution facial images. In particular, a pre-trained FER model was employed as a feature extractor, and a generator network G and a discriminator network D are trained with features extracted from low-resolution and corresponding high-resolution images. Generator network G tries to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images. For better classification performance, we also proposed an effective classification-aware loss reweighting strategy based on the classification probability calculated by a fixed FER model to make our model focus more on samples that are prone to misclassification. Experimental results on the Real-World Affective Faces (RAF) Database and Static Facial Expressions in the Wild (SFEW) 2.0 dataset demonstrate that our method achieves satisfying results on various down-sample factors with a single model and has better performance on low-resolution images compared with methods using image super-resolution and expression recognition separately.

Original language	English
Article number	236
Journal	Knowledge-Based Systems
Volume	236
Issue number	107678
DOIs	https://doi.org/10.1016/j.knosys.2021.107678
Publication status	Published - 25 Jan 2022

Access to Document

10.1016/j.knosys.2021.107678

https://www.sciencedirect.com/science/article/pii/S0950705121009400?via%3Dihub

Cite this

@article{536fe726bae04d50a2ea840de913da1d,

title = "Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images",

abstract = "Facial Expression Recognition (FER) for various low-resolution images is an important task and need in applications of analyzing crowd scenes (station, classroom, etc.). Due to the discriminative feature loss caused by reduced resolution, classifying various low-resolution facial images into the right category is still a challenging task. In this work, we proposed a novel generative adversarial network-based feature level super-resolution method for robust facial expression recognition (FSR-FER), which can reduce the chance of privacy leaking without restoring high-resolution facial images. In particular, a pre-trained FER model was employed as a feature extractor, and a generator network G and a discriminator network D are trained with features extracted from low-resolution and corresponding high-resolution images. Generator network G tries to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images. For better classification performance, we also proposed an effective classification-aware loss reweighting strategy based on the classification probability calculated by a fixed FER model to make our model focus more on samples that are prone to misclassification. Experimental results on the Real-World Affective Faces (RAF) Database and Static Facial Expressions in the Wild (SFEW) 2.0 dataset demonstrate that our method achieves satisfying results on various down-sample factors with a single model and has better performance on low-resolution images compared with methods using image super-resolution and expression recognition separately.",

author = "{Fang Nan} and {Wei Jing} and {Feng Tian} and {Jizhong Zhang} and {Zhenxin Hong} and {Qinghua Zheng} and Kuo-Ming Chao",

year = "2022",

month = jan,

day = "25",

doi = "10.1016/j.knosys.2021.107678",

language = "English",

volume = "236",

journal = "Knowledge-Based Systems",

issn = "0950-7051",

publisher = "Butterworth-Heinemann",

number = "107678",

}

TY - JOUR

T1 - Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images

AU - Fang Nan

AU - Wei Jing

AU - Feng Tian

AU - Jizhong Zhang

AU - Zhenxin Hong

AU - Qinghua Zheng

AU - Chao, Kuo-Ming

PY - 2022/1/25

Y1 - 2022/1/25

N2 - Facial Expression Recognition (FER) for various low-resolution images is an important task and need in applications of analyzing crowd scenes (station, classroom, etc.). Due to the discriminative feature loss caused by reduced resolution, classifying various low-resolution facial images into the right category is still a challenging task. In this work, we proposed a novel generative adversarial network-based feature level super-resolution method for robust facial expression recognition (FSR-FER), which can reduce the chance of privacy leaking without restoring high-resolution facial images. In particular, a pre-trained FER model was employed as a feature extractor, and a generator network G and a discriminator network D are trained with features extracted from low-resolution and corresponding high-resolution images. Generator network G tries to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images. For better classification performance, we also proposed an effective classification-aware loss reweighting strategy based on the classification probability calculated by a fixed FER model to make our model focus more on samples that are prone to misclassification. Experimental results on the Real-World Affective Faces (RAF) Database and Static Facial Expressions in the Wild (SFEW) 2.0 dataset demonstrate that our method achieves satisfying results on various down-sample factors with a single model and has better performance on low-resolution images compared with methods using image super-resolution and expression recognition separately.

AB - Facial Expression Recognition (FER) for various low-resolution images is an important task and need in applications of analyzing crowd scenes (station, classroom, etc.). Due to the discriminative feature loss caused by reduced resolution, classifying various low-resolution facial images into the right category is still a challenging task. In this work, we proposed a novel generative adversarial network-based feature level super-resolution method for robust facial expression recognition (FSR-FER), which can reduce the chance of privacy leaking without restoring high-resolution facial images. In particular, a pre-trained FER model was employed as a feature extractor, and a generator network G and a discriminator network D are trained with features extracted from low-resolution and corresponding high-resolution images. Generator network G tries to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images. For better classification performance, we also proposed an effective classification-aware loss reweighting strategy based on the classification probability calculated by a fixed FER model to make our model focus more on samples that are prone to misclassification. Experimental results on the Real-World Affective Faces (RAF) Database and Static Facial Expressions in the Wild (SFEW) 2.0 dataset demonstrate that our method achieves satisfying results on various down-sample factors with a single model and has better performance on low-resolution images compared with methods using image super-resolution and expression recognition separately.

U2 - 10.1016/j.knosys.2021.107678

DO - 10.1016/j.knosys.2021.107678

M3 - Article

SN - 0950-7051

VL - 236

JO - Knowledge-Based Systems

JF - Knowledge-Based Systems

IS - 107678

M1 - 236

ER -