Multi Label Emotion Classification Using Machine Learning And Deep Learning Methods

Emotion detection in online social networks benefits many applications like personalized advertisement services, suggestion systems, etc. Emotion can be identified from various sources like text, facial expressions, images, speeches, paintings, songs, etc. Emotion detection can be done by various techniques in machine learning. Traditional emotion detection techniques mainly focus on multi-class classification while ignoring the co-existence of multiple emotion labels in one instance. This research work is focussed on classifying multiple emotions from data to handle complex data with the help of different machine learning and deep learning methods. Before modeling, first data analysis is done and then the data is cleaned. Data pre-processing is performed in steps such as stop-words removal, tokenization, stemming and lemmatization, etc., which are performed using a Natural Language Processing toolkit (NLTK). All the input variables are converted into vectors by naive text encoding techniques like word2vec, Bag-of-words, and term frequency-inverse document frequency (TF-IDF). This research is implemented using python programming language. To solve multi-label emotion classification problem, machine learning and deep learning methods were used. The evaluation parameters such as accuracy, precision, recall, and F1-score were used to evaluate the performance of the classifiers Naïve Bayes, support vector machine (SVM), Random Forest, K-nearest neighbour (KNN), GRU (Gated Recurrent Unit) based RNN (Recurrent Neural Network) with Adam optimizer and Rmsprop optimizer. GRU based RNN with Rmsprop optimizer achieves an accuracy of 82.3%, Naïve Bayes achieves highest precision of 0.80, Random Forest achieves highest recall score of 0.823, SVM achieves highest F1 score of 0.798 on the challenging SemEval2018 Task 1: E-c multi-label emotion classification dataset. Also, One-way Analysis of Variance (ANOVA) test was performed on the mean values of performance metrics (accuracy, precision, recall, and F1-score) on all the methods.

This book covers deep-learning-based approaches for sentiment analysis, a relatively new, but fast-growing research area, which has significantly changed in the past few years. The book presents a collection of state-of-the-art approaches, focusing on the best-performing, cutting-edge solutions for the most common and difficult challenges faced in sentiment analysis research. Providing detailed explanations of the methodologies, the book is a valuable resource for researchers as well as newcomers to the field.

Affective computing refers to computing that relates to, arises from, or influences emotions. The goal of affective computing is to bridge the gap between humans and machines and ultimately endow machines with emotional intelligence for improving natural human-machine interaction. In the context of human-robot interaction (HRI), it is hoped that robots can be endowed with human-like capabilities of observation, interpretation, and emotional expression. The research on affective computing has recently achieved extensive progress with many fields contributing including neuroscience, psychology, education, medicine, behavior, sociology, and computer science. Current research in affective computing concentrates on estimating human emotions through different forms of signals such as speech, face, text, EEG, fMRI, and many others. In neuroscience, the neural mechanisms of emotion are explored by combining neuroscience with the psychological study of personality, emotion, and mood. In psychology and philosophy, emotion typically includes a subjective, conscious experience characterized primarily by psychophysiological expressions, biological reactions, and mental states. The multi-disciplinary features of understanding “emotion” result in the fact that inferring the emotion of humans is definitely difficult. As a result, a multi-disciplinary approach is required to facilitate the development of affective computing. One of the challenging problems in affective computing is the affective gap, i.e., the inconsistency between the extracted feature representations and subjective emotions. To bridge the affective gap, various hand-crafted features have been widely employed to characterize subjective emotions. However, these hand-crafted features are usually low-level, and they may hence not be discriminative enough to depict subjective emotions. To address this issue, the recently-emerged deep learning (also called deep neural networks) techniques provide a possible solution. Due to the used multi-layer network structure, deep learning techniques are capable of learning high-level contributing features from a large dataset and have exhibited excellent performance in multiple application domains such as computer vision, signal processing, natural language processing, human-computer interaction, and so on. The goal of this Research Topic is to gather novel contributions on deep learning techniques applied to affective computing across the diverse fields of psychology, machine learning, neuroscience, education, behavior, sociology, and computer science to converge with those active in other research areas, such as speech emotion recognition, facial expression recognition, Electroencephalogram (EEG) based emotion estimation, human physiological signal (heart rate) estimation, affective human-robot interaction, multimodal affective computing, etc. We welcome researchers to contribute their original papers as well as review articles to provide works regarding the neural approach from computation to affective computing systems. This Research Topic aims to bring together research including, but not limited to: • Deep learning architectures and algorithms for affective computing tasks such as emotion recognition from speech, face, text, EEG, fMRI, and many others. • Explainability of deep Learning algorithms for affective computing. • Multi-task learning techniques for emotion, personality and depression detection, etc. • Novel datasets for affective computing • Applications of affective computing in robots, such as emotion-aware human-robot interaction and social robots, etc.

Computer understanding of human emotions has become crucial and complex within the era of digital interaction and artificial intelligence. Emotion detection, a field within AI, holds promise for enhancing user experiences, personalizing services, and revolutionizing industries. However, navigating this landscape requires a deep understanding of machine and deep learning techniques and the interdisciplinary challenges accompanying them. Machine and Deep Learning Techniques for Emotion Detection offer a comprehensive solution to this pressing problem. Designed for academic scholars, practitioners, and students, it is a guiding light through the intricate terrain of emotion detection. By blending theoretical insights with practical implementations and real-world case studies, our book equips readers with the knowledge and tools needed to advance the frontier of emotion analysis using machine and deep learning methodologies.

Neural Networks are at the core of computer vision solutions for various applications. With the advent of deep neural networks Facial Expression Recognition (FER) has been a very ineluctable and challenging task in the field of computer vision. Micro-expressions (ME) have been quite prominently used in security, psychotherapy, neuroscience and have a wide role in several related disciplines. However, due to the subtle movements of facial muscles, the micro-expressions are difficult to detect and identify. Due to the above, emotion detection and classification have always been hot research topics. The recently adopted networks to train FERs are yet to focus on issues caused due to overfitting, effectuated by insufficient data for training and expression unrelated variations like gender bias, face occlusions and others. Association of FER with the Speech Emotion Recognition (SER) triggered the development of multimodal neural networks for emotion classification in which the application of sensors played a significant role as they substantially increased the accuracy by providing high quality inputs, further elevating the efficiency of the system. This thesis relates to the exploration of different principles behind application of deep neural networks with a strong focus towards Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) in regards to their applications to emotion recognition. A Motion Magnification algorithm for ME's detection and classification was implemented for applications requiring near real-time computations. A new and improved architecture using a Multimodal Network was implemented. In addition to the motion magnification technique for emotion classification and extraction, the Multimodal algorithm takes the audio-visual cues as inputs and reads the MEs on the real face of the participant. This feature of the above architecture can be deployed while administering interviews, or supervising ICU patients in hospitals, in the auto industry, and many others. The real-time emotion classifier based on state-of-the-art Image-Avatar Animation model was tested on simulated subjects. The salient features of the real-face are mapped on avatars that are build with a 3D scene generation platform. In pursuit of the goal of emotion classification, the Image Animation model outperforms all baselines and prior works. Extensive tests and results obtained demonstrate the validity of the approach.

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective. This book, emerging from the Special Issue of the Sensors journal on “Emotion and Stress Recognition Related Sensors and Machine Learning Technologies” emerges as a result of the crucial need for massive deployment of intelligent sociotechnical systems. Such technologies are being applied in assistive systems in different domains and parts of the world to address challenges that could not be addressed without the advances made in these technologies.

Sentiment analysis is a branch of natural language processing concerned with the study of the intensity of the emotions expressed in a piece of text. The automated analysis of the multitude of messages delivered through social media is one of the hottest research fields, both in academy and in industry, due to its extremely high potential applicability in many different domains. This Special Issue describes both technological contributions to the field, mostly based on deep learning techniques, and specific applications in areas like health insurance, gender classification, recommender systems, and cyber aggression detection.

Ús de diferents tècniques de deep learning per al reconeixement d'emocions a partir d'imatges i videos. Les diferents tècniques s'apliquen, es valoren i comparen amb l'objectiu de fer-les servir conjuntament en una aplicació final.

Deep learning (DL) is a subset of machine learning and artificial intelligence. It has broad adaptability to most types of tasks, including but not limited to text classification and identification of objects in images and videos. It can generate more powerful models compared with legacy machine learning methods. Multi-task Learning (MTL) is an approach that improves generalization by using the domain information which is contained in the training signals of related tasks as an inductive bias. When we apply Multi-task Learning to Deep Learning, the method is called Multi-task Deep Learning. We focus on deep learning for natural language processing, in particular on how multi-task learning can be used to improve the performance on several tasks at the same time. We present two experiments that deploy Multi-task Deep Learning for detecting affect information from texts. In experiment 1, we propose a hard parameter sharing multi-task deep learning model for the task of detecting happiness ingredients. For training Deep Learning classifiers, the two primary classes, "agency" and "social", meaning whether the author is in control or the moment involves other people, are treated as two separate tasks while "concept", meaning the categories of the moment, is serverd as an auxiliary task. Then, we train a multi-task deep learning classifier to see if the shared knowledge among the three tasks can be used to improve the overall results. In addition, we compare several models that use different kinds of word embeddings: different dimensions of the vectors, fixed versus trainable embeddings, initialized randomly or with pre-trained embeddings. In experiment 2, we compare several different multi-task deep learning models on the task of six labels: Information_disclosure, Emotional_disclosure, Support, General_support, Info_support, and \textitEmo_support. The labels mean that the texts contain informational or emotional disclosure of a person, or express informational or emotional supprtiveness, which can also be catchphrases. We propose a novel way to employ the multi-task deep learning model for the task of detecting disclosure and support, called Venn-diagram-based fragment MTL model. We calculate all possible logical relations between the six labels, represented in a Venn diagram. Based on it, the six labels are distributed to multiple fragment layers. Then, a multi-task deep neural network is built on these layers. We showed that our multi-task learning model has a stronger ability to represent multi-label tasks over multiple single-task learning models, and using pre-trained trainable embeddings with auxiliary tasks can get the best results. Furthermore, among different multi-task deep learning structures, our model based on Venn diagrams achieved better performance than regular multi-task deep learning and obtained the best results in the CL-Aff Shared Task 2020 for the disclosure labels.

Abstract: Sentiment analysis has been a very popular technique in recent times for analyzing a person’s behavior. It mainly deals with the process of computationally identifying and categorizing opinions expressed in the form of text, especially to determine the writer’s attitude towards a topic. Categories involved while classifying these sentiments are positive, negative or neutral. In addition to sentiment analysis, some of the other techniques like facial expressions, pupil dilation has also been widely used. Nevertheless, this proves that numerous attempts have been made towards analyzing a person’s perspective, but their overall effectiveness has been on the lower side. The reason behind their inadequate approach is straightforward; user or subject can easily trick the machine by writing fake or sentimental text, also he or she can make false facial expressions to give wrong signals which hence will be interpreted incorrectly. So, instead of analyzing these outer expressions and signals, if we could directly examine the data produced by the brain and inspect its underlying patterns, then that might give us more accurate results. The emphasis of this thesis is to use such brain signals to classify and uncover the underlying emotions in a human brain. The two targeted classes of emotion are valence and arousal. We will be using deep learning models in the form of artificial neural net and convolution neural net to process the electroencephalography or brain waves data and map it to these two classes. The learning approach will be of supervised learning as corresponding labels are given with the training data. The approach ultimately yields a validation score of more than 75% for one and close to 65% for the second, which is acceptable with the amount of data we had for the training of these deep learning models.