With the increasing interest in digital technologies, emotion recognition plays an important role in several applications such as healthcare computer-aided diagnosis, social media analysis, opinion mining and recommendation systems, understanding human behavior and interaction in workplaces, effective communication and linguistic analysis, and cognitive human–machine interaction. This field is receiving a growing interest in recent years. In this paper, we present a thorough review of emotional artificial intelligence through identification and in-depth analysis of existing multimodal datasets along with their related research directions and methodologies. It establishes essential requirements for the development of a multimodal dataset and outlines challenges spanning its entire lifecycle, from recording to deployment. Moreover, a taxonomy of various categories and applications is introduced based on the key characteristics of various multimodal datasets. Finally, the paper concludes with discussions and insights into future directions and prospects for standard schemes to facilitate the efficient development of reliable and reusable benchmark datasets that can help researchers and developers advance this field.