Multimedia Intelligence: Confluence of Multimedia and Artificial Intelligence

By Rakesh Nakod, Softnautics

In contrast to traditional mass media, such as printed material or audio recordings, which feature little to no interaction between users, a multimedia is a form of communication that uses a combination of different content forms such as audio, text, animations, images, or video into a single interactive presentation. This definition now seems outdated because coming to 2022, multimedia has just exploded with more complex forms of interactions. Alexa, Google Assistant, Twitter, Snapchat, Instagram Reels, and many more such apps are becoming a daily part of the common man’s life. Such an explosion of multimedia and the rising need for artificial intelligence are bound to collide, and that is where multimedia intelligence comes into picture. Multimedia market is being driven forward by the increasing popularity of virtual creation in the media and entertainment industries, as well as its ability to create high-definition graphics and real-time virtual worlds. The growth is such that between 2022 to 2030, the global market for AI in media & entertainment is anticipated to expand at a 26.9% CAGR and reach about USD 99.48 billion, as per the Grand View Research, Inc. reports.

What is multimedia intelligence?

The rise and consumption of ever-emerging multimedia applications and services are churning out so much data, giving rise to conducting research and analysis on it. We are seeing great forms of multimedia research already like image/video content analysis, video or image search, recommendations, multimedia streaming, etc. Also, on the other hand, Artificial Intelligence is evolving at a faster pace, making it the perfect time for tapping content-rich multimedia for more intelligent applications.

Multimedia intelligence refers to the eco-system created when we apply artificial intelligence to multimedia data. This eco-system is a 2-way give-and-take relationship. In the first relation, we see how multimedia can boost research in artificial intelligence, enabling the evolution of algorithms and pushing AI toward achieving human-level perception and understanding. In the second relation, we see how artificial intelligence can boost multimedia data to become more inferable and reliable by providing its ability to reason. Like in the case of on-demand video streaming applications use AI algorithms to analyse user demographics and behaviour and recommend content that they enjoy streaming or watching. As a result, these AI-powered platforms focus on providing users with content tailored to their specific interests, resulting in a truly customized experience. Thus, multimedia intelligence is a closed cyclic loop between multimedia and AI, where they mutually influence and enhance each other.

• Evolution and significance

The evolution of multimedia should be credited to the evolution of smartphones. Video calling through applications like skype, and WhatsApp truly marked that multimedia is here to dominate. This was a significant move because they completely revolutionized long distance communication. This has evolved further to even more complex applications like video streaming apps like discord, twitch, etc. Then AR/VR technology took it a step ahead by integrating motion sensing and geo-sensing into audio, and video.

Multimedia contains multimodal and heterogenous data like images, audio, video, text, etc. together. Multimedia data has become very complex, and this will be incremental. Normal algorithms are not capable enough to co-relate and derive insights from such data and this is still an active area of research, even for AI algorithms it’s a challenge to connect and establish a relationship between different modalities of the data.

• Difference between media intelligence and multimedia intelligence

There is a significant difference between media and multimedia intelligence. Text, drawings, visuals, pictures, film, video, wireless, audio, motion graphics, web, and so on are all examples of media. Simply put, multimedia is the combination of two or more types of media to convey information. So, to date, when we talk about media intelligence, we are already seeing applications that exhibit it. Voice Bots like Alexa and Google Assistant are audio intelligent, Chatbots are text intelligent, and drones that recognize and follow hand gestures are video intelligent. There are very few multimedia intelligent applications. To name one: There is EMO – An AI Desktop robot that utilizes multimedia for all its interactions.

• Industrial landscape for multimedia intelligence

Multimedia is closely tied to the media and entertainment industry. Artificial Intelligence enhances and influences everything in multimedia.

Landscape for Multimedia Intelligence

Let’s walk through each stage and see how artificial intelligence is impacting them:

• Media devices

The media devices that have increasingly become coherent with artificial intelligence applications are cameras and microphones. Smart cameras are not just limited to capturing images and videos these days, but they increasingly do more stuff like detecting objects, tracking items, applying various face filters, etc. All these are driven by AI algorithms and come as part of the camera itself. Microphones are also getting smarter where AI algorithms do active noise cancellations and filter out ambient sounds. Wake words are the new norm, thanks to Alexa and Siri like applications that next-gen microphones are having in-built wake-word or key-phrase recognition AI models.

• Image/Audio coding and compression

Autoencoders consists of two components namely encoder, and decoder and are self-supervised machine learning models that use recreating input data to reduce its size. These models are trained as supervised machine learning models and inferred as unsupervised models, hence the name self-supervised models. Autoencoders can be used for image denoising, image compression, and, in some cases, even the generation of image data. This is not limited to images only, autoencoders can be applied to audio data too for the same requirements.

GAN (General Adversarial Networks) are again revolutionary deep neural networks that have made it possible to generate images from texts. OpenAI’s recent project DALLE can generate images from textual descriptions. GFP (Generative Facial Prior)-GAN is another project that can correct and re-create any bad image. AI has shown quite promising results and has proven the feasibility of Deep learning-based image/audio encoding and compression.

• Audio / Video distribution

Video streaming platforms like Netflix and Disney Hotstar extensively use AI for improving their content delivery across a global set of users. AI algorithms dominate personalization and recommendation services for both platforms. AI algorithms are also used for the generation of video meta-data for improving search on their platforms. Predicting content delivery and caching appropriate video content geographically is a challenging task that has been simplified to a good extent by AI algorithms. AI has honestly proven its potential to be a game-changer for the streaming industry by offering effective ways to encode, distribute, and organize data. Not just for video streaming platforms, but also for game streaming platforms like Discord, and Twitch and communication platforms like Zoom, and Webex, AI will become an integrated part of AV distribution.

• Categorization of content

On the internet, data is created in a wide range of formats in just a few seconds. Putting stuff into categories and organizing it could be a huge task. Artificial intelligence (AI) steps in to help with the successful classification of information into relevant categories, enabling users to find their preferred topic of interest faster, improving customer engagement, creating more enticing and effective targeted content, and boosting revenue.

• Regulating and identifying fake content

Several websites generate and spread fake news in addition to legitimate news stories to enrage the public about events or societal issues. AI is assisting with the discovery and management of such content, as well as with the moderation or deletion of such content before distribution on internet platforms like social media sites. All platforms including Facebook, LinkedIn, Twitter, Instagram, etc. employ powerful AI algorithms in most of their features. Targeted ads services, recommendation services, job recommendations, fraud profile detections, harmful content detections, etc. has AI in it.

We have tried to cover how multimedia and artificial intelligence are interrelated and how they are impacting various industries. Still, this is a broad research topic since media intelligence is still in cogs where AI algorithms are still learning from single media, and we build other algorithms to co-relate them. There is still scope for the evolution of AI algorithms that would understand the full multimedia data in a singularity like how a human does it.

Author Bio:

Rakesh is an Associate Principal Engineer at Softnautics, an AI proficient having experience in developing and deploying AI solutions across computer vision, NLP, audio intelligence, and document mining. He also has vast experience in developing AI-based enterprise solutions and strives to solve real-world problems with AI. He is an avid food lover, passionate about sharing knowledge, and enjoys gaming, and playing cricket in his free time.