Artificial Intelligence in Affective Computing

Affect means "touch the feelings of, or move emotionally". Whereas, affective means “relating to moods, feelings, and attitudes”. Thus affective computing is “the study and development of systems and devices that can recognize, interpret, process, and simulate human affects”. Affective computing is an interdisciplinary field which spans computer science, psychology, and cognitive science. Affective computing is sometimes called artificial emotional intelligence, or emotion AI. Emotional intelligence can be defined as “the capacity to be aware of, control, and express one's emotions, and to handle interpersonal relationships judiciously and empathetically”.

Sentiment analysis might be considered a primitive form of affective computing. Sentiment analysis may be defined as “the process of computationally identifying and categorizing opinions expressed in a piece of text”, especially in order to determine whether the writer's attitude is positive, negative, or neutral. Sentiment analysis may also be referred to as opinion mining, or emotion AI. SentiWordNet is a popular lexical resource for opinion mining. It assigns three sentiment scores to WordNet synsets, or sets of synonyms. Natural language processing toolkits are often used for sentiment analysis, such as GATE, LingPipe, NLTK, R-Project, RapidMiner, StanfordNLP, UIMA, and WEKA.

In terms of natural language the 2011 book Affective Computing and Sentiment Analysis: Emotion, Metaphor and Terminology, edited by Khurshid Ahmad, addresses the role of metaphor in affective computing. Metaphor is something that is considered to be representative or symbolic of something else, in other words a paradox of comparing unlike things. Contributor Andrew Goatly looks at metaphor as a resource for conceptualisation and expression of emotion. For instance, emotions may be present in deep lexical semantics. Metaphoricity is the quality of being metaphorical, which contributor Carl Vogel maintains involves sense modulation. In a conversational agent, affect may be transferred by metaphor, forming a kind of artificial or synthetic emotion.

In ‘affective dialog systems’, an ‘affect listener’ is a device which detects and adapts to the affective states of users, facilitating meaningful responses. The SEMAINE project was a well known European Union initiative to create a non-verbally competent ‘sensitive artificial listener’. SAL, the SEMAINE sensitive artificial listener, was in effect a kind of ‘emotional agent’, or ‘emotion agent’, which could be termed an ‘affective interface’.

Automated, or automatic, emotion recognition leverages techniques from signal processing, machine learning, and computer vision. Computers use different methods to interpret emotion, from Bayesian networks to Paul Ekman's ‘Facial Action Coding System’. A number of companies are now working with automatic emotion recognition, including affectiva.com (Emotion Recognition Software), eyeris.ai (Emotional AI and Face Analytics), imotions.com (Emotion Analysis Engine), nviso.ch (Emotion Recognition Software), and visagetechnologies.com (Face Tracking and Analysis).

From this overview, it becomes clear that that there are two main aspects to affective computing, 1) emotion detection, or emotion recognition, and 2) artificial or synthetic emotion, or emotion synthesis. Facial recognition figures prominently in emotion detection, however language can be used for emotion recognition, in particular metaphor. Voice biometrics are also being used for detecting emotion, something like polygraph biofeedback. Emotion may be synthesized facially in the form of an avatar, or talking head, not to mention animatronic head. Emotional body language could be expressed by a humanoid robot. But also natural language can be used for the expression of computational affect, in the form of metaphor generation as a vehicle for emotion.

References:

• 100 Best Emotion Recognition Videos
• 100 Best Sentiment Analysis Videos
• 100 Best Sensitive Artificial Listener Videos
• Affective Dialog Systems
• Affect Listener & Dialog Systems
• Emotional Agents 2016
• SAL (Sensitive Artificial Listener) 2017
• Sentiment Analysis & Dialog Systems 2017
• Sentiment Analysis Tools & Dialog Systems
• SentiWordNet & Dialog Systems

Aesthetics In Artificial Intelligence: An Overview

Here we look at the scholarly literature on aesthetics in artificial intelligence, and by extension robotics, over the past decade. The focus is mainly on natural language processing, and in particular the sub-genre of natural language generation, or generative text.

Aesthetics can be defined as “a set of principles concerned with the nature and appreciation of beauty”. Since artificial intelligence is built on computing, let’s first look at aesthetics in computing. From the perspective of engineering, the traditional definition of aesthetics in computing could be termed structural, such as an elegant proof, or beautiful diagram. Whereas, a broader definition would include more abstract qualities of form and symmetry enhancing pleasure and creative expression.

2006

2006 appears to have been a watershed year for aesthetics in computing with the publication by MIT Press of Paul Fishwick’s book, Aesthetic Computing. In this book, key figures from art, design, and computer science set the foundation for a new discipline that applies art theory to computing.

Nick Montfort published on Natural Language Generation And Narrative Variation In Interactive Fiction in 2006. In it, he demonstrated applying concepts from narratology, such as separating expression from content or discourse from story, to the new field of interactive fiction.

Also in 2006, Liu and Maes concretized a computational model of aesthetics in the form of an “Aesthetiscope”, as presented in Rendering Aesthetic Impressions Of Text In Color Space. Their device was a computer program that portrayed aesthetic impressions of text rendered as color grid artwork, partially based on Jungian aesthetic theory, and reminiscent of abstract expressionism.

2007

In Command Lines (2007), Liu and Douglass addressed “aesthetics and technique in interactive fiction and new media”. They analyse aesthetic developments in the field of interactive fiction, text-based narrative experiences, in the context of implied code versus frustration aesthetics, or the interactor’s mental model versus structural constraints.

2008

Game aesthetics are further explored by Andrew Hutchison in Making the Water Move (2008), examining “techno-historic limits in the game aesthetics of Myst and Doom”. He uses the landmark games as examples of the evolution of game aesthetics across time.

2009

Howe and Soderman talk about the RiTa toolkit for generative writing in The Aesthetics Of Generative Literature (2009). They discuss issues such as surprise, materiality, push-back, and layering within the larger contexts of generative art and writing practice.

In Data Portraits (2009), Dragulescu and Donath address “aesthetics and algorithms”. What they call data portraits would be called online footprint today, primarily social media; however, their data portraits engaged aesthetically with cinematography, typography and animation.

Michael Nixon’s Enhancing Believability (2009) evaluated the application of Delsarte’s aesthetic system in relation to designing virtual humans. Nixon empirically tested Delsarte’s system, and found it promising as a starting point for creating believable characters.

2010

Datta and Wang report on ACQUINE: Aesthetic Quality Inference Engine (2010), a public system which allowed users to upload photographs for automatic rating of aesthetic quality. The first public tool of its kind, this system was based on an SVM classifier which extracted visual features on the fly for real-time classification and prediction.

In Impossible Metaphors (2010), Hanna Kim addresses the premise of metaphorical un-interpretability of aesthetic terms, arguing instead in favor of metaphorical interpretability, based on multi-dimensionality and default dimension. Rafael De Clercq had previously postulated aesthetic terms are metaphorically uninterpretable, due to having no home domain of application.

2011

In 2011, Scott Dexter et al published On the Embodied Aesthetics of Code, broadly discussing the aesthetics of programming itself. They provide empirical evidence placing the “embodied experience of programming” in the context of “embodiment in the production of meaning”.

2012

Innovation theory, aesthetics, and science of the artificial after Herbert Simon (2012) by Helge Godoe broaches the aesthetic experience of generative literature from the user’s and designer’s roles, with reference to AI pioneer and Nobel laureate Herbert Simon in the context of innovation. Godoe suggests aesthetics, serendipity, and imagination form the “soul” of innovation.

Labutov and Lipson in Humor As Circuits In Semantic Networks (2012) present an implementation of their theory for the automatic generation of humor. They mine simple, humorous scripts using the well known ConceptNet, with dual iteration maximized for Victor Raskin’s “Script-based Semantic Theory of Humor”.

2013

In The User's And The Designer's Role And The Aesthetic Experience Of Generative Literature (2013), Carvalho-Pereira and Maciel analyse the aesthetic interaction between, writers, readers, and what they call “wreaders”. In testing, they found users don’t feel like co-authors.

2014

In the 2014 book, Examining Paratextual Theory and its Applications in Digital Culture, Desrochers and Apollon propose a theoretical and practical interdisciplinary framework. Today, Gérard Genette’s original paratext could be thought of as online footprint, or “data portrait”.

Derezinski and Rohanimanesh’s An Information Theoretic Approach to Quantifying Text Interestingness (2014) studies the problem of automatically predicting text “interestingness”. They use a word distribution model that uses Jensen-Shannon divergence to measure text diversity, and demonstrate that correlates with interestingness (which they point out has been used elsewhere for humor identification).

Adesegun Oyedele et al in Individual Assessment Of Humanlike Consumer Robots (2014) examine the aesthetic attributes of products to appeal to consumers’ mental and emotional needs. They apply the “technology acceptance model” to consumer robots.

In this doctoral thesis, Digital Puppetry of Wayang Kulit Kelantan (2014), Khor Kheng Kia presents a study of how visual aesthetics impact on digital preservation of endangered culture, Malaysian shadow puppet theatre. Key-frame animation and motion capture were both used in his experiments.

In Embodied Aesthetics In Auditory Display (2014), Roddy and Furlong address the importance of aesthetics in “auditory display”, or the use of sound for communication between a computer and user. They present arguments for an embodied aesthetic framework, thus rejecting the aesthetic theory of Immanuel Kant.

Debasis Ganguly et al in their paper, Automatic Prediction Of Text Aesthetics And Interestingness (2014), investigate the problem of automated aesthetics prediction, for instance generated from user generated content and ratings, in this case Kindle “popular highlights” data. They use supervised classification to predict text aesthetics with feature vectors for each text passage.

2015

Stefania Sansoni et al look at the role of personal characteristics in attraction, in The Aesthetic Appeal Of Prosthetic Limbs And The Uncanny Valley (2015). This may be the first research into the relationship of aesthetic attraction to devices and their human-likeness.

The 2015 book by Michele Dickey, Aesthetics and Design for Game-based Learning, is about emotionally imbuing participants with motivation and meaning through aesthetic experiences. This is a vital but neglected aspect of game-based learning.

The book Robots that Talk and Listen (2015), edited by Judith Markowitz, delves into the social impact of technology. In particular, David Duffy addresses “android aesthetics”, humanoid robots as works of art.

2016

In her thesis Quality of Aesthetic Experience and Implicit Modulating Factors (2016), Wendy Ann Mansilla refutes the standing assumption that aesthetics in digital media are restricted to the structural or superficial, and instead points to complex implicit variables that contribute to individual user experiences. She investigated various factors, such as use of color in eliciting emotion, presence of familiar characters or alter ego, and food craving versus pleasure technologies.

Takashi Ogata, in Automatic Generation, Creativity, and Production of Narrative Content (2016), considers the generation, creation, and production of automatic narrative or story generation, from the perspective of cognitive science. The artistic and aesthetic problems are considered in terms of their relationships with technology.

In their 2016 book, Computational and Cognitive Approaches to Narratology, Ogata and Akimoto examine among other things the possibility of generating literary works by computer. They focus on the affective or psychological aspects of language engineering, with regard to “intertextuality”.

2017

In his book Aesthetic Origins (2017), Jay Patrick Starliper examines imagination through the work of Pulitzer Prize winning poet Peter Viereck. And through this philosophical deconstruction, Starliper demonstrates why books are bullets, perhaps giving rise to today’s weaponized narrative of bots and fake news.

Fabrizio Guerrini et al discuss an offspring of interactive storytelling applied to movies, called “movietelling”, in Interactive Film Recombination (2017). They propose the integration of narrative generation, AI planning, and video processing to model and construct filmic variants from baseline content.

Burt Kimmelman reconceives the actual in digital poetry and art in his paper, Code and Substrate (2017). Kimmelman argues that since the dynamic of digital poetry and art demands the dissolution of one from the other, this has brought the notion of embodiment to prominence.

Building on their 2014 work (above), Derezinski, Rohanimanesh and Hydrie present an approach based on a probabilistic model of surprise, and the construction of distributional word embeddings in their 2017 paper, Discovering Surprising Documents with Context-Aware Word Representations. They found it to be particularly effective for short documents, even down to the single sentence level.

In the Handbook of Writing, Literacies, and Education in Digital Cultures (2017), Theo van Leeuwen addresses aesthetics and text in the digital age. He argues that originally art, text has devolved into utility and industry. He maintains this devolution from illuminated to black and white was a function of the dour nature of Protestantism, unadorned delivery of the word.

Conclusion

Surveying the past decade or so, a number of figures, concepts and (new) terminologies emerge. As in the history of aesthetic inquiry in general, Immanuel Kant’s rejection of material dualism in favor of personal subjectivism relative to the material world figures prominently. Francois Delsarte’s “applied aesthetics” is introduced to robotic embodiment. Thus, note can be taken from the sensibilities of “interior design” to extend robotic design beyond the structural and superficial.

The watershed emergence of a field of “aesthetic computing” in 2006 really kicks off the modern journey, with the increase in application of art theory and practice to computing. However, the field of “computational aesthetics” - considered a subfield of artificial intelligence, concerned with the computational assessment of beauty - can be traced back as far as 1928 to mathematician George David Birkhoff’s proposed “aesthetic measure”.

Across time, we can see that machine learning is gradually taking over, leading to what may be termed a “neural aesthetic”. This can be seen in numerous recent “artistic hacks”, such as Deepdream, NeuralTalk, and Stylenet. Couching advanced, or deep, machine learning technologies in artistic metaphor helps clarify otherwise obscure jargon, and makes the subject more accessible to people in the real world.