Here we look at the scholarly literature on aesthetics in artificial intelligence, and by extension robotics, over the past decade. The focus is mainly on natural language processing, and in particular the sub-genre of natural language generation, or generative text.
Aesthetics can be defined as “a set of principles concerned with the nature and appreciation of beauty”. Since artificial intelligence is built on computing, let’s first look at aesthetics in computing. From the perspective of engineering, the traditional definition of aesthetics in computing could be termed structural, such as an elegant proof, or beautiful diagram. Whereas, a broader definition would include more abstract qualities of form and symmetry enhancing pleasure and creative expression.
2006
2006 appears to have been a watershed year for aesthetics in computing with the publication by MIT Press of Paul Fishwick’s book, Aesthetic Computing. In this book, key figures from art, design, and computer science set the foundation for a new discipline that applies art theory to computing.
Nick Montfort published on Natural Language Generation And Narrative Variation In Interactive Fiction in 2006. In it, he demonstrated applying concepts from narratology, such as separating expression from content or discourse from story, to the new field of interactive fiction.
Also in 2006, Liu and Maes concretized a computational model of aesthetics in the form of an “Aesthetiscope”, as presented in Rendering Aesthetic Impressions Of Text In Color Space. Their device was a computer program that portrayed aesthetic impressions of text rendered as color grid artwork, partially based on Jungian aesthetic theory, and reminiscent of abstract expressionism.
2007
In Command Lines (2007), Liu and Douglass addressed “aesthetics and technique in interactive fiction and new media”. They analyse aesthetic developments in the field of interactive fiction, text-based narrative experiences, in the context of implied code versus frustration aesthetics, or the interactor’s mental model versus structural constraints.
2008
Game aesthetics are further explored by Andrew Hutchison in Making the Water Move (2008), examining “techno-historic limits in the game aesthetics of Myst and Doom”. He uses the landmark games as examples of the evolution of game aesthetics across time.
2009
Howe and Soderman talk about the RiTa toolkit for generative writing in The Aesthetics Of Generative Literature (2009). They discuss issues such as surprise, materiality, push-back, and layering within the larger contexts of generative art and writing practice.
In Data Portraits (2009), Dragulescu and Donath address “aesthetics and algorithms”. What they call data portraits would be called online footprint today, primarily social media; however, their data portraits engaged aesthetically with cinematography, typography and animation.
Michael Nixon’s Enhancing Believability (2009) evaluated the application of Delsarte’s aesthetic system in relation to designing virtual humans. Nixon empirically tested Delsarte’s system, and found it promising as a starting point for creating believable characters.
2010
Datta and Wang report on ACQUINE: Aesthetic Quality Inference Engine (2010), a public system which allowed users to upload photographs for automatic rating of aesthetic quality. The first public tool of its kind, this system was based on an SVM classifier which extracted visual features on the fly for real-time classification and prediction.
In Impossible Metaphors (2010), Hanna Kim addresses the premise of metaphorical un-interpretability of aesthetic terms, arguing instead in favor of metaphorical interpretability, based on multi-dimensionality and default dimension. Rafael De Clercq had previously postulated aesthetic terms are metaphorically uninterpretable, due to having no home domain of application.
2011
In 2011, Scott Dexter et al published On the Embodied Aesthetics of Code, broadly discussing the aesthetics of programming itself. They provide empirical evidence placing the “embodied experience of programming” in the context of “embodiment in the production of meaning”.
2012
Innovation theory, aesthetics, and science of the artificial after Herbert Simon (2012) by Helge Godoe broaches the aesthetic experience of generative literature from the user’s and designer’s roles, with reference to AI pioneer and Nobel laureate Herbert Simon in the context of innovation. Godoe suggests aesthetics, serendipity, and imagination form the “soul” of innovation.
Labutov and Lipson in Humor As Circuits In Semantic Networks (2012) present an implementation of their theory for the automatic generation of humor. They mine simple, humorous scripts using the well known ConceptNet, with dual iteration maximized for Victor Raskin’s “Script-based Semantic Theory of Humor”.
2013
In The User's And The Designer's Role And The Aesthetic Experience Of Generative Literature (2013), Carvalho-Pereira and Maciel analyse the aesthetic interaction between, writers, readers, and what they call “wreaders”. In testing, they found users don’t feel like co-authors.
2014
In the 2014 book, Examining Paratextual Theory and its Applications in Digital Culture, Desrochers and Apollon propose a theoretical and practical interdisciplinary framework. Today, Gérard Genette’s original paratext could be thought of as online footprint, or “data portrait”.
Derezinski and Rohanimanesh’s An Information Theoretic Approach to Quantifying Text Interestingness (2014) studies the problem of automatically predicting text “interestingness”. They use a word distribution model that uses Jensen-Shannon divergence to measure text diversity, and demonstrate that correlates with interestingness (which they point out has been used elsewhere for humor identification).
Adesegun Oyedele et al in Individual Assessment Of Humanlike Consumer Robots (2014) examine the aesthetic attributes of products to appeal to consumers’ mental and emotional needs. They apply the “technology acceptance model” to consumer robots.
In this doctoral thesis, Digital Puppetry of Wayang Kulit Kelantan (2014), Khor Kheng Kia presents a study of how visual aesthetics impact on digital preservation of endangered culture, Malaysian shadow puppet theatre. Key-frame animation and motion capture were both used in his experiments.
In Embodied Aesthetics In Auditory Display (2014), Roddy and Furlong address the importance of aesthetics in “auditory display”, or the use of sound for communication between a computer and user. They present arguments for an embodied aesthetic framework, thus rejecting the aesthetic theory of Immanuel Kant.
Debasis Ganguly et al in their paper, Automatic Prediction Of Text Aesthetics And Interestingness (2014), investigate the problem of automated aesthetics prediction, for instance generated from user generated content and ratings, in this case Kindle “popular highlights” data. They use supervised classification to predict text aesthetics with feature vectors for each text passage.
2015
Stefania Sansoni et al look at the role of personal characteristics in attraction, in The Aesthetic Appeal Of Prosthetic Limbs And The Uncanny Valley (2015). This may be the first research into the relationship of aesthetic attraction to devices and their human-likeness.
The 2015 book by Michele Dickey, Aesthetics and Design for Game-based Learning, is about emotionally imbuing participants with motivation and meaning through aesthetic experiences. This is a vital but neglected aspect of game-based learning.
The book Robots that Talk and Listen (2015), edited by Judith Markowitz, delves into the social impact of technology. In particular, David Duffy addresses “android aesthetics”, humanoid robots as works of art.
2016
In her thesis Quality of Aesthetic Experience and Implicit Modulating Factors (2016), Wendy Ann Mansilla refutes the standing assumption that aesthetics in digital media are restricted to the structural or superficial, and instead points to complex implicit variables that contribute to individual user experiences. She investigated various factors, such as use of color in eliciting emotion, presence of familiar characters or alter ego, and food craving versus pleasure technologies.
Takashi Ogata, in Automatic Generation, Creativity, and Production of Narrative Content (2016), considers the generation, creation, and production of automatic narrative or story generation, from the perspective of cognitive science. The artistic and aesthetic problems are considered in terms of their relationships with technology.
In their 2016 book, Computational and Cognitive Approaches to Narratology, Ogata and Akimoto examine among other things the possibility of generating literary works by computer. They focus on the affective or psychological aspects of language engineering, with regard to “intertextuality”.
2017
In his book Aesthetic Origins (2017), Jay Patrick Starliper examines imagination through the work of Pulitzer Prize winning poet Peter Viereck. And through this philosophical deconstruction, Starliper demonstrates why books are bullets, perhaps giving rise to today’s weaponized narrative of bots and fake news.
Fabrizio Guerrini et al discuss an offspring of interactive storytelling applied to movies, called “movietelling”, in Interactive Film Recombination (2017). They propose the integration of narrative generation, AI planning, and video processing to model and construct filmic variants from baseline content.
Burt Kimmelman reconceives the actual in digital poetry and art in his paper, Code and Substrate (2017). Kimmelman argues that since the dynamic of digital poetry and art demands the dissolution of one from the other, this has brought the notion of embodiment to prominence.
Building on their 2014 work (above), Derezinski, Rohanimanesh and Hydrie present an approach based on a probabilistic model of surprise, and the construction of distributional word embeddings in their 2017 paper, Discovering Surprising Documents with Context-Aware Word Representations. They found it to be particularly effective for short documents, even down to the single sentence level.
In the Handbook of Writing, Literacies, and Education in Digital Cultures (2017), Theo van Leeuwen addresses aesthetics and text in the digital age. He argues that originally art, text has devolved into utility and industry. He maintains this devolution from illuminated to black and white was a function of the dour nature of Protestantism, unadorned delivery of the word.
Conclusion
Surveying the past decade or so, a number of figures, concepts and (new) terminologies emerge. As in the history of aesthetic inquiry in general, Immanuel Kant’s rejection of material dualism in favor of personal subjectivism relative to the material world figures prominently. Francois Delsarte’s “applied aesthetics” is introduced to robotic embodiment. Thus, note can be taken from the sensibilities of “interior design” to extend robotic design beyond the structural and superficial.
The watershed emergence of a field of “aesthetic computing” in 2006 really kicks off the modern journey, with the increase in application of art theory and practice to computing. However, the field of “computational aesthetics” - considered a subfield of artificial intelligence, concerned with the computational assessment of beauty - can be traced back as far as 1928 to mathematician George David Birkhoff’s proposed “aesthetic measure”.
Across time, we can see that machine learning is gradually taking over, leading to what may be termed a “neural aesthetic”. This can be seen in numerous recent “artistic hacks”, such as Deepdream, NeuralTalk, and Stylenet. Couching advanced, or deep, machine learning technologies in artistic metaphor helps clarify otherwise obscure jargon, and makes the subject more accessible to people in the real world.
Aesthetics can be defined as “a set of principles concerned with the nature and appreciation of beauty”. Since artificial intelligence is built on computing, let’s first look at aesthetics in computing. From the perspective of engineering, the traditional definition of aesthetics in computing could be termed structural, such as an elegant proof, or beautiful diagram. Whereas, a broader definition would include more abstract qualities of form and symmetry enhancing pleasure and creative expression.
2006
2006 appears to have been a watershed year for aesthetics in computing with the publication by MIT Press of Paul Fishwick’s book, Aesthetic Computing. In this book, key figures from art, design, and computer science set the foundation for a new discipline that applies art theory to computing.
Nick Montfort published on Natural Language Generation And Narrative Variation In Interactive Fiction in 2006. In it, he demonstrated applying concepts from narratology, such as separating expression from content or discourse from story, to the new field of interactive fiction.
Also in 2006, Liu and Maes concretized a computational model of aesthetics in the form of an “Aesthetiscope”, as presented in Rendering Aesthetic Impressions Of Text In Color Space. Their device was a computer program that portrayed aesthetic impressions of text rendered as color grid artwork, partially based on Jungian aesthetic theory, and reminiscent of abstract expressionism.
2007
In Command Lines (2007), Liu and Douglass addressed “aesthetics and technique in interactive fiction and new media”. They analyse aesthetic developments in the field of interactive fiction, text-based narrative experiences, in the context of implied code versus frustration aesthetics, or the interactor’s mental model versus structural constraints.
2008
Game aesthetics are further explored by Andrew Hutchison in Making the Water Move (2008), examining “techno-historic limits in the game aesthetics of Myst and Doom”. He uses the landmark games as examples of the evolution of game aesthetics across time.
2009
Howe and Soderman talk about the RiTa toolkit for generative writing in The Aesthetics Of Generative Literature (2009). They discuss issues such as surprise, materiality, push-back, and layering within the larger contexts of generative art and writing practice.
In Data Portraits (2009), Dragulescu and Donath address “aesthetics and algorithms”. What they call data portraits would be called online footprint today, primarily social media; however, their data portraits engaged aesthetically with cinematography, typography and animation.
Michael Nixon’s Enhancing Believability (2009) evaluated the application of Delsarte’s aesthetic system in relation to designing virtual humans. Nixon empirically tested Delsarte’s system, and found it promising as a starting point for creating believable characters.
2010
Datta and Wang report on ACQUINE: Aesthetic Quality Inference Engine (2010), a public system which allowed users to upload photographs for automatic rating of aesthetic quality. The first public tool of its kind, this system was based on an SVM classifier which extracted visual features on the fly for real-time classification and prediction.
In Impossible Metaphors (2010), Hanna Kim addresses the premise of metaphorical un-interpretability of aesthetic terms, arguing instead in favor of metaphorical interpretability, based on multi-dimensionality and default dimension. Rafael De Clercq had previously postulated aesthetic terms are metaphorically uninterpretable, due to having no home domain of application.
2011
In 2011, Scott Dexter et al published On the Embodied Aesthetics of Code, broadly discussing the aesthetics of programming itself. They provide empirical evidence placing the “embodied experience of programming” in the context of “embodiment in the production of meaning”.
2012
Innovation theory, aesthetics, and science of the artificial after Herbert Simon (2012) by Helge Godoe broaches the aesthetic experience of generative literature from the user’s and designer’s roles, with reference to AI pioneer and Nobel laureate Herbert Simon in the context of innovation. Godoe suggests aesthetics, serendipity, and imagination form the “soul” of innovation.
Labutov and Lipson in Humor As Circuits In Semantic Networks (2012) present an implementation of their theory for the automatic generation of humor. They mine simple, humorous scripts using the well known ConceptNet, with dual iteration maximized for Victor Raskin’s “Script-based Semantic Theory of Humor”.
2013
In The User's And The Designer's Role And The Aesthetic Experience Of Generative Literature (2013), Carvalho-Pereira and Maciel analyse the aesthetic interaction between, writers, readers, and what they call “wreaders”. In testing, they found users don’t feel like co-authors.
2014
In the 2014 book, Examining Paratextual Theory and its Applications in Digital Culture, Desrochers and Apollon propose a theoretical and practical interdisciplinary framework. Today, Gérard Genette’s original paratext could be thought of as online footprint, or “data portrait”.
Derezinski and Rohanimanesh’s An Information Theoretic Approach to Quantifying Text Interestingness (2014) studies the problem of automatically predicting text “interestingness”. They use a word distribution model that uses Jensen-Shannon divergence to measure text diversity, and demonstrate that correlates with interestingness (which they point out has been used elsewhere for humor identification).
Adesegun Oyedele et al in Individual Assessment Of Humanlike Consumer Robots (2014) examine the aesthetic attributes of products to appeal to consumers’ mental and emotional needs. They apply the “technology acceptance model” to consumer robots.
In this doctoral thesis, Digital Puppetry of Wayang Kulit Kelantan (2014), Khor Kheng Kia presents a study of how visual aesthetics impact on digital preservation of endangered culture, Malaysian shadow puppet theatre. Key-frame animation and motion capture were both used in his experiments.
In Embodied Aesthetics In Auditory Display (2014), Roddy and Furlong address the importance of aesthetics in “auditory display”, or the use of sound for communication between a computer and user. They present arguments for an embodied aesthetic framework, thus rejecting the aesthetic theory of Immanuel Kant.
Debasis Ganguly et al in their paper, Automatic Prediction Of Text Aesthetics And Interestingness (2014), investigate the problem of automated aesthetics prediction, for instance generated from user generated content and ratings, in this case Kindle “popular highlights” data. They use supervised classification to predict text aesthetics with feature vectors for each text passage.
2015
Stefania Sansoni et al look at the role of personal characteristics in attraction, in The Aesthetic Appeal Of Prosthetic Limbs And The Uncanny Valley (2015). This may be the first research into the relationship of aesthetic attraction to devices and their human-likeness.
The 2015 book by Michele Dickey, Aesthetics and Design for Game-based Learning, is about emotionally imbuing participants with motivation and meaning through aesthetic experiences. This is a vital but neglected aspect of game-based learning.
The book Robots that Talk and Listen (2015), edited by Judith Markowitz, delves into the social impact of technology. In particular, David Duffy addresses “android aesthetics”, humanoid robots as works of art.
2016
In her thesis Quality of Aesthetic Experience and Implicit Modulating Factors (2016), Wendy Ann Mansilla refutes the standing assumption that aesthetics in digital media are restricted to the structural or superficial, and instead points to complex implicit variables that contribute to individual user experiences. She investigated various factors, such as use of color in eliciting emotion, presence of familiar characters or alter ego, and food craving versus pleasure technologies.
Takashi Ogata, in Automatic Generation, Creativity, and Production of Narrative Content (2016), considers the generation, creation, and production of automatic narrative or story generation, from the perspective of cognitive science. The artistic and aesthetic problems are considered in terms of their relationships with technology.
In their 2016 book, Computational and Cognitive Approaches to Narratology, Ogata and Akimoto examine among other things the possibility of generating literary works by computer. They focus on the affective or psychological aspects of language engineering, with regard to “intertextuality”.
2017
In his book Aesthetic Origins (2017), Jay Patrick Starliper examines imagination through the work of Pulitzer Prize winning poet Peter Viereck. And through this philosophical deconstruction, Starliper demonstrates why books are bullets, perhaps giving rise to today’s weaponized narrative of bots and fake news.
Fabrizio Guerrini et al discuss an offspring of interactive storytelling applied to movies, called “movietelling”, in Interactive Film Recombination (2017). They propose the integration of narrative generation, AI planning, and video processing to model and construct filmic variants from baseline content.
Burt Kimmelman reconceives the actual in digital poetry and art in his paper, Code and Substrate (2017). Kimmelman argues that since the dynamic of digital poetry and art demands the dissolution of one from the other, this has brought the notion of embodiment to prominence.
Building on their 2014 work (above), Derezinski, Rohanimanesh and Hydrie present an approach based on a probabilistic model of surprise, and the construction of distributional word embeddings in their 2017 paper, Discovering Surprising Documents with Context-Aware Word Representations. They found it to be particularly effective for short documents, even down to the single sentence level.
In the Handbook of Writing, Literacies, and Education in Digital Cultures (2017), Theo van Leeuwen addresses aesthetics and text in the digital age. He argues that originally art, text has devolved into utility and industry. He maintains this devolution from illuminated to black and white was a function of the dour nature of Protestantism, unadorned delivery of the word.
Conclusion
Surveying the past decade or so, a number of figures, concepts and (new) terminologies emerge. As in the history of aesthetic inquiry in general, Immanuel Kant’s rejection of material dualism in favor of personal subjectivism relative to the material world figures prominently. Francois Delsarte’s “applied aesthetics” is introduced to robotic embodiment. Thus, note can be taken from the sensibilities of “interior design” to extend robotic design beyond the structural and superficial.
The watershed emergence of a field of “aesthetic computing” in 2006 really kicks off the modern journey, with the increase in application of art theory and practice to computing. However, the field of “computational aesthetics” - considered a subfield of artificial intelligence, concerned with the computational assessment of beauty - can be traced back as far as 1928 to mathematician George David Birkhoff’s proposed “aesthetic measure”.
Across time, we can see that machine learning is gradually taking over, leading to what may be termed a “neural aesthetic”. This can be seen in numerous recent “artistic hacks”, such as Deepdream, NeuralTalk, and Stylenet. Couching advanced, or deep, machine learning technologies in artistic metaphor helps clarify otherwise obscure jargon, and makes the subject more accessible to people in the real world.
