Exploring Art and AI in Georgia Tech's School of Literature, Media, and Communication

Painting may seem old school, but it’s a profoundly technological practice. From the tech behind brushes to the mathematics of perspective, artists of all kinds have long placed themselves where science, technology, and art meet.

But it’s never been more challenging to pinpoint where a relentless ocean of technology splashes over the ever-shifting sands of artistic expression amid the rise in artificial intelligence tools such as DALL-E and ChatGPT capable of generating images, text, and even music from relatively simple text prompts.

This landscape has proven fertile ground for School of Literature, Media, and Communication (LMC) resident painter Mark Leibert, who for years has turned to algorithms and computational methods to help inspire and refine his work.

Leibert and collaborators from the Georgia Tech Research Institute (GTRI) have gone even further, recruiting a team of student researchers in the Art & AI Vertically Integrated Project (VIP) to explore the artistic implications of AI and the technological implications of art. In fact, they were well ahead of the recent public buzz over artificial intelligence tools, having launched four years ago — before DALL-E and ChatGPT became household names.

“We have a really unique situation here where we can think about how these techniques can be used for artistic means and how to use those creative neurons to approach some of these new technologies,” said Leibert, an LMC professor of the practice.

The Art & AI VIP is one of many arts-and-technology efforts housed entirely or partly in LMC, a unit of the Ivan Allen College of Liberal Arts. The most recent initiative is Interactive Media Arts, a partnership between the Ivan Allen College and the Georgia Tech Library, that features many LMC faculty.

Other projects include Professor Brian Magerko’s widely acclaimed EarSketch coding-through-music platform; work examining the shared DNA behind quilting and computing spearheaded by Anne Sullivan, an LMC assistant professor whose doctorate is in computer science, and Professor Janet Murray’s continued exploration of the impact of digital technology on storytelling. The School also recently added a new artist-in- residency program, in collaboration with the GT library, to further examine the nexus of art and tech. It also has invited musicians to a residency program with similar aims, that one in conjunction with the School of Music.

“LMC has always been about looking at how technology is shaping literature, art, and storytelling in general,” said Kelly Ritter, chair of the School. “The Art and AI VIP is one way we’re doing this, but whether it be a new center dedicated to exploring the nexus between media arts and research, the extraordinary work of faculty such as Brian Magerko and his team, who are helping teach young students to code, or how technology and its impact is suffused throughout our writing and communication program, this sense of interplay between technology, textual study, and media arts is central to what we do.” 

DALL-E’s AI-generated take on Leibert leading a class.

The VIP draws students from across campus, many interested in exploring the creative potential of technological degrees or in scratching a creative itch they aren’t always able to satisfy amid a heavy courseload in computing or engineering.

“Tech has great resources for technology and amazing resources for literature, media, and communication. This seemed like one of the few bastions in which the two were really intermingling,” said Jude McClaren, a second-year computational media student.

Leibert formed the VIP team in 2020, initially partnering with now-retired GTRI researcher Betty Whitaker. He’s now working with Ethan Trewhitt, another GTRI researcher, who helps students dig deeper into the sometimes mystifying ways in which AI and artistic intent interact.

“AI systems, in some cases, are solutions looking for a problem, and giving them new outlets helps you recognize where the limitations of those systems are,” said Trewhitt, a senior research engineer at GTRI. “Then the creative motivations of the people using these systems can create new ways to solve problems that can then be used toward what you would consider more practical applications that aren’t necessarily art. So it’s kind of a feedback loop.” 

In its four years, students in the Art and AI VIP have explored the intersection of the two fields through various projects. In recent years, students have tried using AI to write horror, create moving portraits, animate children’s books, explore new takes on old music, creative interactive, emotion-sensing art, and tools to visualize poetry and music. 

In the Spring 2023 semester, an interdisciplinary group of students with experience in coding, illustration, 3D-modeling, computer animation, website design, marketing, and AI systems worked on creating a human-like extension to the ChatGPT AI tool.

You can see a gallery of some of the recent work they have done in a database maintained by the Art & AI VIP.

Their techniques include generative adversarial networks, diffusion models, transformer models, and natural language processing models, as seen in publicly available tools such as DALL-E, EbSynth, and ChatGPT.

“I would say the bulk of the students are working in the deep-learning space, which tends to be a little more black-box kind of functionality,” Trewhitt said. “The fun thing about this is there really isn’t such a thing as failure, so it’s safe to really play with things that can surprise you.”

While Trewhitt is still working through ways to dive deep into the technology behind such systems in a way that works for a team with a range of backgrounds, some more technical than others, some teams have surpassed his knowledge in some areas.

“We had students working on a music video project recently, and by the end of the semester, they definitely understood it better than I did. I had dabbled in it, but they had spent hours really getting it working the way they wanted, peeling back the layers of a complex system to get it to do what they wanted it to do.”

Michelle Lee, an Art & AI VIP member and second-year computer science student from New York, is working on a music visualization program. She said the VIP has allowed her to use her computing knowledge in a creative way that she hopes will help after graduation.

“I hope to bring creativity wherever I go,” Lee said. “I hope to work at a tech company, and I’d really like to bring this kind of creativity to make my work different and unique.” 

Tools such as DALL-E and ChatGPT have made headlines in recent months. The coverage ranges from astonishment at the tools’ ability to render true-to-form, if sometimes absurd, images or text to outright alarm at the potential the tools have to deceive or perhaps even put artists and other creatives out of work.

It’s a topic of discussion in class, of course. Leibert, Trewhitt, and the students have discussed how emergent technologies, such as the AI tools used in class, can scramble methodologies, disciplines, and careers. They’ve also discussed how working with art creates opportunities to be nimble, reframe conventional thinking, and surface novel ways to solve problems. Leibert says that can, in turn, lead to the creation of new tools or even new careers. Case in point: the rising demand for people skilled in “prompt engineering” — in other words, able to coax quality results from a tool that can sometimes be mysterious and even erratic.

That kind of work is something creative people have done for a long time, Leibert said.

“Artists and writers have worked with generative prompts and games for much of the last 100 years,” he said. “Sometimes it has been the flip of a coin, consulting I Ching, or selecting from a deck of prompt cards. Tools like Photoshop and Illustrator have been a part of my process for at least two decades, and working with this group has increased my appreciation for the work computer scientists do.”

Ethical issues also come up, of course, such as who is doing the creating — the human, the machine, or some mixture of the two — and who can claim ownership over the resulting works.

“This is a very new and fast-moving field, so we spend a lot of time talking about the technologies and the creative and conceptual potential of AI in art. But the nice thing about what we are doing is that we are able to have these conversations about ethics in an informal manner as we work, so this kind of ethical thinking becomes as much a part of what we do as the technical inputs or the artistic outputs.”

Students in the VIP are well aware of the ethical issues and committed to using AI for good. And many view the use of AI to make or remix art as natural, just another tool in an artists’ toolkit.

“AI is a legitimate way of making art,” said Marin Hyatt, a third-year computer science student from New York. “I think, like anything else, AI is just a tool, similar to Photoshop. It’s just a different way to make art.”