161: How do we evaluate creativity? Part 1

Initiate results from the survey; a Runway character animation test

There are so many wonderful ideas out there.

How and why do some break the surface and face the light of day, while others lurk in wastebaskets and on hard drives? How do creative and marketing organizations decide, “that one!?” Today’s post begins a series looking into the inner workings of creative evaluation.

But first, and thanks for the tip, Joe…

Sound editing is everything.

Kudos to BBDO and the Snicker’s client team for A) establishing the value of an insightful tagline (or end line if you’re British); then B) evolving a body of work over decades which illuminates that concept so well, you don’t even need to say it here—yet it’s said. Nicely done by director Pete Marquis with film production company Good Behavior, editing by Pamela Petruski for Mackcut, and the all-important audio engineering by Tom Jucarone, with Dana Villereal, producer, at Sound Lounge.

The business of ideas is ancient, diverse and hugely subjective. But somehow decisions have been made and all manner of headlines, visual solutions, graphic design, user experiences, casting, production, editing, coding and utility have been approved. See the ad above. You have to imagine the two creative directors, Sara Carr and Jason Goldberg, had an inkling of a bar, some sense of criteria which worldwide chief creative officer Chris Beresford-Hill, and executive creative directors Peter Kain and Marcelo Nogueira—never mind the marketing clients at Mars—had articulated.

But is that the same criteria for all clients at BBDO? Or another agency? Another marketer? How might someone new to the business of creativity discern the why and how of evaluation?

How do we evaluate creativity? Part 1

You’d think this would be easy.

And yet, when I tried to answer this question for a student, I found myself scrambling for unified conviction. How do the diversity of agencies, let alone clients, evaluate and approve creativity? Is there one standard for evaluating advertising headlines? Or visual solutions? What about how we evaluate the boldness never mind utility of a fresh user experience? How should someone on the cusp of a creative career prepare to share, present and receive evaluation of their work?

Of course, I am not the first to ask questions like these.

RESEARCH ON CREATIVE EVALUATION

Creativity contains multitudes, which helps explain the range of academic opinions on its evaluation. Some of this research is actionable. “What is the role of expertise in evaluating creative products? Novices and experts do not assess creativity similarly, indicating domain-specific knowledge’s role in judging creativity,” write Baer, Cropley, and Reiter-Palmon in their 2013 research. “Domain-specific knowledge” is something we can work with. Other perspectives are less useful. “Evaluative skill is the ability to accurately judge ideas on creativity (or originality), which is a critical component of creativity,” establish Guo, Lin, Williams, Zeng, and Clark in their March 2023 study. I suppose we could wrestle with the word “accurately.” But the truth appears to be, “While a handful of recent studies suggest that novelty is more important than usefulness to evaluations of creativity, little is known about the contextual and interpersonal factors that affect how people weigh these two components when making an overall creativity judgment,” suggest Cox, Pickering, and Bhattacharya in their 2022 report. Right on! Evaluation, like creativity itself, exists in the fog.

YOU MIGHT HAVE OPINIONS

To try and wrestle with my student’s question, I distributed a survey earlier this fall among a few hundred agency creatives and strategists, as well as account and marketing leads inside brands.

Let’s be clear: I have not captured lightening in a bottle, or solved an ancient riddle. If I’ve accomplished anything I suspect it is a lesson for all—prior to initiating creativity, you might want to have a chat about how it will be evaluated. Simple as that.

PART 1: EVALUATION INSIDE AN AGENCY

The first question illuminates the challenge of clarifying, let alone establishing known knowns.

🎯 Does your current agency's creative leadership have a known method or process for evaluating creativity? In other words, how do writers, designers and art directors at the shop discern what their leaders consider “good” versus “great” creativity? (If you freelance, pick a recent agency you collaborated with.)

A handful of survey responses paints the difficulty but also the value of talking through the process and culture of creative evaluation. The question to hold in mind is: Do you concur, and if so, why?

“There is no process I know of. It’s a gut feeling. The creative ideas that rise to the top are typically ones that are sticky, like a brain burr. Ideas you can't ignore. Ideas that have never been done before. Or that make people say, we HAVE to do that.”

“This is a tough question because it requires that a client or agency actually invest dollars into acquiring reliable results…and in some cases, those can be skewed. It’s usually based on the numbers though. Did it drive impressions or engagement, or did it boost or expand awareness, aided awareness, market share, sales or profits? But you can’t always get at brand preference or even brand loyalty which I believe are key indicators as well.

I think there is also value in evaluating what it does internally for an organization. Does it help to align? Does it motivate the employees or give them reasons to believe. Does it build confidence? Long story short…there are measurable and non-measureable metrics to consider.”

No. It changes with every regime, but they usually use Cannes as the gold standard.

[An in-house CD writes] “We have a series of visual and writing guidelines that combine intangibles like tone and voice with tangibles like brand colors. My rule for the team is, ‘make it compelling to [our clearly defined audiences, we’re a manufacturer in the B2B realm],’ and ask them to think of one of our customers they’ve met when they create the work. I do the same when I evaluate it.”

“Yes, we have a scale that we use to get people's minds wrapped around what ‘good creative’ means with examples and such. Once people have that in their brains we often just refer to ‘the scale’ and where does anyone in the ICR see this creative fitting on the scale.”

“First and foremost I pose the question how does this CREATIVE set us apart? Make us unique? Makes others say ‘I wish I thought of that!’”

From a pragmatic standpoint, this question and these responses remind me of the “Big C and little c” context.

One way a creative organization could better standardize its evaluation process is by defining generalized categories for ideas. Some need to be audacious, and evaluated in that context. Others are simply pragmatic and ought to be reviewed thusly.

Reconsider the plight of any creative at the onset of their next creative task. What inkling, what sense of evaluation criteria can you provide which might speed their journey, or reduce frustration?

You might benefit by having a chat about it.

---

Runway Character Animation Update

Last week I mentioned Runway’s new AI character animation pipeline. This week I gave Act One a go. Here’s my process:

First, I generated two images in Midjourney to serve as the style of character I wanted to animate—one photorealistic, the other 2D illustration. I planned to animate both separately.

Second, I recorded a reference video of myself to supply the text, initial vocal performance, and motion capture of my face for expression.

Then I uploaded the photorealistic image and reference video into Runway’s Gen-3 Alpha to test out Act One.

I’m excited by the visual result—the facial animation here is subtle but not bad! Now, I’d prefer to use a voice that’s not mine and sounds like it was recorded in a studio—so, here’s that first pass of animation with my reference audio.

Then I took that audio into ElevenLabs’ Voice Changer, which uses my text and inflection but converts it to an audio character of my choosing (in this case, “Minerva.”)

And here’s where it helps to have some video editing skill. I need to align the Minerva audio under the character animation video. I did that in Adobe Premiere.

Fun, right?

After this test, I tried animating the 2D illustration example above and ran into a snag—Runway was “unable to detect a face” in the example. 😆 So I ran back to Midjourney and developed an alternative, generated the animation in runway, then paired it with the ElevenLabs voice in Adobe Premiere. Here’s the second test, which I like even more.

Granted, there’s some weird extra “stuff” (hand gestures?) that Runway added at the end of the second clip. But let’s be clear—I invested maybe an hour total to work through all of these examples. If my intent was to convey a direction, to help align a team around a creative approach, these examples would more than fulfill the assignment.

What have you got to lose by trying?

---