People Prefer AI-Generated to Human Content, MIT Study Finds

Pundits who confidently state that generative artificial intelligence (AI) like ChatGPT can never replace human writers now have to rethink that position. New research at MIT shows human readers preferred AI-generated content to that created by professional human writers.

The researchers tasked both human experts and ChatGPT 4 with creating two kinds of content: advertising product descriptions and persuasive content for ad campaigns. They evaluated four variations:

Human-only content
AI-only content
Human content edited by AI (“Augmented AI”)
AI content edited by humans (“Augmented Human”)

Shockingly, perhaps to some, the AI content – either written entirely by AI or with a final edit by AI – was preferred by readers.

A second key finding is that disclosing that content was AI-written did not change the preference for the AI content.

AI: The Great Persuader

Marketers and leaders in general need to change the behavior of customers, employees, and others. We think of being persuasive as a uniquely human skill, but this research shows that AI can do better than humans within a tight set of constraints.

The researchers used this initial instruction to the human experts for the persuasive content task:

“Your task is to write persuasive content for a campaign in fewer than 100 words. Your goal is to persuade people to change their behavior after seeing your content.”

Sounds like the kind of task marketers deal with every day, right? The second instruction offered more specifics:

“Please create persuasive content for a client (e.g., an NGO doing a campaign) to convince people to [perform the action advocated by the campaign] in fewer than 100 words.”

The prompts to ChatGPT were “nearly identical.”

The human subjects who read the content were asked, “To what extent you are convinced by the above content to [perform the action advocated by the campaign] on a scale from 1 to 7”? (1 represented “not convinced at all” and 7 meant “being very convinced.”)

The product description task included a question about “willingness to pay.”

Across the tasks, the quality of the AI-created content was judged to be better.

Do People Care If AI Wrote The Content?

In this study, the researchers also tested perception of content with its AI or human origin disclosed. They found the subjects did not reduce their evaluation of the AI content when they were told it had been AI-generated.

Content written by humans got a small boost when the subjects were told of its origin.

In this limited task, at least, the AI content scored well whether or not the subjects knew it was AI-generated. No aversion to AI or bias against AI was found.

The Key Takeaway

The important takeaway from this study isn’t that AI-created content will be better than what humans create. The human subjects created their content in a vacuum – no discussion, no editorial input, no rumination and revision, etc.

The most important finding for marketers is that AI can produce a result at least as good as the average professional human writer when both are assigned a simple, well-defined persuasive writing task like the ones in this study.

As digital marketing expert Chris Penn notes, these findings mean that for content creators, “the use of AI isn’t going to harm your marketing.” Penn also thinks fears of AI creating a “sea of garbage” content are overblown. (Perhaps it won’t be garbage, but I think we’re already seeing the start of a tsunami of mediocre content aimed at boosting search traffic.)

Should Marketers Turn Content Creation Over To ChatGPT?

Writing 100 words of copy is a fairly restricted task. Creating and evaluating campaign concepts and fleshing out taglines, graphics, etc. is a much bigger challenge. And, rather than one person writing the text for a major ad campaign there will likely be many people involved in crafting and fine-tuning the messaging.

It’s doubtful that current AI can create novel and highly persuasive campaigns like those crafted by, say, ad legend David Ogilvy. (Of course, many human agencies might find that difficult, too.)

Companies shouldn’t fire their marketing writers, at least not yet.

Where Can Marketers Apply AI Today?

Most marketing writing is fairly mundane. Creating product descriptions for websites and catalogs. Writing a weekly newsletter. Creating topical article content that benefits customers and improves search engine traffic. Unlike a major new ad campaign, most of this content is written and published with minimal discussion.

Producing this mundane content is where AI can be a force multiplier for human writers, saving them time and money.

Human involvement is still necessary. AI must be prompted properly if optimal results are desired. Of course, prompts can be reused once they are found to be effective. Automated workflows can be created. Over time, the front-end work for humans can be reduced.

Equally important, AI-created content still needs checking. Hallucinations, i.e., invented facts and nonexistent citations, remain a problem for all the models. The models also have difficulty understanding concepts – ask ChatGPT a question about an article or study, and it may offer an answer that sounds plausible but misinterprets the original.

Human writers and editors can save time by prompting the AI they use to check its work, or by giving the content to a different AI to check. Future AI models will no doubt get better at providing results free from hallucinations and conceptual mistakes.

Today, marketers using AI need fewer basic content writers to create the same output. The oft-used maxim about AI-driven job loss applies here: writers won’t lose their jobs to AI. Rather, they’ll lose them to writers effectively leveraging AI to write better content faster.

(This article was entirely human-written. By me. Clearly, I’m not following my own advice, at least not yet.)