Generative Ai In The Development & Validation Of Advertising Materials - Chatgpt & Co In Practical Testing

We've been using ChatGPT for a while now. We use it to help with desk research. It helps us find sources and existing studies on topics that matter to us. We also use it to shape our study designs and build the structure for our questionnaires. ChatGPT helps us write and fix syntax commands too. We use these commands for descriptive data analysis. But we don’t only rely on ChatGPT. Alongside it, we also use Copilot and Claude from Anthropic. Each tool supports different parts of our work.

We wanted to find out if generative AI tools could also help us create and test advertising materials. Our main goal was to make advertising that fits specific target groups. We also wanted to spot chances for improvement early in the process. This would allow us to discuss these improvements with the agency and the campaign management team before the ads go live.

That is why, over the past few months, we have been testing how useful large language models can be in advertising effectiveness. We tried different ways to see how AI tools might support us in both creating ads and checking how well they work.

Using Language Models to Improve Advertising Research

We found that ChatGPT, in particular, was very helpful and reliable. But it is important to remember that humans still need to be involved. This is called “human-in-the-loop.” It means that people guide the AI and also check the results afterward.

First, market researchers need to give very clear instructions to ChatGPT. Second, we must check what the AI says using traditional market research methods. Our tests showed that AI tools have both strong points and weak points.

For example, the results can change from one day to the next. Not every tool works the same way every time. Some tools do not always give good results. That’s why we must look at everything the AI tells us in a careful and thoughtful way.

AI models like ChatGPT are trained on large sets of data. This training data includes many different types of content from the internet and other sources. But the data also has some bias. Bias means that the data may lean toward common or popular opinions.

This means that these tools often show the views of the majority. Because of that, their answers may not always be useful when we are working with very specific or new ideas. Especially when we are focusing on a certain group of people.

Still, AI can show us new ideas that we may not have thought of. It can also confirm things we already believed. But in the end, experts who understand market research need to make the final decisions. Now, we will explain some examples of how we used these tools. These examples are from tests done by our Research & Insights team. We will show how large language models helped in these cases.

1. Ad Pretesting – Fast Feedback and Finding Ways to Improve

We used ChatGPT and other large language models to test ads. We also used tools like Claude, Copilot, and Gemini. We uploaded the ads and asked the tools to analyze them. We looked at how the tools judged the ads based on several different things.

We learned one simple rule: the more clearly we describe the ad and the goal, the better the AI works. If we tell the AI what the ad is about and who it is for, it can give better answers. For example, if we tell it that the ad is for Gen Z, it will look at the ad from that group’s point of view. Gen Z people have different needs than older people.

In our tests, ChatGPT 4o gave the best results. One reason is that it allows us to upload more than one image at the same time. The other tools we tested only allow one image. This makes it easier and faster to get feedback on a full campaign with ChatGPT.

ChatGPT was also very good at understanding humor, hidden meanings, and irony in ads. It can look at what the ad is saying directly and also what it is saying indirectly. It gave us detailed feedback on the ad's strong points and weak spots. It told us how the ad fits with the target group. It also gave ideas on how to improve the ad.

This shows us that AI can add value. It does not replace traditional research, but it works well alongside it. We tested ChatGPT using ads that had already been pretested in the normal way. The AI was very quick in summarizing the content. It also pointed out things we could improve, such as the message, the visual design, and the call to action.

It even noticed if the ad was not diverse enough. Then it gave tips on how to make the ad better for different groups. This type of early feedback is very useful. It helps us improve the ad before we spend more time and money.

Another great feature of ChatGPT is its connection with the DALL·E image generator. This lets us create new ad versions very quickly. We can see how small changes might look and test them right away.

To see if ChatGPT can give good results even with complex messages, we also tested ads from other companies. These ads had hidden meanings, humor, or references to news events. One ad was from a rental car company. It showed the leader of the train drivers' union. The ad referred to a rail strike and hinted that people should rent a car instead. ChatGPT understood who the person was. It also got the joke and understood the deeper meaning.

Claude, another language model, did not understand the joke. It could not link the person to the strike. So, it missed the main message of the ad. Another test ad was from a toy company. It showed two stuffed animals copying the famous scene from the movie “Titanic.” ChatGPT understood the reference. It explained that the scene was based on that film.

Claude, however, misunderstood the whole point. It said that the message was that you could also buy DVDs along with toys. It did not get the joke or the cultural reference. These tests showed us that AI has a lot of potential in advertising research. It helps find ways to make ads better. It does this faster than traditional methods. But we still need to test the ad with real people. That’s how we find out how people feel when they see the ad.

AI gives us a clear and logical view. But human testing shows the emotional side. We need both for good results. Also, remember that ChatGPT only knows things that were available when it was trained. It does not know what happened recently unless we tell it. That means we need to include new trends and events in our prompts.

Still, the tool can help us spot strong points and weak areas in an ad. This gives us a good starting point for improving the ad. At the same time, not all AI tools give the same results. That is why we must always look at the output carefully. Market researchers must use their skills to understand and explain the AI’s suggestions. When people and AI work together, we can do stronger, more useful ad research.

The classic market research is not replaced by AI. Instead, AI adds meaningful support to it. 

2. A/B Testing with Language Models – Useful But Not Perfect

Next, we wanted to find out if AI can help us choose the best ad from a group of good options. Often, we have two or more ads that could work. We want to know which one has the best chance to succeed with the audience.

Normally, we use an A/B test. We show different ads to an online panel and see which one works best. This gives us data and helps us decide. We used this same idea to test how well ChatGPT could help.

First, we did a normal A/B test. One ad came out as the clear winner. Then, we gave the same ads to ChatGPT and asked it to pick a winner. At first, ChatGPT was unsure. It didn’t want to pick one ad over the other.

So, we changed the prompt. We told it to choose a clear winner. After that, ChatGPT chose the same ad that had won in our real A/B test. ChatGPT also told us that a traditional A/B test is still important. This is because we need real numbers like click-through rate, recall, and likeability to fully understand how well an ad works.

One key factor in getting good results with AI is giving a clear definition of the target audience. Without this, ChatGPT had trouble choosing. But when we gave it the audience details, it gave useful insights.

For example, ChatGPT said: “Ad A speaks better to younger people because of its modern design,” and “Ad B fits older people better because it focuses on comfort and safety.”

To see if ChatGPT gives stable results, we repeated the same test 25 times. We used the same ads and the same instructions each time. ChatGPT picked the same winner every time. It gave similar reasons in each test. That shows the AI is consistent.

We also asked other tools, like Claude and Copilot, to check how similar the results were. We asked them to rate the similarity from 1 (exactly the same) to 10 (very different). Copilot gave a 2, saying the results were almost the same. ChatGPT gave itself a 3. Claude also gave a 3. This confirmed that the tool gave stable answers.

This was a good sign. But we found a problem in another test. We did an A/B test with two ads that only changed one thing — the gender of the person shown. In one ad, it was a man. In the other, it was a woman.

In this case, ChatGPT gave mixed answers. It often said the reason for picking one ad was based on gender roles or how men or women might see the ad. This shows that AI might have bias. That is something we must be careful about.

So, even though A/B testing with AI can give useful results, we must check everything closely. We should always test the ads with real people to be sure. Still, using AI this way gives us a good base of information. It helps us plan better and make stronger ads.

3. Expanded Uses with Multi-Format Support

We also looked into how these AI tools work with different types of media. This means using them not just for image ads, but also for other formats like video.

Among all the large language models we tested, only ChatGPT can analyze video formats using YouTube links. This is a unique feature that helps a lot. When we give ChatGPT a link to a video, it can quickly list the main messages of the video. It also identifies the intended audience. It points out what works well and what doesn't. Then it offers ideas for improving the video.

This is helpful not only for checking our own TV ads, but also for analyzing ads made by our competitors. With this, we can make better choices early in the campaign planning. We believe future versions of ChatGPT may also include tools to create videos. That would be another helpful feature.

However, one thing ChatGPT cannot do right now is analyze audio files. For example, it cannot listen to a radio ad and give feedback. This is a current limitation. So, while the tool works well with text, images, and video links, it still cannot help us with sound-only formats.

4. Finding the Key Areas of Attention in an Ad

We also wanted to know how well ChatGPT could help us understand which parts of an ad catch people’s attention first. The layout of an ad is very important. Some parts draw attention immediately. Others may get ignored.

To test this, we used what are called AOIs – "Areas of Interest." These are parts of an ad like the logo, headline, subtext, or a person’s face. We marked and numbered these areas. Then we asked ChatGPT to predict how people might look at them.

We know that ChatGPT is a text-based model. It cannot see images like a human does. So, it cannot do actual visual analysis. But it does know general rules from advertising psychology. Based on this knowledge, it can guess which parts of the ad people are likely to notice first.

The results were useful. ChatGPT identified different AOIs and gave good reasons for each. For example, it said people might look at the headline first because it has big letters. Then they might notice the face, followed by the subtext.

These kinds of insights help us improve our ad designs. We can make changes so that important elements get more attention. ChatGPT also suggested ways to make certain parts stand out more. It recommended changes to things like font size, colors, or contrast. This helps make sure viewers see the most important parts of the ad.

5. Synthetic Heatmaps That Highlight Attention Areas

One more feature of ChatGPT is that it can create synthetic heatmaps. These are visual tools that show which parts of an ad might attract the most attention. The heatmaps use color to show this: warm colors (like red or orange) mean high attention, and cool colors (like blue) mean less attention.

These heatmaps are based on what psychology tells us about how people look at things. They don’t come from actual eye-tracking studies. That means they don’t measure real emotions or behavior. They also don’t show how each individual might react differently.

But even though they are not exact, these synthetic heatmaps are still useful. They give us early clues about which parts of an ad might work well and which parts might be ignored. They help us think about how to improve the structure and design of the ad. We can use this tool as one more way to check the ad, along with other methods.

Still, we should always remember the limits. These heatmaps give only a rough idea. They don’t replace real testing with real people. But they help start the conversation about what to improve.

Please keep in mind:

bias, data freshness, and data protection

AI tools like ChatGPT can give helpful insights for advertising research. But there are a few important things you should watch out for.

AI bias: ChatGPT often gives answers based on general or average opinions. So, less common views might not show up unless you ask for them directly. This can lead to missing or unbalanced insights, especially for certain groups. That’s why it’s important to review the results carefully. If needed, try using more specific prompts to explore different perspectives.

Data freshness: ChatGPT 4o only knows data up to October 2023. This means newer trends, updates, or recent events might be missing. If you want current insights, make sure to include those updates in your prompt. You’ll need to guide the tool with the latest context.

Data protection: Protecting data is very important when using AI. Never use sensitive or private company data in open or insecure systems. If you need to handle such data, always use secure servers and infrastructure—ideally within the EU.

Conclusion:

Potential of AI Systems in Advertising Research

AI systems like ChatGPT also have a lot of potential in the field of advertising research. These systems can help researchers by giving a good starting point. They don’t replace traditional methods, but they offer early support. This support can help guide the next steps in research using older, trusted ways like surveys or focus groups.

When someone uses an AI system in the beginning, it can quickly show early patterns or clues. These early results can point out where ads may need better targeting. They may also show if the design of the ad could be improved. This helps researchers focus on what really matters when testing ads later.

Even though AI can give useful ideas fast, it’s still very important to check the results carefully. People need to look at what the AI says and ask, “Does this make sense?” AI can help a lot, but it should not be trusted without human thinking and review.

AI is not meant to replace traditional market research. Instead, it adds something valuable to it. When we use both AI and traditional research together, the results can be much stronger. The AI gives quick ideas, and the human researchers use their skills to check, test, and build on those ideas.

If used the right way, AI systems can support advertising research in a smart and helpful manner. They can give fast feedback, help with early decisions, and offer ways to make ads better. When human experts work alongside AI, this teamwork creates a solid and reliable base. It can lead to advertising campaigns that are both clear and focused on the right people.