Courts don’t take survey evidence on faith alone. They want facts and proof. Wondering how courts evaluate the reliability of consumer surveys? Basically, judges keep watch under Rule 702 and Daubert. A judge will ask whether the survey rests on reliable methods and actually helps the fact-finder decide the legal issue (likelihood of confusion, secondary meaning, dilution, or deception). In all situations, the standard applies to any expert testimony and not just “hard science.”

The Legal Yardsticks Judges Use

Judges use two important yardsticks to measure reliability in consumer surveys.

• Federal Rule of Evidence 702 requires that an expert’s opinions be firmly grounded in sufficient facts or data. The opinion must also have been produced by reliable principles and methods. In all situations, reliability is applied to the facts of the case.
• Daubert (and Kumho Tire) confirm the court’s “gatekeeping” role by excluding opinions that lack a trustworthy foundation or don’t fit the dispute. Basically, there is no forcing a square peg into a round hole. Everything is built on a solid foundation and must fit the situation.

Recent commentary on the 2023 updates to Rule 702 drives home the importance of the fact that the proponent must show reliability by a preponderance of the evidence. If there is no reliability, then the survey falls short, sometimes very short.

What “Reliability” Looks Like in Survey Work

Judges look for the basics when examining reliability in survey work:

• The right universe which are people who actually buy or influence the purchase.
• Representative sampling
• Randomized exposure
• Appropriate controls
• Neutral questions that mirror real-world shopping.

In addition, judges also examine whether the design fits the legal question.

Examples would be as follows: measuring net confusion for a likelihood of confusion claim, or testing the association for secondary meaning.

The Federal Judicial Center’s Reference Guide on Survey Research is a frequent touchstone for these criteria and is used to judge when evaluating the reliability of consumer surveys.

Format Choice and the Need to Customize

Courts recognize accepted formats like Eveready, which is often used when the senior mark is well known, and Squirt/lineup, which can prove useful when consumers encounter brands side-by-side.

However, those survey formats serve only as starting points, not as a one-size-fits-all solution.

Instead, judges go a step further by asking whether the format matches how buyers actually meet the marks, such as on a shelf, a product page, or an app. As likely happens in virtually all cases, when the marketplace reality differs, a custom design is often the most reliable choice and prudent choice, which is where Keegan & Donato can help with their custom surveys.

How Surveys Get Into Trouble

Courts discount or exclude surveys for what are referred to as classic flaws. These include the following: the wrong universe, leading questions, missing or weak controls, unrealistic stimuli, or a format that doesn’t reflect the buying context.

• In Universal City Studios v. Nintendo, the court rejected a survey that used a skewed audience and a leading question to tie “Donkey Kong” to “King Kong.”
• In Kargo Global v. Advance Magazine, a Southern District of New York opinion criticized stimulus choices and methodological gaps.

These cases illustrate what judges won’t accept, and why method matters.

What a Court Wants to See in Your Report

Beyond design, judges expect transparent documentation:

• Who was surveyed and why
• Sampling and incidence
• Questionnaires, stimuli
• Fieldwork controls
• Exclusion criteria (e.g., speeders/straight-liners), and all analyses needed to compute net results.

They also look for a clear tie-back between each design choice and the legal theory, so the court can follow the logic step by step.

The FJC guide emphasizes purpose-built surveys that answer the relevant legal question directly.

Why Customization Can Strengthen Admissibility

Because reliability hinges on fit, a tailored approach often fares better than a template. If consumers shop primarily on mobile PDPs, a lineup on a white background may miss the mark if awareness is low. In such a situation, Eveready’s setup may not capture the likelihood of confusion realistically.

Courts reward designs that recreate the buying moment and measure the correct outcome.

That’s why Keegan & Donato frequently customize surveys to the facts by building courtroom-ready instruments that reflect the marketplace you’re actually litigating.

How do courts evaluate the credibility of consumer surveys? They look for proven methods, a tight fit to the legal issue, and a design that mirrors the marketplace. When results must hold up under scrutiny, choose proven survey experts. Get started with Keegan & Donato Consulting today by contacting us online or calling (914) 967-9421.