How Do You Know It Will Work? Why Clean Data Comes Before Every Good Test

Honestly, we don’t know whether a test panel will perform well against a control panel. We can make informed recommendations based on how similar organizations have responded to the same test. But no one can promise a result with 100% certainty.

That raises a fair question: why should we pay for two versions of a campaign?

It’s a question many nonprofits ask, and they should. The value of a fundraising consultancy isn’t a guarantee. It’s knowing which tests are worth running first, and how to reach a clear answer faster than a team that is understandably too close to the day-to-day to see the 10,000-foot view.

Seth Godin, celebrated author, entrepreneur, podcast host, and marketing expert, makes a similar point1: asking whether a test is sure to work is the wrong question. Better questions are whether the problem is worth solving, whether the cost of the test is reasonable, whether testing could negatively affect the campaign, and whether it’s the most sensible thing to try next.

There is one condition, though. A test is only as trustworthy as the data used to measure it.

As an analyst, I treat every client’s data as if it were my own and ask, “What would I want to see here?”

First, I want clean, well-organized fields that make sense, no duplicates, and detailed gift information. I know that can be a big ask. Many nonprofits rely on volunteers for gift entry, and automated tools like scanline processing aren’t always in the budget.

Fortunately, we have tools and processes that clean data for mass upload and show where data entry could improve. Here are five ways to assess your own CRM, without increasing costs. They’ll help you decide whether you need trained professionals or just a few adjustments.

1. Check for changed donor IDs.

Pull a report of records where the donor, constituent, or member ID has changed. If there are more than a handful that have changed, someone may be reassigning IDs based on internal criteria. That makes it hard to track who received which segment, and it distorts each donor’s giving history.

2. Look for unlinked households.

Pull a report of records with similar names and addresses that aren’t grouped as a household. This is common and isn’t always a problem, but it can cause two issues. During a test, one household member may receive the control while another receives the test version, which confuses the ask and muddies the results. Volunteers may also enter a gift on the wrong record at that address, which skews segmentation and lifetime value.

3. Confirm gifts carry appeal or source codes.

Pull a report of the past 12 months of gifts with blank or generic appeal codes. A gift that can’t be tied to the mailing that generated it can’t be credited to that mailing. When that happens often enough, a winning package can look like a loser, and you may cut a campaign that was actually working.

4. Compare gift dates to entry dates.

If gifts are dated the day they were keyed in rather than the day they were received, or if entry runs weeks behind, response timing and campaign-to-date reporting will be off. Decisions made mid-campaign will rest on an incomplete picture.

5. Review address quality.

Count active records flagged for returned mail and records whose address hasn’t been updated in several years. Undeliverable mail costs money, and it quietly shrinks the audience your test results are based on.

 

What to do with what you find 

If the numbers of records you come across in your assessment of the five points mentioned are small, a few adjustments may be enough: written data entry guidelines, a short refresher for staff and volunteers, and a regular duplicate review. If the counts are large, that’s a sign to get help before investing in new tests, because otherwise you won’t be able to trust the results, good or bad.

Not sure where your data stands? 

If you’ve run these reports and aren’t sure what the numbers mean, we’d be glad to help. Request our free Data Health Worksheet, fill in your counts (no donor files needed), and connect with us if you’d like to discuss your results and how we can help. Together we’ll look at where your data is today, what it would take to make it test-ready, and which fixes will pay off first. You’ll leave with a one-page summary you can share with your team.

You can’t predict the outcome of a test. But with clean data, you can trust what the results tell you.

Need more help with understanding your data inside and out? Contact us to learn how we can help.