Seattle Zestimates are off by $40,000; now hundreds of data crunchers vie to improve Zillow’s model

If you have a house or you’ve looked for one, you’ve probably checked its Zestimate – Zillow’s best guess as what a house is worth today.

The number might help you determine whether you should sell your house, or if the home you’re trying to buy is a rip-off or a bargain. Or you might just use it to gawk at what your friend or neighbor’s home is worth.

But how accurate is that number?

Long a source of complaints, Zillow’s online valuations are far from perfect, the company acknowledges: For instance, according to its own data, the Zestimate on the typical single-family house sold in Seattle today is off by an average of nearly $40,000.

More than 200 teams have started working on new home value algorithms in the 48 hours since the Seattle-based company launched a new contest to improve its Zestimates model – with a top prize of $1 million.

To win, teams will use Zillow data to create an algorithm that better predicts sales prices for about 110 million U.S. homes listed on the site. Each team’s members and their scores are publicly available on the Kaggle platform, which is hosting the contest: The leading teams’ models have improved slightly just from making tweaks over a 24-hour period, but still lag behind Zillow’s predictions.

One hundred finalists will be picked next January, and the winners will be awarded in January 2019.

Zillow says its Zestimates have improved significantly since launching 11 years ago and now have a median error rate of 5.6 percent. So half of homes end up selling within 5 to 6 percent of what the Zestimate said it was worth.

But that means half are sold for a price that’s not so close to what the Zestimate predicted. In some cases, the error can be huge.

Most notably, and most embarrassingly for the company, Zillow CEO Spencer Rascoff sold his Seattle home last year for $1.05 million – 40 percent less than what the Zestimate said it was worth. Then a couple months later, Rascoff bought a house in Los Angeles for $20 million – or $1.6 million more than the Zestimate called for.

The error rate varies by location, but even the average error can amount to a lot of money in a pricey market. The median single-family house in Seattle now goes for $722,000, and the local Zestimate error rate here is 5.4 percent. That means the Zestimate on a Seattle house selling today will be off by an average of about $39,000.

Across the wider Seattle metro area, Zillow says more than one-fourth of homes have a Zestimate that’s off by more than 10 percent. For 1 in every 13 homes, the Zestimate is wrong by more than 20 percent.

“That’s really not that accurate, especially in this market,” said Mark Corocoran, a local broker for Windermere.

He says about 90 percent of his clients will quote the Zestimate when they go to set the listing price, though he doesn’t take it into consideration. “It is something that everybody is looking at. But they don’t go into the homes. I just don’t trust it.”

Zillow acknowledges that real estate agents are still slightly better at predicting sales prices than its computers, but says the gap between man and machine is narrowing.

Data from Redfin shows that the listing price put together by agents is usually only about 1 percent different than the final sales price in the Seattle region, though in the last couple of months, homes have sold for an average of 4 percent above the list price. (There is a theory, however, that some agents deliberately list homes for less than they think it’s worth to draw in more buyers and drive up competition).

Zillow says it has the best online home value estimations, though other companies with similar tools, like Seattle-based Redfin, disagree.

Zillow chief analytics officer Stan Humphries, who created the Zestimate, said its team of 15 data scientists and machine learning engineers have been able to drop the home value predictor’s error rate significantly, from 14 percent when it launched in 2006.

They now run about 7.5 million statistical models nightly to update the Zestimates, using things like comparable home sales, local assessed values, home size and features, and hundreds of other data points. Lately, they’ve used machine learning to analyze home pictures from listings to determine how fancy various parts of the home are, taking the analysis beyond data like size and listing descriptions.

Humphries says the way to make the Zestimate better now is by using as much hyper-local data as possible, which lends itself better to a nationwide contest than to further research and tweaks from its Seattle-based team.

“At this point we think that future gains will come from widening the circle of ideas to the global data scientist community,” Humphries said.

“I believe we’ll be able to drive (the error rate) lower, I think a lot lower,” he said. “Neither a human nor a computer will get to zero. People will disagree with human and computer opinions, and that’s OK. But we think more opinions make consumers more comfortable.”

Although Zillow was the first to set up an online home estimate tool, in 2006, nowadays the Zestimate is just one of many ways to get a sense of your home value. Sites like Redfin and homes.com have their own algorithms, while local banks and real estate agencies can give you quotes almost instantly.

It’s also easy now to look up actual home sales data from comparable properties online – something that, just a decade ago, was information kept only by real estate professionals.

But the Zestimate is probably still the most-used computer estimate out there. It’s the first thing that shows up when you Google “How much is my home worth,” and Zillow is the most trafficked real estate information site in the United States, with 166 million average monthly users.

The algorithm has also been the subject of lawsuits. Just last week, the company was sued by suburban Chicago home builders who argue the numbers are being illegally passed off as an official appraisal, which are typically done by licensed professionals who visit homes up for sale. Zillow says the home value should be used as a starting point – just one piece of data to consider – and not an official appraisal.