Verification of Redfin s Claims about Superior Notification Speed Performance for Listed Properties

Transcription

1 Verification of Redfin s Claims about Superior Notification Speed Performance for Listed Properties Prepared for Redfin, a residential real estate company that provides webbased real estate database and brokerage services Prepared by Aniruddha Banerjee, Ph.D., SVP, Advanced Analytics, SSRS Stephen Smith, Senior Analyst, SSRS Yvonne Shands, Account Executive, SSRS June 9, 2017

2 Introduction and Summary Redfin commissioned a study by SSRS to learn about its performance in notifying subscribers about new home listings versus other real estate websites. The websites compared with Redfin.com in this study are Zillow.com, Trulia.com, and Realtor.com. 1,2 When properties for sale are listed, real estate companies (such as the four in this study) alert their subscribers about the availability of those properties along with their status (e.g., New Listing, Price Change, Sale Pending, etc.). The websites send out these alerts in the form of s, and each is associated with a timestamp, i.e., the precise date and time (hour, minute, and second) at which the is received. The timestamps of the s can be used to compare the speed at which each website notifies its subscribers. If two or more of the websites notify about the same property and alert type, then the respective timestamps of their notification s can reveal which of those websites was the fastest to inform its subscribers. SSRS structured this study in two parts. In the first part, SSRS used parsing and reporting procedures to create a database of 184,206 properties for which both Redfin and one or more of the other websites sent out notifications during the period of study (March 23 April 5, 2017). In this database, each property retained for study was matched between Redfin and at least one other website, but many properties were matched between Redfin and all three other websites. In the second part of the study, SSRS used notification timestamps for the properties in the database to measure the relative speed performance of the websites that matched. Each timestamp was first converted into the number of seconds since January 1, 1960, at 00:00:00. 3 The difference between any two timestamps then provided the difference in notification speed. A positive speed difference between websites X and Y was then interpreted as Y being faster than X to notify, while the opposite was implied by a negative speed difference. SSRS computed various summary statistics (medians, robust means, and their associated 95% confidence intervals) and distributions of the speed differences to determine relative speed performance in head-to-head matchups between Redfin and each of the other three websites. At Redfin s request, SSRS also conducted the analysis of relative speed performance for 20 Metropolitan Statistical Areas (MSAs). Summary statistics for notification speeds 1 Homes.com was considered as well but not included because it failed to send any notifications despite being requested to do so. 2 In the rest of this report, the websites are referred to as simply Redfin, Zillow, Trulia, and Realtor. 3 For example, a timestamp of March 23, 2017, 11:10:01 converted to 1,805,886,601 seconds since January 1, 1960, 00:00:00. 2

3 were then computed for Redfin and the other websites within each selected MSA and over the complete sample of matches. Based on the analysis, SSRS finds that Redfin is fastest to notify subscribers about properties for sale among the four websites. While, on occasion, the other websites were faster to notify than Redfin for specific properties, Redfin s performance was measurably superior on average. Using the median of the notification speed difference as the yardstick, both Zillow and Realtor were about three hours slower than Redfin, while Trulia was over 18 hours slower. Furthermore, the distributions of the speed differences revealed that Redfin was faster than Zillow for over 94% of matched properties and faster than Trulia and Realtor for over 99% of matched properties. Taken together, these findings provide overwhelming evidence of Redfin s superior notification speed performance. Data Collection SSRS collected and parsed s to create a sizable database of properties for which both Redfin and one or more of the other websites sent out notifications during the period of study (March 23 April 5, 2017). The following three sections describe the procedures used to create two reports: the Matching Report and the New Listing Report. These sections describe the path taken by any given through the system --- starting on Gmail s servers and ending in the reports. Section A: Gmail Downloads For the Gmail analysis, software was first created to log into Gmail s backend database and download all s from a Gmail account. The software was run until a large enough sample had accumulated across 20 Gmail accounts to suit the requirements of the reports. In all, 95,527 s were downloaded across the four websites and 20 MSAs, as shown in Tables 1 and 2, respectively. 4 Table 1. Distribution of Downloaded Counts, by Website Website Number of s Redfin 51,959 Zillow 38,469 Trulia 3,240 Realtor 1,859 4 An can contain more than one property alert. 3

4 Table 2. Distribution of Downloaded Counts, by MSA MSA Number of s Atlanta 7,140 Bay Area 4,289 Boston 5,999 Chicago 5,706 Cleveland 3,617 Dallas 5,437 Denver 3,932 Detroit 998 Houston 3,211 Los Angeles 4,201 Miami 4,082 Minneapolis 3,121 New York 9,963 Orlando 3,282 Philadelphia 5,482 Phoenix 5,047 Portland 2,694 San Diego 6,344 Seattle 3,193 Washington DC 7,789 Section B: Parsing and Removal of Duplicates At the next step, almost 20,000 s were pulled from the resulting database of Gmail downloads, and these s were analyzed to determine the basic structure of each website, along with the level of internal consistency within each website and across the sample over time. While the s were parsed in chronological order, identified properties were tested to determine if they were duplicates of properties previously identified. Duplicates were disqualified from entry into the parsing table. The duplicate checking process varied among the websites. Generally speaking, a duplicate is a property that has exactly the same website, alert type, address, city, state, and 5-digit Zip code as a previously identified property. For duplicate checking of Zillow, Zip codes were not available and, thus, not used. For Redfin properties, duplicates were removed by using the unique property IDs provided by Redfin. If the property ID was blank, the MLS number was used (only Redfin provided MLS numbers). If, however, the MLS field was also blank, the general address information pattern was used instead. Data cleaning was performed in real time in order to both identify duplicates and match properties for analysis. Normalization of address fragments was employed to further assist in this process. Additional automated data cleaning routines were also performed 4

5 once the parser had completed its work. The parser then ran additional steps to fill in missing information on Zillow s missing Zip codes, normalize county names, correct spelling errors, and normalize common abbreviations. The next step was to remove duplicate alerts. Seventy three duplicate notifications (as determined by website, alert type, and address) were removed manually. Among duplicates, the earliest notifications were retained and later notifications of the same listings were omitted. Section C: Reporting At the final step, two reports were created. The first report ( Listing Report ) listed every unique property (as received from the four websites) that had an alert type equal to New Listing. This report included pertinent property fields. It also contained four columns; one for each website, to indicate which websites had sent an notification about the properties listed. A larger report ( Matching Report ) was also created, in which all matching for Trulia and Realtor properties was performed using SQL (a standard data manipulation language). Zillow required a fourth app to perform matching due to a reclassification of alert types. To perform the matching, a base for comparison was first created. This was done by placing all Redfin properties in a new Performance table. All disqualified Redfin properties were removed from that table. Disqualification occurred for unique Redfin alert types which would never match alert types on any other website. Several passes were made to match the Redfin notifications to the properties notified by other websites. The first pass used the most stringent parameters, attempting to match on a combination of alert type and the following five fields: address, city, state, zip, and county. The majority of properties was matched on this first pass. 5 In Table 3 below, these matches are named County & City (because these two fields were used to provide optional precision). On the second pass, the County parameter was removed, producing matches named City below. On the third pass, the County parameter was restored and the City and State parameters were removed, producing matches named County below. On the final pass, all parameters except for alert type, address, and zip were removed, yielding a small number of remaining matches. Table 3. Number of Properties by Property Match Precision and Website Zillow Match Precision Number of Properties Trulia Match Precision Number of Properties Realtor Match Precision Number of Properties County & City 17,537 County & City 10,006 County & City 24,678 City 475 City City 2,335 County 5,917 County 981 County 1,123 Address/Zip only 49 Address/Zip only 38 Address/Zip only % of Zillow properties, 81% of Trulia properties, and 87% of Realtor properties. 5

6 As the data were matched between Redfin and the other three websites, additional data were added to the Performance table, such as matching precision (as seen above), the URL to the property on a website, alert type (if provided), the Gmail date of the property notification, and pointers to preserve a path back through the parser and farther back into the originating table. Data collection resulted in the compilation of notification timestamps for new listings (as well as for other alert types) by website. These timestamps indicated the date, hour, minute, and second at which notification was received from each website. With the goal of comparing notification speeds of Redfin against the other websites, the notification timestamp data were then further processed in four steps: Convert all timestamps into seconds since January 1, 1960, 00:00:00. 6 Identify all matching notifications (by alert type) between Redfin and each of the other websites (on a pairwise basis) and between Redfin and all three other websites simultaneously (whenever all four websites provided notification on the same listing). Compute the difference in notification speeds as the difference between Redfin s notification time (in seconds) and a matched website s notification time (in seconds). The website with the greater notification time has the slower notification speed. Besides those for overall matches, compute the difference in notification speeds by 20 specific MSAs, including Atlanta, Bay Area, Boston, Chicago, Cleveland, Dallas, Denver, Detroit, Houston, Los Angeles, Miami, Minneapolis, New York, Orlando, Philadelphia, Phoenix, Portland, San Diego, Seattle, and Washington DC. 7 These steps resulted in the numbers of matches with notification times that are shown in Tables 4 and 5. Table 4. Number of Overall Matches (Pairwise and All Four Websites) for Comparison of Notification Times Matched Website Comparison Pairwise Matches All Four Websites Matches Zillow vs. Redfin 23,978 2,537 Trulia vs. Redfin 12,283 2,537 Realtor vs. Redfin 28,232 2,537 6 This was accomplished using the clock function in the Stata 12.1 software package from StataCorp, 4905 Lakeway Drive, College Station, TX This step was conducted for pairwise website matches only. Doing so for all four website matches would have resulted in relatively small numbers of cases to analyze. 6

7 Table 5. Number of MSA-Specific Matches (Pairwise) for Comparison of Notification Times MSA Zillow vs. Redfin Trulia vs. Redfin Realtor vs. Redfin Atlanta 2,126 1, Bay Area 1, ,818 Boston 2, ,418 Chicago 1, ,607 Cleveland ,446 Dallas 1, ,826 Denver ,607 Detroit Houston ,686 Los Angeles ,732 Miami ,658 Minneapolis 1, ,165 New York 2,839 1,335 1,133 Orlando ,035 Philadelphia 1, ,636 Phoenix ,580 Portland ,269 San Diego Seattle Washington DC 2,812 1,683 1,812 Overall (20 MSAs) 23,978 12,283 28,232 Analysis With all necessary data collected to perform the notification speeds analysis, SSRS proceeded to first calculate appropriate summary statistics (means, medians, standard errors, and confidence intervals). In place of ordinary arithmetic averages (means), SSRS computed robust (i.e., outlier-adjusted) means. As noted later in this report, this change was made necessary by the seriously outlier-influenced distributions of the differences in pairwise notification speeds. These statistics were computed for overall (both pairwise and all-four-websites) matches in Table 4 and the MSA-specific (pairwise) matches in Table 5. All summary statistics are reported in Tables Tables 6 and 7 report summary statistics for overall comparisons, while Tables 8-10 report those statistics for MSAspecific comparisons. All statistics for speed are reported in hours. In those comparisons, the convention followed is that the difference in notification speeds is the notification time of the first website listed in the comparison less the notification time of the second website listed in the comparison. 7

8 Table 6. Summary Statistics for Difference in Notification Speeds (in Hours) between Zillow and Redfin, Trulia and Redfin, and Realtor and Redfin (Pairwise Matched) Matched Website Comparison (Pairwise) Zillow vs. Redfin Trulia vs. Redfin Realtor vs. Redfin Median Lower 95% Upper 95% Robust Mean Standard Error (Robust Mean) Lower 95% Upper 95% Figure 1 depicts the three pairwise comparisons between Redfin and one other website, using the median and robust mean differences between notification speeds. All median and robust mean differences were statistically significant at the 5% level of the test. All Other Websites are Slower to Notify than Redfin: Based on the Median, Trulia is Slower by over 18 Hours, Followed by Zillow and Realtor (by About 3 Hours Each) TRULIA - REDFIN ZILLOW - REDFIN REALTOR - REDFIN Median Robust Mean Figure 1. Average Difference in Notification Speeds (in Hours) for Matches among Pairs of Websites that Involve Redfin 8

9 Table 7. Summary Statistics for Difference in Notification Speeds (in Hours) between Zillow and Redfin, Trulia and Redfin, and Realtor and Redfin (All Four Websites Matched) Matched Website Comparison (All Websites) Zillow vs. Redfin Trulia vs. Redfin Realtor vs. Redfin Median Lower 95% Upper 95% Robust Mean Standard Error (Robust Mean) Lower 95% Upper 95% For all listings which appear on all four websites, the average differences are, in fact, even greater for Trulia vs. Redfin and marginally greater for Zillow vs. Redfin. This is shown in Figure 2. All Other Websites are Slower to Notify than Redfin: Based on the Median, Trulia is Slower by Over 20 Hours, Followed by Zillow (Over 3 Hours) and Realtor (Almost 3 Hours) TRULIA - REDFIN ZILLOW - REDFIN REALTOR - REDFIN Median Robust Mean Figure 2. Average Difference in Notification Speeds (in Hours) for Matches among All Four Websites In Tables 8-10, median and robust mean differences in notification speeds (and associated statistics) are shown for 20 MSAs where Redfin operates. All median and robust mean differences are positive and statistically significant at the 5% level. Figures 9

10 3-5 correspond to these tables and show median differences, in descending order of magnitude, by MSA. Table 8. Summary Statistics for Difference in Notification Speeds (in Hours) between Zillow and Redfin, by MSA (Pairwise Matched) MSA Median Lower 95% Upper 95% Robust Mean Standard Error (Robust Mean) Lower 95% Upper 95% Atlanta Bay Area Boston Chicago Cleveland Dallas Denver Detroit Houston Los Angeles Miami Minneapolis New York Orlando Philadelphia Phoenix Portland San Diego Seattle Washington DC

11 19.0 Median Difference in Notification Speeds (Hours), By MSA Zillow Notification Time Minus Redfin Notification Time Figure 3. Median Differences between Zillow and Redfin Notification Speeds (in Hours), by MSA Figure 3 shows that Redfin is over 10 times faster to notify than Zillow in San Diego than in Phoenix, and over seven times faster in Seattle than in Phoenix. Table 9. Summary Statistics for Difference in Notification Speeds (in Hours) between Trulia and Redfin, by MSA (Pairwise Matched) MSA Median Lower 95% Upper 95% Robust Mean Standard Error (Robust Mean) Lower 95% Upper 95% Atlanta Bay Area Boston Chicago Cleveland Dallas Denver Detroit Houston Los Angeles Miami Minneapolis

12 New York Orlando Philadelphia Phoenix Portland San Diego Seattle Washington DC Median Difference in Notification Speeds (Hours), By MSA Trulia Notification Time Minus Redfin Notification Time Figure 4. Median Differences between Trulia and Redfin Notification Speeds (in Hours), by MSA Figure 4 shows that Redfin is well more than twice as fast to notify as Trulia in Seattle as it is in Houston. Overall, Redfin is considerably faster to notify than Trulia in all 20 MSAs. Table 10. Summary Statistics for Difference in Notification Speeds (in Hours) between Realtor and Redfin, by MSA (Pairwise Matched) Standard Lower 95% Upper 95% Lower 95% Upper 95% Robust Error MSA Median Mean (Robust Mean) Atlanta

13 Bay Area Boston Chicago Cleveland Dallas Denver Detroit Houston Los Angeles Miami Minneapolis New York Orlando Philadelphia Phoenix Portland San Diego Seattle Washington DC Median Difference in Notification Speeds (Hours), By MSA Realtor Notification Time Minus Redfin Notification Time Figure 5. Median Differences between Realtor and Redfin Notification Speeds (in Hours), by MSA 13

14 Figure 5 shows that Redfin is over 10 times faster to notify than Realtor in San Diego than in Boston, and well over four times faster in Seattle than in Boston. While summary statistics like the median or the robust mean are sufficient measures of central tendency, more insight can be gained from the underlying distributions themselves of the differences in notification speeds. 8 To this end, histograms were constructed for all three comparisons between Redfin and one other website. These histograms reveal that, on average, Redfin s notification speeds are faster than those of the other three websites (considerably so for Trulia) even though there are some individual instances in which Redfin is slower. Moreover, the histograms show that the mass of each distribution is concentrated within a tight band around the zero difference point, but significantly larger differences (both positive and negative) also exist. To understand just how the speed differences are distributed, SSRS constructed a relative frequency (or percentage) distribution of the differences on both sides of zero (i.e., for both positive and negative differences). All speed differences are assigned to 5-hour intervals on both sides of zero, although the intervals over 50 hours or under -50 hours are left open-ended. Tables depict these distributions for the comparison of Redfin with each website. Table 11. Distribution of the Difference of Speeds (in Hours) between Zillow and Redfin (Pairwise Matched) Interval (Hours) Percent Cumulative Percent Range (in Hours) Under Minimum = to Maximum = to to to to to % Zillow is faster in this range -15 to to to to This is particularly useful when those distributions are skewed or leptokurtic (peaked and heavy-tailed), such as when strongly influenced by outliers in one or both tails. In these circumstances, the outlier-adjusted robust mean is preferred to the ordinary arithmetic mean as a measure of central tendency. The median remains useful with these types of distributions. 14

15 0 to to to to to to to % Redfin is faster In this range 35 to to to Over Table 11 shows that the difference of speeds between Zillow and Redfin ranges from -676 hours to 709 hours. However, those speed differences are overwhelmingly positive (over 94% of the time), signifying that Redfin is faster than Zillow more than nine times out of ten. Also, about half of all differences fall within the 0 to 5 hours interval, while almost 28% of those differences fall in the range between 5 and 20 hours. Recall from Figure 1 that, based on the median difference, Redfin is 3.1 hours faster on average than Zillow to notify about properties. Table 12. Distribution of the Difference of Speeds (in Hours) between Trulia and Redfin (Pairwise Matched) Interval (Hours) Percent Cumulative Percent Range (in Hours) Under Minimum = to Maximum = to to to to to % Trulia is faster in this range -15 to to to

16 0 to to to to to to to to % Redfin is faster In this range 35 to to to Over Table 12 shows that the difference of speeds between Trulia and Redfin ranges from -326 hours to 402 hours. Those speed differences are also overwhelmingly positive (over 99% of the time), signifying that Redfin is faster than Trulia almost always. Furthermore, over two-thirds of all differences fall in the range between 10 and 25 hours. Recall from Figure 1 that, based on the median difference, Redfin is 18.3 hours faster on average than Trulia to notify about properties. Table 13. Distribution of the Difference of Speeds (in Hours) between Realtor and Redfin (Pairwise Matched) Interval (Hours) Percent Cumulative Percent Range (in Hours) Under Minimum = to Maximum = to to to to % Realtor is faster in this range -20 to to to

17 -5 to to to to to to to to to % Redfin is faster In this range 35 to to to Over Finally, Table 13 shows that the difference of speeds between Realtor and Redfin ranges from -217 hours to 735 hours. Those speed differences are also overwhelmingly positive (over 99% of the time), signifying that Redfin is faster than Realtor almost always. Furthermore, almost 70% of all differences fall within the 0 to 5 hours interval, while another 18% of all differences fall in the range between 5 and 20 hours. Recall from Figure 1 that, based on the median difference, Redfin is 3 hours faster on average than Realtor to notify about properties. Conclusion At Redfin s request, SSRS conducted a study of notification speeds for Redfin and three other websites (Zillow, Trulia, and Realtor) on matched properties within Redfin s footprint. A large database of matched properties was constructed along with website and alert type information and timestamps for notification s. This database, which was assembled between March 23 and April 5, 2017 using the data collection procedures described in detail in this report, yielded significant numbers of observations on matched properties, both on a pairwise-matched basis and on an all-four-websitesmatched basis. Relative speed performance (of Redfin against each of the other websites) was analyzed for the matched properties, at both the overall footprint level and the MSAspecific level. Timestamps from notification s received from the four websites were converted into notification speed data. Differences in notification speeds between 17

18 websites then became the basis for evaluating relative speed performance. Median and robust mean differences in notification speeds, as well as the actual distributions of those speed differences, were used for the evaluation. To summarize, the study revealed the following patterns: Overall, based on the median speed difference, Redfin is about three hours faster than Zillow and Realtor and over 18 hours faster than Trulia to notify subscribers about new listings. Redfin has a speed advantage over the other three websites nearly always (over 94% of the time over Zillow and over 99% of the time over both Trulia and Realtor). Very large percentages of Zillow s and Realtor s notifications are slower than Redfin s in the narrowest deviation interval (0-5 hours), namely, almost 50% for Zillow and almost 70% for Realtor; moreover, Zillow is slower by more than 25 hours (approximately a full day) over 14% of the time, while Realtor is slower by more than 25 hours just under 9% of the time. In contrast, Trulia is almost never slower than Redfin by 0-5 hours and slower by more than 25 hours almost 31% of the time. Taken together, these findings confirm that Redfin is faster to notify subscribers about newly listed properties than its leading competitor websites, such as Zillow, Trulia, and Realtor. 18