Is Mozilla anti phishing study faulty?

* They didn't make any attempt to score false positives. This is a critical omission, because a filter that produces significant numbers of false positives will quickly train users to ignore its legitimate warnings. (Interestingly, PhishTank's own FAQ agrees with me). IMHO any study that doesn't include false positive data is meaningless.
* Speaking of "doesn't include", the report only looked at IE and Firefox. I would have liked to see some other products (note: not SiteAdvisor) included to give a broader basis of comparison.
* The Firefox report mentions that IE can warn or block, but it doesn't credit IE with any actual warnings. This is a significant omission, although we can't tell how significant because...
* The Firefox report doesn't include any information about the actual URLs used. They promise to publish this data "soon", but without that there's no way to gauge the quality of their data. (I understand that they'll publish the data later today; it'll be interesting to see the raw stuff.) Of course, we published all our URL data in our report.
* Speaking of data: the Firefox team used 1040 phish from Phishtank, a community filtering system, gathered over a two-week period. That's a good number of phish, but the study period was awfully short, and the phish all came from one source. We used multiple sources, including honeypots and user reports, to generate the phish list we used.
* Because they used a community-generated feed of phish, there's no way to tell which of the phish had also (or already) been reported to other systems that may have fed into the "Ask Google" or Microsoft data feeds. By contrast, we took great pains to try to find phish that we knew hadn't been submitted to Microsoft's URL reputation service.

So, my personal opinion is that this study isn't as rigorous as the 3Sharp study or the one done by Dr. Lorrie Cranor et al of Carnegie Mellon Paul's Down-Mozilla releases anti-phishing report more here Firefox 2 Phishing Protection Effectiveness Testing

Linked by shanmuga Tuesday, 14th November 2006 11:11PM