Between September 2024 and January 2025, our research team conducted a comprehensive analysis of deepfake detection performance across video, audio, and image modalities. Our methodology combined a systematic review of academic research with direct evaluation of 12 state-of-the-art detection models using the Deepfake-Eval-2024 dataset compiled through the TrueMedia.org platform and social media moderation forums.

This quarterly deepfake detection benchmark aggregates data from:

We evaluated both open-source and commercial detection models against real-world deepfakes circulating on social media platforms. The results reveal a critical gap between laboratory performance and actual deployment effectiveness.

The following table presents overall detection accuracy rates across the three primary content types where deepfake manipulation occurs:

Overall accuracy sits at just 55%. When confronted with high-quality deepfakes, performance collapses:

This data comes from the most comprehensive meta-analysis of human deepfake detection ever conducted. The systematic review synthesized 56 papers involving 86,155 participants across multiple demographics and training levels.

These findings create a critical dependency on automated detection systems. Yet they also highlight a fundamental challenge: even the best AI models struggle to significantly outperform humans in real-world conditions.

Detection models experience dramatic accuracy drops when deployed against actual deepfakes compared to controlled laboratory testing.

Video detection models experience the most dramatic failure. Models like GenConViT achieve 96% accuracy on academic datasets like FaceForensics++. When tested against contemporary in-the-wild content, accuracy plummets to 63%, a 50% performance drop. AASIST drops from 100% accuracy on ASVspoof2019 to just 43% on real-world audio. RawNet2 falls from 99% to 53%.

The Deepfake-Eval-2024 benchmark revealed the root cause across all major detection architectures:

Academic datasets use outdated manipulation techniques, contain limited content diversity, and feature structured media that differs significantly from social media content.

Models trained on this data fail when confronted with:

Direct comparison between human analysts and automated detection systems reveals clear performance patterns across content types:

Commercial detection systems outperform both human analysts and open-source models across every content type:

Commercial systems don't have better algorithms. They have better data. Superior performance appears primarily attributable to training on more representative datasets rather than architectural innovations. This is proven by open-source models approaching commercial performance when finetuned on in-the-wild data.

For high-quality video deepfakes, human detection collapses to 25% accuracy while commercial systems maintain 71% performance. Non-English audio presents particular challenges for open-source models, with accuracy dropping to 46% compared to commercial systems' 82%.

These findings underscore the critical importance of:

Specific content attributes significantly impact detection accuracy, revealing systematic weaknesses across all model architectures.

Audio detectors lose 26% accuracy when deepfakes contain background music. This represents the single largest vulnerability identified in the benchmark. Background music is common in generation workflows but rare in training datasets. Models simply haven't learned to separate manipulation artifacts from musical overlay effects.

Non-English content reduces audio detection by 7%. This reflects a critical limitation: existing audio datasets typically include only English content, with at most two languages represented.

Real-world threats span 42 languages in contemporary benchmarks.

Social media compression artifacts reduce accuracy by 9% across all modalities. Models struggle to distinguish compression effects from manipulation traces.

Image systems: Text overlays reduce accuracy by 9%. These overlays are prevalent in social media sharing but absent from training data.

Video systems: Non-facial body modifications cause 18% accuracy reductions. Most video detection focuses on facial manipulation, missing full-body deepfakes.

Top-performing commercial models represent the current state-of-the-art for deployed detection technology, significantly outperforming open-source alternatives.

Commercial audio detection achieves 89% accuracy with 93% AUC. This represents the highest performance across all modalities and approaches the estimated 90% accuracy threshold of human forensic analysts.

The balanced metrics show robust performance:

Video detection reaches 78% accuracy with balanced 77% precision and recall. Image detection achieves 82% accuracy despite a lower 71% recall, meaning it misses 29% of manipulated images.

Even top-performing commercial systems maintain 8-12% false positive rates and 16-29% false negative rates. This indicates continued development needs for mission-critical applications requiring higher reliability thresholds, such as:

Commercial systems provide the best available protection but still fall short of the reliability needed for high-stakes decisions. Organizations should implement layered defense strategies combining:

For more information, you can learn more about Ceartas here and contact us through our integrated chat service if you have any questions.