"Maze: a cost-efficient video deduplication system at Web-scale"  

An Qin, Mengbai Xiao, Ben Huang, and Xiaodong Zhang  

Proceedings of 2022 ACM Conference on Multimedia (ACM MM 2022), 
Lisboa, Portugal, October 10-14, 2022.   

Abstract

With the advancement and dominant service of Internet videos, the
content-based video deduplication system becomes an essential and
dependent infrastructure for Internet video service. However, the
explosively growing video data on the Internet challenges the system
design and implementation for its scalability in several ways. (1)
Although the quantization-based indexing techniques are effective 
for searching visual features at a large scale, the costly
re-training over the complete dataset must be done periodically.
(2) The high-dimensional vectors for visual features demand increasingly
large SSD space, degrading I/O performance. (3) Videos
crawled from the Internet are diverse, and visually similar videos
are not necessarily the duplicates, increasing deduplication complexity.
(4) Most videos are edited ones. The duplicate contents
are more likely discovered as clips inside the videos, demanding
processing techniques with close attention to details.
To address above-mentioned issues, we propose Maze, a fullfledged
video deduplication system. Maze has an ANNS layer that
indexes and searches the high dimensional feature vectors. The architecture
of the ANNS layer supports efficient reads and writes and
eliminates the data migration caused by re-training. Maze adopts
the CNN-based feature and the ORB feature as the visual features,
which are optimized for the specific video deduplication task.
The features are compact and fully reside in the memory. Acoustic
features are also incorporated in Maze so that the visually similar
videos but having different audio tracks are recognizable. A
clip-based matching algorithm is developed to discover duplicate
contents at a fine granularity. Maze has been deployed as a production
system for two years. It has indexed 1.3 billion videos and
is indexing about 800 thousand videos per day. For the ANNS layer,
the average read latency is 4 seconds and the average write latency
is at most 4.84 seconds. The re-training over the complete
dataset is no longer required no matter how many new data sets
are added, eliminating the costly data migration between nodes.
Maze recognizes the duplicate live streaming videos with both the
similar appearance and the similar audio at a recall of 98%. Most
importantly, Maze is also cost-effective. For example, the compact
feature design helps save 5800 SSDs and the computation resources
devoted to running the whole system decrease to 250K standard
cores per billion videos.