Hierarchical clustering of large text datasets using Locality-Sensitive Hashing

Abstract

In this paper, we present a hierarchical clustering algorithm of the large text datasets using Locality-Sensitive Hashing (LSH). The main idea of the LSH is to “hash” items several times, in such a way that similar items are more likely to be hashed to the same bucket than dissimilar are. The main drawback of the conventional hierarchical algorithms is a large time complexity (e.g. Single Linkage method has time complexity of O(n^2)) Proposed algorithm reduces the time complexity to O(Pn). Here, P represents the maximum number of items going to the single bucket. P is a small constant as compared to n for the large number of buckets.