calculationMemory

Cosine Similarity for 1 Trillion Pairs of Vectors Motivation ChunkDot Chunk size calculation Memory and speed Usage Conclusion

Introducing ChunkDotpip install -U chunkdotCalculate the 50 most similar and dissimilar items for 100K items.import numpy as npfrom chunkdot import cosine_similarity_top_kembeddings = np.random.randn(100000, 256)# using all you system's memorycosine_similarity_top_k(embeddings, top_k=50)# most dissimilar items using...

Recent posts

Popular categories

ASK DUKE