Image Recognition on Streaming Data

Editor's Notes

1. Google / Facebook leveraging ML to detect violating content. Expensify reading receipts Identifying what objects are in social media posts. Detecting divergence of maps for use in the intelligence communities.
2. You may remember this slide.
3. You can detect who is at your front door You can detect what animal your phone is pointed at You can point your phone at a building and learn attributes about it All of this is possible with MemSQL. Once you have the feature vectors stored in your database you can process and identify those which are the closest to your selected image.
4. Facial recognition is a subject of ongoing research to efficiently extract feature vectors from images using deep learning. For the purpose of this talk, we will assume that this is a somewhat solved problem and we can efficiently extract feature vectors from any incoming image. Once those feature vectors are produced, all you need to do is insert them into a MemSQL table with the following simple schema. Once you have the model produced, you need the tools to process this data at scale. I’m not going to go into how this is done exactly, as there are tons of resources online. I’m going to talk about what happens once this lands in the database.
5. There are two frequently used approaches to measuring the similarity between vectors: cosine similarity (cosine of the angle between the vectors) and Euclidean distance. Cosine similarity is defined as the dot product of the vectors, divided by the product of the vector norms (length of the vectors). If the vectors are normalized, the cosine similarity is simply the dot product of the vectors (since the product of the norms is 1). In this scenario, we choose the approach of normalizing each feature vector by dividing each element in the vector by the length of the vector, such that the scalar length is one. CALL OUT THAT THIS IS A FULL TABLE SCAN
6. Dot Product is an algebraic operation that takes two equal-length sequences of numbers (usually coordinate vectors) and returns a single number. In Euclidean geometry, the dot product of the Cartesian coordinates of two vectors is widely used and often called inner product (or rarely projection product); see also inner product space. Algebraically, the dot product is the sum of the products of the corresponding entries of the two sequences of numbers. Angles between non-unit vectors (vectors with lengths not equal to 1.0) can be calculated either by first normalizing the vectors, or by dividing the dot product of the non-unit vectors by the length of each vector. Taking the dot product of a vectoragainst itself (i.e. The similarity is higher if the dot product of the two vectors is close to one. On the previous slide, we choose a constant of 0.9 for highly similar.
7. People usually try to process this type of information using GPUs, but in this particular use case, the bottleneck is actually the bandwidth of memory. Memory bandwidth is actually roughly 48GB / sec but I’m going to give it the benefit of the doubt we’ll round up to 50 GB/s.
8. How can MemSQL run this faster than memory bandwidth? The answer is compression of columnstore tables. Because the random vectors were normalized, they were able to be compressed from 50GB down to a size that can be read from memory in less than 0.25 seconds. Because you can perform image recognition at in-memory speed, your bottleneck for similarity computation is not necessarily compute. We realize that there are other algorithms that gain efficiency by avoiding the full table scan and only lose a small amount of accuracy. However, you can achieve good practical results with a very straightforward implementation.