{"id":502,"date":"2024-07-28T21:19:34","date_gmt":"2024-07-28T21:19:34","guid":{"rendered":"https:\/\/summergeometry.org\/sgi2024\/?p=502"},"modified":"2024-07-31T01:32:02","modified_gmt":"2024-07-31T01:32:02","slug":"a-deeper-understanding-openais-clip-model","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2024\/a-deeper-understanding-openais-clip-model\/","title":{"rendered":"A Deeper Understanding OpenAI\u2019s CLIP Model"},"content":{"rendered":"\n

Author:<\/strong> Krishna Chebolu<\/a>
Teammates:<\/strong> Bethlehem Tassew<\/a> and Kimberly Herrera<\/a>
Mentor:<\/strong> Dr. Ankita Shukla<\/a><\/h2>\n\n\n\n

Introduction<\/h2>\n\n\n\n

For the past two weeks, our project team mentored by Dr. Ankita Shukla set out to understand the inner workings of OpenAI\u2019s CLIP model. Specifically, we were interested in gaining a mathematical understanding of feature spaces’ geometric and topological properties.\u00a0<\/p>\n\n\n\n

OpenAI’s CLIP<\/a> (Contrastive Language-Image Pre-Training) is a versatile and powerful model designed to understand and generate text and images. CLIP is trained to connect text and images by learning from a large dataset of images paired with their corresponding textual descriptions. The model is trained using a contrastive learning<\/a> approach, where it learns to predict which text snippet is associated with which image from a set of possible pairs. This allows CLIP to understand the relationship between textual and visual information.<\/p>\n\n\n\n

\"\"
Figure 1: OpenAI’s CLIP architecture as it appears in the paper. CLIP pre-trains an image encoder and a text encoder to predict which images were paired with which texts in our dataset. Then OpenAI uses this behavior to turn CLIP into a zero-shot classifier. We convert all of a dataset\u2019s classes into captions such as \u201ca photo of a dog\u201d and predict the class of the caption CLIP estimates best pairs with a given image.<\/figcaption><\/figure>\n\n\n\n

CLIP uses two separate encoders: a text encoder (based on the Transformer architecture) and an image encoder (based on a convolutional neural network or a vision transformer). Both encoders produce embeddings in a shared latent space<\/a> (also called a feature space). By aligning text and image embeddings in the same space, CLIP can perform tasks that require cross-modal understanding, such as image captioning, image classification with natural language labels, and more.<\/p>\n\n\n\n

CLIP is trained on a vast dataset containing 400 million image-text pairs collected online. This extensive training data allows it to generalize across various domains and tasks. One of CLIP\u2019s standout features is its ability to perform zero-shot learning<\/a>. It can handle new tasks without requiring task-specific training data, simply by understanding the task description in natural language. More information can be found in OpenAI\u2019s paper<\/a>.<\/p>\n\n\n\n

In our attempts to understand the inner workings of the feature spaces, we employed tools from UMAP, persistence homology, subspace angles, cosine similarity matrices, and Wasserstein distances. <\/p>\n\n\n\n

Our Study – Methodology and Results<\/h2>\n\n\n\n

All of our teammates started with datasets that contained image-caption pairs. We classified images into various categories using their captions and embedded them using CLIP. Then we used UMAP<\/a> or t-SNE<\/a> plots to visualize their characteristics.<\/p>\n\n\n\n

\"\"
Figure 2: A screenshot of a UMAP of 1000 images from the Flickr 8k dataset<\/a> from Kaggle <\/a>divided into five categories (animal, human, sport, nature, and vehicle) is shown. Here we can also observe that the images (colored) are embedded differently than their corresponding captions (gray). Although not shown here, the captions are also clustered around categories.<\/figcaption><\/figure>\n\n\n\n

After this preliminary visualization, we desire to delve deeper. We introduced noise, a Gaussian blur, to our images to test CLIP\u2019s robustness. We added the noise in increments (for example mean = 0, standard deviation = {1,2,3,4,5}) and encoded them as we did the original image-caption pairs. We then made persistence diagrams<\/a> using ripser<\/a>. We also followed the same procedure within the various categories to understand how noise impacts not only the overall space but also their respective subspaces. These diagrams for the five categories from the Flickr 8k dataset can be found in this Google Colab notebook<\/a>.<\/p>\n\n\n\n

\"\"
Figure 3: The same 1000 images from the Flickr 8k dataset with increasing noise are seen above. Visually, no significant difference can be observed. The standard deviation of the Gaussian blur increases from left to right.<\/figcaption><\/figure>\n\n\n\n

Visually, you can observe that there is no significant difference, which attests to CLIP\u2019s robustness. However, visual assurance is not enough. Thus, we used Scipy\u2019s<\/a> Wasserstein\u2019s distance<\/a> calculation to note how different each persistence diagram is from the other. Continuing the same Flickr 8k dataset, for each category, we obtain the values shown in Figure 4.<\/p>\n\n\n\n

\"\"
Figure 4: Wasserstein distances in each category. You can see the distance between original images with respect to images with Gaussian blur of std. dev = 1 is not high compared to std. dev = 2 or above. This implies that the original set is not as different from the set of images blurred with std. dev. = 1 as compared to std. dev. = 2 which in turn is not as different as the set of blurred images with std. dev. = 3, and so on. This property holds for all five categories.<\/figcaption><\/figure>\n\n\n\n

Another question to understand is how similar are each of the categories to one another.<\/em> This question can be answered by calculating the subspace angles. After embedding, each category can be seen as occupying a space that can often be far away from another category\u2019s space\u2013 we want to quantify how far away, so we use subspace angles<\/a>. Results for the Flickr 8k dataset example are shown in Figure 5. <\/p>\n\n\n\n

\"\"
Figure 5: Subspace angles of each category pair in the five categories introduced earlier in the post. All values are displayed in degrees. You can observe that the angle between human- and animal-category images is ~0.92\u00b0 compared to human and nature: ~2.3\u00b0; which makes sense as humans and animals are more similar than humans compared to nature. It is worthwhile to note that the five categories are simplifying the dataset too much as they do not capture the nuances of the captions. More categories or descriptions of categories would lead to higher accuracy in the quantity of the subspace angle.<\/figcaption><\/figure>\n\n\n\n

Conclusion<\/strong><\/h2>\n\n\n\n

At the start, our team members were novices in the CLIP model, but we concluded as lesser <\/em>novices. Through the two weeks, Dr. Shukla supported us and enabled us to understand the inner workings of the CLIP model. It is certainly thrilling to observe how AI around us is constantly evolving, but at the heart of it is mathematics governing the change. We are excited to possibly explore further and perform more analyses. <\/p>\n","protected":false},"excerpt":{"rendered":"

We attempt to understand the inner workings of OpenAI’s revolutionary CLIP model using tools from geometry and topology.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40,37],"tags":[],"ppma_author":[25],"class_list":["post-502","post","type-post","status-publish","format-standard","hentry","category-math","category-research"],"authors":[{"term_id":25,"user_id":0,"is_guest":1,"slug":"cap-kris-chebo","display_name":"kris.chebo","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","author_category":"","first_name":"","last_name":"","user_url":"","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/502","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/comments?post=502"}],"version-history":[{"count":7,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/502\/revisions"}],"predecessor-version":[{"id":644,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/502\/revisions\/644"}],"wp:attachment":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/media?parent=502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/categories?post=502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/tags?post=502"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/ppma_author?post=502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}