{"id":832,"date":"2024-08-01T19:39:31","date_gmt":"2024-08-01T19:39:31","guid":{"rendered":"https:\/\/summergeometry.org\/sgi2024\/?p=832"},"modified":"2024-08-01T19:42:04","modified_gmt":"2024-08-01T19:42:04","slug":"topology-of-feature-spaces","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2024\/topology-of-feature-spaces\/","title":{"rendered":"Topology of feature spaces"},"content":{"rendered":"\n

CLIP<\/strong> is a system designed to determine which image matches which piece of text in a group of images and texts.<\/p>\n\n\n\n

1. How it works:<\/strong><\/p>\n\n\n\n

\u2022 Embedding Space: Think of this as a special place where both images and text are transformed into numbers.<\/p>\n\n\n\n

\u2022 Encoders: CLIP has two parts that do this transformation:<\/p>\n\n\n\n

\u2013 Image Encoder: This part looks at images and converts them into a set of numbers (called embeddings).<\/p>\n\n\n\n

\u2013 Text Encoder: This part reads text and also converts it into a set of numbers(embeddings).<\/p>\n\n\n\n

2. Training Process:<\/strong><\/p>\n\n\n\n

\u2022 Batch: Imagine you have a bunch of images and their corresponding texts in a group(batch).<\/p>\n\n\n\n

\u2022 Real Pairs: Within this group, some images and texts actually match (like an image of a cat and the word \u201dcat\u201d).<\/p>\n\n\n\n

\u2022 Fake Pairs: There are many more possible combinations that don\u2019t match (like an image of a cat and the word \u201ddog\u201d).<\/p>\n\n\n\n

3. Cosine Similarity:<\/strong> This is a way to measure how close two sets of numbers (embeddings) are. Higher similarity means they are more alike.<\/p>\n\n\n\n

4. CLIP\u2019s Goal: <\/strong>CLIP tries to make the embeddings of matching images and text (real pairs) as close as possible. At the same time, it tries to make the embeddings of non-matching pairs (fake pairs) as different as possible.<\/p>\n\n\n\n

5. Optimization:<\/strong><\/p>\n\n\n\n

\u2022 Loss Function: This is a mathematical way to measure how good or bad the current matchings are.<\/p>\n\n\n\n

\u2022 Symmetric Cross-Entropy Loss: CLIP uses a specific type of loss function that looks at the similarities of both real and fake pairs and adjusts the embeddings to improve the matchings.<\/p>\n\n\n\n

In essence, CLIP learns to accurately match images and texts by continuously improving how it transforms them into numbers so that correct matches are close together and incorrect ones are far apart.<\/p>\n\n\n\n

After learning CLIP, I chose my data set and got to work:<\/p>\n\n\n\n

The Describable Textures Dataset (DTD)<\/strong> is an evolving collection of textural images in the wild, annotated with a series of human-centric attributes, inspired by the perceptual properties of textures.<\/p>\n\n\n\n

The package contains:<\/p>\n\n\n\n

1. Dataset images, train, validation, and test.<\/p>\n\n\n\n

2. Ground truth annotations and splits used for evaluation.<\/p>\n\n\n\n

3. imdb.mat file, containing a struct holding file names and ground truth labels.<\/p>\n\n\n\n

Example images:<\/p>\n\n\n\n

\n
\n
\n
\"\"<\/figure>\n<\/figure>\n<\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div><\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div><\/div>\n<\/div>\n\n\n\n

There are 47 texture classes, with 120 images each for a total of 5640 images in this data set. The above shows \u2018cobwebbed\u2019, \u2018pitted,\u2019 and \u2018banded\u2019. I did the t-SNE<\/strong> visualization by class for all the classes but realized this wasn\u2019t very helpful for analysis. It was the same for UMAP<\/strong>. So I decided to sample 15 classes and then visualize:<\/p>\n\n\n\n

\n
\n
\"\"<\/figure>\n<\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n

In the t-SNE for 15 classes, we see that \u2019polka-dotted\u2019 and \u2019dotted\u2019 are clustered together. This intuitively makes sense. To further our analysis, we computed the subspace angles between the classes. Many pairs of categories have an angle of 0.0, meaning their feature vectors are very close to each other in the feature space. This suggests that these textures are highly similar or share similar feature representations. For instance:<\/p>\n\n\n\n

\u2022 crosshatched and striped<\/p>\n\n\n\n

\u2022 dotted and grid<\/p>\n\n\n\n

\u2022 dotted and polka-dotted<\/p>\n\n\n\n

Then came the tda<\/strong>: A persistence diagram summarizes the topological features of a data set across multiple scales. It captures the birth and death of features such as connected components, holes, and voids as the scale of observation changes. In a persistence diagram, each feature is represented as a point in a two-dimensional space, where the x-coordinate corresponds to the “birth” scale and the y-coordinate corresponds to the “death” scale. This visualization helps in understanding the shape and structure of the data, allowing for the identification of significant features that persist across various scales while filtering out noise.<\/p>\n\n\n\n

I added 3 levels of noise (0.5, 1.0, 2.0) to the images and then extracted features. I visualized these features on a persistence diagram. Here are some examples of those results. We can see that for H_0 at all noise levels, there is one persistent feature so there is one connected component. The death of this persistent feature varies slightly. H_1 at all noise levels there aren\u2019t any highly persistent features, with most points being around the diagonal. The features in H_1 tend to \u201cclump up together\u201d and die quicker as the noise level goes up.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

I then computed the distances between the diagrams<\/strong> with no noise and those with noise. Here are some of those results. Unsurprisingly, with greater levels of noise, there is greater distance.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Finally, we wanted to test the robustness of CLIP <\/strong>so we classified images with respect to noise. The goal was to see if the results we saw with respect to the topology of the feature space corresponded to the classification<\/strong> results. These were the classification accuracies:<\/p>\n\n\n\n

<\/p>\n\n\n\n

We hope to discuss our results further!<\/p>\n","protected":false},"excerpt":{"rendered":"

CLIP is a system designed to determine which image matches which piece of text in a group of images and texts. 1. How it works: \u2022 Embedding Space: Think of this as a special place where both images and text are transformed into numbers. \u2022 Encoders: CLIP has two parts that do this transformation: \u2013 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"ppma_author":[18],"class_list":["post-832","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"authors":[{"term_id":18,"user_id":0,"is_guest":1,"slug":"cap-kimberlyherrera","display_name":"kimberlyherrera","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","author_category":"","first_name":"","last_name":"","user_url":"","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/832","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/comments?post=832"}],"version-history":[{"count":3,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/832\/revisions"}],"predecessor-version":[{"id":841,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/832\/revisions\/841"}],"wp:attachment":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/media?parent=832"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/categories?post=832"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/tags?post=832"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/ppma_author?post=832"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}