{"id":1551,"date":"2021-09-13T20:56:55","date_gmt":"2021-09-13T20:56:55","guid":{"rendered":"http:\/\/summergeometry.org\/sgi2021\/?p=1551"},"modified":"2021-09-13T20:56:57","modified_gmt":"2021-09-13T20:56:57","slug":"self-similarity-loss-for-shape-descriptor-learning-in-correspondence-problems","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2021\/self-similarity-loss-for-shape-descriptor-learning-in-correspondence-problems\/","title":{"rendered":"Self-similarity loss for shape descriptor learning in correspondence problems"},"content":{"rendered":"\n<p class=\"has-text-align-center wp-block-paragraph\"><strong><em>By <\/em><\/strong><a href=\"https:\/\/oaishi.github.io\/\"><strong><em>Faria Huq<\/em><\/strong><\/a><strong><em>, Kinjal Parikh, <\/em><\/strong><a href=\"https:\/\/valenca.io\"><strong><em><strong><em>&nbsp;<\/em><\/strong><\/em><\/strong><\/a><strong><em><a href=\"https:\/\/valenca.io\"><strong><em>Lucas Valen\u00e7a<\/em><\/strong><\/a><\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">During the 4<sup>th<\/sup> week of SGI, we worked closely with <a rel=\"noreferrer noopener\" href=\"http:\/\/csail.mit.edu\/person\/tal-shnitzer\" data-type=\"URL\" data-id=\"csail.mit.edu\/person\/tal-shnitzer\" target=\"_blank\">Dr. Tal Shnitzer<\/a> to develop an improved loss function for learning functional maps with robustness to symmetric ambiguity. Our project goal was to modify recent weakly-supervised works that generated deep functional maps to make them handle symmetric correspondences better.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction:<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Shape correspondence is a task that has various applications in geometry processing, computer graphics, and computer vision \u2013 quad mesh transfer, shape interpolation, and object recognition, to name a few. It entails computing a mapping between two objects in a geometric dataset. Several techniques for computing shape correspondence exist &#8211; functional maps is one of them.&nbsp;<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>A functional map is a representation that can map between functions on two shapes\u2019 using their eigenbases or features (or both). Formally, it is the solution to \\(\\mathrm{arg}\\min_{C_{12}}\\left\\Vert C_{12}F_1-F_2\\right\\Vert^2\\), where \\(C_{12}\\) is the functional map from shape \\(1\\) to shape \\(2\\) and&nbsp; \\(F_1\\), \\(F_2\\) are corresponding functions projected onto the eigenbases of the two shapes, respectively. <\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore, there is no direct mapping between vertices in a functional map. This concise representation facilitates manipulation and enables efficient inference.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"744\" src=\"http:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog-1024x744.png\" alt=\"\" class=\"wp-image-1552\" srcset=\"https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog-1024x744.png 1024w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog-300x218.png 300w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog-768x558.png 768w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog-1200x872.png 1200w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/basc_for_blog.png 1354w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 1: The approach proposed by Sharma and Ovsjanikov may fail on symmetric regions. Notice the hands, which have not been matched correctly due to their symmetric structure.<br><\/figcaption><\/figure><\/div>\n\n\n\n<ul class=\"wp-block-list\"><li>Recently, unsupervised deep learning methods have been developed for learning functional maps. One of the main challenges in such shape correspondence tasks is learning a map that differentiates between shape regions that are similar (due to symmetry). We worked on tackling this challenge.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Background<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li> We build upon the state-of-the-art work \u201c<a href=\"https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/dfb84a11f431c62436cfb760e30a34fe-Paper.pdf\" data-type=\"URL\" data-id=\"https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/dfb84a11f431c62436cfb760e30a34fe-Paper.pdf\">Weakly Supervised Deep Functional Map for Shape Matching<\/a>\u201d by Sharma and Ovsjanikov, which learns shape descriptors from raw 3D data using a PointNet++ architecture. The network\u2019s loss function is based on regularization terms that enforce bijectivity, orthogonality, and Laplacian commutativity. <\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>This method is weakly supervised because the input shapes must be equally aligned, i.e., share the same 6DOF pose. This weak supervision is required because PointNet-like feature extractors cannot distinguish between left and right unless the shapes share the same pose.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>To mitigate the same-pose requirement, we explored adding another component to the loss function <a href=\"https:\/\/openaccess.thecvf.com\/content_ECCV_2018\/papers\/Roey_Mechrez_The_Contextual_Loss_ECCV_2018_paper.pdf\" data-type=\"URL\" data-id=\"https:\/\/openaccess.thecvf.com\/content_ECCV_2018\/papers\/Roey_Mechrez_The_Contextual_Loss_ECCV_2018_paper.pdf\">Contextual Loss<\/a> by Mechrez et al. Contextual Loss is high when the network learns a large number of similar features. Otherwise, it is low. This characteristic promotes the learning of global features and can, therefore, work on non-aligned data.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Work<\/h2>\n\n\n\n<ol class=\"wp-block-list\"><li><strong>Model architecture overview:<\/strong> As stated above, our basic model architecture is similar to \u201c<a href=\"https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/dfb84a11f431c62436cfb760e30a34fe-Paper.pdf\" data-type=\"URL\" data-id=\"https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/dfb84a11f431c62436cfb760e30a34fe-Paper.pdf\">Weakly Supervised Deep Functional Map for Shape Matching<\/a>\u201d by Sharma and Ovsjanikov. We use the <a href=\"https:\/\/papers.nips.cc\/paper\/2017\/file\/d8bf84be3800d12f74d8b05e9b89836f-Paper.pdf\">basic PointNet++ architecture<\/a> and pass its output through a \\(4\\)-layer ResNet model. We use the output from ResNet as shape features to compute the functional map. We randomly select \\(4000\\) vertices and pass them as input to our PointNet++ architecture.<\/li><li><strong>Data augmentation:<\/strong> We randomly rotated the shapes of the input dataset around the &#8220;up&#8221; axis (in our case, the \\(y\\) coordinate). Our motivation for introducing data augmentation is to make the learning more robust and less dependent on the data orientation.<\/li><li><strong>Contextual loss:<\/strong> We explored two ways of adding contextual loss as a component:<ol><li><strong>Self-similarity:<\/strong> Consider a pair of input shapes (\\(S_1\\), \\(S_2\\)) of features \\(P_1\\) with \\(P_2\\) respectively. We compute our loss function as follows:<br>\\(L_{CX}(S_1, S_2) = -log(CX(P_1, P_1)) &#8211; log(CX(P_2, P_2))\\),<br>where \\(CX(x, y)\\) is the contextual similarity between every element of \\(x\\) and every element of \\(y\\), considering the context of all features in \\(y\\).<br>More intuitively, the contextual loss is applied on each shape feature with itself (\\(P_1\\) with \\(P_1\\) and \\(P_2\\) with \\(P_2\\)), thus giving us a measure of \u2018self-similarity\u2019 in a weakly-supervised way. This measure will help the network learn unique and better descriptors for each shape, thus alleviating errors from symmetric ambiguities.<\/li><li><strong>Projected features<\/strong>: We also explored another method for employing the contextual loss. First, we project the basis \\(B_1\\) of \\(S_1\\) onto \\(S_2\\), so that \\(B_{12} = B_ 1 \\cdot C_{12}\\). Similarly, \\(B_{21} = B_2 \\cdot C_{21}\\). Note that the initial bases \\(B_1\\) and \\(B_2\\) are computed directly from the input shapes. Next, we want the projection \\(B_{12}\\) to get closer to \\(B_2\\) (the same applies for \\(B_{21}\\) and \\(B_1\\)). Hence, our loss function becomes:<br>\\(L_{CX}(S_1, S_2) = -log(CX(B_{21}, B_1)) &#8211; log(CX(B_{12}, B_2))\\).<br>Our motivation for applying this loss function is to reduce symmetry error by encouraging our model to map the eigenbases using \\(C_{12}\\) and \\(C_{21}\\) more accurately.<\/li><\/ol><\/li><li><strong>Geodesic error:<\/strong> For evaluating our work, we use the metric of average geodesic error between the vertex-pair mappings predicted by our models and the ground truth vertex-pair indices provided with the dataset.<\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Result<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We trained six different models on the <a href=\"http:\/\/faust.is.tue.mpg.de\/\" data-type=\"URL\" data-id=\"http:\/\/faust.is.tue.mpg.de\/\">FAUST dataset<\/a> (which contains \\(10\\) human shapes at the same  \\(10\\) poses each). Our training set includes \\(81\\) samples, leaving out one full shape (all of its \\(10\\) poses) and one full pose (so the network never sees any shape doing that pose). These remaining  \\(19\\) inputs are the test set. Additionally, during testing we used <a href=\"https:\/\/arxiv.org\/pdf\/1904.07865.pdf\" data-type=\"URL\" data-id=\"https:\/\/arxiv.org\/pdf\/1904.07865.pdf\">ZoomOut<\/a> by Melzi et al.<\/p>\n\n\n\n<figure class=\"wp-block-table aligncenter\"><table><tbody><tr><td><em>Model<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><em>Data Augmentation<\/em><\/td><td><em>Contextual Loss<\/em><\/td><\/tr><tr><td>Base<\/td><td class=\"has-text-align-left\" data-align=\"left\">False<\/td><td>None<\/td><\/tr><tr><td>BA<\/td><td class=\"has-text-align-left\" data-align=\"left\">True<\/td><td>None<\/td><\/tr><tr><td>SS<\/td><td class=\"has-text-align-left\" data-align=\"left\">False<\/td><td>Self Similarity<\/td><\/tr><tr><td>SSA<\/td><td class=\"has-text-align-left\" data-align=\"left\">True<\/td><td>Self Similarity<\/td><\/tr><tr><td>PF<\/td><td class=\"has-text-align-left\" data-align=\"left\">False<\/td><td>Projected Features<\/td><\/tr><tr><td>PFA<\/td><td class=\"has-text-align-left\" data-align=\"left\">True<\/td><td>Projected Features<\/td><\/tr><\/tbody><\/table><figcaption>Table 1: We trained 7 models using the settings described in this table<\/figcaption><\/figure>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"755\" src=\"http:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-1024x755.png\" alt=\"\" class=\"wp-image-1553\" srcset=\"https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-1024x755.png 1024w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-300x221.png 300w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-768x566.png 768w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-1536x1132.png 1536w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-2048x1510.png 2048w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-1200x885.png 1200w, https:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/zoomed_plot-1980x1460.png 1980w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 2: The AUC (area under curve) curves and values with the average geodesic error for each model.<br><\/figcaption><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">For qualitative comparison purposes, the following GIF displays a mapping result. Overall, the main noticeable visual differences between our work and the one we based ourselves on appeared when dealing with symmetric body parts (e.g., hands and feet).<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"696\" height=\"424\" src=\"http:\/\/summergeometry.org\/sgi2021\/wp-content\/uploads\/2021\/09\/animated_blog.gif\" alt=\"\" class=\"wp-image-1554\"\/><figcaption>Figure 3: Comparison between shape correspondence produced by the models described in Table 1.<\/figcaption><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Still, as it can be seen, the symmetric body parts remain the less accurate mappings in our work too, meaning there\u2019s still much to improve.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We thoroughly enjoyed working on this project during SGI and are looking forward to investigating this problem further, to which end we will continue to work on this project after the program. We want to extend our deepest gratitude to our mentor, Dr. Tal Shnitzer, for her continuous guidance and patience. None of this would be possible without her.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thank you for taking the time to read this post. If there are any questions or suggestions, we&#8217;re eager to hear them!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Faria Huq, Kinjal Parikh, &nbsp;Lucas Valen\u00e7a During the 4th week of SGI, we worked closely with Dr. Tal Shnitzer to develop an improved loss function for learning functional maps with robustness to symmetric ambiguity. Our project goal was to modify recent weakly-supervised works that generated deep functional maps to make them handle symmetric correspondences [&hellip;]<\/p>\n","protected":false},"author":21,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,25],"tags":[97,93,100],"class_list":["post-1551","post","type-post","status-publish","format-standard","hentry","category-demo","category-sgi-research-projects","tag-deep-learning","tag-functional-map","tag-weakly-supervised-learning"],"_links":{"self":[{"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/posts\/1551","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/comments?post=1551"}],"version-history":[{"count":4,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/posts\/1551\/revisions"}],"predecessor-version":[{"id":1560,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/posts\/1551\/revisions\/1560"}],"wp:attachment":[{"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/media?parent=1551"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/categories?post=1551"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2021\/wp-json\/wp\/v2\/tags?post=1551"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}