{"id":1586,"date":"2024-08-18T20:49:37","date_gmt":"2024-08-18T20:49:37","guid":{"rendered":"https:\/\/summergeometry.org\/sgi2024\/?p=1586"},"modified":"2024-12-01T04:58:42","modified_gmt":"2024-12-01T04:58:42","slug":"loop-subdivision-for-tetsphere-splatting","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2024\/loop-subdivision-for-tetsphere-splatting\/","title":{"rendered":"Loop Subdivision for Tetsphere Splatting"},"content":{"rendered":"\n

SGI Fellows: <\/strong>Harini Rammohan, Dianlun Luo, Idil Sulo<\/p>\n\n\n\n

Project Mentor:<\/strong> Minghao Guo <\/p>\n\n\n\n

SGI Volunteer:<\/strong> Shanthika Naik<\/p>\n\n\n\n

Introduction:<\/h3>\n\n\n\n

TetSphere Splatting<\/strong> [Guo et al., 2024] is a cutting-edge technique for high-quality 3D shape reconstruction using tetrahedral meshes. This method stands out by delivering superior geometry without relying on neural networks or post-processing. Unlike traditional Eulerian methods, TetSphere Splatting excels in applications such as single-view 3D reconstruction and text-to-3D generation.<\/p>\n\n\n\n

Our project aimed to enhance TetSphere Splatting through two key improvements:<\/p>\n\n\n\n

    \n
  1. Geometry Optimization<\/strong>: We integrated subdivision and remeshing techniques to refine the geometry optimization process, enabling the capture of finer details and better tessellation.<\/li>\n\n\n\n
  2. Adaptive TetSphere Modification<\/strong>: We developed adaptive mechanisms for splitting and merging tetrahedral spheres, enhancing flexibility and detail in the optimization process.<\/li>\n<\/ol>\n\n\n\n

    Setting up the Code Base:<\/h3>\n\n\n\n

    Our mentor, Minghao, facilitated the setup of the codebase in a cloud environment equipped with GPU support. Using a VSCode tunnel, we were able to work remotely and make modifications within the same environment. Following the instructions in the TetSphere Splatting GitHub README file, we successfully initialized the TetSpheres and executed the TetSphere Splatting process for our input images, including those of a white dog and a cartoon boy.<\/p>\n\n\n\n

    \n
    \"\"
    Input Image of a White Dog<\/figcaption><\/figure>\n\n\n\n
    \"\"
    TetSphere Initialization<\/figcaption><\/figure>\n\n\n\n
    \"\"
    Tetsphere Splatting<\/figcaption><\/figure>\n<\/figure>\n\n\n\n
    \n
    \"\"
    Input Image of a Cartoon Boy<\/figcaption><\/figure>\n\n\n\n
    \"\"
    Tetsphere Initialization<\/figcaption><\/figure>\n\n\n\n
    \"\"
    Tetsphere Splatting<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

    Geometry Optimization:<\/h3>\n\n\n\n

    To improve the geometry optimization in TetSphere Splatting, we explored various algorithms for adaptive remeshing and subdivision. Initially, we considered parametrization-based remeshing using Centroidal Voronoi Tessellation (CVT) but opted for direct surface remeshing due to its efficiency and simplicity. For subdivision, we chose the Loop subdivision algorithm, as it is specifically designed for triangle meshes and better suited for our application compared to Catmull-Clark.<\/p>\n\n\n\n

    Implementing Loop Subdivision:<\/h3>\n\n\n\n

    In the second week of the project, I worked on implementing the loop subdivision algorithm and incorporating it into the Geometry Optimization pipeline. Below is a detailed explanation of the algorithm and the method of implementation using the example of the example of the white dog mesh after TetSphere splatting.<\/p>\n\n\n\n

    1. Importing the required libraries, loading and initializing the mesh:<\/h5>\n\n\n\n

    Instead of working throughout with numpy arrays, I chose to use defaultdict, a subclass of Python dictionaries to reduce the running time of the program. unique_edges defined below ensures that the list of edges doesn’t count each edge twice, once as (e1<\/sub>, e2<\/sub>) and once as (e2<\/sub>, e1<\/sub>).<\/p>\n\n\n\n

    import trimesh\nimport numpy as np\nfrom collections import defaultdict\n\ninit_mesh = trimesh.load(\".\/results\/a_white_dog\/final\/final_surface_mesh.obj\")\nV = init_mesh.vertices\nF = init_mesh.faces\nE = init_mesh.edges\n\ndef unique_edges(edges): \n    sorted_edges = np.sort(edges, axis=1)\n    unique_edges = np.unique(sorted_edges, axis=0)\n    return unique_edges<\/code><\/pre>\n\n\n\n
    2. Edge-Face Adjacency Mapping<\/strong><\/h5>\n\n\n\n
    This maps edges to the faces they belong to, which will later help to identify boundary and interior edges.<\/p>\n\n\n\n
    def compute_edge_face_adjacency(faces):\n    edge_face_map = defaultdict(list)\n    for i, face in enumerate(faces):\n        v0, v1, v2 = face\n        edges = [(min(v0, v1), max(v0, v1)),\n                 (min(v1, v2), max(v1, v2)),\n                 (min(v2, v0), max(v2, v0))]\n        for edge in edges:\n            edge_face_map[edge].append(i)\n    return edge_face_map<\/code><\/pre>\n\n\n\n
    3. Computing New Vertices:<\/strong><\/h5>\n\n\n\n
    If an edge is a boundary edge (it belongs to only one face), then the new vertex is simply the midpoint of the two vertices at the ends of the edge. If not, the edge belongs to two faces and the new vertex is found to be a weighted average of the four vertices which make the two faces, with a weight of 3\/8 for the vertices on the edge and 1\/8 for the vertices not on the edge. The function also produces indices for the new vertices.<\/p>\n\n\n
    \n
    \"\"
    (a) The weights for an interior edge; (b) the weights for a boundary edge.                 Image from https:\/\/www.pbr-book.org\/3ed-2018\/Shapes\/Subdivision_Surfaces<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n
    def compute_new_vertices(edge_face_map, vertices, faces):\n    new_vertices = defaultdict(list)\n    i = vertices.shape[0] - 1\n    for edge, facess in edge_face_map.items():\n        v0, v1 = edge\n        if len(facess) == 1:  # Boundary edge\n            i += 1\n            new_vertices[edge].append(((vertices[v0] + vertices[v1]) \/ 2, i))\n        elif len(facess) == 2:  # Internal edge\n            i += 1\n            adjacent_vertices = []\n            for face_index in facess:\n                face = faces[face_index]\n                for vertex in face:\n                    if vertex != v0 and vertex != v1:\n                        adjacent_vertices.append(vertex)\n            v2 = adjacent_vertices[0]\n            v3 = adjacent_vertices[1]\n            new_vertices[edge].append(((1 \/ 8) * (vertices[v2] + vertices[v3]) + \n                                       (3 \/ 8) * (vertices[v0] + vertices[v1]), i))\n    return new_vertices<\/code><\/pre>\n\n\n\n
    4. Create a dictionary of updated vertices:<\/h5>\n\n\n\n
    This generates updated vertices that include both old and newly created vertices.<\/p>\n\n\n\n
    def generate_updated_vertices(new_vertices, vertices, edges):\n    updated_vertices = defaultdict(list)\n    for i in range(vertices.shape[0]):\n        vertex = vertices[i, :]\n        updated_vertices[i].append(vertex)\n    for i in range(edges.shape[0]):\n        e1, e2 = edges[i, :]\n        edge = (min(e1, e2), max(e1, e2))\n        vertex, index = new_vertices[edge][0]\n        updated_vertices[index].append(vertex)\n    return updated_vertices<\/code><\/pre>\n\n\n\n
    5. Construct New Faces:<\/h5>\n\n\n\n
    This constructs new faces using the updated vertices. Each old face is split into four new faces.<\/p>\n\n\n
    \n
    \"\"
    The four child faces created, ordered such that the ith child face is adjacent to the ith vertex of the original face and the fourth child face is in the center of the subdivided face. Image from https:\/\/www.pbr-book.org\/3ed-2018\/Shapes\/Subdivision_Surfaces<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n
    def create_new_faces(faces, new_vertices):\n    new_faces = []\n    for face in faces:\n        v0, v1, v2 = face\n        edge_0 = tuple(sorted((v0, v1)))\n        edge_1 = tuple(sorted((v1, v2)))\n        edge_2 = tuple(sorted((v2, v0)))\n\n        e0 = new_vertices[edge_0][0][1]\n        e1 = new_vertices[edge_1][0][1]\n        e2 = new_vertices[edge_2][0][1]\n\n        new_faces.append([v0, e0, e2])\n        new_faces.append([v1, e1, e0])\n        new_faces.append([v2, e2, e1])\n        new_faces.append([e0, e1, e2])\n    \n    return np.array(new_faces)<\/code><\/pre>\n\n\n\n
    6. Modifying Old Vertices Based on Adjacency:<\/h5>\n\n\n\n
    First, the function compute_vertex_adjacency finds the neighbors of each vertex. Then modify_vertices weights each of the neighbor vertices of each old vertex by a weight\u00a0\u03b2 (defined in the code below) and weights the old vertex by 1-n\u03b2, where n is the degree of the old vertex. It does not change the new vertices.<\/p>\n\n\n
    \n
    \"\"
    Computing the new position of the old vertex v. Image from https:\/\/www.pbr-book.org\/3ed-2018\/Shapes\/Subdivision_Surfaces<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n
    def compute_vertex_adjacency(faces):\n    vertex_adj_map = defaultdict(set)\n    for face in faces:\n        v0, v1, v2 = face\n        vertex_adj_map[v0].add(v1)\n        vertex_adj_map[v0].add(v2)\n        vertex_adj_map[v1].add(v0)\n        vertex_adj_map[v1].add(v2)\n        vertex_adj_map[v2].add(v0)\n        vertex_adj_map[v2].add(v1)\n    return vertex_adj_map\n\ndef modify_vertices(vertices, updated_vertices, vertex_adj_map):\n    modified_vertices = defaultdict(list)\n    for i in range(len(updated_vertices)):\n        if i in range(vertices.shape[0]):\n            vertex = vertices[i,:]\n            neighbors = vertex_adj_map[i]\n            n = len(neighbors)\n            beta = (5 \/ 8 - (3 \/ 8 + 1 \/ 4 * np.cos(2 * np.pi \/ n)) ** 2)\/n\n            weight_v = 1 - beta * n\n            modified_vertex = weight_v * vertex\n            for neighbor in neighbors:\n                neighbor_point = updated_vertices[neighbor][0]\n                modified_vertex += beta * neighbor_point\n            modified_vertices[i].append(modified_vertex)\n        else:\n            modified_vertices[i].append(updated_vertices[i][0])\n    return modified_vertices<\/code><\/pre>\n\n\n\n
    7. Generating the Final Subdivided Mesh:<\/h5>\n\n\n\n
    Now, all that is left is to collect the modified vertices and faces as numpy arrays and create the final subdivided mesh. The function loop_subdivision_iter simply applies the loop subdivision iteratively ‘n’ times, further refining the loop each time.<\/p>\n\n\n\n
    def create_vertices_array(modified_vertices):\n    vertices_list = [modified_vertices[i][0] for i in range(len(modified_vertices))]\n    return np.array(vertices_list)\n\ndef loop_subdivision(vertices, faces, edges):\n    edges = unique_edges(edges)\n    edge_face_map = compute_edge_face_adjacency(faces)\n    new_vertices = compute_new_vertices(edge_face_map, vertices, faces)\n    updated_vertices = generate_updated_vertices(new_vertices, vertices, edges)\n    new_faces = create_new_faces(faces, new_vertices)\n    vertex_adj_map = compute_vertex_adjacency(new_faces)\n    modified_vertices = modify_vertices(vertices, updated_vertices, vertex_adj_map)\n    vertices_array = create_vertices_array(modified_vertices)\n    return vertices_array, new_faces\n\ndef loop_subdivision_iter(vertices, faces, n):\n    def unique_edges(faces): \n            edges = np.vstack([\n            np.column_stack((faces[:, 0], faces[:, 1])),\n            np.column_stack((faces[:, 1], faces[:, 2])),\n            np.column_stack((faces[:, 2], faces[:, 0]))\n                ])\n            sorted_edges = np.sort(edges, axis=1)\n            unique_edges = np.unique(sorted_edges, axis=0)\n            return unique_edges\n    if n == 0:\n        edges = unique_edges(faces)\n        return loop_subdivision(V, F, E)\n    else:\n        edges = unique_edges(faces)\n        vertices, faces = loop_subdivision(vertices, faces, edges)\n        output = loop_subdivision_iter(vertices, faces, n-1)\n        return output\n\nnew_V, new_F = loop_subdivision_iter(V, F, 1)\nsubdivided_mesh = trimesh.Trimesh(vertices=new_V, faces=new_F)\nsubdivided_mesh.export(\".\/a_white_dog_subdivided.obj\")\n<\/code><\/pre>\n\n\n\n
    Results:<\/h3>\n\n\n\n
    I used the above code to obtain the subdivided meshes for four input meshes – a white dog, a cartoon boy, a camel, and a horse. The results for each of these are displayed below.<\/p>\n\n\n\n
    \n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n
    Subdivision of the white dog mesh<\/figcaption><\/figure>\n\n\n\n
    \n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n
    Subdivision of the cartoon boy mesh<\/figcaption><\/figure>\n\n\n\n
    \n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n
    Subdivision of the camel mesh<\/figcaption><\/figure>\n\n\n\n
    \n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n
    Subdivision of the horse mesh<\/figcaption><\/figure>\n\n\n\n
    Conclusion: <\/h3>\n\n\n\n
    The project successfully enhanced the TetSphere Splatting method through geometry optimization and adaptive TetSphere modification. The implementation of Loop subdivision refined mesh detail and smoothness, as evidenced by the improved results for the test meshes. The adaptive mechanisms introduced greater flexibility, contributing to more precise and detailed 3D reconstructions. <\/p>\n\n\n\n
    Through this project I learned a lot about TetSphere Splatting, Geometry Optimization and Loop Subdivion. I am very grateful to Minghao and Shanthika who supported and guided us in our progress.<\/p>\n\n\n\n
    References:<\/h3>\n\n\n\n
    {1} Guo, Minghao, et al. “TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes”, https:\/\/doi.org\/10.48550\/arXiv.2405.20283<\/a><\/p>\n\n\n\n
    {2} Kerbl, Bernhard, et al. “3D Gaussian Splatting 
    for Real-Time Radiance Field Rendering” SIGGRAPH 2023<\/em>,     https:\/\/doi.org\/10.48550\/arXiv.2308.04079<\/a>.<\/p>\n\n\n\n
    {3} “Remeshing I”, https:\/\/graphics.stanford.edu\/courses\/cs468-12-spring\/LectureSlides\/13_Remeshing1.pdf<\/a><\/p>\n\n\n\n
    {4} “Subdivision”, https:\/\/www.cs.cmu.edu\/afs\/cs\/academic\/class\/15462-s14\/www\/lec_slides\/Subdivision.pdf<\/a><\/p>\n\n\n\n
    {5} Pharr, Matt, et al. “Physically Based Rendering: From Theory To Implementation<\/em> – 3.8 Subdivision Surfaces”, https:\/\/www.pbr-book.org\/3ed-2018\/Shapes\/Subdivision_Surfaces<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
    SGI Fellows: Harini Rammohan, Dianlun Luo, Idil Sulo Project Mentor: Minghao Guo SGI Volunteer: Shanthika Naik Introduction: TetSphere Splatting [Guo et al., 2024] is a cutting-edge technique for high-quality 3D shape reconstruction using tetrahedral meshes. This method stands out by delivering superior geometry without relying on neural networks or post-processing. Unlike traditional Eulerian methods, TetSphere Splatting […]<\/p>\n","protected":false},"author":73,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37],"tags":[],"ppma_author":[17],"class_list":["post-1586","post","type-post","status-publish","format-standard","hentry","category-research"],"authors":[{"term_id":17,"user_id":0,"is_guest":1,"slug":"cap-harini","display_name":"harini","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","author_category":"","first_name":"","last_name":"","user_url":"","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/1586","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/users\/73"}],"replies":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/comments?post=1586"}],"version-history":[{"count":5,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/1586\/revisions"}],"predecessor-version":[{"id":3646,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/1586\/revisions\/3646"}],"wp:attachment":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/media?parent=1586"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/categories?post=1586"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/tags?post=1586"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/ppma_author?post=1586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}