{"id":3520,"date":"2024-11-21T20:43:46","date_gmt":"2024-11-21T20:43:46","guid":{"rendered":"https:\/\/summergeometry.org\/sgi2024\/?p=3520"},"modified":"2024-11-21T20:45:13","modified_gmt":"2024-11-21T20:45:13","slug":"pseudo-rendering-a-novel-deep-learning-approach-for-2d-cortical-mesh-segmentation","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2024\/pseudo-rendering-a-novel-deep-learning-approach-for-2d-cortical-mesh-segmentation\/","title":{"rendered":"Pseudo Rendering: A Novel Deep Learning Approach for 2D Cortical Mesh Segmentation"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>Mentor:<a href=\"https:\/\/sites.google.com\/site\/karthikharitz\/\">Dr. Karthik Gopinath<\/a><\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Volunteer Mentor:<\/em> <em>Kyle <\/em>Onghai.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Fellows: Sergius Justus Nyah, Nicolas Pigadas, Mutiraj Laksanawisit<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"824\" height=\"263\" src=\"https:\/\/summergeometry.org\/sgi2024\/wp-content\/uploads\/2024\/11\/image-3.png\" alt=\"\" class=\"wp-image-3635\" srcset=\"https:\/\/summergeometry.org\/sgi2024\/wp-content\/uploads\/2024\/11\/image-3.png 824w, https:\/\/summergeometry.org\/sgi2024\/wp-content\/uploads\/2024\/11\/image-3-300x96.png 300w, https:\/\/summergeometry.org\/sgi2024\/wp-content\/uploads\/2024\/11\/image-3-768x245.png 768w\" sizes=\"auto, (max-width: 824px) 100vw, 824px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Picture 1: A high-level of the 2D projection descriptor pipeline we propose. (1) Extraction of multiple views and multiple descriptors from the 3D shape; (2) The extracted descriptors can be: normals, depth and curvature; (3) Input of the descriptors into a multi-view network; (4) Segmentation of the views (5) 3D reconstruction of the 3D segmented shape.<\/em><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Abstract<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Labeling brain surfaces is vital to many aspects of neuroscience and medicine, but to do so manually is laborious and time-consuming.&nbsp; An automated method would streamline the process, enabling more efficient analysis and interpretation of neuroimaging data. Such a task aligns with the longstanding inquiry in computer vision: is it more effective to use 3D shape representations or would 2D projection descriptor approaches yield better understanding of the shape? In this work, we explore the 2D approach of this question. We propose an automated end-to-end pipeline, structured into four main phases: selection of views for 2D projection extraction, rendering of the cortical mesh from multiple perspectives, segmentation of these projections and inverse rendering to map 2D segmentations back to 3D, integrating multiple views (Picture 1)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Definitions of Basic Project-Related Terms<\/strong>:<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Pseudo-Rendering<\/strong>:&nbsp; This is a process whereby a 3D model, such as a cortical mesh (in our case), is projected into 2D images from multiple perspectives. This process involves a transformation from the 3D coordinate space to the 2D image plane. The perspectives can be defined by a virtual camera&#8217;s position and orientation relative to the 3D model. The resulting 2D images retain depth information from the 3D model, hence bringing forth the perception of three-dimensionality.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>2D Segmentation<\/strong>: 2D Segmentation is the process of dividing a 2D image into distinct regions based on pixel characteristics such as color, intensity, or texture. The segmentation method, which can include techniques like thresholding, clustering, watershed, and edge-based methods, determines how these regions are defined. For example, in an image of an airplane in the sky, one region might be the blue sky and another the white airplane. Similarly, an image of a chair could be segmented into regions representing the chair&#8217;s legs, seat, and backrest. Post-processing steps may be applied to refine these regions. The success of the segmentation can be evaluated using metrics like pixel accuracy, Intersection over Union (IoU), and Dice coefficient (a measure of the performance of Segmentation algorithms).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Cortical Mesh<\/strong>: This is a 3D model that represents the outer surface of the brain (cerebral cortex), usually obtained from Magnetic Resonance Imaging data.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Parcellation<\/strong>: This refers to the process of dividing cortical meshes, typically derived from brain imaging data, into distinct regions or parcels. These parcels often represent functionally or structurally distinct areas of the brain. Its purpose here is to simplify the analysis of brain imaging data by reducing the complexity of the data and focusing on regions of interest. <br><\/li>\n<\/ul>\n\n\n\n<h5 class=\"wp-block-heading\">Initial Steps: <\/h5>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Load and visualize Brain surfaces with FreeSurfer [<a href=\"https:\/\/surfer.nmr.mgh.harvard.edu\/fswiki\/DownloadAndInstall\">Link<\/a>]&nbsp;<img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"253\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXegt1M0nek9oSIHiveLTBrCHZv1jEL6ed7bhlB9E9RaMbn8cpUs0hAuL2aHE6hjBiV6_XlkTG50Rnuyf2x3wkL8lb-tKFNbWs2PC7s_R1QZGchmZPfoYPYd0xUwKBC8VrGOsVOc8Da_vdWDKW5yQqgBUJwt?key=8CPym_mI_TuFGxTS3jna6A\"><\/li>\n\n\n\n<li>Compute Normals, Curvature, and Point clouds from the Mesh surface<img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"305\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXc1x-0KHyqmRAU5gwOjovbSWs1SSjkpMSO76vxjY1fCyJgJnrR7c11oXzkQj1xsNHjjGx5_YoUOwanLAe7uSPLGk54EZ-bAssRT3aZYyREyMUHpola_kuDNClpRFEAgJx0V-7JriIxivE0trioErTR1IGhg?key=8CPym_mI_TuFGxTS3jna6A\"><\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\">Method: <\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><em><strong>Selecting camera views to Extract 2D Images.<\/strong><\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The selection of camera views is a critical step which involves determining the optimal perspectives from which to project the 3D cortical mesh onto 2D planes. Our goal here is to capture the most informative views that will facilitate accurate segmentation and subsequent inverse rendering.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We will start adopting a systematic approach to select six canonical views: Front, Bottom, Top, Right, Back, and Left. These views are chosen to ensure comprehensive coverage of the cortical surface, capturing its intricate geometry from multiple angles.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The selection process begins by computing the intrinsic matrix for the camera, which is used to simulate the camera&#8217;s perspective. The intrinsic matrix is calculated using the following function in Python:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def compute_intmat(img_width, img_height):\n\n    intmat = np.eye(3)\n\n    # Fill the diagonal elements with appropriate values\n    np.fill_diagonal(intmat, &#091;-(img_width + img_height) \/ 1, -(img_width + img_height) \/ 1, 1])\n\n    # Set the last column of the matrix for image centering\n    intmat&#091;:,-1] = &#091;img_width \/ 2, img_height \/ 2, 1]\n\n    return intmat<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Next, we&#8217;ll create external transformation matrices to align the camera with the six predefined views. These matrices help us generate rays for ray casting, allowing us to simulate what the camera would see from each perspective. We&#8217;ll use the pinhole camera model to generate these rays.<br><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def generate_maps(mesh, labels, intmat, extmat, img_width, img_height, rotation_matrices, recompute_normals):\n    assert isinstance(mesh, o3d.t.geometry.TriangleMesh)\n    assert isinstance(labels, np.ndarray) and labels.shape == (mesh.vertex.normals.shape&#091;0],)\n    assert isinstance(intmat, np.ndarray) and intmat.shape == (3, 3)\n    assert isinstance(extmat, np.ndarray) and (extmat.shape == (1, 4, 4) or extmat.shape == (6, 4, 4))\n    assert isinstance(img_width, int) and img_width &gt; 0\n    assert isinstance(img_height, int) and img_height &gt; 0\n\n    if recompute_normals:\n        mesh.vertex.normals = mesh.vertex.normals@np.transpose(rotation_matrices&#091;0]&#091;:3,:3].astype(np.float32))\n        mesh.triangle.normals = mesh.triangle.normals@np.transpose(rotation_matrices&#091;0]&#091;:3,:3].astype(np.float32))\n\n    scene = o3d.t.geometry.RaycastingScene()\n    scene.add_triangles(mesh)\n\n    output_maps, labels_maps, ids_maps, vertex_maps = &#091;], &#091;], &#091;], &#091;]\n\n    for i in range(rotation_matrices.shape&#091;0]):\n        rays = scene.create_rays_pinhole(intmat, extmat&#091;i], img_width, img_height)\n        cast = scene.cast_rays(rays)\n        ids_map = np.array(cast&#091;'primitive_ids'].numpy(), dtype=np.int32)\n        ids_maps.append(ids_map)\n        hit_map = np.array(cast&#091;'t_hit'].numpy(), dtype=np.float32)\n        weights_map = np.array(cast&#091;'primitive_uvs'].numpy(), dtype=np.float32)\n        label_ids = np.argmax(np.concatenate((weights_map, 1 - np.sum(weights_map, axis=2, keepdims=True)), axis=2), axis=2)\n\n        normal_map = np.array(mesh.triangle.normals&#091;ids_map.clip(0)].numpy(), dtype=np.float32)\n        normal_map&#091;ids_map == -1] = &#091;0, 0, -1]\n        normal_map&#091;:, :, -1] = -normal_map&#091;:, :, -1].clip(-1, 0)\n        normal_map = normal_map * 0.5 + 0.5\n\n        vertex_map = np.array(mesh.triangle.indices&#091;ids_map.clip(0)].numpy(), dtype=np.int32)\n        vertex_map&#091;ids_map == -1] = &#091;-1]\n        vertex_maps.append(vertex_map)\n\n        inverse_distance_map = 1 \/ hit_map\n        coded_map_inv = normal_map * inverse_distance_map&#091;:, :, None]\n        output_map = (coded_map_inv - np.min(coded_map_inv)) \/ (np.max(coded_map_inv) - np.min(coded_map_inv))\n        output_maps.append(output_map)\n\n        labels_map = labels&#091;vertex_map.clip(0)]\n        labels_map&#091;vertex_map == -1] = -1\n        labels_map = labels_map&#091;np.arange(labels_map.shape&#091;0])&#091;:, np.newaxis], np.arange(labels_map.shape&#091;1]), label_ids]\n        labels_map = labels_map.astype('float64')\n        labels_maps.append(labels_map)\n\n    return np.array(output_maps), np.array(labels_maps)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">By casting rays from these six perspectives, we can project the entire cortical surface onto 2D planes, making sure we capture all the important features. This multi-view approach strengthens the segmentation process by reducing the chances of occlusions and giving us a more complete picture of the 3D structure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The following images illustrate the six canonical views we used in our pipeline:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdjyUkRPAnbxCoWHDG3K2HX5fVxTHXO-VP1ypjSmUqYWrerQUeI8evVvRc0lx9-f9JnST3Eo66oFYhJCtK9p7R9NSQP3mX5D5ZvukUheWtqHSqfplG6EKjRKenCPWc6wmeeqdwGzGX5U8h2FelC9syzUoxM?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now, We will accompany each 2D projection with annotations (ground truth).<br><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXfcKxqSjoTEbmJxbuMU35rmS5Bb1hB1hJ2tQ111E7QKMcsjfdOVN-085keCwBQBlUFtuSk_IUlrKUfHOCGBblx2MBskByw13c_B403pRtUcq1AuMGw5g9aX6_PZaQ-KCLYWiyCaB8o0O5CI0pRxZRurIhz7?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">These views are crucial for the next steps in our pipeline, such as rendering, segmentation, and inverse rendering. By carefully choosing and using these perspectives, we improve both the accuracy and efficiency of the automated labeling process. The six camera positions ensure that every part of the cortical surface is captured, giving us a complete set of 2D projections that can be accurately mapped back to the 3D structure. <\/p>\n\n\n\n<h4 class=\"wp-block-heading\">2D Projections: Annotations and Curvature<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">To continue, we take the 2D projections obtained from the six camera views and perform annotations and curvature calculations, which are essential for understanding the cortical surface&#8217;s geometry and features.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Process:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Calculate the curvature of the cortical surface from the 2D projections, which helps in identifying important features and understanding the surface&#8217;s geometry.<\/li>\n\n\n\n<li>Annotate the 2D projections with relevant labels, such as different brain regions or anatomical landmarks.<\/li>\n\n\n\n<li>Use these annotations to the CNN (As seen below) for automated labeling.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Code: <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Compute per-face curvature using gpytoolbox (angle defect)\ncurvature = gpy.angle_defect(vertices, faces)\n\n\n# Debugging: Print raw curvature values\nprint(\"Raw Curvature Values:\")\nprint(curvature)\nprint(\"Curvature min:\", np.min(curvature))\nprint(\"Curvature max:\", np.max(curvature))\n\n\n# Percentile-based normalization\nlower_percentile = np.percentile(curvature, 1)\nupper_percentile = np.percentile(curvature, 99)\n\n\n# Clipping the curvature values to the 1st and 99th percentiles to diminish the effect of outliers\ncurvature_clipped = np.clip(curvature, lower_percentile, upper_percentile)\n\n\n# Normalize the clipped curvature values between 0 and 1\ncurvature_normalized = (curvature_clipped - lower_percentile) \/ (upper_percentile - lower_percentile)\n\n\n# Debugging: Print normalized curvature values\nprint(\"Normalized Curvature Values:\")\nprint(curvature_normalized)\n\n\n# Select color map\ncolor_map = plt.get_cmap('viridis')\ncurvature_colors = color_map(curvature_normalized)&#091;:, :3] # Ignore alpha channel\n\n\n# Create Open3D mesh\nmesh = o3d.geometry.TriangleMesh()\nmesh.vertices = o3d.utility.Vector3dVector(vertices)\nmesh.triangles = o3d.utility.Vector3iVector(faces)\nmesh.vertex_colors = o3d.utility.Vector3dVector(curvature_colors) # Apply colors to vertices\n\n\n# Compute normals to improve lighting in visualization\nmesh.compute_vertex_normals()\n\n\n# Visualize the mesh with curvature coloring\no3d.visualization.draw_geometries(&#091;mesh], window_name='Mesh with Curvature Colors')\n\n\n# Visualize the normalized curvature values as a histogram\nplt.figure()\nplt.hist(curvature_normalized, bins=50, color='blue', alpha=0.7)\nplt.title(\"Histogram of Normalized Curvature Values\")\nplt.xlabel(\"Normalized Curvature\")\nplt.ylabel(\"Frequency\")\nplt.show()<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Result (As seen also in (2) above):<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXc1x-0KHyqmRAU5gwOjovbSWs1SSjkpMSO76vxjY1fCyJgJnrR7c11oXzkQj1xsNHjjGx5_YoUOwanLAe7uSPLGk54EZ-bAssRT3aZYyREyMUHpola_kuDNClpRFEAgJx0V-7JriIxivE0trioErTR1IGhg?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Training the multi-view CNN<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Now we use the annotated 2D projections to train a multi-view Convolutional Neural Network (CNN). This multi-view CNN leverages the different perspectives to improve the accuracy of the labeling process.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">Process:<\/h5>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Data Preparation<\/strong>: \n<ul class=\"wp-block-list\">\n<li>Prepare the annotated 2D projections as input data for the CNN.<\/li>\n\n\n\n<li>Split the data into training, validation, and test sets.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>import os\nimport nibabel as nib\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\n\ndef load_data(data_dir):\n\n    data = &#091;]\n    labels = &#091;]\n\n    for subject_dir in os.listdir(data_dir):\n        surf_dir = os.path.join(data_dir, subject_dir, 'surf')\n        label_dir = os.path.join(data_dir, subject_dir, 'label')\n\n        if os.path.isdir(surf_dir) and os.path.isdir(label_dir):\n            # Load surface data\n            surf_file = os.path.join(surf_dir, 'lh_aligned.surf')\n            if os.path.exists(surf_file):\n                surf_data = nib.freesurfer.read_geometry(surf_file)&#091;0]\n                data.append(surf_data)\n\n            # Load label data\n            label_file = os.path.join(label_dir, 'lh.annot')\n            if os.path.exists(label_file):\n                label_data = nib.freesurfer.read_annot(label_file)&#091;0]\n                labels.append(label_data)\n\n    return np.array(data), np.array(labels)\n\n\n# Load actual data\ntrain_data, train_labels = load_data('\/home\/sergy\/cortical-mesh-parcellation\/10brainsurfaces (1)')\nval_data, val_labels = load_data('\/home\/sergy\/cortical-mesh-parcellation\/10brainsurfaces (1)')\n\n\n# Convert data to PyTorch tensors\ntrain_data = torch.tensor(train_data, dtype=torch.float32)\ntrain_labels = torch.tensor(train_labels, dtype=torch.long)\nval_data = torch.tensor(val_data, dtype=torch.float32)\nval_labels = torch.tensor(val_labels, dtype=torch.long)\n\n\n# Define custom dataset class\nclass ExampleDataset(Dataset):\n    def __init__(self, data, labels):\n        self.data = data\n        self.labels = labels\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, index):\n        return self.data&#091;index], self.labels&#091;index]\n\n\n# Create DataLoader for training and validation data\ntrain_dataset = ExampleDataset(train_data, train_labels)\nval_dataset = ExampleDataset(val_data, val_labels)\n\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2<\/strong>. <strong>Model Architecture<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Design a CNN architecture that can handle multi-view inputs.<\/li>\n\n\n\n<li>Use techniques like data augmentation to improve the model&#8217;s robustness.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>from trainCNN import MultiViewCNN\nimport torch.nn as nn\n\n# Initialize the model\nmodel = MultiViewCNN()\n\n# Define the loss function and optimizer\ncriterion = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>3. Training the CNN:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Train the CNN using the prepared data.<\/li>\n\n\n\n<li>Monitor the training process using metrics like accuracy and loss.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code># Training loop\nfor epoch in range(5):  # 5 epochs\n    model.train()\n    for i, (inputs, labels) in enumerate(train_loader):\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = criterion(outputs, labels)\n        loss.backward()\n        optimizer.step()\n\n        if i % 100 == 0:\n            print(f\"Epoch &#091;{epoch+1}\/5], Step &#091;{i+1}\/{len(train_loader)}], Loss: {loss.item():.4f}\")\n\n    # Validation loop\n    model.eval()\n    val_loss = 0.0\n    with torch.no_grad():\n        for inputs, labels in val_loader:\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            val_loss += loss.item()\n\n    val_loss \/= len(val_loader)\n    print(f\"Validation Loss after Epoch &#091;{epoch+1}\/5]: {val_loss:.4f}\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Training result: <\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXctUxysJZMUJGuOojjsnlhC2eVQ52w1UHWiqGpal9Wv1wVqfsm5lBqY2Rk0R1vV-f4JL8ZvHSKsghwTkx5zoMMyf_sjTc6DnIrtwYRm3av0ilq1JiwzkJ2LWi2ISR5HUclP9w9q?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Analysis of Training Results<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em><strong>Initial Loss<\/strong>: In the first epoch, the initial loss is 2.3174. This indicates that the model is starting to learn from the data, but there is still a significant difference between the predicted and actual labels.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em><strong>Subsequent Epochs<\/strong>: From the second epoch onwards, the loss drops to 0.0000. This suggests that the model has quickly learned to minimize the loss and make accurate predictions.<\/em><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3D Reconstruction: Annotations and Curvature:&nbsp;<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">To conclude, we will map the 2D annotations and curvature back to the 3D structure, which provides a comprehensive view of the cortical surface with detailed annotations and curvature information. We can divide this into 3 steps; <\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Mapping Annotations:<\/strong><\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Firstly, we will map the annotations predicted by the Multi-View CNN back to the 3D cortical surface, which involves projecting the 2D annotations onto the 3D mesh.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import numpy as np\ndef map_annotations_to_3d(annotations_2d, vertices, faces):\n    # Initialize a 3D array to store the annotations\n    annotations_3d = np.zeros(vertices.shape&#091;0])\n    # Iterate over each face and map the 2D annotations to the 3D vertices\n    for i, face in enumerate(faces):\n        for vertex in face:\n            annotations_3d&#091;vertex] = annotations_2d&#091;i]\n    return annotations_3d\n# Example usage\nannotations_2d = np.random.randint(0, 10, size=(faces.shape&#091;0],))  # Replace with actual 2D annotations\nannotations_3d = map_annotations_to_3d(annotations_2d, vertices, faces)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">2. <strong><strong>Mapping Curvature:<\/strong><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We then map the calculated curvature values from the 2D projections back to the 3D surface. This will help us in visualizing the curvature on the 3D model.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\ndef map_curvature_to_3d(curvature_2d, vertices, faces):\n    # Initialize a 3D array to store the curvature values\n    curvature_3d = np.zeros(vertices.shape&#091;0])\n    \n    # Iterate over each face and map the 2D curvature to the 3D vertices\n    for i, face in enumerate(faces):\n        for vertex in face:\n            curvature_3d&#091;vertex] = curvature_2d&#091;i]\n    \n    return curvature_3d\n\n# Define vertices and faces\nvertices = np.array(&#091;&#091;0, 0, 0], &#091;1, 0, 0], &#091;1, 1, 0], &#091;0, 1, 0]])\nfaces = np.array(&#091;&#091;0, 1, 2], &#091;0, 2, 3]])<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">3. <strong>&nbsp;Integration and Visualization<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, we will integrate the annotations and curvature into a single 3D model and use a visualization tool (Polyscope, in this case) to display the final annotated and curved 3D structure.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\n# Load annotations and curvature data\nannotations_path = '10brainsurfaces (1)\/100206\/label\/lh.annot'\ncurvature_path = 'curvature_array.npy'\nannotations_3d = nib.freesurfer.read_annot(annotations_path)&#091;0]\ncurvature_3d = np.load(curvature_path)\n# Ensure curvature_3d has the correct shape\nif curvature_3d.shape&#091;0] != vertices.shape&#091;0]:\n    curvature_3d = curvature_3d&#091;:vertices.shape&#091;0]]\n# Initialize Polyscope\nps.init()\n\n# Register the 3D mesh with Polyscope\nmesh = ps.register_surface_mesh(\"annotated_brain\", vertices, faces)\n# Add the annotations and curvature as scalar quantities\nmesh.add_scalar_quantity(\"annotations\", annotations_3d, defined_on=\"vertices\", cmap=\"viridis\")\nmesh.add_scalar_quantity(\"curvature\", curvature_3d, defined_on=\"vertices\", cmap=\"coolwarm\")\n# Show the visualization\nps.show()<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Final Results from Annotations and Mappings: <\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Our pipeline successfully achieved its goals:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The trainCNN script trained the MultiViewCNN model and saved the state dictionary.<\/li>\n\n\n\n<li>The projections.py script visualized the cortical mesh with annotations and curvature, as seen below<\/li>\n\n\n\n<li>The trained model extracted features from the 2D projections, enhancing our understanding of the cortical surface.<\/li>\n\n\n\n<li>Our goal of Pseudo Rendering was achieved. <\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Figure 1:<\/strong> Front view of brain section with labeled annotations: <\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXduHwvMyGW8QbB8uEDk2pIN-0H25mX-iI-ouPnYXiW0IwPp53kIITc2oIgqRWdgsWHz1Owl9xXuIzcKuUbs-kMpZ40LnYd0Zs6LMMm9oGBX9M30zRSL4uerezsIE3gaCOzcNJBiag?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Figure 2:<\/strong> Back view of brain section with labeled annotations: <\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdYRlCntJC9wMjy9rZpGqHQe6oWnHzzle5oJzJ60CfBI6r3TCOQDOND_z_bttkpnqksloec34Qh9HgMvy5saqmcHJAg6BTXnJ7lmc_Enb5CDD4SHyNxTp2dtqNrmiAoh7fOlgW3xg?key=8CPym_mI_TuFGxTS3jna6A\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">To conclude this long post, permit us discuss the practical use-scopes of Pseudo rendering, across diverse feilds in Science, healthcare, and Research: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pseudo-rendering enhances the visualization of complex anatomical structures, aiding in better diagnosis and treatment planning by providing detailed 3D models of organs and tissues.<\/li>\n\n\n\n<li>Enables efficient analysis of 3D data by generating 2D projections from multiple camera angles, like in the case of cortical mesh parcellation.<\/li>\n\n\n\n<li>Reduces computational resources required for rendering complex 3D models, making the process less resource-intensive.<\/li>\n\n\n\n<li>Supports interactive exploration of 3D models, allowing users to manipulate 2D projections to explore different views and perspectives. <\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Closing Remarks: <\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">At this point, we would like to express our gratitude to our amazing mentor, Dr. Karthik Gopinath, and volunteer mentor, Kyle Onghai, for their unwavering support and guidance throughout the project. Their effective guidance enabled us to rapidly develop our ideas and foster a deep passion for the project. We look forward to continuing our work on this brilliant research idea as soon as possible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thank you for reading this far! <em>\ud83c\udf89 <\/em> <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Long Live the SGI! <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em><br>  <\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Mentor:Dr. Karthik Gopinath Volunteer Mentor: Kyle Onghai. Fellows: Sergius Justus Nyah, Nicolas Pigadas, Mutiraj Laksanawisit Picture 1: A high-level of the 2D projection descriptor pipeline we propose. (1) Extraction of multiple views and multiple descriptors from the 3D shape; (2) The extracted descriptors can be: normals, depth and curvature; (3) Input of the descriptors into [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"ppma_author":[8],"class_list":["post-3520","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"authors":[{"term_id":8,"user_id":0,"is_guest":1,"slug":"cap-sergius-justus","display_name":"sergius.justus","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","author_category":"","first_name":"","last_name":"","user_url":"","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/3520","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/comments?post=3520"}],"version-history":[{"count":6,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/3520\/revisions"}],"predecessor-version":[{"id":3641,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/posts\/3520\/revisions\/3641"}],"wp:attachment":[{"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/media?parent=3520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/categories?post=3520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/tags?post=3520"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/summergeometry.org\/sgi2024\/wp-json\/wp\/v2\/ppma_author?post=3520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}