{"id":1113,"date":"2022-09-15T20:53:44","date_gmt":"2022-09-15T20:53:44","guid":{"rendered":"http:\/\/summergeometry.org\/sgi2022\/?p=1113"},"modified":"2022-09-16T13:46:12","modified_gmt":"2022-09-16T13:46:12","slug":"how-to-train-your-dragon-using-a-siren","status":"publish","type":"post","link":"https:\/\/summergeometry.org\/sgi2022\/how-to-train-your-dragon-using-a-siren\/","title":{"rendered":"How to Train Your Dragon Using a SIREN"},"content":{"rendered":"\n

A theoretical walkthrough INRs methods for unoriented point clouds<\/em><\/strong><\/p>\n\n\n\n

By Alisia Lupidi, Ahmed Elhag and Krishnendu Kar<\/em><\/p>\n\n\n\n

Hello SGI Community, \ud83d\udc4b
We are the Implicit Neural Representation (INR) team, Alisia Lupidi, Ahmed Elhag, and Krishnendu Kar, under the supervision of Dr. Dena Bazazian and Shaimaa Monem Abdelhafez. In this article, we would like to introduce you to the theoretical background for our project.
We based our work on two papers: one about Sinusoidal Representation Network<\/em> (SIREN[1]<\/sup> ) and one about Divergence Guided Shape Implicit Neural Representation for unoriented point clouds<\/em> (DiGS[2]<\/sup>), but before presenting these we added a (simple and straightforward) prerequisites section where you can find all the theoretical background needed to understand the papers.
Enjoy!<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

0.<\/strong> Prerequisites<\/strong><\/p>\n\n\n\n

0.1 Implicit Neural Representation (INR)<\/strong><\/p>\n\n\n\n

Various paradigms have been used to represent a 3D object predicted by neural networks – including voxels, point clouds, and meshes. All of these methods are discrete and therefore pose several limitations – for instance, they require a lot of memory which limits the resolution of the 3D object predicted.

The concept of implicit neural representation (INR) for 3D representation has recently been introduced and uses a signed distance function (SDF) to represent 3D objects implicitly. It is a type of regression problem and consists of encoding a continuous differential signal within a neural network. With INR, we can provide a high resolution for a 3D object with a lower memory footprint, despite the limitations of discrete methods.<\/p>\n\n\n\n


0.2 Signed Distance Function (SDF)<\/strong><\/p>\n\n\n\n

The signed distance function[5]<\/sup> of a set in \u03a9 is a metric space that determines the distance of a given point \ud835\udccd from the boundary of \u03a9.
The function has positive values for \u2200\ud835\udccd \u2208 \u03a9, and this value decreases as \ud835\udccd approaches the boundary of \u03a9. On the boundary, the signed distance function is exactly 0 and it takes negative values outside of \u03a9.

0.3<\/strong> Rectified Linear Unit (ReLU)<\/strong><\/p>\n\n\n\n

The Rectified Linear Unit (ReLU, Fig. 1) is the most commonly used activation function in deep learning models. The function returns 0 if it receives any negative input, but for any positive value \ud835\udccd, it returns that value. It is defined by f(\ud835\udccd) = max(0, \ud835\udccd)<\/em>, where \ud835\udccd<\/a> is an input to a neuron.<\/p>\n\n\n

\n
\"\"
Figure 1: ReLU Plo<\/strong>t[3]<\/sup><\/figcaption><\/figure>\n<\/div>\n\n\n
ReLU pros<\/strong><\/td>ReLU cons<\/strong><\/td><\/tr>
Extremely easy to implement (just a max function), unlike the Tanh and the Sigmoid activation functions that require exponential calculation<\/td>
Dying ReLU: for all negative inputs, ReLU gives 0 as output – a ReLU neuron is \u201cdead\u201d if it is stuck in the negative side (not recoverable).

<\/td><\/tr>
The rectifier function behaves like a linear activation function<\/td>Non-differentiable at zero<\/td><\/tr>
Can output a true zero value<\/td>Not zero-centered and unbounded<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n


0.4<\/strong> Multilayer Perceptron (MLP)<\/strong> <\/p>\n\n\n\n

A multilayer perceptron (MLP)[6]<\/sup> is a fully connected class of feedforward<\/a> artificial neural network<\/a> (ANN) that utilizes backpropagation<\/a> for training (supervised learning). It has at least three layers<\/a> of nodes: an input layer<\/a>, a hidden layer<\/a> and an output layer<\/a>, and except for the first layer, each node is a neuron that uses a nonlinear activation function<\/a>.

A perceptron is a linear classifier <\/em>that takes in input \ud835\udcce = \u2211<\/a> W\u1d62<\/em>\ud835\udccd\u1d62<\/em> + b\u1d62<\/em> <\/em>where \ud835\udccd<\/em><\/a> is the feature vector, W<\/em><\/a> are the weights, and b<\/em><\/a> the bias. This input, in some cases such as SIREN, is passed to an activation function to produce the bias (see equation 5).<\/p>\n\n\n\n

1.SIREN<\/strong><\/p>\n\n\n\n

1.1 Sinusoidal Representation Network (SIREN)<\/strong><\/p>\n\n\n\n

SIRENs[1]<\/sup> are neural networks (NNs) used for signals\u2019 INRs. In comparison with other networks\u2019 architectures, SIRENs can represent a signal\u2019s spatial and temporal derivatives, thus conveying a greater level of detail when modeling a signal, thanks to their sine-based activation function.<\/p>\n\n\n\n

1.2 How does SIREN work?<\/strong><\/p>\n\n\n\n

We want to learn a function \u03a6 that satisfies the following equation and is limited by a set of constraints[1]<\/sup>,<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

This above is the formulation of an implicit neural representation where F<\/em> is our NN architecture and it takes as input the spatio-temporal coordinates \ud835\udccd \u2208 \u211dm<\/sup> and the derivatives of \u03a6 with respect to \ud835\udccd. This feasibility problem can be summarized as[1]<\/sup>,<\/p>\n\n\n\n

<\/p>\n\n\n\n

As we want to learn \u03a6<\/a>, it makes sense to cast this problem in a loss function to see how well we accomplish our goal. This loss function penalizes deviations from the constraints on their domain \u03a9m<\/sub><\/em>[1]<\/sup>,<\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Here we have the indicator function<\/em> 1\u03a9m<\/sub><\/sub>= 1 if \ud835\udccd \u2208 \u03a9m<\/sub><\/em> and otherwise equal to 0.<\/em><\/p>\n\n\n\n

All points \ud835\udccd\u1d62<\/em><\/em> are mapped to \u03a6(\ud835\udccd<\/em>)<\/a> in a way that minimizes its deviation from the sampled value of the constraint a\u1d62(\ud835\udccd)<\/em>. The dataset D = {(\ud835\udccd\u1d62, a\u1d62(\ud835\udccd))}\u1d62<\/em> is sampled dynamically at training time using Monte Carlo integration. The aim of doing this is to improve the approximation of the loss L<\/em> as the number of samples grows.<\/p>\n\n\n\n

To calculate \u03a6<\/a>, we use \u03b8<\/em><\/a> parameters and solve the resulting optimization problem using gradient descent.[1]<\/sup><\/p>\n\n\n\n

<\/p>\n\n\n\n

Here \u03b8<\/em>i<\/em><\/sub> \u2236 \u211d\u2192 \u211dNi<\/sub> <\/sup> <\/em>is the ith<\/sup><\/em><\/a> layer of the network. It consists of the affine transform defined by the weight matrix W<\/em>i<\/em><\/sub> \u2208<\/em> \u211dNi<\/sub> <\/em>x Mi<\/sub><\/em><\/sup> and the biases b<\/i>i<\/em><\/sub> \u2208<\/em> \u211dNi<\/sub><\/em><\/sup><\/a> applied on the input \ud835\udccd\u1d62<\/em><\/em> \u2208<\/em> \u211dMi<\/sub><\/em><\/sup><\/a>. The sine acts as the activation function and it was chosen to achieve greater resolution: a sine activation function ensures that all derivatives are never null regardless of their order (the sine\u2019s derivative is a cosine, whereas a polynomial one would approach zero in a number of derivations related to its grade). A polynomial would thus lose higher order frequencies and therefore render a model with less information.<\/p>\n\n\n\n

1.3 Examples and Experiments<\/strong><\/p>\n\n\n\n

In the paper[1]<\/sup>, the authors compared SIREN performances to the most popular network architectures used-like Tanh, ReLU, Softplus, etc. Conducting experiments on different signals (picture, sound, video, etc), we can see that SIREN outperforms all the other methods by a significant margin (i.e., it reconstructs the signal with higher accuracy, Fig. 2). <\/p>\n\n\n

\n
\"\"
Figure 2: SIREN performance compared to other INRs.[1]<\/sup><\/figcaption><\/figure>\n<\/div>\n\n\n

To appreciate all of SIREN\u2019s potential, it is worth mentioning that we can look at the experiment the authors did on an oriented point cloud representing a 3D statue and on shape representation with differentiable signed distance functions (Fig. 3).<\/p>\n\n\n

\n
\"\"
Figure 3: SIREN reconstruction of a high detailed statue compared to ReLU ones.[5]<\/sup><\/figcaption><\/figure>\n<\/div>\n\n\n

By applying SIREN to this problem, they were able to reconstruct the whole object while accurately reproducing even the finer details on the elephant and woman\u2019s bodies.<\/p>\n\n\n\n

The whole scene is stored in the weights of a single 5-layer NN.
Compared to recent procedures (like combining voxel grids with neural implicit representations) with SIREN, we have no 2D or 3D convolutions and fewer parameters.<\/p>\n\n\n\n

What gives SIREN an edge over other architectures is that: it does not compute SDFs using supervised GT SDF or occupancy values. Instead, it requires supervision in the gradient domain, so even though it is a harder problem to solve, it results in a better model and more efficient approach.<\/p>\n\n\n\n

1.4 Conclusion on SIREN<\/strong><\/p>\n\n\n\n

SIREN is a new and powerful method in the INR world that allows 3D image reconstructions with greater precision, an increased amount of details, and smaller memory usage.
All of this has been possible thanks to a smart choice of the activation function: using the sine gives an edge on the other networks (Tanh, ReLU, Sigmoid, etc) as its derivatives are never null and retrieve information even at higher frequency levels (higher order derivatives).<\/p>\n\n\n\n

2.<\/strong> DiGS<\/strong><\/p>\n\n\n\n

2.1<\/strong> Divergence Guided Shape Implicit Neural Representation for unoriented point clouds (DiGS)<\/strong><\/p>\n\n\n\n

Now that we know what SIREN is, we present DiGS[2]<\/sup>: a method to reconstruct surfaces from point clouds that have no normals.<\/p>\n\n\n\n

Why are normals important for INR?<\/em> When normal vectors are available for each point in the cloud, a higher fidelity representation can be learned. Unfortunately, most of the time normals are not provided with the raw data. DiGS is significant because it manages to render objects with high accuracy in all cases, thus bypassing the need for normals as a prior.<\/p>\n\n\n\n

2.2<\/strong> Training SIRENs without normal vectors<\/strong><\/p>\n\n\n\n

The problem has now changed into training a SIREN without supplying the shape\u2019s normals beforehand. <\/p>\n\n\n\n

Looking at the gradient vector field of the signed distance function produced by the network, we can see that it has low divergence nearly everywhere except for sink points – like in the center of the circle in Fig. 4. By incorporating this geometric prior as a soft constraint in the loss function, DiGS reliably orients gradients to match the unknown normals at each point and in some cases this produces results that are even better than those produced by approaches that use directly GT normals.<\/p>\n\n\n

\n
\"\"
Figure 4: DiGS applied to a 2D example. [2]<\/sup><\/figcaption><\/figure>\n<\/div>\n\n\n

To do this, DiGS requires a network architecture that has continuous second derivatives<\/em>, such as SIRENs, and an initial zero level set<\/em> that is approximately spherical. To achieve the latter, two new geometric initialization methods are proposed: the initialization to a sphere and the Multi-Frequency Geometric Initialization (MFGI). The MFGI is better than the first because the first keeps all activations within the first period of the sinusoidal activation function and this will not generate high-frequency outputs, thus losing some detail. The second one, however, introduced controlled high frequencies into the first layer of the NN, and even though is a noisier method it solves the aforementioned problem.<\/p>\n\n\n\n

To train SIRENS with DiGS, firstly we need a precise geometric initialization. Initializing the shape to a sphere biases the function to start with an SDF that is positive away from the object and negative in the center of the object\u2019s bounding box, while keeping the model\u2019s ability to have high frequencies (in a controllable manner).
Progressively, finer and finer details are added into consideration, passing from a smooth surface to a coarse shape with edges. This step-by-step approach allows the model to learn a function that has smoothly changing normals and that is interpolated as much as possible with the original point cloud samples.<\/p>\n\n\n\n

The original SIREN loss function includes:<\/p>\n\n\n\n