Free ell

= {{\mathcal {L}}}({\textbf{y}_j^I},{\textbf{y}_k^T}) + {\mathcal {L}}({\textbf{y}_j^I},{\textbf{y}_k^I}) + {{\mathcal {L}}}({\textbf{y}_j^T},{\textbf{y}_k^T}) + \chi {\ell _r}, \end{aligned}$$ (7) here the hyper-parameter \(\chi\) controls the contributions of \({\ell _r}\), and \({\ell _r}\) is a constraint that is introduced into CSAN to enforce the modality consistency of the multiple linear transforms. And \({\ell _r}\) is formulated as$$\begin{aligned} {\ell _r} = \frac{1}{n}\Vert {{\textbf{Y}}_I} - {{\textbf{Y}}_T}\vert {\vert _F}. \end{aligned}$$ (8) Different from \({{\mathcal {L}}}({{\textbf{v}}_j},{{\textbf{v}}_k})\), the loss \({\ell _r}\) not only minimizes the discrepancy between pairwise samples, but also eliminates the gap between unpaired samples with similar semantics. Therefore, the proposed CSAN further pushes multi-modality discrimination into the potential common subspace. Overall, the loss function for the forward generators is shown as$$\begin{aligned} {{{\mathcal {L}}}_f} = {{{\mathcal {L}}}_G} + \alpha {{{\mathcal {L}}}_{\textrm{s}}} + \beta {{{\mathcal {L}}}_C}. \end{aligned}$$ (9) In Eq. 9, the hyper-parameters \(\alpha\) and \(\beta\) are set to guarantee different loss functions within the same magnitude.3.4 Cross-coupled semantic architectureIn this paper, the cross-coupled semantic architecture is presented mainly to learn two mapping functions, so that the ‘style’ information between different modality domains can transfer to each other. As illustrated in Fig. 3, the proposed cross-coupled semantic architecture is equipped with one forward generator \({G_m}\) and one couple-generator \(G_m^c\).Fig. 3Cross-coupled semantic architecture. The scaling operation indicates that the dimension of the forward generator and the coupling generator are matched by exploiting a single linear layer to scale the representationsFull size imageIn the proposed CSAN, the learned mapping functions are trained to be cycle-consistent, that is, the mapped representations \(\textbf{X}\) from one forward generator should carry abundant domain information from another modality, while the other modality generator should also be able to bring these representations \(\textbf{X}\) backward to the original modality representations. By employing the weight sharing strategy, the couple-generator and the corresponding modality forward generator unify the learned parameters

\(M_\text {V}^T\) accordingly. Additionally let \(r_\ell ,t_\ell \in \mathbb {R}^3\), \(\ell \in {\mathcal {V}}\) be the centers of gravity (COGs) of the different ventricles, i.e., \(r_\text {LLV}\) is the COG of \(M^R_\text {LLV}\), \(t_\text {LLV}\) is the COG of \(M^T_\text {LLV}\), etc.For the registration, we then minimize the following objective function w.r.t. to deformation vector field y:$$\begin{aligned} \begin{aligned} J(R, T(y))&= \text {NGF}(R, T(y)) + \frac{\alpha }{2} \sum _{k=1}^3\Vert \Delta y_k\Vert _{L^2(\Omega )}^2 \\&\quad + \beta \int _{\Omega }^{}\psi (\det \nabla y(x))dx \\&\quad + \frac{\gamma }{2} \Big (\Vert M^T_\text {BP}(y)-M^R_\text {BP}\Vert ^2_{L^2(\Omega )} \\&\quad + \Vert M^T_\text {V}(y)-M^R_\text {V}\Vert ^2_{L^2(\Omega )}\Big ) \\&\quad + \frac{\delta }{2}\sum _{\ell \in {\mathcal {V}}} \Vert y(r_\ell )-t_\ell \Vert _2^2 \end{aligned} \end{aligned}$$ (1) with weights \(\alpha , \beta , \gamma , \delta > 0\), NGF distance measure$$\begin{aligned} \text {NGF}(R,T) = \frac{1}{2} \int _{\Omega }^{} 1 - \left( \frac{ \left\langle \nabla R(x),\nabla T(x)\right\rangle _{\varepsilon _R \varepsilon _T} }{ \Vert \nabla T(x)\Vert _{\varepsilon _T} \, \Vert \nabla R(x)\Vert _{\varepsilon _R} } \right) ^2 dx \end{aligned}$$ (2) where \( \left\langle x,y\right\rangle _{\varepsilon } := x^\top y+\varepsilon \), \(\Vert x\Vert _\varepsilon := \sqrt{\langle x,x\rangle _{\varepsilon ^2}}\) and \(\varepsilon _R,\varepsilon _T>0\) are the so-called edge-parameters controlling influence of noise in the images. The weights are fixed and were determined empirically. In addition to penalizing the second-order (Laplacian) derivatives by the so-called curvature regularization, we add an additional term penalizing the Jacobians of the deformation, respectively, volume changes with the function \(\psi (t)=(t-1)^2/t\) for \(t>0\) and \(\psi (t):=\infty \) for \(t\le 0\). Note that \(\psi (1)=0\) and \(\psi (t)=\psi (1/t)\) and thus volume growth or shrinkage are penalized symmetrically, and \(\psi (t)=\infty \) for \(\det \nabla y \le 0\) prevents local changes in the topology and thus unwanted mesh folds.The optimization is done by using a multi-level approach with L-BFGS.Multi-modal atlas registrationIn general, our approach builds on a single MR atlas that is transferred to CT as described before. However, to achieve better performance and coverage of anatomical variations, we bootstrap the MR atlas to a multi-modal MR-CT multi-atlas. To this end, all CT images in our data set (220 cases) were registered with the MR atlas intensity image and labels were propagated from MR to CT, so that we obtained a label image \(\text {CT}^\text {Label}\) for each CT scan. Afterward, we manually selected three CT images along with the propagated label images that had the highest ventricular Dice values (\(\ge

RegistrationAs the robust multi-modal, inter-subject, non-rigid registration of medical images is an extremely challenging task, we incorporate multiple structure and landmark guidance into our solution. In our method, we combine mono-modal with multi-modal atlas registration. We assumed that the creation of CT atlas images through CT-MR registration could be beneficial over the mere use of multi-modal registration, as a mono-modal approach is known to be less prone to errors, especially for inter-patient scenarios as discussed here. However, our starting point is an MRI atlas that consists of an intensity image and corresponding labels, such that \(\text {MR}(x)\) is the intensity and \(\text {MR}^\text {Label}(x)\) is the anatomical label at position x. Then, we use multi-modal registration to propagate the labels to CT. To this end, we register the intensity images and subsequently warp the labels from MR to CT. That is, we compute a deformation vector field y such that \(\text {CT}(x)\approx \text {MR}(y(x))\) and we define \(\text {CT}^\text {Label}(x):= \text {MR}^\text {Label}(y(x))\).Registration approachWe use a variational registration scheme that builds on normalized gradient fields (NGF) image similarity measure, second-order curvature and volume regularization of the deformation vector field. NGF has been proven to be a reliable distance measure in multi-modal CT-MR [19] as well as mono-modal CT-CT registration scenarios [20]. Furthermore, to improve robustness and accuracy we incorporate additional knowledge by adding penalty terms that enforce the alignment of the corresponding masks for brain and ventricles and centers of gravity (COG) of the ventricles, similar to [21, 22]. The COG could lie outside of the ventricle volume, but we are not searching for an anatomical meaningful landmark but rather a sensible reference point that can be extracted out of the ground truth that we have right now.To be specific, in our setting the CT image is the so-called fixed or reference image R and the MR is the so-called moving or template image T that shall be aligned on a domain \(\Omega \subset \mathbb {R}^3\) modeling the field-of-view of R. Furthermore, we assume corresponding segmentations for brain parenchyma (BP), left and right lateral ventricle (LLV, RLV) and fourth ventricle (FV), that are given as binary masks \(M^R_\ell , M^T_\ell \) for \(\ell =\text {BP}, \text {LLV},\text {RLV},\text {FV}\). Moreover, we consider combined masks for all ventricles, i.e., we set \(M^R_\text {V} := \sum _{\ell \in {\mathcal {V}}}M^R_\ell \) for ventricle labels \({\mathcal {V}}:=\{\text {LLV}, \text {RLV}, \text {FV}\}\) and