Volume 41, Issue 4

Cancer diagnosis, prognosis, mymargin and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion , an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion .

中文摘要翻译待生成

Most methods for medical image segmentation use U-Net or its variants as they have been successful in most of the applications. After a detailed analysis of these “traditional” encoder-decoder based approaches, we observed that they perform poorly in detecting smaller structures and are unable to segment boundary regions precisely. This issue can be attributed to the increase in receptive field si...

中文摘要翻译待生成

Deep learning-based histopathology image classification is a key technique to help physicians in improving the accuracy and promptness of cancer diagnosis. However, the noisy labels are often inevitable in the complex manual annotation process, and thus mislead the training of the classification model. In this work, we introduce a novel hard sample aware noise robust learning method for histopatho...

中文摘要翻译待生成

Computed tomography (CT) images are often impaired by unfavorable artifacts caused by metallic implants within patients, which would adversely affect the subsequent clinical diagnosis and treatment. Although the existing deep-learning-based approaches have achieved promising success on metal artifact reduction (MAR) for CT images, most of them treated the task as a general image restoration proble...

中文摘要翻译待生成

Lung cancer is the leading cause of cancer deaths worldwide. Accurately diagnosing the malignancy of suspected lung nodules is of paramount clinical importance. However, to date, the pathologically-proven lung nodule dataset is largely limited and is highly imbalanced in benign and malignant distributions. In this study, we proposed a Semi-supervised Deep Transfer Learning (SDTL) framework for ben...

中文摘要翻译待生成

Precise segmentation of teeth from intra-oral scanner images is an essential task in computer-aided orthodontic surgical planning. The state-of-the-art deep learning-based methods often simply concatenate the raw geometric attributes (i.e., coordinates and normal vectors) of mesh cells to train a single-stream network for automatic intra-oral scanner image segmentation. However, since different ra...

中文摘要翻译待生成

An increased pulse pressure, due to arteries stiffening with age and cardiovascular disease, may lead to downstream brain damage in microvessels and cognitive decline. Brain-wide imaging of the pulsatility propagation from main feeding arteries to capillaries in small animals could improve our understanding of the link between pulsatility and cognitive decline. However, it requires higher spatiote...

中文摘要翻译待生成

Despite the promise of Convolutional neural network (CNN) based classification models for histopathological images, it is infeasible to quantify its uncertainties. Moreover, CNNs may suffer from overfitting when the data is biased. We show that Bayesian–CNN can overcome these limitations by regularizing automatically and by quantifying the uncertainty. We have developed a novel technique to utiliz...

中文摘要翻译待生成

Digital reconstruction of neuronal morphologies in 3D microscopy images is critical in the field of neuroscience. However, most existing automatic tracing algorithms cannot obtain accurate neuron reconstruction when processing 3D neuron images contaminated by strong background noises or containing weak filament signals. In this paper, we present a 3D neuron segmentation network named Structure-Gui...

中文摘要翻译待生成

While Electrical Impedance Tomography (EIT) has found many biomedicine applications, better image quality is needed to provide quantitative analysis for tissue engineering and regenerative medicine. This paper reports an impedance-optical dual-modal imaging framework that primarily targets at high-quality 3D cell culture imaging and can be extended to other tissue engineering applications. The fra...

中文摘要翻译待生成

Image-guided radiation therapy (IGRT) is the most effective treatment for head and neck cancer. The successful implementation of IGRT requires accurate delineation of organ-at-risk (OAR) in the computed tomography (CT) images. In routine clinical practice, OARs are manually segmented by oncologists, which is time-consuming, laborious, and subjective. To assist oncologists in OAR contouring, we pro...

中文摘要翻译待生成

Coronary calcification is a strong indicator of coronary artery disease and a key determinant of the outcome of percutaneous coronary intervention. We propose a fully automated method to segment and quantify coronary calcification in intravascular OCT (IVOCT) images based on convolutional neural networks (CNN). All possible calcified plaques were segmented from IVOCT pullbacks using a spatial-temp...

中文摘要翻译待生成

Machine learning and deep learning methods have become essential for computer-assisted prediction in medicine, with a growing number of applications also in the field of mammography. Typically these algorithms are trained for a specific task, e.g., the classification of lesions or the prediction of a mammogram’s pathology status. To obtain a comprehensive view of a patient, models which were all t...

中文摘要翻译待生成

Spherical matrix arrays represent an advantageous tomographic detection geometry for non-invasive deep tissue mapping of vascular networks and oxygenation with volumetric optoacoustic tomography (VOT). Hybridization of VOT with ultrasound (US) imaging remains difficult with this configuration due to the relatively large inter-element pitch of spherical arrays. We suggest a new approach for combini...

中文摘要翻译待生成

The minimum histological criterion for the diagnostics of Alzheimer’s disease (AD) in tissue is the presence of senile plaques and neurofibrillary tangles in specific brain locations. The routine procedure of morphological analysis implies time-consuming and laborious steps including sectioning and staining of formalin-fixed paraffin-embedded (FFPE) tissue. We developed a multispectral Stokes pola...

中文摘要翻译待生成

This paper presents U-LanD, a framework for automatic detection of landmarks on key frames of the video by leveraging the uncertainty of landmark prediction. We tackle a specifically challenging problem, where training labels are noisy and highly sparse. U-LanD builds upon a pivotal observation: a deep Bayesian landmark detector solely trained on key video frames, has significantly lower predictiv...

中文摘要翻译待生成

Uniquely capable of simultaneous imaging of the hemoglobin concentration, blood oxygenation, and flow speed at the microvascular level in vivo, multi-parametric photoacoustic microscopy (PAM) has shown considerable impact in biomedicine. However, the multi-parametric PAM acquisition requires dense sampling and thus a high laser pulse repetition rate (up to MHz), which sets a strict limit on the ap...

中文摘要翻译待生成

Dark-field radiography of the human chest is a promising novel imaging technique with the potential of becoming a valuable tool for the early diagnosis of chronic obstructive pulmonary disease and other diseases of the lung. The large field-of-view needed for clinical purposes could recently be achieved by a scanning system. While this approach overcomes the limited availability of large area grat...

中文摘要翻译待生成

Imaging applications tailored towards ultrasound-based treatment, such as high intensity focused ultrasound (FUS), where higher power ultrasound generates a radiation force for ultrasound elasticity imaging or therapeutics/theranostics, are affected by interference from FUS. The artifact becomes more pronounced with intensity and power. To overcome this limitation, we propose FUS-net, a method tha...

中文摘要翻译待生成

We present a volumetric mesh-based algorithm for parameterizing the placenta to a flattened template to enable effective visualization of local anatomy and function. MRI shows potential as a research tool as it provides signals directly related to placental function. However, due to the curved and highly variable in vivo shape of the placenta, interpreting and visualizing these images is difficult...

中文摘要翻译待生成

We propose a novel pairwise distance measure between image keypoint sets, for the purpose of large-scale medical image indexing. Our measure generalizes the Jaccard index to account for soft set equivalence (SSE) between keypoint elements, via an adaptive kernel framework modeling uncertainty in keypoint appearance and geometry. A new kernel is proposed to quantify the variability of keypoint geom...

中文摘要翻译待生成

We had released MoNuSAC2020 as one of the largest publicly available, manually annotated, curated, multi-class, and multi-instance medical image segmentation datasets. Based on this dataset, we had organized a challenge at the International Symposium on Biomedical Imaging (ISBI) 2020. Along with the challenge participants, we had published an article summarizing the results and findings of the cha...

中文摘要翻译待生成

The MoNuSAC 2020 challenge was hosted at the ISBI 2020 conference, where the winners were announced. Challenge organizers, in addition to the leaderboard, released the evaluation code and visualisations of the prediction masks of the “top 5” teams. This shows a very high level of transparency, and provides a unique opportunity to better understand the challenge results. Our analysis of the code an...

中文摘要翻译待生成

Describes the above-named upcoming conference event. May include topics to be covered or calls for papers.

中文摘要翻译待生成

Presents the table of contents for this issue of this publication.

中文摘要翻译待生成