Deep Learning-based Histological Segmentation Differentiates Cavitation Patterns in Knee and Digit Joints
•
0 likes•3 views
Unsupervised segmentation (unlabeled regions of interest, ROIs) and autoencoder (AE)-based classification were used to classify differences in cavitation patterns in knees and digits using the stained images (n=20-30 images/group). Each image was divided into 256 x 256 pixel patches, and a convolutional neural network (CNN)-based unsupervised segmentation was used to identify ROIs. These patches were subsequently fed into a CNN-based AE whose latent space layer was connected to a classifier for input patch classification. The AE was trained using the ROIs identified by the unsupervised segmentation, and the image classes were used to train the classifier. Whole image classifications were determined by maximum voting of the patch results and evaluated by accuracy.
![[1] Archer+ 1999, [2] Kim+ 2023, [3] Kim+ 2023. This study was supported by the NIH K01 AR078387 and Rowan-Virtua Seed Grant.
• Joint cavitation is a developmental process to create a fluid-filled joint cavity, permitting unhindered joint
motion1. Our recent study showed that dense fibroblast-like mesenchymal cells in the interzone produce a
robust matrix (i.e., pericellular coat), mostly hyaluronan (HA)2.
• Cavitation initiates with the appearance of microcavities between cells. The microcavities accumulate, coalesce,
and eventually form a single cavity. This physical separation involves HA cleavage into its fragments by
hyaluronidase, in turn reducing cell-matrix interactions. While synovial joints undergo this process, the cavitation
timing and patterns are distinct in the knee and digit joints3.
• Cavitation starts and ends within 6-12h (E15-15.5) in knee joints, while digits take 72-84h (E15-18.5).
Interestingly, the growing femur and tibia progressively flexed (angulated) at the knee during cavitation, but the
digit joint changed minimally. Further, the cells undergoing cavitation in knees show highly stretched and
elongated cell nuclei, whereas those in digits remain round, suggesting joint cavitation is differentially regulated
in these joints. However, it remains unclear what originates the differences2.
• We used a deep learning technique to determine (1) whether artificial intelligence (AI) perceives the
spatiotemporal process of synovial joint development and cavitation, (2) whether it could distinguish between
knee and digit joints by learning their developmental process using histological data sets, and (3) ultimately,
what factor(s) are involved to regulate the knee and digit joint development and cavitation.
Deep Learning-based Histological Segmentation
Differentiates Cavitation Patterns in Knee and Digit Joints
Introduction
Materials and Methods
Results
Discussion
References & Acknowledgments
Significance
Minwook Kim1, Heejong Kim2, Wookjin Choi3
1Department of Molecular Biology, Rowan-Virtua School of Translational Biomedical Engineering & Sciences, School of Osteopathic Medicine, Stratford, NJ,
2University of Pennsylvania, Philadelphia, PA, 3Thomas Jefferson University, Philadelphia, PA
KIM
LAB
• All animal procedures were approved by Rowan University IACUC. CD-1 mouse hindlimbs (E14.5 – E18.5 at 6-
24h intervals) were embedded, sectioned, and stained with Alcian blue.
• Unsupervised segmentation (unlabeled regions of
interest, ROIs) and autoencoder (AE)-based
classification were used to classify differences in
cavitation patterns in knees and digits using the
stained images (n=20-30 images/group).
• Each image was divided into 256 x 256 pixel
patches, and a convolutional neural network
(CNN)-based unsupervised segmentation was
used to identify ROIs. These patches were
subsequently fed into a CNN-based AE whose
latent space layer was connected to a classifier
for input patch classification.
• The AE was trained using the ROIs identified by
the unsupervised segmentation, and the image
classes were used to train the classifier. Whole
image classifications were determined by
maximum voting of the patch results and
evaluated by accuracy.
• The unsupervised segmentation perceives relevant ROIs and guides the AE and the classifiers to identify the
spatiotemporal process of synovial joint development and cavitation. Interestingly, matrix (i.e., hyaluronan, HA1,
2, and 3) and microcavity (MC1, 2, and 3) in the interzone are recognized as three classes, respectively.
• Distribution of the matrix (HA) and microcavity (MC) are colocalized in the interzone along the cavitation line.
The perceived changes in matrix (HA) and microcavity (MC) regarding the size (individual and summed) and
distribution with developmental time are consistent with our previous findings that microcavity increases in
number and size with increase of HA production and hyaluronidase activity.
• The two classified latent variables differentiate knee and digit joints, and the variational AE (VAE)-based model
reconstructs the original image with significance. The reconstructed image recapitulates most features, such as
cell and matrix distributions and intensity, except for blurriness and cell morphology.
• Future studies will investigate how the segmentation pattern changes as cell morphology and molecular
compositions change with developmental time.
• Deep learning techniques identify spatiotemporal processes of synovial joint development and cavitation and
differentiate knee and digit joints. The determined factors (classes) recapitulate the original images.
• This study will provide novel insights regarding what potential factor(s) are involved in differentiating cavitation
timing and patterns in knee and digit joints and how they contribute to synovial joint development and cavitation.
• Unsupervised segmentation perceives the spatiotemporal process
of joint development and cavitation and differentiates classes.
Prolif.
Cart
(future
‘Bone’)
fe
ti
itz
fe
ti
me
itz fe
ti
me
itz
ti
fe
itz
me
fe
ti
itz
fe
ti
me
itz fe
ti
me
itz
ti
fe
itz
me
mp
pp
mp
pp
mp
pp
mp
pp
mp
pp
mp
pp
mp
pp
mp
pp
itz
itz itz itz
itz
itz itz itz
Interzone
(Cartilage)
Alcian
blue-stained
Image
(Original)
Prolif.
Cart
(future
‘Bone’)
Interzone
(Cartilage)
Segmentation
E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5
Knee Digit
Interzone
(cavitating)
Interzone
(cavitating)
Individual
Area
in
Interzone
(µm
2
)
0
10
20
30
40
0
50
100
150
200
0
50
100
150
200
0
1000
2000
3000
4000
0
10
20
30
40
0
1000
2000
3000
4000
0
1 0
2 0
3 0
4 0
0
10
20
30
40
E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5
Summed
Area
in
Interzone
(µm
2
)
0
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
HA1 (Hyaluronan/ matrix 1): Segmented group 1 cavitating/cavitated in interzone MC1 (Microcavity 1): Segmented group 1 in cavitating/cavitated interzone
HA2 (Hyaluronan/ matrix 2): Segmented group 2 in cavitating/cavitated interzone
HA3 (Hyaluronan/ matrix 3): Segmented group 3 in cavitating/cavitated interzone
MC2 (Microcavity 2): Segmented group 2 in cavitating/cavitated interzone
MC3 (Microcavity 3): Segmented group 3 in cavitating/cavitated interzone
E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5
hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity
50µm
10µm
• Classified two latent variables identify knee and digit cavitation
patterns, and the VAE-based model reconstructs the original image.
fe
ti
itz
fe
ti
me
itz
ti
fe
itz
me
fe
ti
me
itz
mp
pp
mp
pp
mp
pp
mp
pp
itz
itz itz itz
Prolif.
Cart
(future
‘Bone’)
Interzone
(Cartilage)
Segmentation
with
Classification
Interzone
(cavitating)
fe
ti
itz
fe
ti
me
itz fe
ti
me
itz
ti
fe
itz
me
mp
pp
mp
pp
mp
pp
mp
pp
itz
itz itz itz
Prolif.
Cart
(future
‘Bone’)
Interzone
(Cartilage)
Reconstruction
from
Latent
Space
&
Segmentation
Interzone
(cavitating)
Differentiation
(two
potential
variables)
Reconstruction
Knee
Digit
Knee
Digit
Knee
Digit
knee
digit
actual predicted
Selected
(p<0.005)
Class
Frequency
Efficient Histological Image Analysis Model Architecture
II. Multilayer perceptron (MLP) classification
I. Unsupervised
Segmentation
256x256
128x128
64
2
32
2
16
2
8
2
128 128
3 48 128 128 48+3
Share attention weights of
unsupervised segmentation
Encoder – ShuffleNet v2 Decoder
Bottleneck and
Latent space
III. Image reconstruction from
Latent space and Segmentation
24
24
48
96
192 1024 192 2 192
96+96
24
48+48
24+24
24
1x1 Conversion
Inverted Residual /
Inverted Residual Transpose
3x3 Conv / 3x3 Conv Transpose
2x2 Max Pool / x2 Up sample
8x8 Global Pool
Skip Connection
Attention
Image / Segmentation
Feature / Decoded Feature
• Significance was determined by one or two-way ANOVA with Tukey’s post hoc (p<0.05).](https://image.slidesharecdn.com/kim-240214012213-2f5d97da/85/deep-learningbased-histological-segmentationdifferentiates-cavitation-patterns-in-knee-and-digit-joints-1-320.jpg)
Recommended
Recommended
More Related Content
Similar to Deep Learning-based Histological Segmentation Differentiates Cavitation Patterns in Knee and Digit Joints
Similar to Deep Learning-based Histological Segmentation Differentiates Cavitation Patterns in Knee and Digit Joints (20)
More from Wookjin Choi
More from Wookjin Choi (20)
Recently uploaded
Recently uploaded (20)
Deep Learning-based Histological Segmentation Differentiates Cavitation Patterns in Knee and Digit Joints
- 1. [1] Archer+ 1999, [2] Kim+ 2023, [3] Kim+ 2023. This study was supported by the NIH K01 AR078387 and Rowan-Virtua Seed Grant. • Joint cavitation is a developmental process to create a fluid-filled joint cavity, permitting unhindered joint motion1. Our recent study showed that dense fibroblast-like mesenchymal cells in the interzone produce a robust matrix (i.e., pericellular coat), mostly hyaluronan (HA)2. • Cavitation initiates with the appearance of microcavities between cells. The microcavities accumulate, coalesce, and eventually form a single cavity. This physical separation involves HA cleavage into its fragments by hyaluronidase, in turn reducing cell-matrix interactions. While synovial joints undergo this process, the cavitation timing and patterns are distinct in the knee and digit joints3. • Cavitation starts and ends within 6-12h (E15-15.5) in knee joints, while digits take 72-84h (E15-18.5). Interestingly, the growing femur and tibia progressively flexed (angulated) at the knee during cavitation, but the digit joint changed minimally. Further, the cells undergoing cavitation in knees show highly stretched and elongated cell nuclei, whereas those in digits remain round, suggesting joint cavitation is differentially regulated in these joints. However, it remains unclear what originates the differences2. • We used a deep learning technique to determine (1) whether artificial intelligence (AI) perceives the spatiotemporal process of synovial joint development and cavitation, (2) whether it could distinguish between knee and digit joints by learning their developmental process using histological data sets, and (3) ultimately, what factor(s) are involved to regulate the knee and digit joint development and cavitation. Deep Learning-based Histological Segmentation Differentiates Cavitation Patterns in Knee and Digit Joints Introduction Materials and Methods Results Discussion References & Acknowledgments Significance Minwook Kim1, Heejong Kim2, Wookjin Choi3 1Department of Molecular Biology, Rowan-Virtua School of Translational Biomedical Engineering & Sciences, School of Osteopathic Medicine, Stratford, NJ, 2University of Pennsylvania, Philadelphia, PA, 3Thomas Jefferson University, Philadelphia, PA KIM LAB • All animal procedures were approved by Rowan University IACUC. CD-1 mouse hindlimbs (E14.5 – E18.5 at 6- 24h intervals) were embedded, sectioned, and stained with Alcian blue. • Unsupervised segmentation (unlabeled regions of interest, ROIs) and autoencoder (AE)-based classification were used to classify differences in cavitation patterns in knees and digits using the stained images (n=20-30 images/group). • Each image was divided into 256 x 256 pixel patches, and a convolutional neural network (CNN)-based unsupervised segmentation was used to identify ROIs. These patches were subsequently fed into a CNN-based AE whose latent space layer was connected to a classifier for input patch classification. • The AE was trained using the ROIs identified by the unsupervised segmentation, and the image classes were used to train the classifier. Whole image classifications were determined by maximum voting of the patch results and evaluated by accuracy. • The unsupervised segmentation perceives relevant ROIs and guides the AE and the classifiers to identify the spatiotemporal process of synovial joint development and cavitation. Interestingly, matrix (i.e., hyaluronan, HA1, 2, and 3) and microcavity (MC1, 2, and 3) in the interzone are recognized as three classes, respectively. • Distribution of the matrix (HA) and microcavity (MC) are colocalized in the interzone along the cavitation line. The perceived changes in matrix (HA) and microcavity (MC) regarding the size (individual and summed) and distribution with developmental time are consistent with our previous findings that microcavity increases in number and size with increase of HA production and hyaluronidase activity. • The two classified latent variables differentiate knee and digit joints, and the variational AE (VAE)-based model reconstructs the original image with significance. The reconstructed image recapitulates most features, such as cell and matrix distributions and intensity, except for blurriness and cell morphology. • Future studies will investigate how the segmentation pattern changes as cell morphology and molecular compositions change with developmental time. • Deep learning techniques identify spatiotemporal processes of synovial joint development and cavitation and differentiate knee and digit joints. The determined factors (classes) recapitulate the original images. • This study will provide novel insights regarding what potential factor(s) are involved in differentiating cavitation timing and patterns in knee and digit joints and how they contribute to synovial joint development and cavitation. • Unsupervised segmentation perceives the spatiotemporal process of joint development and cavitation and differentiates classes. Prolif. Cart (future ‘Bone’) fe ti itz fe ti me itz fe ti me itz ti fe itz me fe ti itz fe ti me itz fe ti me itz ti fe itz me mp pp mp pp mp pp mp pp mp pp mp pp mp pp mp pp itz itz itz itz itz itz itz itz Interzone (Cartilage) Alcian blue-stained Image (Original) Prolif. Cart (future ‘Bone’) Interzone (Cartilage) Segmentation E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5 Knee Digit Interzone (cavitating) Interzone (cavitating) Individual Area in Interzone (µm 2 ) 0 10 20 30 40 0 50 100 150 200 0 50 100 150 200 0 1000 2000 3000 4000 0 10 20 30 40 0 1000 2000 3000 4000 0 1 0 2 0 3 0 4 0 0 10 20 30 40 E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5 Summed Area in Interzone (µm 2 ) 0 1000 2000 3000 4000 5000 0 1000 2000 3000 4000 5000 HA1 (Hyaluronan/ matrix 1): Segmented group 1 cavitating/cavitated in interzone MC1 (Microcavity 1): Segmented group 1 in cavitating/cavitated interzone HA2 (Hyaluronan/ matrix 2): Segmented group 2 in cavitating/cavitated interzone HA3 (Hyaluronan/ matrix 3): Segmented group 3 in cavitating/cavitated interzone MC2 (Microcavity 2): Segmented group 2 in cavitating/cavitated interzone MC3 (Microcavity 3): Segmented group 3 in cavitating/cavitated interzone E14.5 E15 E15+6h E15.5 E15 E15.5 E16.5 E18.5 hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity hyaluronan microcavity 50µm 10µm • Classified two latent variables identify knee and digit cavitation patterns, and the VAE-based model reconstructs the original image. fe ti itz fe ti me itz ti fe itz me fe ti me itz mp pp mp pp mp pp mp pp itz itz itz itz Prolif. Cart (future ‘Bone’) Interzone (Cartilage) Segmentation with Classification Interzone (cavitating) fe ti itz fe ti me itz fe ti me itz ti fe itz me mp pp mp pp mp pp mp pp itz itz itz itz Prolif. Cart (future ‘Bone’) Interzone (Cartilage) Reconstruction from Latent Space & Segmentation Interzone (cavitating) Differentiation (two potential variables) Reconstruction Knee Digit Knee Digit Knee Digit knee digit actual predicted Selected (p<0.005) Class Frequency Efficient Histological Image Analysis Model Architecture II. Multilayer perceptron (MLP) classification I. Unsupervised Segmentation 256x256 128x128 64 2 32 2 16 2 8 2 128 128 3 48 128 128 48+3 Share attention weights of unsupervised segmentation Encoder – ShuffleNet v2 Decoder Bottleneck and Latent space III. Image reconstruction from Latent space and Segmentation 24 24 48 96 192 1024 192 2 192 96+96 24 48+48 24+24 24 1x1 Conversion Inverted Residual / Inverted Residual Transpose 3x3 Conv / 3x3 Conv Transpose 2x2 Max Pool / x2 Up sample 8x8 Global Pool Skip Connection Attention Image / Segmentation Feature / Decoded Feature • Significance was determined by one or two-way ANOVA with Tukey’s post hoc (p<0.05).