Processing math: 0%
New website getting online, testing
Breast tumor grading network based on adaptive fusion and microscopic imaging
  • Abstract

    Tumor grading based on microscopic imaging is critical for the diagnosis and prognosis of breast cancer, which demands excellent accuracy and interpretability. Deep networks with CNN blocks combined with attention currently offer better induction bias capabilities but low interpretability. In comparison, the deep network based on ViT blocks has stronger interpretability but less induction bias capabilities. To that end, we present an end-to-end adaptive model fusion for deep networks that combine ViT and CNN blocks with integrated attention. However, the existing model fusion methods suffer from negative fusion. Because there is no guarantee that both the ViT blocks and the CNN blocks with integrated attention have acceptable feature representation capabilities, and secondly, the great similarity between the two feature representations results in a lot of redundant information, resulting in a poor model fusion capability. For that purpose, the adaptive model fusion approach suggested in this study consists of multi-objective optimization, an adaptive feature representation metric, and adaptive feature fusion, thereby significantly boosting the model's fusion capabilities. The accuracy of this model is 95.14%, which is 9.73% better than that of ViT-B/16, and 7.6% better than that of FABNet; secondly, the visualization map of our model is more focused on the regions of nuclear heterogeneity (e.g., mega nuclei, polymorphic nuclei, multi-nuclei, and dark nuclei), which is more consistent with the regions of interest to pathologists. Overall, the proposed model outperforms other state-of-the-art models in terms of accuracy and interpretability.

    Keywords

  • 本文的主要贡献如下:

    为使乳腺癌肿瘤分级结果同时具有良好的可解释性和高精度,本文提出一种基于自适应模型融合的端到端深度网络 (deep network with adaptive model fusion, AMFNet)。由于现有的模型融合方法bagging[], boosting[]和stacking[]存在负融合现象[],原因:1) ViT块参数量是AMCNN块的十倍,学习率过大,ViT块容易错过局部最优解,学习率过小,AMCNN块会陷入“鞍部”。因此,ViT块和AMCNN块无法同时具备良好的特征表示能力;2) ViT块和AMCNN块所学的特征表示相似度高,导致融合特征矩阵的冗余度高,并且现有方法去冗余能力较弱,故AMFNet融合ViT块和AMCNN块的能力不足[]。因此,为了解决AMFNet在融合ViT块和AMCNN块时出现负融合现象,本文提出一种自适应模型融合(adaptive model fusion, AMF)方法,包括多目标优化 (multi-objective optimization, MOO)、自适应特征表示度量(adaptive feature representation metric, AFRM)和自适应特征融合(adaptive feature fusion, AFF)三部分。其中,多目标优化是保证ViT块和AMCNN块能够同时具有良好的特征表示能力;而自适应特征表示度量和自适应特征融合是提高AMFNet的去冗余能力,进而提升模型的融合能力。

    虽然数字病理全玻片图像技术发展迅速,但基于显微镜成像的方式仍然是病理科诊断的主要手段,这是因为全玻片机昂贵的价格和生成数字全玻片图像的速度较慢,无法满足病理科大量的病例诊断需求。现有肿瘤分级仍十分依赖病理专家的经验和精力,因为病理图像的肿瘤级别之间差异小,分级指标复杂且带有很强的主观性[]。因此,研究计算机肿瘤分级算法来辅助病理专家进行肿瘤分级的工作显得十分重要。

    在癌症组织病理显微镜成像分析中,基于集成注意力机制的卷积神经网络(attention mechanism based on convolutional neural networks,AMCNN)块的深度学习模型辅助病理专家进行诊断已取得快速发展[-]。Yang等人[]采用一种引导注意力机制来提高CNN模型识别乳腺癌组织病理图像的能力。BenTaieb等人[]提出一种基于循环注意力机制的检测模型,实现癌症数字全玻片图像中对有无肿瘤病变进行预测。Tomita等人[]采用注意力机制改进ResNet18来提高食道癌病理图像分类性能。Yao等人[]提出孪生注意力机制和全连接层从数字全玻片中识别癌变。Huang[]和Zhou[]等人构建了一种融合注意机制的改进CNN模型,提高了喉癌病理图像肿瘤分级的性能。虽然,以上基于AMCNN块的深度学习模型的识别性能和归纳偏差能力较好,但是其可解释性较差,导致模型结果的临床可信度不高[-]

    2) 提出一种自适应特征表示度量方法,能降低ViT块和AMCNN块所学特征表示之间的相似性和冗余度,进而提高AMFNet的模型融合能力;

    3) 提出一种自适应特征融合方法,能够自适应和更加稀疏地去除融合特征矩阵中的冗余信息,从而提高AMFNet的模型融合能力。

    1) 提出一种多目标优化方法,能同时保证ViT块和AMCNN块具有良好的特征表示能力,从而提高AMFNet的模型融合能力;

    乳腺癌是女性中最常见的癌症[]。乳腺癌肿瘤的组织学分级已被证明是一个强有力的诊断和预后指标,其主要依靠病理专家评估组织切片的显微镜成像中组织和细胞的形态学状况,例如腺体分化、细胞核异型和有丝分裂计数等[]。已有研究文献表明苏木精-伊红(Hematoxylin-Eosin, HE)染色的显微成像样本和孕激素受 体(Estrogen Receptor, ER)的免疫组化(Immunohistochemically, IHC)染色的显微成像样本存在较强统计相关性,即ER状态与肿瘤组织分级有着较强的相关性[-]。其中,Desai等人[]的研究表明ER状态值的下降与肿瘤恶性程度有一定的相关性。Zafrani等人[]发现ER状态值和肿瘤分级有着很强的统计相关性(P < 10−4)。Fuqua等人[]发现ER状态值和肿瘤组织分级有着明显的相关性。Vagunda等人[]的研究也发现ER值与肿瘤分级有统计学相关性。Baqai和Shousha等人[]的工作表明ER阴性与肿瘤高级别有着明显相关性 (P < 0.001)。Sofi等人[]发现ER在66.3%的病例中呈阳性,随着年龄的增长而增加,高级别病变和较大尺寸的肿瘤更可能是 ER阴性。Azizun-Nisa等人[]研究表明高级肿瘤中的 ER表达显著降低 (ER 5.6% vs 10.5%)。因此,在基于ER IHC染色的病理组织显微镜成像中进行肿瘤分级有一定的统计意义,可以作为临床诊断和预后的一个参考指标。

    基于Vision Transformer (ViT)块的深度学习模型流行于计算机视觉领域,在癌症组织病理图像处理方向也有一些应用[-]。Gao等人[]提出一种基于示例学习的ViT模型从病理图像中学习更加鲁棒的特征表示。Wang等人[]提出一种混合ViT模型来探索图像固有特性和在特定域捕获新的知识。Li等人[]采用一种可变型的ViT模块,来提高模型的归纳偏差能力。Chen等人[]提出一种多尺度可视化ViT模型,用于胃部组织病理图像的分类。Zou等人[]提出一种双ViT网络模型,可通过同时捕获图像局部和全局信息识别癌症图像。尽管ViT块采用多Attention机制的串、并联结构,使其具有较好的可解释性。但是其缺乏旋转不变性、共享权重和尺度不变性等优势,导致基于ViT块的深度学习模型的归纳偏差能力较差[]

    为实现对ER IHC染色的乳腺癌病理组织显微镜成像进行高精度和可解释性的肿瘤分级,本文通过有效地融合ViT块和AMCNN块,提出了一种基于自适应模型融合的深度网络,如图1所示。该方法主要包括五部分:基于ER IHC染色的乳腺癌病理组织显微镜成像部分、ViT块、AMCNN块、ViT-AMCNN块和AMF方法(包括MOO、AFRM和AFF)。其中,ViT块采用ViT-B/16[],AMCNN块使用FABNet模型[]。依据图1,对本文算法流程进行简要的介绍。首先从图1(a)中采集显微镜成像数据分别输入到图1(b) AMFNet的AMCNN块、 ViT块、 ViT-AMCNN块中得到对应的特征矩阵( {{{{\cal{Y}}}}}_{\text{AMCNN}},{\boldsymbol{\mathcal{Y}}}_{\text{ViT}}\mathcal{和}{\boldsymbol{\mathcal{Y}}}_{\text{f-inter}} )。第二,将 {\boldsymbol{\mathcal{Y}}}_{\text{f-inter}} 输入到AFF方法中去,计算每个特征通道的权重分数,然后乘以权重分数进行自适应地去除特征矩阵的冗余信息,得到 {\boldsymbol{\mathcal{Y}}}_{\text{f}} 。第三,将每个块所学到的特征表示({\boldsymbol{\mathcal{F}}}_{\text{AMCNN}}=\left\{{\boldsymbol{\mathcal{Y}}}_{\text{AMCNN}},{{\boldsymbol{\ell }}}\right\},{\boldsymbol{\mathcal{F}}}_{\text{ViT}}=\left\{{\boldsymbol{\mathcal{Y}}}_{\text{ViT}},{\boldsymbol{\ell }}\right\}\mathcal{}{\boldsymbol{\mathcal{F}}}_{\text{ViT-AMCNN}}=\{{\boldsymbol{\mathcal{Y}}}_{\text{f}}, {\boldsymbol{\ell }} \}, {\boldsymbol{\ell }} 表示标签矩阵)一起输入到AFRM方法中去,采用深度度量学习来度量特征表示之间的联合分布距离,拉开 {\boldsymbol{\mathcal{F}}}_{\text{ViT}}\mathcal{和}{\boldsymbol{\mathcal{F}}}_{\text{AMCNN}} 的距离,从而降低 {\boldsymbol{\mathcal{F}}}_{\text{ViT-AMCNN}} 的冗余性。第四,采用MOO方法对ViT、AMCNN、AFRM和AFF的输出目标同时进行多目标和多学习率优化。最后采用Grad-CAM[]和Rollout[]技术可视化模型所学知识。

    Figure 1. Block diagram of the algorithm in this paper.  (a) ER IHC staining based microscopic imaging procedure for breast cancer pathology; (b) AMFNet
    Full-Size Img PowerPoint

    Block diagram of the algorithm in this paper. (a) ER IHC staining based microscopic imaging procedure for breast cancer pathology; (b) AMFNet

    2) 组织脱水:采用脱水仪器Leica ASP300S对组织病理标本进行脱水处理,整个脱水过程有13个步骤,每个步骤都需要将脱水仪器设置真空、加压或两者交替进行的脱水模式。而脱水过程主要用到的试剂有:AF液、75%酒精、85%酒精、95%酒精、无水酒精、二甲苯和石蜡;

    6) 显微镜成像:采用Olympus BX41多头显微镜对病理组织玻片进行成像,其中物镜分别设置为20X (N.A. 0.4,W.D. 1.2 Spring)和40X (N.A. 0.65,W.D. 0.6 Spring)。齐焦距为45 mm,观察镜筒的倾角为30°,使用柯勒照明器作为内置透射光源头。

    4) 组织切片:采用Leica RM2235石蜡切片机进行组织切片,其选配的8° X/Y轴精确定位系统带校准调节功能,有助于快速定位切过的样品(重切)。定位系统随后快速返回到有2个红色指示标记的基准0刻度位。垂直行程最长达70 mm,不仅能使用超大包埋盒,并且由于加大了距离刀刃的工作距离;

    1) 组织标本病理取材:对送检的手术大样本按标准进行逐一剖开,并充分固定(>12 h);

    3) 组织包埋:采用Leica EG1150 H石蜡包埋机进行组织包埋,其蜡缸的容量为3L,在工作日和工作时间设置仪器自动运行。包埋盒和模具的加热槽可互换,以适应变化的包埋流程。Leica EG1150 H的冷台维持–5 °C恒温,实现包埋蜡块和模具的快速冷却。该过程主要耗材是包埋盒和石蜡等;

    5) 免疫组化染色封片:采用Autostainer XL Leica ST5010染色封片系统、医嘱所需的免疫组化试剂、酒精、二甲苯、玻片、封片胶带等耗材对组织切片进行免疫组化染色并封装。该过程分为23个步骤,需要精准地设置每个步骤的时间和温度;

    基于ER IHC染色的乳腺癌病理组织显微镜成像流程主要包括六个步骤:组织标本病理取材、组织脱水、组织包埋、组织切片、免疫组化染色封片和显微镜成像,如图1(a)所示。

    式中: \text{t} 表示损失函数系数,M表示模型输入张量的batch size,{{{{{\boldsymbol{\ell}} }} }}表示真实的标签矩阵, {\mathcal{Y}}^{{i}} 表示模型输出的特征矩阵, \delta 表示Softmax激活函数。

    为有效地融合ViT块和AMCNN块,来提高AMFNet的模型融合能力,使模型的肿瘤分级具有高精度和可解释性的特性。本文提出一个多目标优化的损失函数 \mathcal{L} 来实现上述目的,同时也优化AMF方法中的各个部分。

    L=iLAMCNN+rLViT+jLAFF+uLAFRM,
    LCE=t i=1Milog(δ(Yi)),

    式中: {\boldsymbol{\mathcal{L}}}_{\text{A}\text{MCNN}} {\boldsymbol{\mathcal{L}}}_{\text{ViT}} 分别优化AMCNN块和ViT块,保证它们同时具有良好特征表示的能力,且保证AMFNet模型的肿瘤分级结果具有高精度和可解释性。 {\boldsymbol{\mathcal{L}}}_{\text{A}\text{FF}} 用于优化AFF方法部分,其输出为模型最终的肿瘤分级结果。 {\boldsymbol{\mathcal{L}}}_{\text{AFRM}} 用于优化AFRM方法部分。其中 {\boldsymbol{\mathcal{L}}}_{\text{A}\text{MCNN}} , {\boldsymbol{\mathcal{L}}}_{\text{ViT}} , {\boldsymbol{\mathcal{L}}}_{\text{A}\text{FF}} 使用了交叉熵损失函数( {\boldsymbol{\mathcal{L}}}_{\rm{C}\rm{E}} ),如式(2)所示。 {i}\text{,}\, {r}{,}\,{j}\text{,}\, {u} 是对应子损失函数的权重系数。由于AMCNN块和ViT块参数量差异较大,本文设置了多学习率优化策略: {\boldsymbol{\mathcal{L}}}_{\text{A}\text{MCNN}} {\boldsymbol{\mathcal{L}}}_{\text{A}\text{FF}} 的学习率设置为1~50 epochs: \text{1×}{\text{10}}^{{-}\text{4}} ,51~75 epochs: 2\times {10}^{-5} ,76~100 epochs: 1\times {10}^{-6} {\boldsymbol{\mathcal{L}}}_{\text{ViT}} 为1~50 epochs: 1\times {10}^{-5} ,51~75 epochs: 5\times {10}^{-6} ,76~100 epochs: 1\times {10}^{-6} {\boldsymbol{\mathcal{L}}}_{\text{AFRM}} 没有设置专门对应的可学习参数,而是在优化过程中 {\boldsymbol{\mathcal{L}}}_{\text{AFRM}} 会同时影响ViT块,AMCNN块和AFF方法所对应的可学习参数。

    为去 除ViT块和AMCNN块采用特征拼接(feature stacking,按特征矩阵的第二维度进行拼接)方法融合后的特征矩阵中存在较多冗余信息,而传统特征选择方法无法实现自适应和稀疏性的去冗余。因此,本文提出一种AFF方法,采用编解码的神经网络和Softmax概率权重的思路,分别实现自适应和稀疏性求解各个特征的权重系数。从而有效地提高了AMFNet的模型融合能力,让模型的肿瘤分级结果能够具有更高的准确率和更好的可解释性。实现细节如图2和式(3)所示。给出AMFNet的AMCNN块、ViT块、ViT-AMCNN块中得到对应的特征矩阵( {\mathcal{Y}}_{\text{AMCNN}},{\mathcal{Y}}_{\text{ViT}}{,\mathcal{Y}}_{\text{f-inter}} ),自适应融合后的最终特征矩阵为 {\mathcal{Y}}_{\text{f}}

    Yf(B,1792)=δ[θ(Yf-inter(B,1792)×Wl1(1792,112)+Bl1(1,112))×Wl2(112,1792)+Bl2(1,1792)],
    Figure 2. Detail diagram of the AFF method implementation
    Full-Size Img PowerPoint

    Detail diagram of the AFF method implementation

    式中: \delta 表示Softmax激活函数, \theta 表示ReLU激活函数, {\mathcal{W}}_{l1},{\mathcal{B}}_{l1} 表示Linear 1层(编码层)的权重和偏置项, {\mathcal{W}}_{l2},{\mathcal{B}}_{l2} 表示Linear 2层(解码层)的权重和偏置项。

    Figure 3. AFRM method implementation schematic
    Full-Size Img PowerPoint

    AFRM method implementation schematic

    H=argHmin{1NiNLCE(H(Yf),)+gD(P(Yf),P(YAMCNN))gD(P(Yf),P(YViT))+(1g)D(P(|Yf),P(|YAMCNN))(1g)D(P(|Yf),P(|YViT))},
    LAFRM=μ{1BiBϕ(Yf(i,1792))1BiBϕ(YAMCNN(i,1024))221BiBϕ(Yf(i,1792))1BiBϕ(YViT(i,768))22}+1μG{jG1B(G)iB(G)ϕ(Yf(i,1792))1B(G)iB(G)ϕ(YAMCNN(i,1024))22jG1B(G)iB(G)ϕ(Yf(i,1792))1B(G)iB(G)ϕ(YViT(i,768))22}

    为了上述目标实现自适应的优化,本文为自适应特征表示度量设计了专门的损失函数( {\mathcal{L}}_{\text{AFRM}} ),如式(5)所示。

    式中:N表示训练的次数, \mathcal{H} 表示AMFNet的张量函数, {\mathcal{H}}^{\mathcal{*}} 表示最优的张量函数, \mathcal{P}\left(\mathcal{*}\right) 表示边缘概率分布, \mathcal{P}\left(\mathcal{*}\right|\mathcal{*}) 表示条件概率分布, \text{g} 表示分布的权重系数, \mathcal{D} 表示在高斯空间的欧式距离。

    解决ViT和AMCNN块融合后冗余信息过多的另外一个方法是降低两个模块所学特征表示之间的相似度。为此,本文提出一种AFRM的方法,通过深度度量学习,来拉开ViT和AMCNN块的特征表示之间的距离(特征表示:由特征矩阵和所对应的标签矩阵构成联合概率分布, 其中图3中的{{\boldsymbol{\mathcal{F}}}}_{{\rm{ViT}}}, {\mathcal{F}}_{{\rm{AMCNN}}} {\mathcal{F}}_{{\rm{ViT}}-{\rm{AMCNN}}} 分别表示ViT块、AMCNN块和ViT-AMCNN块所学的特征表示),从而降低融合特征矩阵的冗余度,提高AMFNet的模型融合能力,进而提升模型肿瘤分级结果的准确率和可解释性。为了实现这个目标,必须要设置一个目标函数(如式(4)所示),采用梯度反向传播的方式进行多次优化,从而得到一个最优的AMFNet权重参数,实现细节如式(4)和图3所示。

    式中:AMFNet的AMCNN块、ViT块和ViT-AMCNN块所得到的特征矩阵分别是{\mathcal{Y}}_{\text{AMCNN}}, {\mathcal{Y}} _{\text{ViT}}{\mathcal{Y}}_{\text{f-inter}}, {\text{} \text{}{B}}^{\text{(}{{{G}}}\text{)}}表示每个肿瘤级别的对应的样本数, {G} 表示肿瘤的分级数, \phi 表示高斯核映射。

    {i}\text{=1.0}\text{,}\text{}\,{r}\text{=0.5,}{j}\text{=0.2,}\,{u}\text{=0.2,} Epoch=100,Batch size =16。AMFNet的输入尺寸为224×224×3。实验的GPU是NVIDIA GeForce RTX3060 12 GB。并且本文采用类别准确度(Grade I Acc、Grade II Acc、Grade III Acc)、平均准确度(Average Acc)、特异性(Percision,P)、灵敏性(Recall,R)、F1得分(F1)和ROC曲线下面积(AUC)对模型性能进行定量分析。

    Distribution of the number of ER IHC datasets for breast cancer

    乳腺癌ER IHC数据集数量分布表

    Datasets Parameter
    Grade IGrade IIGrade IIIImage sizeTotal
    Training set268355360224×224983
    Validation set90118120224×224328
    Testing set90119120224×224329
    Total448592600224×2241640
    CSV Show Table

    基于ER IHC染色的乳腺癌病理组织显微镜成像流程数据集[]包含414张分辨率为1300 pixels×1030 pixels的图像。该数据集包括I级、II级和III级的肿瘤病理显微镜图像。本文依此顺序裁剪出1640张patch图像,并将乳腺癌patch图像的分辨率调整为224 pixels×224 pixels。 此外,1640张patch图像被分为983、328和329张图像,分别用于训练、验证和测试,具体如表1所示。所提出的AMFNet是在PyTorch框架中使用Rmsprop进行优化。该网络模型中使用的超参数具体如下:

    本文的乳腺癌ER IHC染色的组织病理图像分为三个等级(I级、II级和III级),对于类别准确度和平均准确度的计算准则是:取置信度为0.5,大于0.5的算判对,最终将判对的除以总数得到准确度百分比。本文对应的是多分类,采用了类别数量权重来计算三种类别的平均肿瘤分级准确率,其次本文采用了Marco方式来计算三种类别的平均P、R、AUC和F1等指标。

    R=TPTP+FN.

    当对乳腺癌ER IHC染色的组织病理图像进行自动分类时,会出现分类混淆矩阵(如表1所示)中的各种情况。为了分析真正的阳性样本中有多大比例被预测正确,本文用召回率来分析,见式(6)。

    表2所示,为了确定预测为阳性的样本中有多少比例是真正的阳性样本,本文使用特异性指标来进行分析,其计算如式(7)所示。

    Classification confusion matrix

    分类混淆矩阵

    RealityForecast result
    PositiveNegative
    PositiveTrue positive (TP)False negative (FN)
    NegativeFalse positive (FP)True negative (TN)
    CSV Show Table
    P=TPTP+FP.

    F1分数,也被称为平衡F分数(balanced F score),被定义为精度和召回率的平均值。 其计算式如式(8)所示。

    F1=2×PRP+R.
    FPR=FPTN+FP.
    TPR=TPTP+FN.

    本文使用ROC曲线和AUC (area under ROC curve,AUC)来评价模型性能。根据分类器的预测结果的置信度对样本进行排序,将样本作为阳性样本按此顺序进行预测,每次计算真阳性率(true positive rate,TPR)如式(9),假阳性率(false positive rate,FPR)如式(10),TPR作为ROC曲线的纵轴,FPR作为ROC曲线的横轴。

    为进一步研究AMF方法对网络模型的影响,本文分别对其进行消融实验验证。从表3可知,在AMF方法中同时使用MOO、AFRM和AFF方法的平均准确度达到了95.14%,优于没有使用AMF方法的69.00%,优于只使用了MOO方法的92.40%,也优于使用了MOO和AFRM方法的93.92%。

    Ablation of AMF method in ER IHC pathological microimaging of breast cancer

    在ER IHC染色的乳腺癌病理组织显微镜成像中对AMF方法进行消融 (✗表示无,✓表示有)

    Model AMF methodAverage acc/%PRF1AUC
    MOOAFRMAFF
    AMFNet (ViT-AMCNN blocks)69.000.69340.69000.69030.7643
    92.400.92400.92400.92400.9423
    93.920.94030.93920.93940.9532
    95.140.95200.95140.95130.9617
    CSV Show Table

    Tumor grading accuracy of breast cancer ER IHC histopathology microscopic imaging

    乳腺癌ER IHC染色的病理组织显微成像的肿瘤分级准确率

    Model Grade I acc/%Grade II acc/%Grade III acc/%Average acc/%PRF1AUC
    Inception V3[]74.7178.1978.0177.200.77210.77200.77170.8290
    Xception V3[]71.8268.4076.4272.340.72260.72340.72260.7925
    ResNet50[]72.9673.7776.0874.470.75240.74470.74390.8085
    DenseNet121[]82.8077.5381.6380.550.80590.80550.80470.8541
    DenseNet121+Nonlocal[]85.8884.3981.9783.890.83960.83890.83910.8791
    DenseNet121+SENet[]79.5583.2784.3982.670.82720.82670.82660.8700
    DenseNet121+CBAM[]82.7681.2084.8082.980.83070.82980.82940.8723
    DenseNet121+HIENet[]86.2185.5985.4885.710.85850.85710.85720.8928
    FABNet[]86.0591.2984.9087.540.87650.87540.87520.9036
    ViT-S/16[]54.0255.8769.2060.1860.3760.180.60230.6970
    ViT-B/16[]85.8786.3284.1785.410.85460.85410.85410.8913
    ViT-B/32[]68.6070.0073.9871.120.71140.71120.71070.7799
    ViT-L/16[]78.9879.1781.2379.940.81130.79940.79870.8430
    ViT-L/32[]50.3561.2159.8358.360.60180.58360.57740.6770
    AMFNet (ours)92.6697.0295.1295.14 \uparrow 7.60.95200.95140.95130.9617
    CSV Show Table

    表4可知,本文提出的AMFNet模型的平均分级精度为95.14%,明显高于FABNet的87.54%、ViT-L/16的79.94%、ViT-B/16的85.41%、DenseNet+HIENet的85.71%、DenseNet+Nonlocal的83.89%、DenseNet+CBAM的82.98%和DenseNet+SENet的82.67%。因此,本文提出的AMFNet比其他先进方法(State Of the Art,SOTA)有更好的肿瘤分级性能。此外,由表3可知,本文的AMF方法可以有效地提高AMFNet的模型融合能力。

    图4观察可知,本文提出的AMFNet模型所学知识的可视化图更关注肿瘤细胞的异型核,如多核、巨核、多形核和深色核。而病理肿瘤分级主要依赖于细胞核的异型程度。而其他SOTA模型(如FABNet)只是关注部分异型核区和许多非典型异型细胞核区,这导致了整体可解释性差。因此,与SOTA模型相比,本文提出的AMFNet得到的可解释性的肿瘤分级结果与病理理论更加一致。

    Figure 4. Visually interpretative results comparisons with SOTA and our method on the breast cancer IHC microscopic imaging
    Full-Size Img PowerPoint

    Visually interpretative results comparisons with SOTA and our method on the breast cancer IHC microscopic imaging

    此外,本文的目的是融合ViT和AMCNN模块的优点,在优化过程中融合特征表示较为倾向ViT,所以ViT-AMCNN所学的融合特征表示的归纳偏差能力较强,同时具备较好的可解释性。显然这样的学习策略导致AMFNet的可解释性优于AMCNN模型而弱优于ViT模型。这里可以从图4中看出,我们发现在四幅可视化图中,AMFNet的可解释性均优于传统的AMCNN模型,而只有第一幅图优于ViT-B/16模型,其余三幅是持平的。但是,从可解释性和肿瘤分级的总体性能来说,本文模型都远优于ViT和AMCNN模型。

    由2.2和2.3节实验分析可知:1) MOO方法能够同时保证ViT块和AMCNN块具有良好特征表示的能力,从而提高AMFNet的模型融合能力;2) AFF方法能够实现自适应和稀疏性地去除ViT块和AMCNN块融合后的特征矩阵的冗余信息,从而提高AMFNet的模型融合能力;3) AFRM方法通过深度度量学习,能够有效地拉开ViT和AMCNN块所学特征表示的联合概率分布之间的距离,从而降低融合特征矩阵的冗余度,进而提高AMFNet模型的融合能力。

    此外,MOO方法充分地保证了ViT块和AMC块具有良好的特征表示能力,是解决模型负融合现象的根本,而AFF方法在MOO方法的基础上,自适应和稀疏地去除模型内部融合特征矩阵中的冗余信息,进一步提高了模型融合能力和缓解负融合现象,最后进行的AFRM方法有效地拉开了ViT块和AMC块所学特征表示之间的相似度,进一步提升了模型融合能力。总之,由MOO、AFF和AFRM所组成的AMF方法有效地解决了AMFNet的负融合问题,从而提升AMFNet模型融合能力,进而提升了AMFNet的肿瘤分级准确率。

    由2.4节的实验分析可知,与SOTA模型相比,本文提出的端到端AMFNet模型更加关注细胞核异型区域,例如多核、巨核、多形核和深色核等。这与病理医生所关注的区域更加吻合,且更加符合病理的相关分析。因此,本文所提出的AMFNet模型能同时具备较好的可解释性和肿瘤分级准确性。故本文的AMFNet模型一定程度上解决了现有病理图像的计算机辅助诊断算法的可解释性不足的问题。

    Distribution of the number of brain cancer histopathology image datasets

    脑癌组织病理图像数据集数量分布表

    Datasets Grade IGrade IIGrade IIIGrade IVTotal
    Training set2764928738912532
    Validation set91164290296841
    Testing set91164290296841
    Total458820145314834214
    CSV Show Table

    Comparison table of tumor grading accuracy of histopathological images of brain cancer

    脑癌组织病理图像的肿瘤分级准确率的对比表

    Model Metrics
    Grade I acc/%Grade II acc/%Grade III acc/%Grade IV acc/%Average acc/%PRF1AUC
    Inception V3[]84.1668.1180.6278.0577.760.77790.77760.77660.8575
    Xception V3[]84.9572.3478.3180.8078.830.79760.78830.78740.8588
    ResNet50[]82.0065.6266.4371.1269.800.69630.69800.69610.8091
    DenseNet121[]87.0574.7574.8681.7578.950.79780.78950.78580.8625
    DenseNet121+Nonlocal[]91.8481.9086.7089.3987.400.87630.87400.87270.9195
    DenseNet121+SENet[]96.2284.9788.0590.2589.180.89250.89180.89110.9279
    DenseNet121+CBAM[]92.7182.0089.0486.7787.400.87500.87400.87260.9162
    DenseNet121+HIENet[]95.7085.9987.8187.5088.230.88510.88230.88200.9244
    FABNet[]93.6887.7090.9790.8290.610.90720.90610.90570.9391
    ViT-S/16[]65.9846.9865.9769.7364.210.63750.64210.63590.7489
    ViT-B/16[]83.9080.0087.8386.8385.610.86180.85610.85530.9028
    ViT-B/32[]81.4862.5978.9377.2075.740.75500.75740.75410.8319
    ViT-L/16[]70.7954.4972.0573.5669.320.68970.69320.69020.7808
    ViT-L/32[]75.4958.0273.7675.0871.700.71520.71700.71340.8084
    AMFNet (ours)98.3293.3894.3093.9094.41 \uparrow 3.80.94510.94410.94420.9611
    CSV Show Table

    本文为了验证AMFNet方法的泛化能力和稳定性,在更大的脑癌组织病理图像数据集[](4214张病理图像,具体如表5所示)上进行了实验。其中训练集、测试集和验证集仍是按照6:2:2的比例进行划分,其中训练集2532张,验证集841张,测试集841张,其余的实验条件与乳腺癌组织病理数据集实验保持一致。脑癌组织病理图像的实验结果如表6图5所示,可以看出,本文所提出的AMFNet在更大的脑癌数据集上,相比其他13种前沿先进模型,仍有最佳的肿瘤分级准确率和可解释性。

    Figure 5. Comparison of visualization results of histopathological images of brain cancer
    Full-Size Img PowerPoint

    Comparison of visualization results of histopathological images of brain cancer

    基于ER IHC染色的病理组织显微成像对乳腺癌肿瘤进行分级是一项重要而具有挑战性的工作,它依赖于病理学家的经验和时间。特别是,现有的辅助诊断算法的肿瘤分级结果不能同时具有良好的准确性和可解释性。因此,本文提出了一种基于自适应模型融合的端到端深度网络,其保证了ViT和AMCNN块同时具有良好的特征表示能力,可以提高模型融合和冗余去除能力。大量的实验结果表明,本文提出的AMFNet不仅比其他SOTA模型具有更高的肿瘤分级准确性,而且能够对肿瘤分级结果进行更好的可视化解释,特别是模型对异型核的关注与病理分析更加一致。通过大量的实验证实本文提出的AMFNet在一定程度上提高了现有乳腺癌辅助诊断算法的临床可信度。

    所有作者声明无利益冲突

  • References

    [1]

    Parkin D M, Bray F, Ferlay J, et al. Estimating the world cancer burden: globocan 2000[J]. Int J Cancer, 2001, 94(2): 153−156.

    DOI: 10.1002/ijc.1440

    CrossRef Google Scholar

    [2]

    Robertson S, Azizpour H, Smith K, et al. Digital image analysis in breast pathology—from image processing techniques to artificial intelligence[J]. Transl Res, 2018, 194: 19−35.

    DOI: 10.1016/j.trsl.2017.10.010

    CrossRef Google Scholar

    [3]

    Desai S B, Moonim M T, Gill A K, et al. Hormone receptor status of breast cancer in India: a study of 798 tumours[J]. Breast, 2000, 9(5): 267−270.

    DOI: 10.1054/brst.2000.0134

    CrossRef Google Scholar

    [4]

    Zafrani B, Aubriot M H, Mouret E, et al. High sensitivity and specificity of immunohistochemistry for the detection of hormone receptors in breast carcinoma: comparison with biochemical determination in a prospective study of 793 cases[J]. Histopathology, 2000, 37(6): 536−545.

    DOI: 10.1046/j.1365-2559.2000.01006.x

    CrossRef Google Scholar

    [5]

    Fuqua S A W, Schiff R, Parra I, et al. Estrogen receptor β protein in human breast cancer: correlation with clinical tumor parameters[J]. Cancer Res, 2003, 63(10): 2434−2439.

    Google Scholar

    [6]

    Vagunda V, Šmardová J, Vagundová M, et al. Correlations of breast carcinoma biomarkers and p53 tested by FASAY and immunohistochemistry[J]. Pathol Res Pract, 2003, 199(12): 795−801.

    DOI: 10.1078/0344-0338-00498

    CrossRef Google Scholar

    View full references list
  • Cited by

    Periodical cited type(2)

    1. 侯琳,林抟宇,陈窕璇,黄海健,黄伟鹏. 酰胺质子转移加权成像与人工智能融合技术在乳腺癌中的应用进展. 分子影像学杂志. 2025(05): 633-638 .
    2. 叶世杰,王永雄. 基于图神经网络的WSI癌症生存预测方法. 光电工程. 2024(04): 66-78 . 本站查看

    Other cited types(1)

  • Author Information

  • Copyright

    The copyright belongs to the Institute of Optics and Electronics, Chinese Academy of Sciences, but the article content can be freely downloaded from this website and used for free in academic and research work.
  • About this Article

    DOI: 10.12086/oee.2023.220158
    Cite this Article
    Huang Pan, He Peng, Yang Xing, Luo Jiayang, Xiao Hualiang, Tian Sukun, Feng Peng. Breast tumor grading network based on adaptive fusion and microscopic imaging. Opto-Electronic Engineering 50, 220158 (2023). DOI: 10.12086/oee.2023.220158
    Download Citation
    Article History
    • Received Date July 07, 2022
    • Revised Date September 05, 2022
    • Accepted Date September 05, 2022
    • Available Online December 27, 2022
    • Published Date January 24, 2023
    Article Metrics
    Article Views(8832) PDF Downloads(1001)
    Share:
  • Related Articles

  • Datasets Parameter
    Grade IGrade IIGrade IIIImage sizeTotal
    Training set268355360224×224983
    Validation set90118120224×224328
    Testing set90119120224×224329
    Total448592600224×2241640
    View in article Downloads
  • RealityForecast result
    PositiveNegative
    PositiveTrue positive (TP)False negative (FN)
    NegativeFalse positive (FP)True negative (TN)
    View in article Downloads
  • Model AMF methodAverage acc/%PRF1AUC
    MOOAFRMAFF
    AMFNet (ViT-AMCNN blocks)69.000.69340.69000.69030.7643
    92.400.92400.92400.92400.9423
    93.920.94030.93920.93940.9532
    95.140.95200.95140.95130.9617
    View in article Downloads
  • Model Grade I acc/%Grade II acc/%Grade III acc/%Average acc/%PRF1AUC
    Inception V3[36]74.7178.1978.0177.200.77210.77200.77170.8290
    Xception V3[37]71.8268.4076.4272.340.72260.72340.72260.7925
    ResNet50[38]72.9673.7776.0874.470.75240.74470.74390.8085
    DenseNet121[39]82.8077.5381.6380.550.80590.80550.80470.8541
    DenseNet121+Nonlocal[40]85.8884.3981.9783.890.83960.83890.83910.8791
    DenseNet121+SENet[41]79.5583.2784.3982.670.82720.82670.82660.8700
    DenseNet121+CBAM[42]82.7681.2084.8082.980.83070.82980.82940.8723
    DenseNet121+HIENet[43]86.2185.5985.4885.710.85850.85710.85720.8928
    FABNet[15]86.0591.2984.9087.540.87650.87540.87520.9036
    ViT-S/16[27]54.0255.8769.2060.1860.3760.180.60230.6970
    ViT-B/16[27]85.8786.3284.1785.410.85460.85410.85410.8913
    ViT-B/32[27]68.6070.0073.9871.120.71140.71120.71070.7799
    ViT-L/16[27]78.9879.1781.2379.940.81130.79940.79870.8430
    ViT-L/32[27]50.3561.2159.8358.360.60180.58360.57740.6770
    AMFNet (ours)92.6697.0295.1295.14 \uparrow 7.60.95200.95140.95130.9617
    View in article Downloads
  • Datasets Grade IGrade IIGrade IIIGrade IVTotal
    Training set2764928738912532
    Validation set91164290296841
    Testing set91164290296841
    Total458820145314834214
    View in article Downloads
  • Model Metrics
    Grade I acc/%Grade II acc/%Grade III acc/%Grade IV acc/%Average acc/%PRF1AUC
    Inception V3[36]84.1668.1180.6278.0577.760.77790.77760.77660.8575
    Xception V3[37]84.9572.3478.3180.8078.830.79760.78830.78740.8588
    ResNet50[38]82.0065.6266.4371.1269.800.69630.69800.69610.8091
    DenseNet121[39]87.0574.7574.8681.7578.950.79780.78950.78580.8625
    DenseNet121+Nonlocal[40]91.8481.9086.7089.3987.400.87630.87400.87270.9195
    DenseNet121+SENet[41]96.2284.9788.0590.2589.180.89250.89180.89110.9279
    DenseNet121+CBAM[42]92.7182.0089.0486.7787.400.87500.87400.87260.9162
    DenseNet121+HIENet[43]95.7085.9987.8187.5088.230.88510.88230.88200.9244
    FABNet[15]93.6887.7090.9790.8290.610.90720.90610.90570.9391
    ViT-S/16[27]65.9846.9865.9769.7364.210.63750.64210.63590.7489
    ViT-B/16[27]83.9080.0087.8386.8385.610.86180.85610.85530.9028
    ViT-B/32[27]81.4862.5978.9377.2075.740.75500.75740.75410.8319
    ViT-L/16[27]70.7954.4972.0573.5669.320.68970.69320.69020.7808
    ViT-L/32[27]75.4958.0273.7675.0871.700.71520.71700.71340.8084
    AMFNet (ours)98.3293.3894.3093.9094.41 \uparrow 3.80.94510.94410.94420.9611
    View in article Downloads
[1]

Parkin D M, Bray F, Ferlay J, et al. Estimating the world cancer burden: globocan 2000[J]. Int J Cancer, 2001, 94(2): 153−156.

DOI: 10.1002/ijc.1440

CrossRef Google Scholar

[2]

Robertson S, Azizpour H, Smith K, et al. Digital image analysis in breast pathology—from image processing techniques to artificial intelligence[J]. Transl Res, 2018, 194: 19−35.

DOI: 10.1016/j.trsl.2017.10.010

CrossRef Google Scholar

[3]

Desai S B, Moonim M T, Gill A K, et al. Hormone receptor status of breast cancer in India: a study of 798 tumours[J]. Breast, 2000, 9(5): 267−270.

DOI: 10.1054/brst.2000.0134

CrossRef Google Scholar

[4]

Zafrani B, Aubriot M H, Mouret E, et al. High sensitivity and specificity of immunohistochemistry for the detection of hormone receptors in breast carcinoma: comparison with biochemical determination in a prospective study of 793 cases[J]. Histopathology, 2000, 37(6): 536−545.

DOI: 10.1046/j.1365-2559.2000.01006.x

CrossRef Google Scholar

[5]

Fuqua S A W, Schiff R, Parra I, et al. Estrogen receptor β protein in human breast cancer: correlation with clinical tumor parameters[J]. Cancer Res, 2003, 63(10): 2434−2439.

Google Scholar

[6]

Vagunda V, Šmardová J, Vagundová M, et al. Correlations of breast carcinoma biomarkers and p53 tested by FASAY and immunohistochemistry[J]. Pathol Res Pract, 2003, 199(12): 795−801.

DOI: 10.1078/0344-0338-00498

CrossRef Google Scholar

[7]

Baqai T, Shousha S. Oestrogen receptor negativity as a marker for high-grade ductal carcinoma in situ of the breast[J]. Histopathology, 2003, 42(5): 440−447.

DOI: 10.1046/j.1365-2559.2003.01612.x

CrossRef Google Scholar

[8]

Sofi G N, Sofi J N, Nadeem R, et al. Estrogen receptor and progesterone receptor status in breast cancer in relation to age, histological grade, size of lesion and lymph node involvement[J]. Asian Pac J Cancer Prev, 2012, 13(10): 5047−5052.

DOI: 10.7314/apjcp.2012.13.10.5047

CrossRef Google Scholar

[9]

Azizun-Nisa, Bhurgri Y, Raza F, et al. Comparison of ER, PR and HER-2/neu (C-erb B 2) reactivity pattern with histologic grade, tumor size and lymph node status in breast cancer[J]. Asian Pac J Cancer Prev, 2008, 9(4): 553−556.

Google Scholar

[10]

Gurcan M N, Boucheron L E, Can A, et al. Histopathological image analysis: a review[J]. IEEE Rev Biomed Eng, 2009, 2: 147−171.

DOI: 10.1109/RBME.2009.2034865

CrossRef Google Scholar

[11]

Yang H, Kim J Y, Kim H, et al. Guided soft attention network for classification of breast cancer histopathology images[J]. IEEE Trans Med Imaging, 2020, 39(5): 1306−1315.

DOI: 10.1109/TMI.2019.2948026

CrossRef Google Scholar

[12]

BenTaieb A, Hamarneh G. Predicting cancer with a recurrent visual attention model for histopathology images[C]//21st International Conference on Medical Image Computing and Computer Assisted Intervention, 2018: 129–137. https://doi.org/10.1007/978-3-030-00934-2_15.

https://doi.org/10.1007/978-3-030-00934-2_15.

" target="_blank">Google Scholar

[13]

Tomita N, Abdollahi B, Wei J, et al. Attention-based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides[J]. JAMA Netw Open, 2019, 2(11): e1914645.

DOI: 10.1001/jamanetworkopen.2019.14645

CrossRef Google Scholar

[14]

Yao J W, Zhu X L, Jonnagaddala J, et al. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks[J]. Med Image Anal, 2020, 65: 101789.

DOI: 10.1016/j.media.2020.101789

CrossRef Google Scholar

[15]

Huang P, Tan X H, Zhou X L, et al. FABNet: Fusion attention block and transfer learning for laryngeal cancer tumor grading in p63 IHC histopathology images[J]. IEEE J Biomed Health Inform, 2022, 26(4): 1696−1707.

DOI: 10.1109/JBHI.2021.3108999

CrossRef Google Scholar

[16]

Zhou X L, Tang C W, Huang P, et al. LPCANet: classification of laryngeal cancer histopathological images using a CNN with position attention and channel attention mechanisms[J]. Interdiscip Sci, 2021, 13(4): 666−682.

DOI: 10.1007/s12539-021-00452-5

CrossRef Google Scholar

[17]

刘敬璇, 樊金宇, 汪权, 等. SS-OCTA对黑色素瘤皮肤结构和血管的成像实验[J]. 光电工程, 2020, 47(2): 190239.

DOI: 10.12086/oee.2020.190239

Liu J X, Fan J Y, Wang Q, et al. Imaging of skin structure and vessels in melanoma by swept source optical coherence tomography angiography[J]. Opto-Electron Eng, 2020, 47(2): 190239.

DOI: 10.12086/oee.2020.190239

CrossRef Google Scholar

[18]

Abnar S, Zuidema W. Quantifying attention flow in transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 4190–4197. https://doi.org/10.18653/v1/2020.acl-main.385.

https://doi.org/10.18653/v1/2020.acl-main.385.

" target="_blank">Google Scholar

[19]

Chefer H, Gur S, Wolf L. Transformer interpretability beyond attention visualization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 782–791. https://doi.org/10.1109/CVPR46437.2021.00084.

https://doi.org/10.1109/CVPR46437.2021.00084.

" target="_blank">Google Scholar

[20]

梁礼明, 周珑颂, 陈鑫, 等. 鬼影卷积自适应视网膜血管分割算法[J]. 光电工程, 2021, 48(10): 210291.

DOI: 10.12086/oee.2021.210291

Liang L M, Zhou L S, Chen X, et al. Ghost convolution adaptive retinal vessel segmentation algorithm[J]. Opto-Electron Eng, 2021, 48(10): 210291.

DOI: 10.12086/oee.2021.210291

CrossRef Google Scholar

[21]

刘侠, 甘权, 刘晓, 等. 基于超像素的联合能量主动轮廓CT图像分割方法[J]. 光电工程, 2020, 47(1): 190104.

DOI: 10.12086/oee.2020.190104

Liu X, Gan Q, Liu X, et al. Joint energy active contour CT image segmentation method based on super-pixel[J]. Opto-Electron Eng, 2020, 47(1): 190104.

DOI: 10.12086/oee.2020.190104

CrossRef Google Scholar

[22]

Gao Z Y, Hong B Y, Zhang X L, et al. Instance-based vision transformer for subtyping of papillary renal cell carcinoma in histopathological image[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 299–308. https://doi.org/10.1007/978-3-030-87237-3_29.

https://doi.org/10.1007/978-3-030-87237-3_29.

" target="_blank">Google Scholar

[23]

Wang X Y, Yang S, Zhang J, et al. TransPath: Transformer-based self-supervised learning for histopathological image classification[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 186–195. https://doi.org/10.1007/978-3-030-87237-3_18.

https://doi.org/10.1007/978-3-030-87237-3_18.

" target="_blank">Google Scholar

[24]

Li H, Yang F, Zhao Y, et al. DT-MIL: deformable transformer for multi-instance learning on histopathological image[C]//24th International Conference on Medical Image Computing and Computer Assisted Intervention, 2021: 206–216. https://doi.org/10.1007/978-3-030-87237-3_20.

https://doi.org/10.1007/978-3-030-87237-3_20.

" target="_blank">Google Scholar

[25]

Chen H Y, Li C, Wang G, et al. GasHis-transformer: a multi-scale visual transformer approach for gastric histopathological image detection[J]. Pattern Recognit, 2022, 130: 108827.

DOI: 10.1016/j.patcog.2022.108827

CrossRef Google Scholar

[26]

Zou Y, Chen S N, Sun Q L, et al. DCET-Net: dual-stream convolution expanded transformer for breast cancer histopathological image classification[C]//2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021: 1235–1240. https://doi.org/10.1109/BIBM52615.2021.9669903.

https://doi.org/10.1109/BIBM52615.2021.9669903.

" target="_blank">Google Scholar

[27]

Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale[C]//9th International Conference on Learning Representations, 2021.

Google Scholar

[28]

Breiman L. Bagging predictors[J]. Mach Learn, 1996, 24(2): 123−140.

DOI: 10.1023/A:1018054314350

CrossRef Google Scholar

[29]

Schapire R E. The boosting approach to machine learning: an overview[M]//Denison D D, Hansen M H, Holmes C C, et al. Nonlinear Estimation and Classification. New York: Springer, 2003: 149–171. https://doi.org/10.1007/978-0-387-21579-2_9.

https://doi.org/10.1007/978-0-387-21579-2_9.

" target="_blank">Google Scholar

[30]

Ting K M, Witten I H. Stacking bagged and Dagged models[C]//Proceedings of the Fourteenth International Conference on Machine Learning, 1997: 367–375. https://doi.org/10.5555/645526.657147.

https://doi.org/10.5555/645526.657147.

" target="_blank">Google Scholar

[31]

Ke W C, Fan F, Liao P X, et al. Biological gender estimation from panoramic dental X-ray images based on multiple feature fusion model[J]. Sens Imaging, 2020, 21(1): 54.

DOI: 10.1007/s11220-020-00320-4

CrossRef Google Scholar

[32]

周志华. 机器学习[M]. 北京: 清华大学出版社, 2016: 171–196.

Zhou Z H. Machine Learning[M]. Beijing: Tsinghua University Press, 2016: 171–196.

Google Scholar

[33]

Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, 2017: 618–626.

Google Scholar

[34]

Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 618–626. https://doi.org/10.1109/ICCV.2017.74.

https://doi.org/10.1109/ICCV.2017.74.

" target="_blank">Google Scholar

[35]

Kostopoulos S, Glotsos D, Cavouras D, et al. Computer-based association of the texture of expressed estrogen receptor nuclei with histologic grade using immunohistochemically-stained breast carcinomas[J]. Anal Quant Cytol Histol, 2009, 31(4): 187−196.

Google Scholar

[36]

Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 2818–2826. https://doi.org/10.1109/CVPR.2016.308.

https://doi.org/10.1109/CVPR.2016.308.

" target="_blank">Google Scholar

[37]

Chollet F. Xception: deep learning with depthwise separable convolutions[C]//30th IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1800–1807. https://doi.org/10.1109/CVPR.2017.195.

https://doi.org/10.1109/CVPR.2017.195.

" target="_blank">Google Scholar

[38]

He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. https://doi.org/10.1109/CVPR.2016.90.

https://doi.org/10.1109/CVPR.2016.90.

" target="_blank">Google Scholar

[39]

Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2261–2269. https://doi.org/10.1109/CVPR.2017.243.

https://doi.org/10.1109/CVPR.2017.243.

" target="_blank">Google Scholar

[40]

Wang X L, Girshick R, Gupta A, et al. Non-local neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7794–7803. https://doi.org/10.1109/CVPR.2018.00813.

https://doi.org/10.1109/CVPR.2018.00813.

" target="_blank">Google Scholar

[41]

Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks[J]. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011−2023.

DOI: 10.1109/TPAMI.2019.2913372

CrossRef Google Scholar

[42]

Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 3–19. https://doi.org/10.1007/978-3-030-01234-2_1.

https://doi.org/10.1007/978-3-030-01234-2_1.

" target="_blank">Google Scholar

[43]

Sun H, Zeng X X, Xu T, et al. Computer-aided diagnosis in histopathological images of the endometrium using a convolutional neural network and attention mechanisms[J]. IEEE J Biomed Health Inform, 2020, 24(6): 1664−1676.

DOI: 10.1109/JBHI.2019.2944977

CrossRef Google Scholar

[44]

Glotsos D, Kalatzis I, Spyridonos P, Kostopoulos S, Daskalakis A, Athanasiadis E, Ravazoula P, Nikiforidis G, and Cavouras D. Improving accuracy in astrocytomas grading by integrating a robust least squares mapping driven support vector machine classifier into a two level grade classification scheme[J]. Comput Methods Programs Biomed, 2008, 90(3): 251−261.

DOI: 10.1016/j.cmpb.2008.01.006

CrossRef Google Scholar

Related Articles
Show full outline

Catalog

    Corresponding author: Feng Peng, coe-fp@cqu.edu.cn

    1. On this Site
    2. On Google Scholar
    3. On PubMed
    Breast tumor grading network based on adaptive fusion and microscopic imaging
    • Figure  1

      Block diagram of the algorithm in this paper. (a) ER IHC staining based microscopic imaging procedure for breast cancer pathology; (b) AMFNet

    • Figure  2

      Detail diagram of the AFF method implementation

    • Figure  3

      AFRM method implementation schematic

    • Figure  4

      Visually interpretative results comparisons with SOTA and our method on the breast cancer IHC microscopic imaging

    • Figure  5

      Comparison of visualization results of histopathological images of brain cancer

    • Figure  1
    • Figure  2
    • Figure  3
    • Figure  4
    • Figure  5
    Breast tumor grading network based on adaptive fusion and microscopic imaging
    • Datasets Parameter
      Grade IGrade IIGrade IIIImage sizeTotal
      Training set268355360224×224983
      Validation set90118120224×224328
      Testing set90119120224×224329
      Total448592600224×2241640
    • RealityForecast result
      PositiveNegative
      PositiveTrue positive (TP)False negative (FN)
      NegativeFalse positive (FP)True negative (TN)
    • Model AMF methodAverage acc/%PRF1AUC
      MOOAFRMAFF
      AMFNet (ViT-AMCNN blocks)69.000.69340.69000.69030.7643
      92.400.92400.92400.92400.9423
      93.920.94030.93920.93940.9532
      95.140.95200.95140.95130.9617
    • Model Grade I acc/%Grade II acc/%Grade III acc/%Average acc/%PRF1AUC
      Inception V3[36]74.7178.1978.0177.200.77210.77200.77170.8290
      Xception V3[37]71.8268.4076.4272.340.72260.72340.72260.7925
      ResNet50[38]72.9673.7776.0874.470.75240.74470.74390.8085
      DenseNet121[39]82.8077.5381.6380.550.80590.80550.80470.8541
      DenseNet121+Nonlocal[40]85.8884.3981.9783.890.83960.83890.83910.8791
      DenseNet121+SENet[41]79.5583.2784.3982.670.82720.82670.82660.8700
      DenseNet121+CBAM[42]82.7681.2084.8082.980.83070.82980.82940.8723
      DenseNet121+HIENet[43]86.2185.5985.4885.710.85850.85710.85720.8928
      FABNet[15]86.0591.2984.9087.540.87650.87540.87520.9036
      ViT-S/16[27]54.0255.8769.2060.1860.3760.180.60230.6970
      ViT-B/16[27]85.8786.3284.1785.410.85460.85410.85410.8913
      ViT-B/32[27]68.6070.0073.9871.120.71140.71120.71070.7799
      ViT-L/16[27]78.9879.1781.2379.940.81130.79940.79870.8430
      ViT-L/32[27]50.3561.2159.8358.360.60180.58360.57740.6770
      AMFNet (ours)92.6697.0295.1295.14 \uparrow 7.60.95200.95140.95130.9617
    • Datasets Grade IGrade IIGrade IIIGrade IVTotal
      Training set2764928738912532
      Validation set91164290296841
      Testing set91164290296841
      Total458820145314834214
    • Model Metrics
      Grade I acc/%Grade II acc/%Grade III acc/%Grade IV acc/%Average acc/%PRF1AUC
      Inception V3[36]84.1668.1180.6278.0577.760.77790.77760.77660.8575
      Xception V3[37]84.9572.3478.3180.8078.830.79760.78830.78740.8588
      ResNet50[38]82.0065.6266.4371.1269.800.69630.69800.69610.8091
      DenseNet121[39]87.0574.7574.8681.7578.950.79780.78950.78580.8625
      DenseNet121+Nonlocal[40]91.8481.9086.7089.3987.400.87630.87400.87270.9195
      DenseNet121+SENet[41]96.2284.9788.0590.2589.180.89250.89180.89110.9279
      DenseNet121+CBAM[42]92.7182.0089.0486.7787.400.87500.87400.87260.9162
      DenseNet121+HIENet[43]95.7085.9987.8187.5088.230.88510.88230.88200.9244
      FABNet[15]93.6887.7090.9790.8290.610.90720.90610.90570.9391
      ViT-S/16[27]65.9846.9865.9769.7364.210.63750.64210.63590.7489
      ViT-B/16[27]83.9080.0087.8386.8385.610.86180.85610.85530.9028
      ViT-B/32[27]81.4862.5978.9377.2075.740.75500.75740.75410.8319
      ViT-L/16[27]70.7954.4972.0573.5669.320.68970.69320.69020.7808
      ViT-L/32[27]75.4958.0273.7675.0871.700.71520.71700.71340.8084
      AMFNet (ours)98.3293.3894.3093.9094.41 \uparrow 3.80.94510.94410.94420.9611
    • Table  1

      Distribution of the number of ER IHC datasets for breast cancer

        1/6
    • Table  2

      Classification confusion matrix

        2/6
    • Table  3

      Ablation of AMF method in ER IHC pathological microimaging of breast cancer

        3/6
    • Table  4

      Tumor grading accuracy of breast cancer ER IHC histopathology microscopic imaging

        4/6
    • Table  5

      Distribution of the number of brain cancer histopathology image datasets

        5/6
    • Table  6

      Comparison table of tumor grading accuracy of histopathological images of brain cancer

        6/6