• •

### An Automatic Classification Pipeline for the Complex Synaptic Structure Based on Deep Learning

SHEN Lijun1, MA Chao1, LUO Jie2, HONG Bei2

1. 1. Faculty of Information Technology, Macau University of Science and Technology, Macau 999078, China;
2. Faculty of Mathematics and Statistics, Hubei University, Wuhan 430062, China
• 收稿日期:2020-12-04 修回日期:2021-06-02 出版日期:2022-08-25 发布日期:2022-08-02
• 作者简介:SHEN Lijun,Email:shenlj.must@gmail.com;MA Chao,Email:chao.ma.must@gmail.com;LUO Jie,Email:luoj0901@163.com;HONG Bei,Email:hong bei1994@163.com
• 基金资助:
This research was supported by the Science and Technology Development Fund, Macau SAR under Grant No. 0024/2018/A1, the Research Fund of Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent MicroNano Optoelectronic Technology under Grant No. 2020B1212030010, the National Science Foundation of China under Grant Nos. 61673381, 61201050, 61306070, 61701497, 11771130, 61871177, Special Program of Beijing Municipal Science & Technology Commission under Grant No. Z181100000118002, Strategic Priority Research Program of Chinese Academy of Science under Grant No. XDB32030200, and Scientific research instrument and equipment development project of Chinese Academy of Sciences under Grant No. YZ201671.

SHEN Lijun, MA Chao, LUO Jie, HONG Bei. An Automatic Classification Pipeline for the Complex Synaptic Structure Based on Deep Learning[J]. 系统科学与复杂性, 2022, 35(4): 1398-1414.

SHEN Lijun, MA Chao, LUO Jie, HONG Bei. An Automatic Classification Pipeline for the Complex Synaptic Structure Based on Deep Learning[J]. Journal of Systems Science and Complexity, 2022, 35(4): 1398-1414.

### An Automatic Classification Pipeline for the Complex Synaptic Structure Based on Deep Learning

SHEN Lijun1, MA Chao1, LUO Jie2, HONG Bei2

1. 1. Faculty of Information Technology, Macau University of Science and Technology, Macau 999078, China;
2. Faculty of Mathematics and Statistics, Hubei University, Wuhan 430062, China
• Received:2020-12-04 Revised:2021-06-02 Online:2022-08-25 Published:2022-08-02
• Supported by:
This research was supported by the Science and Technology Development Fund, Macau SAR under Grant No. 0024/2018/A1, the Research Fund of Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent MicroNano Optoelectronic Technology under Grant No. 2020B1212030010, the National Science Foundation of China under Grant Nos. 61673381, 61201050, 61306070, 61701497, 11771130, 61871177, Special Program of Beijing Municipal Science & Technology Commission under Grant No. Z181100000118002, Strategic Priority Research Program of Chinese Academy of Science under Grant No. XDB32030200, and Scientific research instrument and equipment development project of Chinese Academy of Sciences under Grant No. YZ201671.
As a hallmark of brain complexity, synapses in the nervous system have always received extensive attentions. The diversity of the synaptic structure reflects various functions and mechanisms, some research indicates that, as one of the complex synaptic structures, multiple synapses can strengthen the synaptic connection, what's more, it is closely associated with the procedure of memory and learning. Accompanied by the fast advancement of electron microscopy (EM) technology, it is possible to detect the composition of multiple synapse with high resolution. On this basis, there have been various meaningful studies concerning the relationship between the multiple synapse and cognitive abilities. Despite the extensive studies have been made by different researchers on multiple synapse, no attention has been paid to the classification accuracy of the type of multiple synapse. The current research puts forward an effective method for the automatic classification of multiple synapse, which should be performed in three steps, namely the segmentation of synaptic clefts, the segmentation of vesicle bands, as well as the segmentation of multiple synapses. According to experimental results based on four data sets, the mean classification rate of the method is approximately 97%. In addition, the experimental result on the public dataset shows that the accuracy can reach 96.5%. The classification results provide a basis for quantitative statistics of follow-up studies. Moreover, this automatic classification method can reduce the time in artificial statistics, and thus researchers can focus more attention on the analysis of statistical results.
As a hallmark of brain complexity, synapses in the nervous system have always received extensive attentions. The diversity of the synaptic structure reflects various functions and mechanisms, some research indicates that, as one of the complex synaptic structures, multiple synapses can strengthen the synaptic connection, what's more, it is closely associated with the procedure of memory and learning. Accompanied by the fast advancement of electron microscopy (EM) technology, it is possible to detect the composition of multiple synapse with high resolution. On this basis, there have been various meaningful studies concerning the relationship between the multiple synapse and cognitive abilities. Despite the extensive studies have been made by different researchers on multiple synapse, no attention has been paid to the classification accuracy of the type of multiple synapse. The current research puts forward an effective method for the automatic classification of multiple synapse, which should be performed in three steps, namely the segmentation of synaptic clefts, the segmentation of vesicle bands, as well as the segmentation of multiple synapses. According to experimental results based on four data sets, the mean classification rate of the method is approximately 97%. In addition, the experimental result on the public dataset shows that the accuracy can reach 96.5%. The classification results provide a basis for quantitative statistics of follow-up studies. Moreover, this automatic classification method can reduce the time in artificial statistics, and thus researchers can focus more attention on the analysis of statistical results.
()
 [1] Yang Y, Liu D, Huang W, et al., Selective synaptic remodeling of amygdalocortical connections associated with fear memory, Nature Neuroscience, 2016, 19(10): 1348-1355.[2] Zhang X and Poo M, Progress in neural plasticity, Science China Life Sciences, 2010, 53(3): 322-329.[3] Medvedev N I, Dallrac G, Popov V I, et al., Multiple spine boutons are formed after long-lasting LTP in the awake rat, Brain Structure and Function, 2014, 219(1): 407-414.[4] Hruska M, Henderson N, Le Marchand S J, et al., Synaptic nanomodules underlie the organizationand plasticity of spine synapses, Nature Neuroscience, 2018, 21: 671-682.[5] Reilly J E, Hanson H H, Fernndez-Monreal M, et al., Characterization of MSB synapses in dissociated hippocampal culture with simultaneous pre- and postsynaptic live microscopy, PLoS One, 2011, 6(10): e26478.[6] Toni N, Buchs P A, Nikonenko I, et al., LTP promotes formation of multiple spine synapses between a single axon terminal and a dendrite, Nature Neuroscience, 1999, 402(6760): 421-425.[7] Nagerl U V, Kostinger G, Anderson J C, et al., Protracted synaptogenesis after activity-dependent spinogenesis in hippocampal neurons, J Neurosci, 2007, 27(30): 8149-8156.[8] Medvedev N I, Dallrac G, Popov V I, et al., Multiple spine boutons are formed after long-lasting LTP in the awake rat, Brain Structure and Function, 2014, 219(1): 407-414.[9] Geinisman Y, Berry R W, Disterhoft J F, et al., Associative learning elicits the formation of multiple-synapse boutons, J Neurosci, 2001, 21(15): 5568-5573.[10] Knott G W, Holtmaat A, Wilbrecht L, et al., Spine growth precedes synapse formation in the adult neocortex in vivo, Nature Neuroscience, 2006, 9(9): 1117-1124.[11] Black J E, Isaacs K R, Anderson B J, et al., Learning causes synaptogenesis, whereas motor activity causes angiogenesis, in cerebellar cortex of adult rats, Proceedings of the National Academy of Sciences USA, 1990, 87(14): 5568-5572.[12] Crair M C, Gillespie D C, and Stryker M P, The role of visual experience in the development of columns in cat visual cortex, Science, 1998, 279(5350): 566-570.[13] Kim H W, Oh S, Lee S H, et al., Different types of multiple-synapse boutons in the cerebellar cortex between physically enriched and ataxic mutant mice, Microscopy Research and Technique, 2019, 82(2): 25-32.[14] Wu L F, Xu Y W, Jian M, et al., Face liveness detection scheme with static and dynamic features, International Journal of Wavelets Multiresolution and Information Processing, 2018, 16(2): 1840001.[15] Jiang Y, Zhuo J Y, Zhang J, et al., The optimization of parallel convolutional RBM based on Spark, International Journal of Wavelets Multiresolution and Information Processing, 2019, 17(2): 1902001.[16] Cui Y, Zhang G, Liu Z, et al., A deep learning algorithm for one-step contour aware nuclei segmentation of histopathological images, Medical & Biological Engineering & Computing, 2019, 57: 2027-2043.[17] Xiao C, Liu J, Chen X, et al., Deep contextual residual network for electron microscopy image segmentation in connectomics, IEEE 15th International Symposium on Biomedical Imaging, 2018, 378-381.[18] Quan T M, Hildebrand D G C, and Jeong W K, FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics. arXiv preprint, 2016, arXiv: 1612.05360.[19] Li W F, Deng H, Rao Q, et al., An automated pipeline for mitochondrial segmentation on ATUMSEM Stacks, Journal of Bioinformatics and Computational Biology, 2017, 15(3): 1750015.[20] Xiao C, Chen X, Li W F, et al., Automatic mitochondria segmentation for EM data using a 3d supervised convolutional network, Frontiers in Neuroanatomy, 2018, 12: 92, DOI: 10.3389/fnana.2018.00092.[21] Xie Q W, Chen X, Shen L J, et al., Brain microstructure reconstruction based on deep learning, Systems Engineering — Theory & Practice, 2018, 38(2): 482-491.[22] Sigdel M, Dinc I, Sigdel M S, et al. Feature analysis for classification of trace fluorescent labeled protein crystallization images, BioData Mining, 2017, 10(1): 14, DOI: 10.1186/s13040-017-0133-9.[23] Kaltdorf K V, Theiss M, Markert S M, et al., Automated classification of synaptic vesicles in electron tomograms of C. elegans using machine learning, PLoS One, 2018, 13(10): e0205348.[24] Pang S, Du A, Orgun M A, et al., A novel fused convolutional neural network for biomedical image classification, Medical & Biological Engineering & Computing, 2019, 57: 107-121.[25] Zhao J, Zhang M, Zhou Z, et al., Automatic detection and classification of leukocytes using convolutional neural networks, Medical & Biological Engineering & Computing, 2017, 55: 1287- 1301.[26] Wu Z, Shen C, and Van Den Hengel A, Wider or deeper: Revisiting the resnet model for visual recognition, Pattern Recognition, 2019, 90: 119-113.[27] Szegedy C, Vanhoucke V, Ioffe S, et al., Rethinking the inception architecture for computer vision, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 2818- 2826.[28] Huang G, Liu Z, Van Der Maaten L, et al., Densely connected convolutional networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 4700-4708.[29] Shan G C, Wang H Y, and Liang W, Robust encoder-decoder learning framework towards offline handwritten mathematical expression recognition based on multi-scale deep neural network, Science China: Information Sciences, 2021, 64(3): 139101.[30] Wang H Y and Shan G C, Recognizing handwritten mathematical expressions as LaTex sequences using a multiscale robust neural network, 2020, arXiv: 2003.00817.[31] Luo J, Hong B, Jiang Y, et al., Automatic classification for the type of multiple synapse based on deep learning, 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019, 40-43.[32] He K M, Gkioxari G, Dollar P, et al., Mask R-CNN, arXiv preprint, 2018, arXiv: 1703.06870.[33] Luo J, Raw picture, figshare, 10.6084/m9.figshare.10731641.[34] Schmid B, Schindelin J, Cardona A, et al., A high-level 3d visualization api for java and imagej, BMC Bioinformatics, 2010, 11: 1-7.[35] Dataset sources: https://neurodata.io/data/kasthuri15/.[36] Kasthuri N, Hayworth K J, Berger D R, et al., Saturated reconstruction of a volume of neocortex, Cell, 2015, 162(3): 648-661.[37] Keras: Deep learning library for theano and tensorflow, 2015, http://keras.io/.[38] SynapseWeb, Kristen M. Harris, PI, https://synapseweb.clm.utexas.edu/em-cleft.[39] Hong B, Liu J, Li W F, et al., Fully automatic synaptic cleft detection and segmentation from EM images based on deep learning, Advances in Brain Inspired Cognitive Systems, 2018, 64-74.[40] Li W F, Liu J, Xiao C, et al., A fast forward 3D connection algorithm for mitochondria and synapse segmentations from serial EM images, BioData Mining, 2018, 11(1): 24, DOI: 10.1186/s13040- 018-0183-7.[41] Walsby A E, Gas Vesicles, Microbiological Reviews, 1994, 58(1): 94-144.[42] CREMI challenge: https://cremi.org.
 [1] Kamal H.Yasir;Yun TANG. CLASSIFICATION OF CUBIC PARAMETERIZED HOMOGENEOUS VECTOR FIELDS[J]. Journal of Systems Science and Complexity, 2002, 15(2): 125-138. [2] LI Banghe;ZHANG Ping. ON ISOTOPY CLASSIFICATION OF EMBEDDINGS OF n-MANIFOLDS IN 2n-MANIFOLDS[J]. Journal of Systems Science and Complexity, 1993, 6(1): 61-069.