A generic high-throughput microstructure classification and quantification method for regular SEM images of complex steel microstructures combining EBSD labeling and deep learning

doi:10.1016/j.jmst.2021.04.009

Abstract

Abstract:

We present an electron backscattered diffraction (EBSD)-trained deep learning (DL) method integrating traditional material characterization informatics and artificial intelligence for a more accurate classification and quantification of complex microstructures using only regular scanning electron microscope (SEM) images. In this method, EBSD analysis is applied to produce accurate ground truth data for guiding the DL model training. An U-Net architecture is used to establish the correlation between SEM input images and EBSD ground truth data using only small experimental datasets. The proposed method is successfully applied to two engineering steels with complex microstructures, i.e., a dual-phase (DP) steel and a quenching and partitioning (Q&P) steel, to segment different phases and quantify phase content and grain size. Alternatively, once properly trained the method can also produce quasi-EBSD maps by inputting regular SEM images. The good generality of the trained models is demonstrated by using DP and Q&P steels not associated with the model training. Finally, the method is applied to SEM images with various states, i.e., different imaging modes, image qualities and magnifications, demonstrating its good robustness and strong application ability. Furthermore, the visualization of feature maps during the segmenting process is utilised to explain the mechanism of this method's good performance.

Key words: Microstructure quantification, Deep learning, Electron backscatter diffraction, Small sample problem

Chunguang Shen, Chenchong Wang, Minghao Huang, Ning Xu, Sybrandvan der Zwaag, Wei Xu. A generic high-throughput microstructure classification and quantification method for regular SEM images of complex steel microstructures combining EBSD labeling and deep learning[J]. J. Mater. Sci. Technol., 2021, 93: 191-204.

Figures/Tables 14

Fig. 1. The schematic diagram of the EBSD-trained DL method and its applications.

Table 1. Composition of the alloys used in this work. Compositions are in weight percentages.

	Fe	C	Si	Mn	Ti	Nb	Cr	Ni
DP / DP-validation steel	Bal.	0.02	0.52	1.23	0.09	-	17.60	9.22
Q&P steel	Bal.	0.25	1.70	2.00	0.03	0.02	-	-
Q&P-validation steel	Bal.	0.27	1.64	2.22	0.01	0.01	0.01	-

Fig. 2. The processing routes for the selected steels: (a) DP steel, (b) Q&P steel; the typical microstructures: (c) BSE image of DP steel, (f) SE images of Q&P steel; (d) EBSD ground truth from (c) BSE image; (g) EBSD ground truth from (f) SE image; (e, h) original EBSD maps.

Fig. 3. The U-Net network architecture. The numbers on the top (or bottom) and at the right (or left) of each block correspond to the number of filters and the size of the feature map, respectively.

Fig. 4. The segmentation results of DP steel: (a) the distribution of the evaluation indices PA and MIoU; (b, e) BSE images of two areas taken from the testing images, (c, f) corresponding EBSD ground truth and (d, g) corresponding segmentation results.

Fig. 5. Quantitative analysis of martensite content in DP steel: (a) original BSE image, (b) segmentation result and (c) ground truth for calculating martensite content; (d) distribution of deviations of quantitative results between EBSD and the present method.

Fig. 6. The segmentation results of Q&P steel: (a) the distribution of the evaluation indices PA and MIoU; (b, e) SE images of two areas taken from the testing images, (c, f) corresponding EBSD ground truth and (d, g) corresponding segmentation results.

Fig. 7. The quantitative analysis of Q&P steel. (a) the SE image, (b) EBSD ground truth and (c) segmentation result; (d) phase content calculated based on present method and EBSD; (e) the distribution of area of RA from present method and EBSD.

Fig. 8. Segmentation and quantitative analysis of validation steels: (a) BSE image, (b) EBSD phase map and (c) segmentation result of DP-validation steel; (d) SE image, (e) EBSD result and (f) segmentation result of Q&P-validation steel.

Fig. 9. (a) The grayscale distribution of BSE images with various qualities; (b) the segmentation results of BSE images with various qualities by the DLDP model trained on high-quality images; the detailed cases of images with various qualities and ground truth: (c) BSE image and (d) segmentation result for an imaging rate of 100 ns/pixel; (e) BSE image and (f) segmentation result for an imaging rate of 200 ns/pixel; (g) BSE image and (h) segmentation result for an imaging rate of 500 ns/pixel; (i) BSE image and (j) segmentation result for an imaging rate of 1000 ns/pixel; (k) ground truth.

Fig. 10. The SE images at the magnification of (a) 500 ×, (b) 1000 × and (c) 4000 ×; (d) the deviations of evaluation indexes between images with magnifications of 500 ×, 1000 × and 4000 × and the reference image with 2000 × magnification; two segmentation cases for the BSE images at the magnification of (e-g) 500 × and (h-j) 4000 ×: (f, i) EBSD ground truth, (g, j) segmentation results.

Fig. 11. Comparison between the proposed method and a traditional binary image method: (a) BSE image; (b) EBSD phase map; (c) segmentation result; (d) result of binary image; (e) variation in measured martensite content as the threshold value changes. (f) The binary image with noise in austenite; (g) comparative result for the number of noises in austenite between the binary method and the present method; (h) the binary image with noise in martensite; (i) comparative result for the number of noises in martensite between the binary method and the present method; Segmentation result of low-quality images using binary method: (j) the imaging rate of 100 ns/pixel; (k) the imaging rate of 200 ns/pixel; (l) the imaging rate of 500 ns/pixel; (m) the imaging rate of 1000 ns/pixel.

Fig. 12. (a) Visualization of feature maps in the U-Net architecture, a case for Q&P steel; the original feature map and the enhanced feature map from the “skip layer” in (b) Upconv1 and (c) Upconv2; feature maps of Conv1 and Bottleneck for input images with imaging rates of (d) 100 ns/pixel and (e) 1000 ns/pixel, respectively.

Fig. 13. Comparison of computational efficiency and classification accuracy among present method, EBSD and binary method. The number marked on points represents the step size in EBSD region and threshold value in the region of binary method, respectively.

References 56

[1]	P. Gao, M. Fu, M. Zhan, Z. Lei, Y. Li, J. Mater. Sci. Technol., 39(2020), pp. 56-73.
[2]	E.C.H.C. O’ Brien, H.K. Yeddu, J. Mater. Sci. Technol., 49(2020), pp. 157-165.
[3]	D.P. Yang, P.J. Du, D. Wu, H.L. Yi, J. Mater. Sci. Technol., 75(2021), pp. 205-215.
[4]	T. Chen, L. He, M.H. Cullison, C. Hay, J. Burns, Y. Wu, L. Tan, Acta Mater, 195(2020), pp. 433-445.
[5]	Z. Guo, W. Sha, D. Vaumousse, Acta Mater, 51(2003), pp. 101-116.
[6]	Y. Zhang, D. Zhan, X. Qi, Z. Jiang, J. Mater. Sci. Technol., 35(2019), pp. 1240-1249.
[7]	L.E. Samuels, Light Microscopy of Carbon Steels, ASM International, OH(1999).
[8]	Y. Wang, Y. Tomota, T. Ohmura, S. Morooka, W. Gong, S. Harjo, Acta Mater, 184(2020), pp. 30-40.
[9]	D. An, S.-.I. Baik, Q. Ren, M. Jiang, M. Zhu, D. Isheim, B.W. Krakauer, D.N. Seidman, Mater. Charact., 162(2020), Article 110207.
[10]	D. Schryvers, P. Boullay, P.L. Potapov, R.V. Kohn, J.M. Ball, Int. J. Solids Struct., 39(2002), pp. 3543-3554.
[11]	S. Morito, H. Tanaka, R. Konishi, T. Furuhara, T. Maki, Acta Mater, 51(2003), pp. 1789-1799.
[12]	A. Navarro-López, J. Hidalgo, J. Sietsma, M.J. Santofimia, Mater. Charact., 128(2017), pp. 248-256.
[13]	G.Y.P. Medinaa, E.H. Delgadoa, A.F.M. Péreza, H.L. Ferreirab, Mater. Res., 20(2017), pp. 1161-1165
[14]	J.A. Taillon, C. Pellegrinelli, Y.-.L. Huang, E.D. Wachsman, L.G. Salamanca-Riba, Ultramicroscopy, 184(2018), pp. 24-38.
[15]	F.J. Humphreys, Scr. Mater., 51(2004), pp. 771-776.
[16]	A. Winkelmann, G. Cios, T. Tokarski, G. Nolze, R. Hielscher, T. Kozieł, Acta Mater, 188(2020), pp. 376-385.
[17]	F. Long, D. Kerr, G. Domizzi, Q. Wang, M.R. Daymond, Acta Mater, 129(2017), pp. 450-461.
[18]	R. Fialho Tomaz, D. Brandão Santos, K. Camey, R. Barbosa, M. Spangler Andrade, D. Pérez Escobar, J. Mater. Res. Technol., 8(2019), pp. 2423-2431.
[19]	S.L. Shrestha, A.J. Breen, P. Trimby, G. Proust, S.P. Ringer, J.M. Cairney, Ultramicroscopy, 137(2014), pp. 40-47.
[20]	L. Ryde, Mater. Sci. Technol., 22(2006), pp. 1297-1306.
[21]	M. Díaz-Fuentes, A. Iza-Mendia, I. Gutiérrez, Metall. Mater. Trans. A, 34(2003), pp. 2505-2516.
[22]	F.J. Humphreys, J. Microsc., 195(1999), pp. 170-185.
[23]	B. Gaskey, L. Hendl, X. Wang, M. Seita, Acta Mater, 194(2020), pp. 558-564.
[24]	J. Gola, J. Webel, D. Britz, A. Guitar, T. Staudt, M. Winter, F. Mücklich, Comput. Mater. Sci., 160(2019), pp. 186-196.
[25]	J. Gola, D. Britz, T. Staudt, M. Winter, A.S. Schneider, M. Ludovici, F. Mücklich, Comput. Mater. Sci., 148(2018), pp. 324-335.
[26]	B.L. DeCost, E.A. Holm, Comput. Mater. Sci., 110(2015), pp. 126-133.
[27]	M. Guillaumin, J. Verbeek, C. Schmid,Computer Vision, 2009 IEEE 12th International Conference on, IEEE (2009), pp.498-505.
[28]	A. Mignon, F. Jurie,Computer Vision and Pattern Recognition(CVPR), 2012 IEEE Conference on, IEEE (2012), pp.2666-2672.
[29]	T. Okuyama, T. Gonsalves, J. Upadhay,Intelligent Autonomous Systems (ICoIAS), 2018 International Conference on, IEEE (2018), pp.201-205.
[30]	C. Liu, K. Chusap, Z. Li, Z. Chen, D. Rogers, F. Song,Software Maintenance and Evolution (ICSME), 2019 IEEE International Conference on, IEEE (2019), p. 363.
[31]	A. Rajkomar, E. Oren, K. Chen, A.M. Dai, N. Hajaj, M. Hardt, P.J. Liu, X. Liu, J. Marcus, M. Sun, P. Sundberg, H. Yee, K. Zhang, Y. Zhang, G. Flores, G.E. Duggan, J. Irvine, Q. Le, K. Litsch, A. Mossin, J. Tansuwan, D. Wang, J. Wexler, J. Wilson, D. Ludwig, S.L. Volchenboum, K. Chou, M. Pearson, S. Madabushi, N.H. Shah, A.J. Butte, M.D. Howell, C. Cui, G.S. Corrado, J.Dean, npj Digit. Med., 1(2018), pp. 1-10.
[32]	Z. Che, Y. Liu, Data Mining Workshops (ICDMW), 2017 IEEE International Conference on, IEEE (2017), pp.1100-1109.
[33]	A. Mogelmose, M.M. Trivedi, T.B. Moeslund, IEEE T Intell. Transp., 13(2012), pp. 1484-1497.
[34]	Y. Wu, Y. Liu, J. Li, H. Liu, X. Hu, Neural Networks (IJCNN), The 2013 International Joint Conference on, IEEE(2013), pp. 1-7.
[35]	S.M. Azimi, D. Britz, M. Engstler, M. Fritz, F. Mucklich, Sci Rep 8(2018) 2128.
[36]	F. Ajioka, Z.-.L. Wang, T. Ogawa, Y. Adachi, ISIJ Int, 60(2020), pp. 954-959.
[37]	B. Ma, X. Ban, H. Huang, Y. Chen, W. Liu, Y. Zhi, Symmetry-Basel, 10 (2018), p.107.
[38]	B.L. DeCost, B. Lei, T. Francis, E.A. Holm, Microsc. Microanal., 25(2019), pp. 21-29.
[39]	M. Müller, D. Britz, F. Mücklich, Adv. Mater. Process, 179(2021), pp. 16-21.
[40]	P. Yang, R. Guo, P. Guo, Z. Fang,Transportation, Mechanical, and Electrical Engineering (TMEE), Proceedings 2011 International Conference on, IEEE(2011), pp. 1465-1468.
[41]	C. Hofer, S. Primig, H. Clemens, F. Winkelhofer, R. Schnitzer, Adv. Eng. Mater., 19(2017), Article 1600658.
[42]	N. Ibtehaz, M.S. Rahman, Neural Netw, 121(2020), pp. 74-87.
[43]	G. Litjens, T. Kooi, B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, J.A.W.M. van der Laak, B. van Ginneken, C.I. Sánchez, Med. Image Anal., 42(2017), pp. 60-88.
[44]	O. Ronneberger, P. Fischer, T. Brox, Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, Springer International Publishing(2015), pp. 234-241.
[45]	Z. Chen, L. Deng, G. Li, J. Sun, X. Hu, L. Liang, Y. Ding, Y. Xie, IEEE T Neur. Net. Lear., 1(2020), pp. 348-362.
[46]	N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, J. Mach. Learn. Res., 15(2014), pp. 1929-1958.
[47]	J.C. Duchi, E. Hazan, Y. Singer, J. Mach. Learn. Res., 12(2011), pp. 2121-2159.
[48]	P. De Boer, D.P. Kroese, S. Mannor, R.Y. Rubinstein, Ann. Oper. Res., 134(2005), pp. 19-67.
[49]	R.K. Yadav, Anubhav, Cogn. Syst. Res., 64(2020), pp. 191-199.
[50]	J. Long, E. Shelhamer, T. Darrell, 2015 IEEE Conference on, IEEE (2015), pp.3431-3440.
[51]	A. Baskaran, G. Kane, K. Biggs, R. Hull, D. Lewis, Comput. Mater. Sci., 177 (2020), p.109593.
[52]	X. Hu, X. Sun, K. Raghavan, R.J. Comstock, Y. Ren, Mater. Sci. Eng. A, 780(2020), Article 139176.
[53]	F. Badkoobeh, A. Nouri, H. Hassannejad, H. Mostaan, Mater. Sci. Eng. A, 790(2020), Article 139703.
[54]	M.D. Zeiler, R. Fergus, Computer Vision - ECCV 2014, Springer International Publishing (2014), pp. 818-833.
[55]	I. Díaz, A. Diez, A. A.C. Vega, G. Dorffner, H. Bischof, K. Hornik(Eds.), Artificial Neural Networks - ICANN 2001, Berlin,Heidelberg, Springer(2001), pp. 443-449.
[56]	Y.-.F. Shen, R. Pokharel, T.J. Nizolek, A. Kumar, T. Lookman, Acta Mater, 170(2019), pp. 118-131.