Mexican Sign Language word recognition using RGB-D information
DOI:
https://doi.org/10.32870/recibe.v10i2.209Keywords:
EspañolAbstract
Sign Language is the primary alternative method of communication between people with hearing or speech impairment. However, most of the population that does not suffer from this disability cannot understand or interact with them. Consequently, communication of the signatories with their social environment becomes almost impossible. This paper presents progress towards constructing a system to translate words from the Mexican Sign Language into text, by the signatory's hands motion recognition from a 3D trajectory using a Kinect sensor. A corpus of 53 words was built and considered only words belonging to eleven semantic fields. Intermediate points were added, the KNN algorithm was used for filtering to eliminate possible inconsistencies and noise in the extracted pattern. In addition, the descriptor method used divides the pattern into two sections according to the cusp of its trajectory, and the representative 3D positions of both sections are obtained utilizing the arithmetic mean. From the general pattern, its width, height, depth, and orientation are also obtained. For the classification of the words in the corpus, an Artificial Neural Network of the Multi-Layer Perceptron type is used. This network was trained with the Backpropagation algorithm, and for the validation of the recognizing system, it was performed using the K-Fold Cross Validation method. The percentage of mean precision achieved by this implementation was 93.46%.References
INEGI Censo población 2020. (2021). Población. Discapacidad. Cuentame.inegi.org.mx. Recuperado en 5 de mayo de 2021, de: http://www.cuentame.inegi.org.mx/poblacion/discapacidad.aspx?tema=P.
Sensor Kinect. (2021). Microsoft Kinect for Windows Specs y Prices. CNET. Retrieved 3 June 2021, from https://www.cnet.com/products/microsoft-kinect-for-windows/.
Microsoft Kinect SDK. (2021). Download Kinect for Windows SDK v1.0 from Official Microsoft Download Center. Microsoft.com. Retrieved 8 July 2021, from https://www.microsoft.com/en-us/download/details.aspx?id=28782.
Jimenez, J., Martin, A., Uc, V. y Espinosa, A. (2017). Mexican Sign Language Alphanumerical Gestures Recognition using 3D Haar-like Features. IEEE Latin America Transactions, 15(10), 2000–2005. https://doi.org/10.1109/TLA.2017.8071247
Pérez, L. M., Rosales, A. J., Gallegos, F. J., y Barba, A. V. (2017). LSM static signs recognition using image processing, 14th International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE 2017), pp. 1-5, doi: 10.1109/ICEEE.2017.8108885.
Carmona‐Arroyo, G., Rios‐Figueroa, H. V., y Avendaño‐Garrido, M. L. (2021). Mexican Sign‐Language Static‐Alphabet Recognition Using 3D Affine Invariants. In M. Malarvel, S. R. Nayak, P. K. Pattnaik, y S. N. Panda (Eds.), Machine Vision Inspection Systems, Volume 2 (1st ed., pp. 171–192). Wiley. https://doi.org/10.1002/9781119786122.ch9
Lahamy, H., y Lichti, D. (2012). Towards Real-Time and Rotation-Invariant American Sign Language Alphabet Recognition Using a Range Camera. Sensors, 12(11), 14416–14441. MDPI AG. Retrieved from http://dx.doi.org/10.3390/s121114416
Agarwal, A. y Thakur, M. K. (2013). Sign language recognition using Microsoft Kinect. 2013 Sixth International Conference on Contemporary Computing (IC3), 181–185. https://doi.org/10.1109/IC3.2013.6612186
Luis-Pérez, F. E., Trujillo-Romero, F., y Martínez-Velazco, W. (2011). Control of a Service Robot Using the Mexican Sign Language. Advances In Soft Computing, 419-430. https://doi.org/10.1007/978-3-642-25330-0_37
Estrivero-Chavez, C., Contreras-Teran, M., Miranda-Hernandez, J., Cardenas-Cornejo, J., Ibarra-Manzano, M., y Almanza-Ojeda, D. (2019). Toward a Mexican Sign Language System using Human Computer Interface. 2019 International Conference On Mechatronics, Electronics And Automotive Engineering (ICMEAE). https://doi.org/10.1109/icmeae.2019.00010
LeapMotion. (2021). LeapMotion Datasheet. Ultraleap.com. Retrieved 5 June 2021, from https://www.ultraleap.com/datasheets/Leap_Motion_Controller_Datasheet.pdf.
Garcia-Bautista, G., Trujillo-Romero, F., y Caballero-Morales, S. (2017). Mexican sign language recognition using kinect and data time warping algorithm. 2017 International Conference On Electronics, Communications And Computers (CONIELECOMP). https://doi.org/10.1109/conielecomp.2017.7891832
Tazhigaliyeva, N., Kalidolda, N., Imashev, A., Islam, S., Aitpayev, K., Parisi, G., y Sandygulova, A. (2017). Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS). 2017 IEEE International Conference On Robotics And Automation (ICRA). https://doi.org/10.1109/icra.2017.7989526
Hazari, S., Asaduzzaman, Alam, L., y Goni, N. (2017). Designing a sign language translation system using kinect motion sensor device. 2017 International Conference On Electrical, Computer And Communication Engineering (ECCE). https://doi.org/10.1109/ecace.2017.7912929
Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X., y Zhou, M. (2013, April). Sign language recognition and translation with kinect. In IEEE Conf. on AFGR (Vol. 655, p. 4).
Ghotkar, A., y Kharate, G. (2015). Dynamic Hand Gesture Recognition for Sign Words and Novel Sentence Interpretation Algorithm for Indian Sign Language Using Microsoft Kinect Sensor. Journal of Pattern Recognition Research, 10(1), 24–38. https://doi.org/10.13176/11.626
Garcia-Bautista, G., Trujillo-Romero, F., y Diaz-Gonzalez, G. (2016). Advances to the development of a basic Mexican sign-to-speech and text language translator (A. G. Tescher, Ed.; p. 99713E). https://doi.org/10.1117/12.2238281
Sosa-Jimenez, C., Rios-Figueroa, H., Rechy-Ramirez, E., Marin-Hernandez, A., y Gonzalez-Cosio, A. (2017). Real-time Mexican Sign Language recognition. 2017 IEEE International Autumn Meeting On Power, Electronics And Computing (ROPEC). https://doi.org/10.1109/ropec.2017.8261606
Molchanov, P., Gupta, S., Kim, K. y Kautz, J. (2015). Hand gesture recognition with 3D convolutional neural networks. 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1–7. https://doi.org/10.1109/CVPRW.2015.7301342
Oyewole, O. G., Nicholas, G., Oludele, A., y Samuel, O. (2018). Bridging communication gap among people with hearing impairment: An application of image processing and artificial neural network. International Journal of Information and Communication Sciences, 3(1), 11.
Gürpınar, C., Uluer, P., Akalın, N. et al. Sign Recognition System for an Assistive Robot Sign Tutor for Children. Int J of Soc Robotics 12, 355–369 (2020). https://doi.org/10.1007/s12369-019-00609-9
Zhang, Z., Su, Z., y Yang, G. (2019). Real-Time Chinese Sign Language Recognition Based on Artificial Neural Networks*. 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 1413–1417. https://doi.org/10.1109/ROBIO49542.2019.8961641
Jiang, S., Sun, B., Wang, L., Bai, Y., Li, K., y Fu, Y. (2021). Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3413-3423).
Fregoso, J., Gonzalez, C. I., y Martinez, G. E. (2021). Optimization of Convolutional Neural Networks Architectures Using PSO for Sign Language Recognition. Axioms, 10(3), 139. MDPI AG. Retrieved from http://dx.doi.org/10.3390/axioms10030139
Calvo, M.T. (2004). Diccionario Español - Lengua de Señas Mexicana (DIELSEME): estudio introductorio. Dirección de Educación Especial: México.
Serafín de Fleischmann, M., González Pérez, R. (2011). Manos con voz, Diccionario de Lenguaje de Señas Mexicana. Primera edición, Libre Acceso, A.C., ISBN 978-607-9134-0I-3
Código estándar Mex-Esp. (2021). ISO 639 — Language codes. ISO. Retrieved 14 Mayo 2021, from https://www.iso.org/iso-639-language-codes.html.
Zhang, Z. (2012). Microsoft Kinect Sensor and Its Effect. IEEE MultiMedia, 19(2), 4–10. https://doi.org/10.1109/MMUL.2012.24
Dal Mutto, C., Zanuttigh, P., y Cortelazzo, G. (2012). Time-of-flight cameras and Microsoft Kinect. Springer.
Rossenblatt, F. (1957). The perceptron, a perceiving and recognizing automation. Cornell Aeronautical Laboratory. Report No. 85-460-1.
Werbos, P. (1974). Beyond Regression: New tools for prediction and analysis in the behavioral sciences (Ph.D). Harvard University.
Rumelhart, D. E., Hinton, G., y Williams, R. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536. https://doi.org/10.1038/323533a0
BSL Corpus. (2021). Home | BSL Corpus Project. British Sign Language Corpus Project. Retrieved 13 March 2021, from http://www.bslcorpusproject.org/.
GSL Corpus. (2021). German Sign Language Korpus. Retrieved 14 March 2021, from: http://www.sign-lang.unihamburg.de/dgs-korpus/index.php/welcome.html
LSE Corpus. (2021). Corpus de la lengua de signos española. Corpuslse.es. Retrieved 14 July 2021, from https://www.corpuslse.es/.
LSP Corpus. (2021). Repositorio Digital de la Lengua de Señas Peruana - Grupo Señas Gramaticales. Grupo Señas Gramaticales. Retrieved 10 March 2021, from https://investigacion.pucp.edu.pe/grupos/senasgramaticales/proyecto/repositorio-digital-de-la-lengua-de-senas-peruana/.
