Bidirectional Braille-speech Communication System for Deafblind Students
DOI:
https://doi.org/10.7160/eriesj.2024.170206Keywords:
Accessibility, Assistive Technology, Braille, Convolutional Neural Networks, Deafblindness, EducationAbstract
Deafblindness is a type of dual disability wherein both visual and auditory capabilities are significantly impaired. Special communication methods have been developed for the deafblind community, yet these methods require that both people involved have prior knowledge and training to successfully communicate; limiting deafblind people’s social interactions, particularly in academic settings. This paper describes the development of a device that enables two-way communication between a severe to profoundly deafblind user and a hearing person with no prior knowledge of Braille and no additional intermediaries. A Convolutional Neural Network (CNN) scheme for speech recognition was designed and implemented along with the development of an algorithm capable of developing both text-to-speech and Finger-Braille-to-text conversion. Lastly, a system integration via 3D modelling and additive manufacturing was carried out to deliver a functional prototype. The resulting device aims to allow deafblind students to send and receive information entirely in finger Braille, using buttons and vibrotactile feedback, while the hearing tutor receives auditory messages and speaks to reply, making the educational experience as familiar as possible for both parties. Users testing the device achieved an average typing accuracy of over 95% and demonstrated understanding of commands transmitted through the device’s components.
References
Ask Larsen, F. and Damen, S. (2014) ‘Definitions of deaf-blindness and congenital deaf-blindness’, Research in Developmental Disabilities, Vol. 35, No. 10, pp. 2568–2576. https://dx.doi.org/10.1016/j.ridd.2014.05.029
Ayton, L., Galvin, K., Johansen, L., O’Hare, F. and Shepard, E. (2023) ‘Awareness of Usher syndrome and the need for multidisciplinary care: A cross-occupational survey of Allied Health Clinicians’, Journal of Multidisciplinary Healthcare, Vol. 16, pp. 1927–1936. https://dx.doi.org/10.2147/jmdh.s411306
Banerjee, K., Prasad, V., Gupta, R., Vyat, K., Anushree H. and Mishra, B. (2021) ‘Exploring alternatives to Softmax function’, Proceedings of the 2nd International Conference on Deep Learning Theory and Applications, (DeLTA, 2021), Paris, Vol. 1, pp. 81–86.
Blenkhorn, P. (1995) ‘A system for converting braille into print’, IEEE Transactions on Rehabilitation Engineering, Vol. 3, No. 2, pp. 215–221. https://dx.doi.org/https://doi.org/10.1109/86.392366
Bowen, S.K. and Probst, K.M. (2023) ‘Deaf and hard of hearing students with disabilities: An evolving landscape’, Education Sciences, Vol. 13, No. 7, pp. 752–767. https://dx.doi.org/10.3390/educsci13070752
Castiglione, A. and Möller, C. (2022) ‘Usher syndrome’, Audiology Research, Vol. 12, No. 1, pp. 42–65. https://dx.doi.org/10.3390/audiolres12010005
CDHCM (2020) It is necessary to consider measures and strategies that include deaf-blind people in the new normal | CDHCM, [Online], Available: https://cdhcm.org.mx/2020/06/cdhcm-es-necesario-que-en-la-nueva-normalidad-se-contemplen-medidas-y-estrategias-que-incluyan-a-personas-sordociegas/ [19 Feb 2024].
CONADIS (2018) Deaf-blindness in México: Facts for June the 27th, International Deaf-blindness Day | Government of Mexico, [Online], Available at: https://www.gob.mx/conadis/es/articulos/la-sordoceguera-en-/mexico-datos-por-el-27-de-junio-dia-internacional-de-la-/sordoceguera?idiom=es [9 Jul 2023].
CONAPRED (2008) ‘Una Esperanza Para Ángel’ [Hope for Ángel] | National Council for Preventing Discrimination, [Online], Available: https://www.conapred.org.mx/index.php?contenido=noticias&id=1854&id_opcion=&op=447 [10 Jul 2023].
Ding, X. (ed.) (2012) Advances in Character Recognition: Finger braille recognition system, Beijing: IntechOpen.
Dogbe, D.S.Q. and Anku, F.K. (2024) ‘Prospects and challenges of educating a deaf-blind student in a university in Ghana’, European Journal of Education Studies, Vol. 11, No. 1, pp. 118–134. http://dx.doi.org/10.46827/ejes.v11i1.5157
Dongare, A., Kharde, R. and Kachare, A. (2012) ‘Introduction to artificial neural network’, International Journal of Engineering and Innovative Technology, Vol. 2, No. 1, pp. 189–194.
He, J., Li, H., Xu, J. and Zheng, C. (2018) ‘Relu deep neural networks and linear finite elements’, Journal of Computational Mathematics, Vol. 38, No. 3, pp. 502–527. https://dx.doi.org/10.4208/jcm.1901-m2018-0160
Hersh, M. (2013) ‘Deaf-blind people, Communication, independence, and isolation’, Journal of Deaf Studies and Deaf Education, Vol. 18, No. 4, pp. 446–463. https://dx.doi.org/10.1093/deafed/ent022
INEGI (2020) [Population with disability, with limitations for daily activity and some mental problem or condition, by federated state and quinquennial age groups according to sex | Interactive-generic tabulations, [Online], Available: https://www.inegi.org.mx/app/tabulados/interactivos/?pxq=Discapacidad_Discapacidad_01_29827fe7-b1cd-4bd2-81d6-9d08bda47df8 [9 Jul 2023].
Kassem, K., Caramazza, P., Mitchell, K., Miller, M., Emadi, A. and Faccio, D. (2022) ‘Real-time scene monitoring for deaf-blind people’, Sensors, Vol. 22, No. 19, p. 7136. https://dx.doi.org/10.3390/s22197136
Kingma, D.P. and Ba, J. (2015) ‘ADAM: A method for stochastic optimization’, Proceedings of the 3rd International Conference for Learning Representations (ICLR, 2015), San Diego.
Li, J., Cheng, J.-H., Shi, J.-Y. and Huang, F. (2012) ‘Brief introduction of back propagation (bp) neural network algorithm and its improvement’, Advances in Computer Science and Information Engineering, Vol. 169, pp. 553–558. https://dx.doi.org/10.1007/978-3-642-30223-7_87
Liu, W., Yu, W., Li, K., Zhou, S., Wang, Q. and Yu, H. (2023) ‘Enhancing blind-dumb assistance through a self-powered tactile sensor-based braille typing system’, Nano Energy, Vol. 116, p. 108795. https://dx.doi.org/10.1016/j.nanoen.2023.108795
Manga, T. and Masuku, K.P. (2020) ‘Challenges of teaching the deaf-blind learner in an education setting in Johannesburg: Experiences of educators and assistant educators’, South African Journal of Communication Disorders, Vol. 67, No. 1. https://dx.doi.org/10.4102/sajcd.v67i1.649
Mazouch, P., Ptáčková, V., Fischer, J. and Hulík, V. (2018) ‘Students who have unsuccessfully studied in the past – analysis of causes’, Journal on Efficiency and Responsibility in Education and Science, Vol. 11, No. 3, pp. 65–71. https://dx.doi:10.7160/eriesj.2018.110303
McFee, B., McVicar, M., Raffel, C., Liang, D., Nieto, O., Moore, J., Ellis, D., Repetto, D., Viktorin, P., Santos, J.F. and Holovaty, A. (2015) ‘Librosa: v0.4.0.’, Zenodo. https://dx.doi.org/10.5281/zenodo.18369
McKinney, E. (2022) ‘Meeting the Needs of a Student with Deaf-blindness in the Academic Classroom: A Model That Works’, Visual Impairment and Deaf-blind Education Quarterly, Vol. 67, No. 4, pp. 17–43.
Montgomery, D.C. (1991) Introduction to Statistical Quality Control, 2nd Edition, New York: John Wiley and Sons.
Moore, C. and Murray, I. (2001) ‘An electronic design of a low-cost braille typewriter’, Proceedings of The Seventh Australian and New Zealand Intelligent Information Systems Conference (ANZIIS, 2001), Adelaide, pp. 153–157.
Morgan, D.S., Carhart, D.A., Dmitriyev, S., Gunderson, B.J., Patadia, S., Pulik, L. and Yanz, L.A. (2011) Braille Writer: US Patent 20090274505A1, [Online], Available: https://patents.google.com/patent/US20090274505A1/en [16 Sep 2023].
NCDB (2013) State Deaf-Blind projects, State Deaf-Blind Projects | National Center on Deaf-Blindness, [Online], Available: https://www.nationaldb.org/state-deaf-blind-projects/ [11 Jul 2023].
NFB (2021) Deaf-blind Resources, Deaf-blind Resources | National Federation of the Blind, [Online], Available: https://nfb.org/resources/deaf-blind-resources [11 Jul 2023].
Olayi, J.E., Igba, I.U. and Ewa, J.A. (2023) ‘Communication strategies for teaching deaf-blind learners in an inclusive classroom’, Special needs education from the lens of interdisciplinary dialogue: A Festschrift in honour of Prof. Emeka D. Ozoji, Vol. 1, No. 1, pp. 269–278.
Riccobono, D. and Morrow, S.M. (2022) ‘State Deaf-blind Projects: National Survey Results’, Visual Impairment and Deaf-blind Education Quarterly, Vol. 67, No. 4, pp. 109–125.
Rodríguez-Jiménez, M., Pérez-Jorge, D., Puerta-Araña, I. and Ariño-Mateo, E. (2022) ‘Quality of life in Deaf-blind people and its effect on the processes of educational adaptation and social inclusion in Canary Islands, Spain’, Education Sciences, Vol. 12, No. 7, p. 490. https://dx.doi.org/10.3390/educsci12070490
Rusiecki, A. (2019), ‘Trimmed categorical cross-entropy for deep learning with label noise’, Electronic Letters, Vol. 55, No. 6, pp. 319–320
Schulz, S., Harzheim, L., Hübner, C., Lorke, M., Jünger, S. and Woopen, C. (2023) ‘Patient-centered empirical research on ethically relevant psychosocial and cultural aspects of cochlear, glaucoma and cardiovascular implants – a scoping review’, BMC Medical Ethics, Vol. 24, No. 1. https://dx.doi.org/10.1186/s12910-023-00945-6
Smolen, E.R. and Paul, P.V. (2023) ‘Perspectives on identity and d/deaf and hard-of-hearing students’, Education Sciences, Vol. 13, No. 8, p. 782. https://dx.doi.org/10.3390/educsci13080782
Traer, J. and McDermott, J.H. (2016) Statistics of natural reverberation enable perceptual separation of sound and space | IR Survey (Traer and McDermott), [Online], Available: https://mcdermottlab.mit.edu/Reverb/IR_Survey.html [17 Sep 2023].
Turunen-Taheri, S., Hagerman, A., Hellström, S., Skjönsberg, Å. and Backenroth, G. (2023) ‘Combined severe-to-profound hearing and vision impairment—experiences of daily life and need of support, an interview study’, PLOS ONE, Vol. 18, No. 6, p. e0280709. https://dx.doi.org/10.1371/journal.pone.0280709
Vincent, C., Wittich, W., Bergeron, F., Hotton, M. and Achou, B. (2021) ‘Shopping when you are Deaf-blind: A pre-technology test of new methods for face-to-face communication—deaf-blindness and face-to-face communication’, Societies, Vol. 11, No. 4, p. 131. https://dx.doi.org/10.3390/soc11040131
WFDB (2024) World Federation of the Deaf-blind | Personas con sordoceguera y educación, [Online], Available: https://wfdb.eu/es/wfdb-report-2018/deaf-blindness-and-education/ [19 February 2024].
WFDB (2023) Second global report on the situation of persons with deaf-blindness | World Federation of the Deaf-blind, [Online], Available: https://wfdb.eu/wp-content/uploads/2023/03/ENG_WFDB-2nd-Global-Report_-FINAL-V6.pdf [19 Feb 2024].
Zdravkova, K., Krasniqi, V., Dalipi, F. and Ferati, M. (2022) ‘Cutting-edge communication and learning assistive technologies for disabled children: An artificial intelligence perspective’, Frontiers in Artificial Intelligence, Vol. 5, p. 970430. https://dx.doi.org/10.3389/frai.2022.970430
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Ana Paula Pérez-Aguirre, Iván Arturo Morales-Pérez, Jorge Allan Gómez-Mercado, Rodrigo Alberto Gutiérrez-Martínez, Iván Matehuala-Moran, Rubén Fuentes-Alvarez
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors declare with this manuscript intended for publication to ERIES Journal that:
- all co-authors agree with the publication of the manuscript even after amendments arising from peer review;
- all co-authors agree with the posting of the full text of this work on the web page of ERIES Journal and to the inclusion of references in databases accessible on the internet;
- no results of other researchers were used in the submitted manuscript without their consent, proper citation, or acknowledgement of their cooperation or material provided;
- the results (or any part of them) used in the manuscript have not been sent for publication to any other journal nor have they already been published (or if so, that the relevant works are cited in this manuscript);
- submission of the manuscript for publication was completed in accordance with the publishing regulations pertaining to place of work;
- experiments performed comply with current laws and written consent of the Scientific Ethics Committee / National Animal Care Authority (as is mentioned in the manuscript submitted);
- grant holders confirm that they have been informed of the submitted manuscript and they agree to its publication.
Authors retain copyright and grant ERIES Journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the published work with an acknowledgement of its initial publication in ERIES Journal. Moreover, authors are able to post the published work in an institutional repository with an acknowledgement of its initial publication in ERIES Journal. In addition, authors are permitted and encouraged to post the published work online (e.g. institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
