year 16, Issue 1 (January - February 2022)                   Iran J Med Microbiol 2022, 16(1): 49-55 | Back to browse issues page


XML Persian Abstract Print


Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:

Saeidnia H R, Karajizadeh M, Mohammadzadeh Z, Abdoli S, Hassanzadeh M. Usability Evaluation of the Mask Mobile Application: The Official Application of the Iranian Government. Iran J Med Microbiol 2022; 16 (1) :49-55
URL: http://ijmm.ir/article-1-1404-en.html
1- Department of Knowledge and Information Science Tarbiat Modares University, (TMU) Tehran, Iran
2- Department of Health Information Management, Health Human Resources Research Center, School of Management & Medical Information Sciences, Shiraz University of Medical Sciences, Shiraz, Iran
3- Health Information Technology, Kashan University of Medical Sciences and Health Services, Kashan, Iran
4- Information Science Department, Allameh Tabataba'i Univesity, Tehran, Iran
5- Department of Knowledge and Information Science, Tarbiat Modares University (TMU), Tehran, Iran , hasanzadeh@modares.ac.ir
Full-Text [PDF 508 kb]   (906 Downloads)     |   Abstract (HTML)  (4708 Views)
Full-Text:   (1383 Views)
Introduction

These days, the COVID-19 pandemic is the biggest problem, leading to increased mortality, morbidity, and lockdowns in cities globally (1). Although Coronavirus mortality has decreased, the number of people affected by the disease is still increasing (2).

To fight against Coronavirus, most countries, especially high-income and developed countries, used technological advances, including artificial intellig-ence, big data, and the Internet of Things (3). During this time, mobile-based applications have received much attention from governments, and many have tried to unveil their official mobile applications (4).
Most of these mobile applications use contact tracing technology (5). According to World Health Organization (WHO) advice and the experience of China and South Korea, contact tracing is essential to control Coronavirus disease (6). One of the most important mobile applications is COVID Alert, an official application of the Canadian government that uses contact tracking technology and artificial intelligence, or COVID Safe, an official application of the Australian government that uses technologies like COVID Alert (7, 8).
The Iranian government, like other countries, designed and developed a mobile-based application called Mask. This mobile application uses contact tracking technology and supports public access to evidence-based information about COVID-19 (9). The Mask mobile application was developed by a volun-teer team of professors and graduates from Sharif University of Technology, Shahid Beheshti University, and Amirkabir University of Technology (9, 10).

 

 Figure 1. Mask app pages-Android version (2.1.1).

Figure 1. Mask app pages-Android version (2.1.1).

 

According to application developers' claims, the Mask can show a map indicating individuals afflicted with COVID-19. It is also possible to assess the possibility of contact with Coronavirus by regularly answering this application's questions (10).
During the COVID-19 pandemic, mobile-based applications can be valuable if they can be used by almost anyone in the community (11). Consequently, for widespread download, the user interface and user experience of these mobile applications must be developed to satisfy all users (11, 12). This issue about everyone benefiting from software (e.g., mobile appli-cations) is so fundamental that it is emphasized as a human right and social justice (13). Therefore, usabi-lity evaluation of mobile applications is one of the ways that can determine whether developers and des-igners in the design of user experience mobile appli-cations have observed social justice or not? (14, 15).
Numerous studies have evaluated mobile appli-cations developed during the Coronavirus era. Saeidnia et al. (2021) evaluated the HealthBuddy mobile application. They used the heuristic evaluation method, the system usability scale (SUS) tool, and Jacob Nielsen's 10 general principles (12) to evaluate this application. Scherr et al. (2021) assessed the MyCOVIDKey mobile application. This research was conducted in three phases: pilot study design, ParaData collection, and post-pilot survey (16). In their study, Bente et al. (2021) evaluated the Dutch application CoronaMelder. They assessed the application using scenario-based, think-aloud usability tests and additional interviews (17).
Accordingly, in this study, we seek to answer whether the user experience of the Mask application is designed as standard or not? And consequently, this study aimed to evaluate the Mask application using heuristic and user-based testing methods.


 

Materials and Methods

This cross-sectional study was conducted in two phases in 2020. The first phase evaluated the Mask mobile application's interactive design by considering five experts' opinions and Nelson Jacob's 10 principles tools, while the second phase evaluated usability with 124 users' participation. Furthermore, data analysis in both phases was performed using Microsoft Excel software.

Phases 1: Heuristic Method

We used Jacob Nielsen's 10 general principles for evaluating the interactive design in this phase. During the evaluation process by the heuristic method, the application was introduced to five experts (3 IT experts and 2 health information experts) who had master's degrees and Ph.D. in health informatics and health information management with experience in mobile application UX evaluation. To begin with, we required the experts to install the Mask mobile application, Android version (2.1.1) and allocated one hour to check it out. Then experts requested to grade Jacob Nielsen's 10 general principles on Nielsen's severity ranking scale, based on the 5-points (18). Nielsen's 5-point severity ranking is as follows:

  • 0 = I do not agree that this is a usability issue.

  • 1 = Not to be fixed unless extra time is available for the project.

  • 2 = This should be fixed at the lowest priority.

  • 3 = Must be fixed, so it should be given high priority.

  • 4 = This must be fixed before the product can be released.

Phases 2: Users-based Testing

We enrolled Mask mobile application users for our study through purposive sampling in this phase. Consequently, we could recruit at least 100 people (users) for users-based testing (19, 20). These users were selected by sending a call for cooperation to different social networking groups and pages. The inclusion criteria involved personal volunteering between 18 and 50 and using the Mask mobile application for more than three months. Finally, in total number, 124 users enrolled. To obtain an overall impression of subjective evaluations of the Mask mobile application, we required users to answer the SUS (system usability scale) questionnaire. In short, it could be that SUS was considered a quantitative example of a qualitative user experience (21). This questionnaire includes 10 questions rated by the 5-Likert scale (1 = strongly disagree, 5 = strongly agree). Eventually, using SUS proprietary formula (21), the application's usability is determined in the form of grade and adjective rating (Table 1).
In quarantine conditions, we introduced the research team and project aims in the online workshop, and their commitment to the project was explained. In the online workshop, we requested users to answer the SUS questionnaire designed by Google Form.

Table 1. Interpreting System Usability Scale (SUS) Score (21)

SUS Score Grade Adjective Rating
> 80.3 A Excellent
68 – 80.3 B Good
68 C Okay
51 – 68 D Poor
< 51 F Awful


 

Results

Usability Evaluation using Heuristic Method

According to the results of a heuristic evaluation of the Mask mobile application, two principles: error prevention and flexibility and efficiency criteria, have earned points 2 to 3. It means that the Mask mobile application has been weak in these two aspects. Table 2 represents the results where ΣA refers to the total number of point scale for each item Jacob Nielsen's 10 general principles. ΣB is the same as ΣA; however, it was classified by eliminating duplicate point scale, which finally, total ΣA 62 and ΣB 32 (Table 2).
Usability Evaluation using Users-based Testing
In the user-based test, 124 users, including 49 women and 75 men with a mean age of 37 years, participated (Table 3).


Table 2. Distribution of Frequency and Severity Scores by 10 Usability Heuristics

Heuristic Principle ΣA1 ΣB2 13 2 3 4 5
Visibility of system status 4 3 0 1 2 1 0
Match between system and the real world 2 2 0 0 0 0 2
User control and freedom 4 3 1 1 0 2 0
Consistency and standards 2 1 0 1 0 0 1
Error prevention 12 5 2 2 2 3 3
Recognition rather than recall 3 1 0 0 1 1 1
Flexibility and efficiency of use 12 5 2 2 3 2 3
Aesthetic and minimalist design 7 6 1 0 1 2 3
Error identification, diagnosis, and recovery 3 1 0 1 0 1 1
Help and documentation 13 5 2 2 3 3 3
Total 62 32 8 10 12 15 17

1-A=Number of point scale; 2-B=Number of kinds of point scale (Number of point scale with the removal of duplicates) ;3- Each expert's number
 
Table 3. Demographic characteristics of the participants

Characteristic Value
Age (years), n (%)  
18-28 45 (36.30%)
29-38 42 (33.87%)
39-50 37 (29.83%)
Gender, n (%)  
Male 49 (39.51%)
Female 75 (60.48%)
Duration of benefit from the Mask app (month), n (%)  
3-5 42 (33.87%)
6-11 36 (29.03%)
11 - up to 46 (37.09%)


According to the findings, the overall usability score of the Mask mobile application for fighting COVID-19 was 89 out of 100 in the evaluation by users. This score is higher than 83 (> 80.3), so based on the SUS formula, such usability of the Mask mobile application can be described: Average SUS score = 89 = Excellent = Grade A.

 
 

Discussion

Our study aim was on usability evaluation of the Mask mobile application that an official application of the Iranian government for fighting against Corona-virus.
In the phase heuristic, the error prevention item was a major usability problem. User errors are generally categorized into error slips and mistakes (22). Slips are common errors if users do not pay full attention to something or have poor memory lapses (22). Mistakes occur when the user creates a mental model of the user interface and creates a goal that is not appropriate for the situation (12). To fix these problems in the error slip to the matter of lessening loads on users and guiding users when accuracy is required (22) and for errors, Mistakes planning in light of the user's experience and assisting the user with building a decent mental model of your interface (22, 23). This heuristic method permits user customization, not prescriptive about center obligation steps, and unobtrusive accelerators (24). The application should have shortcuts that speed up users. The non-appearance of this component demonstrates a breach of the flexibility and efficiency of use. Although these shortcuts are often identified and used by proficient users (24, 25), their absence in the system is known as a usability problem; problems related to this index also existed in many parts of the system. In addition, when providing error messages, there were not enough details to inform users of the type of error and how to fix it. This application lacks the flexibility of being able to customize specific parts of it (e.g., theme, font), which is also a problem when it comes to flexibility. It was necessary to zoom in on some parts information, which the app did not provide. This application received high scores for accessibility of system status, the connection between application and reality, consistency and standards, evidence-based content.
In the users-based testing phase, the overall usability score of the Mask mobile application for fighting COVID-19 was calculated by users. The average SUS score was 89 out of 100, which can be considered 89, equal to an excellent level (> 80.3) and grade A.
In a similar study, Saeidnia et al. (2021) evaluated the HealthBuddy mobile application by using the heuristic evaluation method using Jacob Nielsen's 10 general principles and the system usability scale (SUS). The evaluation result was that the HealthBuddy mobile application failed to meet two principles: error prevention, flexibility, and efficiency. The average SUS score of the HealthBuddy mobile application was 77.5 out of 100, a rating that can be interpreted as a B (12). In another similar study, Scherr et al. (2021) research has been conducted to evaluate mobile-based applications to fight and prevent COVID-19. This research was conducted in three phases of pilot study design, Para data collection, and post-pilot survey and intended to evaluate a MyCOVIDKey application. The tools used in the Scherr et al. research was a questionnaire consisting of 59 questions and a system usability scale (SUS). Finally, Scherr et al. concluded that a MyCOVIDKey application is a valuable tool for call tracking when fighting COVID-19 (16). We also conducted the present study in two phases with two same tools (SUS and Jakob Nielsen's 10 general principles). This study also used expert and user evaluations.
In their research, Kondylakis et al. reviewed COVID-19 mobile applications. They concluded that provided mobile applications were a safe and confident method of receiving healthcare, obtaining credible inform-ation, and monitoring activity and symptoms (26). The Mask mobile application contained features that enabled the user to access educational materials, track activities, and self-assess their health. Based on our findings, the application was considered an acceptable tool for receiving health care services. Therefore, using the Mask application can make it possible to access health care services and educational resources, which is a unique feature when it comes to effective m-Health applications (27-30).
The study had limitations, such as the quarantine conditions, which made it difficult to have a wide community of users, and it was also very challenging to organize the community. However, all users were active, and we spent more time and energy trying to alleviate these limitations. It is suggested to use the findings of this study in future updates of the Mask mobile application. Also, it is recommended to perform future investigations into the effectiveness of these applications in fighting COVID-19.


 

Conclusion

In the context of the Coronavirus pandemic, mobile health application developers, in this case, can use the heuristic method to evaluate their applications within a few hours and with minimal resources. But the use of the user-based testing method gives a wider view of the applicability of the application. In this study, we used a mix-method, the heuristic, and the questi-onnaire's user-based testing method. While the heuristic evaluation in this study was conducted after the application had been implemented, it is also beneficial to consider usability heuristics during these applications' design, selection, and implementation process. However, early heuristic evaluations of application prototypes are beneficial for identifying usability issues and improving them. Usability and user interface properties are differentiation factors key for mobile health applications. Mobile health applications can significantly impact finding inform-ation, self-care, self-assessment, and prevention about the coronavirus (COVID-19) pandemic. Still, important challenges such as usability, quality of user interface, and the efficacy of applications must be addressed, requiring a global collaborative effort.


Acknowledgment

The authors would like to thank all professors and Tarbiat Modares University staff, all who developed the Mask mobile application, and participants of this study.


Ethical Approval

The study was conducted, obtaining the approval of the Ethics Committee of Tarbiat Modares University, Tehran, Iran (IR.MODARES.REC.1399.142).

 

Funding

No specific organization contributed to the funding of this research.

 

Conflicts of Interest

The authors declared no conflicts of interest.


 

Type of Study: Original Research Article | Subject: Computer Vision
Received: 2021/07/26 | Accepted: 2021/12/5 | ePublished: 2022/01/20

References
1. Paterlini M. On the front lines of coronavirus: the Italian response to covid-19. Bmj. 2020;368. [DOI:10.1136/bmj.m1065] [PMID]
2. WHO. WHO Coronavirus (COVID-19) Dashboard: WHO; 2021 [Available from: https://covid19.who.int/.]
3. Ting DSW, Carin L, Dzau V, Wong TY. Digital technology and COVID-19. Nature medicine. 2020;26(4):459-61. [DOI:10.1038/s41591-020-0824-5] [PMID] [PMCID]
4. Abbas R, Michael K. COVID-19 contact trace app deployments: learnings from Australia and Singapore. IEEE Consum Electron Mag. 2020;9(5):65-70. [DOI:10.1109/MCE.2020.3002490]
5. Islam MN, Islam I, Munim KM, Islam AN. A review on the mobile applications developed for COVID-19: an exploratory analysis. IEEE Access. 2020;8:145601-10. [DOI:10.1109/ACCESS.2020.3015102] [PMID] [PMCID]
6. Zens M, Brammertz A, Herpich J, Südkamp N, Hinterseer M. App-based tracking of self-reported COVID-19 symptoms: analysis of questionnaire data. J Med Internet Res. 2020;22(9):e21956. [DOI:10.2196/21956] [PMID] [PMCID]
7. Gómez-Ramírez O, Medeiros P, Wainer R, Iyamu I. 13 Does the 'Canada COVID-19 alert'app stand up to critical scrutiny? A rapid qualitative assessment. British Medical Journal Publishing Group; 2021. [DOI:10.1136/bmjopen-2021-QHRN.13]
8. Abbas HSM, Xu X, Sun C. Role of COVIDsafe app and control measures in Australia in combating COVID-19 pandemic. Transform Gov: People Process Policy. 2021. [DOI:10.1108/TG-01-2021-0004]
9. Mask. Mask app iran2021 [cited 2021 March 10, 2021]. Available from: https://mask.ir/.
10. J G. Regulating electronic means to fight the spread of COVID‐19: Libr Congr; 2020 [Available from: https://www.loc.gov/law/help/coronavirus-apps/iran.php.]
11. Rizzo E. COVID-19 contact tracing apps: the 'elderly paradox'. Public Health. 2020;185:127. [DOI:10.1016/j.puhe.2020.06.045] [PMID] [PMCID]
12. Saeidnia HR, Mohammadzadeh Z, Hassanzadeh M. Evaluation of Mobile Phone Healthcare Applications During the Covid-19 Pandemic. Studies in Health Technology and Informatics. 2021;281:1100-1. [DOI:10.3233/SHTI210363] [PMID]
13. Walls DM, editor User experience in social justice contexts. Proceedings of the 34th ACM International Conference on the Design of Communication; 2016. [DOI:10.1145/2987592.2987604]
14. Zahra F, Hussain A, Mohd H, editors. Usability evaluation of mobile applications; where do we stand? AIP Conference Proceedings; 2017: AIP Publishing LLC. [DOI:10.1063/1.5005389]
15. Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: validation study. JMIR mHealth and uHealth. 2018;6(1). [DOI:10.2196/mhealth.8851] [PMID] [PMCID]
16. Scherr TF, DeSousa JM, Moore CP, Hardcastle A, Wright DW. App Use and Usability of a Barcode-Based Digital Platform to Augment COVID-19 Contact Tracing: Postpilot Survey and Paradata Analysis. JMIR Public Health and Surveillance. 2021;7(3):e25859. [DOI:10.2196/25859] [PMID] [PMCID]
17. Bente BE, Roderick van't JWJ, Schreijer MA, Berkemeier L, van Gend JE, Slijkhuis PJH, et al. The Dutch COVID-19 Contact Tracing App (the CoronaMelder): Usability Study. JMIR formative research. 2021;5(3):e27882. [DOI:10.2196/27882] [PMID] [PMCID]
18. Brown J, Nam Kim H, editors. Usability evaluation of Alzheimer's mHealth applications for caregivers. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2018: SAGE Publications Sage CA: Los Angeles, CA. [DOI:10.1177/1541931218621115]
19. Balasubramoniam V, Tungatkar N. Study of user experience (UX) and UX evaluation methods. Int J Adv Res Comput Sci Eng Inf technol. 2013;2(3):1214-9.
20. Barnum C. The state of UX research. J Usability Stud. 2019;15(1):1-7.
21. Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Stud. 2018;34(7):577-90. [DOI:10.1080/10447318.2018.1455307]
22. Paterno F, Santoro C. Preventing user errors by systematic analysis of deviations from the system task model. Int J Hum Comput Stud. 2002;56(2):225-45. [DOI:10.1006/ijhc.2001.0523]
23. Gregoriades A, Sutcliffe A, Shin JE. Assessing the reliability of socio‐technical systems. Systems engineering. 2003;6(3):210-23. [DOI:10.1002/sys.10044]
24. Tiffany B, Blasi P, Catz SL, McClure JB. Mobile apps for oral health promotion: content review and heuristic usability analysis. JMIR mHealth and uHealth. 2018;6(9):e11432. [DOI:10.2196/11432] [PMID] [PMCID]
25. Hutton L, Price BA, Kelly R, McCormick C, Bandara AK, Hatzakis T, et al. Assessing the privacy of mhealth apps for self-tracking: heuristic evaluation approach. JMIR mHealth and uHealth. 2018;6(10):e185. [DOI:10.2196/mhealth.9217] [PMID] [PMCID]
26. Kondylakis H, Katehakis DG, Kouroubali A, Logothetidis F, Triantafyllidis A, Kalamaras I, et al. COVID-19 Mobile Apps: A Systematic Review of the Literature. J Med Internet Res. 2020;22(12):e23170. [DOI:10.2196/23170] [PMID] [PMCID]
27. Saeidnia H, Mohammadzadeh Z, Saeidnia M, Mahmoodzadeh A, Ghorbani N, Hasanzadeh M. Identifying Requirements of a Self-care System on smartphones for preventing coronavirus disease 2019 (COVID-19). Iran J Med MIcrobiol. 2020;14(3):241-6. [DOI:10.30699/ijmm.14.3.241]
28. Gonçalves-Bradley DC, Maria ARJ, Ricci-Cabello I, Villanueva G, Fønhus MS, Glenton C, et al. Mobile technologies to support healthcare provider to healthcare provider communication and management of care. Cochrane Database of Systematic Reviews. 2020(8). [DOI:10.1002/14651858.CD012927.pub2] [PMID] [PMCID]
29. Saeidnia HR, Ghorbi A, Kozak M, Herteliu C. Smartphone-Based Healthcare Apps for Older Adults in the COVID-19 Era: Heuristic Evaluation. Stud Health Technol Inform. 2022; 289:128-131. [DOI:10.3233/SHTI210875] [PMID]
30. Karajizadeh M, Zand F, Vazin A, Sharifian R, Bayati N. Usability of Venous Thromboembolism Prophylaxis Recommender System: A Pilot Study. Stud Health Technol Inform. 2022. ;289:220-223. [DOI:10.3233/SHTI210899] [PMID]

Add your comments about this article : Your username or Email:
CAPTCHA

Send email to the article author


Rights and permissions
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

© 2024 CC BY-NC 4.0 | Iranian Journal of Medical Microbiology

Designed & Developed by : Yektaweb | Publisher: Farname Inc