VALIDITY OF ENGAGEMENT INSTRUMENT DURING ONLINE LEARNING IN MATHEMATICS EDUCATION
Keywords:
Confirmatory Factor Analysis, Engagement, Exploratory Factor Analysis, Rasch Analysis, Z GenerationAbstract
The current study aimed to assess the validity of the engagement instrument during online learning in mathematics education. This study used a survey research design as its approach. The current research participants were 203 Generation Z students in West Nusa Tenggara Barat, Indonesia. Convenience sampling techniques were used to assess who had completed the online survey. Three procedures were used to analyze the data in this research: exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and Rasch analysis. EFA revealed that the online engagement instrument had two sub-components: behavioral engagement and emotional engagement. At the same time, the CFA results showed that the model fit indices established the first- and second-order model's two-factor structure. Finally, the results showed that the online engagement instrument’s item and person reliability were good. The findings indicate a potential for enhancement even though the Rasch analysis largely supported the results of EFA and CFA. The current research’s novelty is that it provides a valid and reliable instrument to assess student`s engagement during online learning in a mathematical education context. Using the current instrument can ensure the accuracy, reliability, and credibility of research on student engagement during online learning in mathematics education.
Downloads
References
Adnan, M., & Anwar, K. (2020). Online learning amid the COVID-19 pandemic: Students' perspectives. Journal of Pedagogical Sociology and Psychology, 1(2), 45–51. https://doi.org/10.33902/jpsp.2020261309
Aljaberi, M. A., et al. (2022). Rasch modeling and multilevel confirmatory factor analysis for the usability of the impact of event scale-revised (IES-R) during the COVID-19 pandemic. Healthcare, 10(10), 1858. https://doi.org/10.3390/healthcare10101858
Asrial, A., Syahrial, S., Sabil, H., Ryan, J., Lamb, M., Cox, R., Emmitt, M., & Rahmi, R. (2024). Science teacher’s book: Analyzing elementary school students’ character values. Jurnal Ilmiah Ilmu Terapan Universitas Jambi, 8(1), 180–187. https://doi.org/10.22437/jiituj.v8i1.32766
Bagdi, H., Bulsara, H. P., Sankar, D., & Sharma, L. (2023). The transition from traditional to digital: factors that propel Generation Z’s adoption of online learning. International Journal of Educational Management, 37(3), 695-717. https://doi.org/10.1108/IJEM-01-2023-0003
Bagriacik Yilmaz, Y., & Banyard, P. (2020). Engagement in distance education settings: A trend analysis. Turkish Online Journal of Distance Education. 101–120. https://doi.org/10.17718/tojde.690362
Bravini, E., Giordano, A., Sartorio, F., Ferriero, G., & Vercelli, S. (2016). Rasch analysis of the Italian lower extremity functional scale: Insights on dimensionality and suggestions for an improved 15-item version. Clinical Rehabilitation, 31(4), 532–543. https://doi.org/10.1177/0269215516647180
Boone, W., Staver, J., & Yale, M.S. (2014). Rasch analysis in the human sciences. Springer.
Buelow, J. R., Barry, T. A., & Rich, L. E. (2018). Supporting learning engagement with online students. Online Learning, 22(4), 313–340. https://doi.org/10.24059/olj.v22i4.1384
Cacheiro-Gonzalez, M. L., Medina-Rivilla, A., Dominguez-Garrido, M. C., & Medina-Dominguez, M. (2019). The learning platform in distance higher education: Students' perceptions. Turkish Online Journal of Distance Education, 20(1), 71–95. https://doi.org/10.17718/tojde.522387
Cadiz, M. C. D., Manuel, L. a. F., Reyes, M. M., Natividad, L. R., & Ibarra, F. P. (2024). Technology Integration in Philippine higher education: A content-based bibliometric analysis. Jurnal Ilmiah Ilmu Terapan Universitas Jambi, 8(1), 35–47. https://doi.org/10.22437/jiituj.v8i1.31807
Callahan, C. M., Azano, A. P., Park, S., Brodersen, A. V., Caughey, M., Bass, E. L., & Amspaugh, C. M. (2020). Validation of instruments for measuring affective outcomes in gifted education. Journal of Advanced Academics, 31(4), 470–505. https://doi.org/10.1177/1932202x20929963
Csobanka, Z. E. (2016). The Z generation. Acta Technologica Dubnicae, 6(2), 63–76. https://doi.org/10.1515/atd-2016-0012
Dogan, U. (2014). Validity and reliability of student engagement scale. Bartin Universitesi Egitim Fakultesi Dergisi, 3(2), 390. https://doi.org/10.14686/BUEFAD.201428190
Dumford, D., & Miller, A. L. (2018). Online learning in higher education: Exploring advantages and disadvantages for engagement. Journal of Computing in Higher Education, 30(3), 452–465. https://doi.org/10.1007/s12528-018-9179-z
Farrell, O., & Brunton, J. (2020). A balancing act: A window into online student engagement experiences. International Journal of Educational Technology in Higher Education, 17(1), 1–19. https://doi.org/10.1186/s41239-020-00199-x
Ferri, F., Grifoni, P., & Guzzo, T. (2020). Online learning and emergency remote teaching: Opportunities and challenges in emergency situations. Societies, 10(4), 1–18. https://doi.org/10.3390/soc10040086
Goretzko, D., Pham, T. T., & Bühner, M. (2019). Exploratory factor analysis: Current use, methodological developments and recommendations for good practice. Current Psychology, 40(7), 3510–3521. https://doi.org/10.1007/s12144-019-00300-2
Harrington, D. (2009). Confirmatory factor analysis. Oxford, UK: Oxford University Press.
Hidayat, R., Liu, S., Mohd Saad, M. R., & Hermandra. (2023). The role of mastery goal on life satisfaction using perma as a mediator for college students. Emerging Science Journal, 7, 238–252. https://doi.org/10.28991/esj-2023-sied2-018
Hidayat, R., Wan Idris, W. I., Qudratuddarsi, H., & Abdul Rahman, M. N. (2021). Validation of the mathematical modeling attitude scale for Malaysian mathematics teachers. Eurasia Journal of Mathematics, Science and Technology Education, 17(12), 1–16. https://doi.org/10.29333/ejmste/11375
Hidayat, R., Zainuddin, Z., & Mazlan, N. H. (2024). The relationship between technological pedagogical content knowledge and belief among preservice mathematics teachers. Acta Psychologica, 249, 104432. https://doi.org/10.1016/j.actpsy.2024.104432
Hoi, V. N., & Le Hang, H. (2021). The structure of student engagement in online learning: A bifactor exploratory structural equation modeling approach. Journal of Computer Assisted Learning, 37(4), 1141–1153. https://doi.org/10.1111/jcal.12551
Howard, M. C. (2015). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human-Computer Interaction, 32(1), 51–62. https://doi.org/10.1080/10447318.2015.1087664
Inda-Caro, M., Maulana, R., Fernández-García, C.-M., Peña-Calvo, J.-V., Rodríguez-Menéndez, M. del, & Helms-Lorenz, M. (2018). Validating a model of effective teaching behavior and student engagement: Perspectives from Spanish students. Learning Environments Research, 22(2), 229–251. https://doi.org/10.1007/s10984-018-9275-z
Irvine, J. (2020). Positively Influencing Student Engagement and Attitude in Mathematics through an Instructional Intervention Using Reform Mathematics Principles. Journal of Education and Learning, 9(2), 48-75. https://doi.org/10.5539/jel.v9n2p48
Jabar, J. M., Hidayat, R., Samat, N. A., Rohizan, M. F. H., Rosdin, N. ‘A., Salim, N., & Norazhar, S. A. (2022). Augmented reality learning in mathematics education: A systematic literature review. Journal of Higher Education Theory and Practice, 22(15). https://doi.org/10.33423/jhetp.v22i15.5570
Jackson, D. L., Gillaspy, J. A., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6–23. https://doi.org/10.1037/a0014694
Joshi, D. R., Adhikari, K. P., Khanal, B., Khadka, J., & Belbase, S. (2022). Behavioral, cognitive, emotional and social engagement in mathematics learning during COVID-19 pandemic. PloS one, 17(11), e0278052. https://doi.org/10.1371/journal.pone.0278052
Kahn, P., Everington, L., Kelm, K., Reid, I., & Watkins, F. (2016). Understanding student engagement in online learning environments: The role of reflexivity. Educational Technology Research and Development, 65(1), 203–218. https://doi.org/10.1007/s11423-016-9484-z
Kannadass, P., Hidayat, R., Siregar, P. S., & Husain, A. P. (2023). Relationship between computational and critical thinking towards modelling competency among pre-service mathematics teachers. TEM Journal, 1370–1382. Portico. https://doi.org/10.18421/tem123-17
Khalid, F. (2019). Students' identities and its relationships with their engagement in an online learning community. International Journal of Emerging Technologies in Learning (iJET), 14(05), 4–18. https://doi.org/10.3991/ijet.v14i05.8196
Knekta, E., Runyon, C., & Eddy, S. (2019). One size doesn’t fit all: Using factor analysis to gather validity evidence when using surveys in your research. CBE—Life Sciences Education, 18(1), rm1. https://doi.org/10.1187/cbe.18-04-0064
Larsen, R., & Warne, R. T. (2010). Estimating confidence intervals for eigenvalues in exploratory factor analysis. Behavior Research Methods, 42(3), 871–876. https://doi.org/10.3758/brm.42.3.871
Lee, J., Song, H.-D., & Hong, A. (2019). Exploring factors, and indicators for measuring students’ sustainable engagement in e-learning. Sustainability, 11(4), 1–12. https://doi.org/10.3390/su11040985
Li, S. (2022). Measuring cognitive engagement: An overview of measurement instruments and techniques. International Journal of Psychology and Educational Studies, 8(3), 63–76. https://doi.org/10.52380/ijpes.2021.8.3.239
Marshall, L., & Wolanskyj-Spinner, A. (2020). Covid-19: Challenges and opportunities for educators and generation Z learners. Mayo Clinic Proceedings, 95(6), 1135–1137. https://doi.org/10.1016/j.mayocp.2020.04.015
Martin, F., & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance of engagement strategies in the Online Learning Environment. Online Learning, 22(1), 205–222. https://doi.org/10.24059/olj.v22i1.1092
Mayer, R. E. (2018). Thirty years of research on online learning. Applied Cognitive Psychology, 33(2), 152–159. https://doi.org/10.1002/acp.3482
McBrien, J. L., Cheng, R., & Jones, P. (2009). Virtual spaces: Employing a synchronous online classroom to facilitate student engagement in online learning. The International Review of Research in Open and Distributed Learning, 10(3), 1–16. https://doi.org/10.19173/irrodl.v10i3.605
Mindrila, D., An, Y., Lawrence, M., & Elemen, J. (2017). Exploratory factor analysis: Applications in school improvement research. NovaScience Publisher.
Mosca, J., & Curtis, K. P. (2019). New approaches to learning for generation Z. Journal of Business Diversity, 19(3), 66–74. https://doi.org/10.33423/jbd.v19i3.2214
Mohd Saad, M. R., Mamat, S., Hidayat, R., & Othman, A. J. (2023). Integrating technology-based instruction and mathematical modelling for STEAM-based language learning: a sociocultural and self-determination theory perspective. International Journal of Interactive Mobile Technologies (IJIM), 17(14), 55–80. https://doi.org/10.3991/ijim.v17i14.39477
Moon, J., & Ke, F. (2020). In-game actions to promote game-based math learning engagement. Journal of Educational Computing Research, 58(4), 863-885. https://doi.org/10.1177/0735633119878611
Mukhtar, K., Javed, K., Arooj, M., & Sethi, A. (2020). Advantages, limitations and recommendations for online learning during COVID-19 pandemic era. Pakistan Journal of Medical Sciences, 36(COVID19-S4). https://doi.org/10.12669/pjms.36.covid19-s4.2785
Nguyen, T. D., Cannata, M., & Miller, J. (2016). Understanding student behavioral engagement: Importance of student interaction with peers and teachers. The Journal of Educational Research, 111(2), 163–174. https://doi.org/10.1080/00220671.2016.1220359
Nordholm, D., Arnqvist, A., & Nihlfors, E. (2020). Principals’ identity – developing and validating an analysis instrument in the Swedish context. International Journal of Leadership in Education, 1–16. https://doi.org/10.1080/13603124.2020.1804621
Ouadoud, M., Rida, N., & Chafiq, T. (2021). Overview of e-learning platforms for teaching and learning. International Journal of Recent Contributions from Engineering, Science & IT (iJES), 9(1), 50–70. https://doi.org/10.3991/ijes.v9i1.21111
Peng, W. (2017). Research on model of student engagement in online learning. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 3127–3142. https://doi.org/10.12973/eurasia.2017.00723a
Qudratuddarsi, H., Hidayat, R., Shah, R. L., Nasir, N., Imami, M. K. W., & bin Nor, R. (2022). Rasch validation of instrument measuring Gen-Z science, technology, engineering, and mathematics (STEM) application in teaching during the pandemic. International Journal of Learning, Teaching and Educational Research, 21(6), 104–121. https://doi.org/10.26803/ijlter.21.6.7
Qudratuddarsi, H., Sathasivam, R. V., & Hutkemri, H. (2019). Difficulties and correlation between phenomenon and reasoning tier of multiple-choice questions: A survey study. Indonesian Research Journal in Education| IRJE|, 249-264. https://doi.org/10.22437/irje.v3i2.6970
Rahayu, W., Putra, M. D., Iriyadi, D., Rahmawati, Y., & Koul, R. B. (2020). A Rasch and factor analysis of an Indonesian version of the student perception of Opportunity Competence Development (SPOCD) questionnaire. Cogent Education, 7(1), 1–18. https://doi.org/10.1080/2331186X.2020.1721633
Redmond, P., Heffernan, A., Abawi, L., Brown, A., & Henderson, R. (2018). An online engagement framework for higher education. Online Learning, 22(1), 183–204. https://doi.org/10.24059/olj.v22i1.1175
Richardson, C., & Newby, T. (2006). The role of students' cognitive engagement in online learning. American Journal of Distance Education, 20(1), 23–37. https://doi.org/10.1207/s15389286ajde2001_3
Roblek, V., Mesko, M., Dimovski, V., & Peterlin, J. (2019). Smart technologies as social innovation and complex social issues of the Z generation. Kybernetes, 48(1), 91–107. https://doi.org/10.1108/k-09-2017-0356
Roche, A., Gervasoni, A., & Kalogeropoulos, P. (2023). Factors that promote interest and engagement in learning mathematics for low-achieving primary students across three learning settings. Mathematics Education Research Journal, 35(3), 525-556. https://doi.org/10.1007/s13394-021-00402-w
Sarsar, F., & Kisla, T. (2016). Emotional presence in online learning scale: A scale development study. Turkish Online Journal of Distance Education, 50–61. https://doi.org/10.17718/tojde.87040
Schreiber, J. B. (2021). Issues and recommendations for exploratory factor analysis and principal component analysis. Research in Social and Administrative Pharmacy, 17(5), 1004–1011. https://doi.org/10.1016/j.sapharm.2020.07.027
Shek, D. T. L., & Yu, L. (2014). Confirmatory factor analysis using Amos: A demonstration. International Journal on Disability and Human Development, 13(2), 191–204. https://doi.org/10.1515/ijdhd-2014-0305
Shrestha, N. (2021). Factor Analysis as a Tool for Survey Analysis. American Journal of Applied Mathematics and Statistics, 9(1), 4–11. https://doi.org/10.12691/ajams-9-1-2
Siew, N. M., & Saidi, S. S. (2019). Reliability and validity analysis of statistical reasoning test survey instrument using the rasch measurement model. International Electronic Journal of Mathematics Education, 14(3). https://doi.org/10.29333/iejme/5755
Singh, V., & Thurman, A. (2019). How many ways can we define online learning? A systematic literature review of definitions of online learning (1988-2018). American Journal of Distance Education, 33(4), 289–306. https://doi.org/10.1080/08923647.2019.1663082
Skinner, E. A., Kindermann, T. A., & Furrer, C. J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children's behavioral and emotional participation in academic activities in the classroom. Educational and psychological measurement, 69(3), 493-525. https://doi.org/10.1177/0013164408323233
Stein, H., Gurevich, I., & Gorev, D. (2020). Integration of technology by novice mathematics teachers–what facilitates such integration and what makes it difficult? Education and Information Technologies, 25(1), 141-161. https://doi.org/10.1007/s10639-019-09950-y
Story, D. A., & Tait, A. R. (2019). Survey research. Anesthesiology, 130(2), 192-202. https://doi.org/10.1097/ALN.0000000000002436
Swami, V., & Barron, D. (2019). Translation and validation of body image instruments: Challenges, good practice guidelines, and reporting recommendations for test adaptation. Body Image, 31, 204–220. https://doi.org/10.1016/j.bodyim.2018.08.014
Testa, I., et al. (2021). Development and validation of an instrument to measure students’ engagement and participation in science activities through factor analysis and Rasch analysis. International Journal of Science Education, 44(1), 18–47. https://doi.org/10.1080/09500693.2021.2010286.
Wara, E., Aloka, P. J., & Odongo, B. C. (2018). Relationship between emotional engagement and academic achievement among Kenyan secondary school students. Academic Journal of Interdisciplinary Studies, 7(1), 107–118. https://doi.org/10.2478/ajis-2018-0011
Watkins, W. (2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807
Watkins, W. (2021). A step-by-step guide to exploratory factor analysis with SPSS. Routledge.
Watson, J. C. (2017). Establishing evidence for internal structure using exploratory factor analysis. Measurement and Evaluation in Counseling and Development, 50(4), 232–238. https://doi.org/10.1080/07481756.2017.1336931
Wei, H.-C., & Chou, C. (2020). Online learning performance and satisfaction: Do perceptions and readiness matter? Distance Education, 41(1), 48–69. https://doi.org/10.1080/01587919.2020.1724768
Wu, X. V., Chan, Y. S., Tan, K. H., & Wang, W. (2018). A systematic review of online learning programs for nurse preceptors. Nurse Education Today, 60, 11–22. https://doi.org/10.1016/j.nedt.2017.09.010
Xu, X. (2024). Student engagement and academic achievement in learning Calculus. (2024). International Journal of New Developments in Education, 6(2). https://doi.org/10.25236/ijnde.2024.060204
Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167–177. https://doi.org/10.1177/1469787410379680
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Riyan Hidayat, Muh Khairul Wajedi Imami, Sibo Liu, Hilman Qudratuddarsi, Mohd Rashid Mohd Saad
This work is licensed under a Creative Commons Attribution 4.0 International License.
