Development and Initial Validation of a Chatbot-Based Conversational Assessment Model for Post-Learning Reflective Affective Assessment
DOI:
https://doi.org/10.61194/education.v4i3.1080Keywords:
affective assessment, conversational assessment, chatbot, large language model, krathwohl's affective taxonomyAbstract
Affective assessment remains challenging in educational practice due to its subjective nature and the difficulty of systematically documenting students' attitudes and values. Although affective learning outcomes are essential for holistic education, assessment practices often rely on teacher observations that may be influenced by personal judgment. This study aimed to develop and conduct an initial validation of Affectra, a chatbot-based conversational assessment model for reflective affective assessment following learning activities. Unlike most educational chatbots designed for learning support or cognitive assessment, Affectra integrates conversational assessment, adaptive probing, Large Language Model (LLM)-based analysis, and Krathwohl's affective taxonomy into a unified assessment framework. A modified Research and Development (R&D) approach was adopted, consisting of needs analysis, model and instrument design, expert validation, alpha testing, and system refinement. The model was developed for secondary school learning, with validation limited to expert review and researcher-controlled alpha testing. Content validity was evaluated using the Index of Item–Objective Congruence (IOC) with six experts in educational assessment and educational technology, while system performance was assessed through functional and scenario-based testing. The instrument achieved an overall IOC score of 0.81, indicating satisfactory content validity, and demonstrated consistent classification across Krathwohl's five affective levels under predefined simulated scenarios. These findings provide preliminary evidence supporting the initial development and internal validation of Affectra. Further classroom validation involving actual students is required to evaluate its practical effectiveness.
References
Abrahams, L., Pancorbo, G., Primi, R., Santos, D., Kyllonen, P. C., John, O. P., & De Fruyt, F. (2019). Social-Emotional Skill Assessment in Children and Adolescents: Advances and Challenges in Personality, Clinical, and Educational Contexts. Psychological Assessment, 31(4), 460–473. https://doi.org/10.1037/pas0000591 DOI: https://doi.org/10.1037/pas0000591
Aldrup, K., Carstensen, B., & Klusmann, U. (2022). Is Empathy the Key to Effective Teaching? A Systematic Review of Its Association With Teacher-Student Interactions and Student Outcomes. Educational Psychology Review, 34, 1177–1216. https://doi.org/10.1007/s10648-021-09649-y DOI: https://doi.org/10.1007/s10648-021-09649-y
Allouch, M., Azaria, A., & Azoulay, R. (2021). Conversational Agents: Goals, Technologies, Vision and Challenges. Sensors, 21(24), 8448. https://doi.org/10.3390/s21248448 DOI: https://doi.org/10.3390/s21248448
Assarroudi, A., Heshmati Nabavi, F., Armat, M. R., Ebadi, A., & Vaismoradi, M. (2018). Directed Qualitative Content Analysis: The Description and Elaboration of Its Underpinning Methods and Data Analysis Process. Journal of Research in Nursing, 23(1), 42–55. https://doi.org/10.1177/1744987117741667 DOI: https://doi.org/10.1177/1744987117741667
Borg, W. R., Gall, M. D., & Gall, J. P. (2007). Educational Research: An Introduction (8th ed.). Pearson.
Boyd, V. A., Woods, N. N., Kumagai, A. K., Kawamura, A. A., Orsino, A., & Ng, S. L. (2022). Examining the Impact of Dialogic Learning on Critically Reflective Practice. Academic Medicine, 97(11 Suppl 2), S71–S79. https://doi.org/10.1097/ACM.0000000000004916 DOI: https://doi.org/10.1097/ACM.0000000000004916
Chen, J., Liu, Z., Huang, X., & others. (2024). When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities. World Wide Web, 27, 42. https://doi.org/10.1007/s11280-024-01276-1 DOI: https://doi.org/10.1007/s11280-024-01276-1
Ferreira, M., Martinsone, B., & Talić, S. (2020). Promoting Sustainable Social Emotional Learning at School Through Relationship-Centered Learning Environment, Teaching Methods and Formative Assessment. Journal of Teacher Education for Sustainability, 22(1), 21–36. https://doi.org/10.2478/jtes-2020-0003 DOI: https://doi.org/10.2478/jtes-2020-0003
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the New World of HCI. Interactions, 24(4), 38–42. https://doi.org/10.1145/3085558 DOI: https://doi.org/10.1145/3085558
Gardner, J., O’Leary, M., & Yuan, L. (2021). Artificial Intelligence in Educational Assessment: “Breakthrough? Or Buncombe and Ballyhoo?” Journal of Computer Assisted Learning, 37(5), 1207–1216. https://doi.org/10.1111/jcal.12577 DOI: https://doi.org/10.1111/jcal.12577
Heritage, M. (2018). Assessment for Learning as Support for Student Self-Regulation. The Australian Educational Researcher, 45(1), 51–63. https://doi.org/10.1007/s13384-018-0261-3 DOI: https://doi.org/10.1007/s13384-018-0261-3
Ho, A., Hancock, J. T., & Miner, A. S. (2018). Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot. Journal of Communication, 68(4), 712–733. https://doi.org/10.1093/joc/jqy026 DOI: https://doi.org/10.1093/joc/jqy026
Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Buckingham Shum, S., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., & Koedinger, K. R. (2022). Ethics of AI in Education: Towards a Community-Wide Framework. International Journal of Artificial Intelligence in Education, 32(3), 504–526. https://doi.org/10.1007/s40593-021-00239-1 DOI: https://doi.org/10.1007/s40593-021-00239-1
Jacob, B. A., & Lefgren, L. (2008). Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education. Journal of Labor Economics, 26(1), 101–136. https://doi.org/10.1086/522974 DOI: https://doi.org/10.1086/522974
Kim, H., Sefcik, J. S., & Bradway, C. (2017). Characteristics of Qualitative Descriptive Studies: A Systematic Review. Research in Nursing & Health, 40(1), 23–42. https://doi.org/10.1002/nur.21768 DOI: https://doi.org/10.1002/nur.21768
Kraft, M. A. (2019). Teacher Effects on Complex Cognitive Skills and Social-Emotional Competencies. Journal of Human Resources, 54(1), 1–36. https://doi.org/10.3368/jhr.54.1.0916.8265R3 DOI: https://doi.org/10.3368/jhr.54.1.0916.8265R3
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of Educational Objectives: Handbook II: Affective Domain. David McKay Company.
Luo, J., Zheng, C., Yin, J., & Teo, H. H. (2025). Design and Assessment of AI-Based Learning Tools in Higher Education: A Systematic Review. International Journal of Educational Technology in Higher Education, 22(1), 42. https://doi.org/10.1186/s41239-025-00540-2 DOI: https://doi.org/10.1186/s41239-025-00540-2
Machost, H., & Stains, M. (2023). Reflective Practices in Education: A Primer for Practitioners. CBE-Life Sciences Education, 22(1), es2. https://doi.org/10.1187/cbe.22-07-0148 DOI: https://doi.org/10.1187/cbe.22-07-0148
Meyliasari, A. R. (2024). Penyusunan Instrumen Penilaian Afektif di Sekolah [Development of Affective Assessment Instruments in Schools]. Muaddib: Jurnal Pendidikan Agama Islam, 2(2), 430–441. https://ejournal.insuriponorogo.ac.id/index.php/muaddib/article/view/6590
Min, B., Ross, H., Sulem, E., Veyseh, A. P. Ben, Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2023). Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey. ACM Computing Surveys, 56(2), 1–40. https://doi.org/10.1145/3605943 DOI: https://doi.org/10.1145/3605943
Mohamad, S. K., & Tasir, Z. (2023). Exploring How Feedback Through Questioning May Influence Reflective Thinking Skills Based on Association Rules Mining Technique. Thinking Skills and Creativity, 47, 101231. https://doi.org/10.1016/j.tsc.2023.101231 DOI: https://doi.org/10.1016/j.tsc.2023.101231
Obigbesan, O., Graham, K., & Benzies, K. (2024). Software Testing of eHealth Interventions: Existing Practices and the Future of an Iterative Strategy. JMIR Nursing, 7, e56585. https://doi.org/10.2196/56585 DOI: https://doi.org/10.2196/56585
Pit, S. W., Hamiduzzaman, M., Schneider, C. R., & Barraclough, F. (2025). Evaluation Framework for Conversational AI Agents in Pharmacy Education: A Scoping Review of Key Characteristics and Outcome Measures. Research in Social and Administrative Pharmacy, 21(10), 729–742. https://doi.org/10.1016/j.sapharm.2025.05.006 DOI: https://doi.org/10.1016/j.sapharm.2025.05.006
Pit-ten Cate, I. M., Hörstermann, T., Krolak-Schwerdt, S., Gräsel, C., Böhmer, I., & Glock, S. (2020). Teachers’ Information Processing and Judgement Accuracy: Effects of Information Consistency and Accountability. European Journal of Psychology of Education, 35(3), 675–702. https://doi.org/10.1007/s10212-019-00436-6 DOI: https://doi.org/10.1007/s10212-019-00436-6
Rogers, G. D., Mey, A., & Chan, P. C. (2017). Development of a Phenomenologically Derived Method to Assess Affective Learning in Student Journals Following Impactive Educational Experiences. Medical Teacher, 39(12), 1250–1260. https://doi.org/10.1080/0142159X.2017.1372566 DOI: https://doi.org/10.1080/0142159X.2017.1372566
Rovinelli, R. J., & Hambleton, R. K. (1977). On the Use of Content Specialists in the Assessment of Criterion-Referenced Test Item Validity.
Shute, V. J., & Rahimi, S. (2017). Review of Computer-Based Assessment for Learning in Elementary and Secondary Education. Journal of Computer Assisted Learning, 33(1), 1–19. https://doi.org/10.1111/jcal.12172 DOI: https://doi.org/10.1111/jcal.12172
Stephens, M., & Ormandy, P. (2019). An Evidence-Based Approach to Measuring Affective Domain Development. Journal of Professional Nursing, 35(3), 216–223. https://doi.org/10.1016/j.profnurs.2018.12.004 DOI: https://doi.org/10.1016/j.profnurs.2018.12.004
Sukhera, J. (2022). Narrative Reviews: Flexible, Rigorous, and Practical. Journal of Graduate Medical Education, 14(4), 414–417. https://doi.org/10.4300/JGME-D-22-00480.1 DOI: https://doi.org/10.4300/JGME-D-22-00480.1
UNESCO. (2022). Recommendation on the Ethics of Artificial Intelligence. UNESCO.
UNESCO. (2023). Guidance for Generative AI in Education and Research. https://doi.org/10.54675/EWZM9535 DOI: https://doi.org/10.54675/EWZM9535
van der Schaaf, M., Baartman, L., Prins, F., Oosterbaan, A., & Schaap, H. (2013). Feedback Dialogues That Stimulate Students’ Reflective Thinking. Scandinavian Journal of Educational Research, 57(3), 227–245. https://doi.org/10.1080/00313831.2011.628693 DOI: https://doi.org/10.1080/00313831.2011.628693
Vistorte, A. O. R., Deroncele-Acosta, A., Ayala, J. L. M., Barrasa, A., Lopez-Granero, C., & Marti-Gonzalez, M. (2024). Integrating Artificial Intelligence to Assess Emotions in Learning Environments: A Systematic Literature Review. Frontiers in Psychology, 15, 1387089. https://doi.org/10.3389/fpsyg.2024.1387089 DOI: https://doi.org/10.3389/fpsyg.2024.1387089
Wang, P., Chen, F., Wang, D., & others. (2025). Enhancing Students’ Dialogic Reflection Through Classroom Discourse Visualisation. International Journal of Computer-Supported Collaborative Learning, 20, 293–315. https://doi.org/10.1007/s11412-024-09443-2 DOI: https://doi.org/10.1007/s11412-024-09443-2
Wang, Q. (2024). The Educational Design Research Approach. In Designing Technology-Mediated Learning Environments. Springer. https://doi.org/10.1007/978-981-96-0680-1_8 DOI: https://doi.org/10.1007/978-981-96-0680-1_8
Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are We There Yet? A Systematic Literature Review on Chatbots in Education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924 DOI: https://doi.org/10.3389/frai.2021.654924
Yun, H., & Cho, J. (2022). Affective Domain Studies of K-12 Computing Education: A Systematic Review From a Perspective on Affective Objectives. Journal of Computers in Education, 9, 477–514. https://doi.org/10.1007/s40692-021-00211-x DOI: https://doi.org/10.1007/s40692-021-00211-x
Yusoff, M. S. B. (2019). ABC of Content Validation and Content Validity Index Calculation. Education in Medicine Journal, 11(2), 49–54. https://doi.org/10.21315/eimj2019.11.2.6 DOI: https://doi.org/10.21315/eimj2019.11.2.6
Yusuf, H., Money, A., & Daylamani-Zad, D. (2025). Pedagogical AI Conversational Agents in Higher Education: A Conceptual Framework and Survey of the State of the Art. Educational Technology Research and Development, 73, 815–874. https://doi.org/10.1007/s11423-025-10447-4 DOI: https://doi.org/10.1007/s11423-025-10447-4
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Sinergi International Journal of Education
This work is licensed under a Creative Commons Attribution 4.0 International License.
