Research on a Job Competency Profiling and Multi-Agent Collaborative Intelligent Training System for Vocational Learners

DOI: https://doi.org/10.62517/jike.202604318

Author(s)

Yinghao Ma1,*, Yifan Li1, Jingjing Zhou1, Yaohui Tian1, Mingming Gong2

Affiliation(s)

1School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, Henan, China 2Xunfei Lingzhi Talent Training Department, iFlytek, Zhengzhou, Henan, China *Corresponding Author

Abstract

Rapid changes in occupational knowledge and skill requirements create challenges for intelligent training systems, including weak alignment between learner competencies and job requirements, insufficient domain grounding of generated resources, and uncontrolled generation for out-of-knowledge requests. This paper proposes a domain-knowledge-enhanced multi-agent intelligent training system for vocational learners. The system constructs a job competency graph linking positions, competency domains, skills, knowledge points, courses, and practical tasks, and organizes learner experience, skill labels, learning behavior, and practical performance into a job-oriented competency profile. Learning diagnosis, job competency analysis, knowledge retrieval, content generation, quality verification, and learning evaluation agents collaborate to organize training resources. Hybrid retrieval, competency-domain filtering, dynamic Top-K, and a deterministic knowledge-boundary gate are introduced to improve generation reliability. Expert evaluation of 24 course resources produced an overall score of 4.097. On 30 held-out boundary tasks, the gate achieved 93.33% accuracy and 93.27% macro-F1, while increasing the strict termination rate for insufficient-knowledge tasks to 100%. Hybrid retrieval achieved 100% Recall@3; after competency-domain filtering and dynamic Top-K were introduced, precision increased from 0.2000 to 0.5104 and the irrelevant-document rate decreased from 80.00% to 48.96%. These results demonstrate that the proposed system improves domain grounding and generation-boundary control while maintaining effective knowledge retrieval.

Keywords

Vocational Learners; Intelligent Training; Multi-Agent System; Domain Knowledge Base; Retrieval-Augmented Generation; Job Competency Graph

References

[1] Anagnostou G, Doumanas D, Kotis K. LLM4KGen: A framework for developing KG-based semantic applications with LLMs, RAG and AI agents.Data & Knowledge Engineering, 2026, 165102626-102626. DOI: 10. 1016/J. DATAK. 2026. 102626. [2] TLILI A, SHEHATA B, ADARKWAH M A, et al. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 2023, 10(1): 15. [3] LO C K. What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 2023, 13(4): 410. [4] OGUNLEYE B, FOSTER C, et al. A systematic review of generative AI in higher education. International Journal of Educational Technology in Higher Education, 2024, 21: 30. [5] DWIVEDI Y K, KSHETRI N, HUGHES L, et al. So what if ChatGPT wrote it? Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI. International Journal of Information Management, 2023, 71: 102642. [6] XI Z, CHEN W, GUO X, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. [7] HONG S, ZHENG X, TANG J, et al. MetaGPT: Meta programming for multi-agent collaborative framework//International Conference on Learning Representations. 2024. [8] QIAN C, LIU X, FENG H, et al. ChatDev: Communicative agents for software development//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024: 15174-15186. [9] Skulskiy N Y, Cheremisina N E ,Kirov F E , et al.Adaptive Automated Response System for Virtual Computer Lab and LMS Moodle Using LLM, RAG, and Serverless Architecture. Physics of Particles and Nuclei, 2026, 57(4): 578-580. [10] WANG L, MA C, FENG X, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024, 18(6): 186345. [11] ASAI A, WU Z, WANG Y, et al. Self-RAG: Learning to retrieve, generate, and critique through self-reflection//International Conference on Learning Representations. 2024. [12] YAN S, GUI H, ZHANG J, et al. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024. [13] GAO Y, XIONG Y, GAO X, et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. [14] ES S, JHA S, et al. RAGAS: Automated evaluation of retrieval augmented generation//Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 2024: 150-158. [15] SAAD-FALCON J, HOU Y, et al. ARES: An automated evaluation framework for retrieval-augmented generation systems// Proceedings of NAACL-HLT. 2024: 338-354. [16] RU D, QIAN Y, et al. RAGChecker: A fine-grained framework for diagnosing retrieval-augmented generation//Advances in Neural Information Processing Systems. 2024, 37. [17] DEEPSEEK-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv: 2501. 12948, 2025.