Development of Operational Level Practical Test Instrument by Using Competency-Based Assessment Model

Authors

  • Irwan Irwan Politeknik Pelayaran Sumatera Barat
  • Sarifuddin Sarifuddin Politeknik Pelayaran Sumatera Barat
  • Riki Wanda Putra Politeknik Pelayaran Sumatera Barat
  • Hasri Devin Politeknik Pelayaran Sumatera Barat

DOI:

https://doi.org/10.37329/cetta.v7i4.3665

Keywords:

Operational Level, Practical Test Instrument, Competency-Based Assessment

Abstract

The lack relevance of practical ability assessment with industry needs causes nautical technology graduates to not be fully prepared to work in the field. Therefore, it is necessary to develop a practical test instrument that is in accordance with Graduate Learning Outcomes. The purpose of this study is to produce an operational level practical test instrument with a valid, reliable, and effective competency-based assessment model. This type of research is development research by using the Wilson Model and the Oriondo and Antonio Model. Validity test was content validity. Reliability test was test-retest and inter-rater reliability. Effectiveness test used the N-Gain percentage. The results of the study prove that the developed instrument is valid with a validity value 3.8 and criterion A. The instrument is reliable because the t-count value is smaller than the t-table value, meaning that there is no difference between the average values ​​of the three raters. The instrument is effective because the N-Gain percentage = 74.6%. The limitations of this study lie in the limited testing, only focusing on the operational level, and the variation of raters who only come from within the institution. The recommendation from this study is that the developed instrument should be tested in other maritime educational institutions. Then, it is necessary to develop similar instruments for the management level in the maritime sector.

References

Abduh, M., Hasnur, J., & Siska, S. Y. (2022). The Effect Of Maritime English Vocabulary For Beginners Module On The Vocabulary Learning Outcomes. Jurnal Pendidikan Vokasi, 12(2), 117-129.

Al-Bahlani, S. M., & Ecke, P. (2023). Assessment Competence And Practices Including Digital Assessment Literacy Of Postsecondary English Language Teachers In Oman. Cogent Education, 10(2), 2239535.

Alt, D., Schneider, N. L., & Weishut, D. J. (2023). Competency-Based Learning And Formative Assessment Feedback As Precursors Of College Students’ Soft Skills Acquisition. Studies in Higher Education, 48(12), 1901-1917.

Andermo, S., Ehrlich, F. K., Larm, F. M., Bergström, L., Siljehag, A. P., & Broberger, E. (2023). Assessing Students Interprofessional Competence Using A Swedish Version Of The Interprofessional Collaborator Assessment Rubric. Journal of Interprofessional Care, 37(4), 605-612.

Angraini, A. R. A., & Okfalisa. (2021). Measuring Information Security Policy Compliance: Content Validity Of Questionnaire. Indonesian Journal of Electrical Engineering and Computer Science, 22(1), 469-475.

Asimakopoulos, I., Valencia, A. L. D., Lützen, M., & Rytter, N. G. M. (2024). Data-Driven Condition Monitoring Of Two-Stroke Marine Diesel Engine Piston Rings With Machine Learning. Ships and Offshore Structures, 19(9), 1241-1253.

Bolstad, T., Wallin, P., Lundheim, L., Larsen, B. B., & Tybell, T. (2021). Emergent Premises In Student Experiences Of A First-Year Electrical Engineering Course. European Journal of Engineering Education, 46(2), 302-317.

Chanmas, G., Taveekitworachai, P., Paliyawan, P., Thawonmas, R., Thawonmas, R., Nukoolkit, C., & Dajpratham, P. (2023). Driving Scenarios And Environmental Settings In Simulator-Based Driving Assessment Systems For Stroke: A Systematic Review. Topics in Stroke Rehabilitation, 30(8), 872-880.

Dragostinov, Y., & Mõttus, R. (2023). Test-Retest Reliability and Construct Validity of the Brief Dark Triad Measurements. Journal of Personality Assessment, 105(2), 143-148.

Eltayar, A. N., Aref, S. R., Khalifa, H. M., & Hammad, A. S. (2022). Do Entrustment Scales Make A Difference In The Inter-Rater Reliability Of The Workplace-Based Assessment?. Medical Education Online, 27(1), 2053401.

Ghosh, S., Emad, G. R., & Ravi, A. (2024). Investigating The Characteristics Of Skills And Competency Frameworks Through A Systematic Literature Review: A Feasibility Study To Revise The Stcw Code For Seafarer Training. Australian Journal of Maritime & Ocean Affairs, 1-17.

Guvey, A. E., & Mermi, F. (2022). Speaking Skills Attitude Scale for Primary School Students: Validity and Reliability Study. In Elementary School Forum (Mimbar Sekolah Dasar), 9(1), 43-57.

Hartikainen, S., Pylväs, L., & Nokelainen, P. (2022). Engineering Students’ Perceptions Of Teaching: Teacher-Created Atmosphere And Teaching Procedures As Triggers Of Student Emotions. European Journal of Engineering Education, 47(5), 814-832.

Holmström, O., & Stjärnhagen, O. (2024). Coordination Of Courses In University Programmes And Students’ Experiences Of Their Studies: Student Perspectives On The Importance Of Course Coordination. Quality in Higher Education, 30(2), 221-238.

Intasoi, S., Junpeng, P., Tang, K. N., Ketchatturat, J., Zhang, Y., & Wilson, M. (2020). Developing An Assessment Framework Of Multidimensional Scientific Competencies. International Journal of Evaluation and Research in Education, 9(4), 963–970.

Kárpáti, A., & Paál, Z. (2022). Assessment Of Visual Sub-Competencies Through Visual Rubrics: Case Studies Based On The Common European Framework Of Reference of Visual Competencies (CEFR-VC). Journal of Visual Literacy, 41(4), 224–246.

Kebede, A., Asgedom, A., & Asfaw, A. (2024). Is linking Technical And Vocational Training With Industry A Bridge To Employability? Lessons From A Literature Review For Ethiopia. Cogent Education, 11(1), 2406721.

Kritkharuehart, S., Vimolsilp, U., & Photipussa, P. (2024). Development Of Competency-Based Assessment Model For KRU RAK THIN Scholarship’Recipients. Cogent Education, 11(1), 2373230.

Kubicki, A., Brika, M., Coquisart, L., Basile, G., Laroche, D., & Mourey, F. (2020). The Frail’Bestest. An Adaptation Of The “Balance Evaluation System Test” For Frail Older Adults. Description, Internal Consistency And Inter-Rater Reliability. Clinical Interventions in Aging, 15, 1249-1262.

Kusmaryono, I., Suyitno, H., Dwijanto, D., & Dwidayati, N. (2018). Developing Assessment Instrument as a Mathematical Power Measurement. Journal of Education and Learning (EduLearn), 12(3), 382-391.

Lai, Y. H., Wu, M. J., Chen, H. H., Lin, S. P., Wu, C. Y., Chin, C. S., Lin, C. H., Shiu, S. I., Lin, Y. C., Chen, H. C., Hou, S. C., & Chang, C. W. (2023). Impacts Of Huddle Intervention On The Patient Safety Culture Of Medical Team Members In Medical Ward: One-Group Pretest-Posttest Design. Journal of Multidisciplinary Healthcare, 16, 3599-3607.

Latifah, M., Saripah, I., Suryana, D., & Sunarya, Y. (2024). Validity and Reliability of Self-Concept Instrument Using Rasch Model. Jurnal Kajian Bimbingan Dan Konseling, 9(1), 26-35.

Mashartanto, A., Pranata, W., & Siska, S. Y. (2024). Development of Learning Media Maritime English Textbook for Ratings Forming. JOLLT: Journal of Languages and Language Teaching, 12(3), 1238–1252.

Masuwai, A., Zulkifli, H., & Hamzah, M. I. (2024). Evaluation Of Content Validity And Face Validity Of Secondary School Islamic Education Teacher Self-Assessment Instrument. Cogent Education, 11(1), 2308410.

Mehta, J., & Connor, O. A. (2023). Test Retest Variability In Stereoacuity Measurements. Strabismus, 31(3), 188–196.

Moore, A. M., Fairhurst, P., Bennett, J. M., Harrison, C., Correia, C. F., & Durk, J. (2024). Assessment And Practical Science: Identifying Generalizable Characteristics Of Written Assessments That Reward And Incentivise Effective Practices In Practical Science Lessons. International Journal of Science Education, 46(7), 643-669.

Mori, Y., & Manuel, M. (2024). An Exploratory Study Of Approaches To Onboard Training In Different Countries. Journal of International Maritime Safety, Environmental Affairs, and Shipping, 8(3), 2386221.

Nada, Q., Herdiana, I., & Andriani, F. (2022). Testing The Validity And Reliability Of The Depression Anxiety Stress Scale (DASS)-21 Instrument For Individuals With Psychodermatology. Psikohumaniora, 7(2), 153–168.

Ofianto, S. N., Ningsih, T. Z., Nababan, S. A., Asri, Z., Aman, & Setiawan, J. (2024). Developing An Online Project-Based Assessment Instrument For Assessing Students’ Historical Thinking Skills. International Journal of Evaluation and Research in Education, 13(6), 3993-4003.

Rahmawati, E., & Harun, H. (2019). Developing Instruments Of Teacher’s Perception Of Critical Thinking In Elementary School. Journal of Education and Learning (EduLearn), 13(4), 559–566.

Raof, S. A., Musta’amal, A. H., Zamzuri, F. K., & Salleh, M. H. (2021). Validity and Reliability of Students Perceptions on OBE Approach in Malaysian VC Using Rasch Model. Journal of Innovation in Educational and Cultural Research, 2(2), 44–50.

Richter, T., & Kjellgren, B. (2024). Engineers Of The Future: Student Perspectives On Integrating Global Competence In Their Education. European Journal of Engineering Education, 49(3), 474–491.

Rohde, A., McCracken, M., Worrall, L., Farrell, A., O’Halloran, R., Godecke, E., David, M., & Doi, S. A. (2022). Inter-Rater Reliability, Intra-Rater Reliability And Internal Consistency Of The Brisbane Evidence-Based Language Test. Disability and Rehabilitation, 44(4), 637–645.

Sabaruddin, S., Nur, M., Fadli, M., & Mazlan, M. (2022). Content Validity of KIP-K Scholarship Instruments at Higher Education Using Aiken’s Coefficient Validity Scale. AL-TANZIM: Jurnal Manajemen Pendidikan Islam, 6(3), 934–947.

Sandal, A. K. (2023). Vocational Teachers` Professional Development In Assessment For Learning. Journal of Vocational Education and Training, 75(4), 654–676.

Sanders, J., Johnson, E., Mirabelli, J., Kunze, A., Vohra, S., & Jensen, K. (2024). Engineering Professor Perceptions Of Undergraduate Engineering Student Stress. European Journal of Engineering Education, 1–21.

Sellberg, C., Lundin, M., & Säljö, R. (2022). Assessment In The Zone Of Proximal Development: Simulator-Based Competence Tests And The Dynamic Evaluation Of Knowledge-In-Action. Classroom Discourse, 13(4), 365–385.

Septianingsih, E., & Jerusalem, M. A. (2021). Developing Instrument Of Academic Potential Test Analogy Verbal Ability For Undergraduate Students. Journal of Education and Learning (EduLearn), 15(2), 234–241.

Sukenti, D., & Tambak, S. (2020). Developing Indonesian Language Learning Assessments: Strengthening the Personal Competence and Islamic Psychosocial of Teachers. International Journal of Evaluation and research In education, 9(4), 1079-1087.

Wakid, M., Sofyan, H., Widowati, A., & Zaida, I. A. (2024). Learning-Oriented Assessment: A Systematic Literature Network Analysis. Cogent Education, 11(1), 2366075.

Yamtinah, S., Dewi, M. C., Nurhayati, N. D., Susilowati, E., Saputro, S., Fakhrudin, I. A., Ramadhani, D. G., & Shidiq, A. S. (2022). Content Validity in Android-Based Augmented Reality Media for High School Science Students on Covalent Bonds Topic: Rasch Model Analysis. Jurnal Pendidikan Sains Indonesia, 10(2), 240–249.

Yildiz, A., Butun, M., Turkdogan, A., & Kocak, E. (2022). Development of a Perception Scale of Private Lesson: A Validity and Reliability Study. Mimbar Sekolah Dasar, 9(3), 517–534.

Yulia, P., & Nasution, E. Y. P. (2024). Geometry and Islamic Values: Validity of Teaching Materials Based on Modified Project-Based Learning Model. Mosharafa: Jurnal Pendidikan Matematika, 13(1), 113–124.

Zain, N. H. M., Aziz, S. R. A., Moketar, N. A., Ishak, N. H., & Hendrayati, H. (2022). Content Validity Study: A Gamification Model To Drive Behavior Change In Defeating Coronavirus Disease 2019 Pandemic. International Journal of Electrical and Computer Engineering, 12(5), 5375-5382.

Downloads

Published

11-11-2024

How to Cite

Irwan, I., Sarifuddin, S., Wanda Putra, R., & Devin, H. (2024). Development of Operational Level Practical Test Instrument by Using Competency-Based Assessment Model. Cetta: Jurnal Ilmu Pendidikan, 7(4), 69–80. https://doi.org/10.37329/cetta.v7i4.3665

Issue

Vol. 7 No. 4 (2024)

Section

Articles

License

Copyright (c) 2024 Irwan Irwan, Sarifuddin Sarifuddin, Riki Wanda Putra, Hasri Devin

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

An author who publishes in the Cetta : Jurnal Ilmu Pendidikan agrees to the following terms:

  • Author retains the copyright and grants the journal the right of first publication of the work simultaneously licensed under the Creative Commons Attribution-ShareAlike 4.0 License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal
  • Author is able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book) with the acknowledgement of its initial publication in this journal.
  • Author is  permitted and encouraged to post his/her  work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of the published work (See The Effect of Open Access).

Read more about the Creative Commons Attribution-ShareAlike 4.0 Licence here: https://creativecommons.org/licenses/by-sa/4.0/.