Professor Katharine Boursicot

Professor Sandra Kemp

Quality Assurance of OSCEs

OSCEs are widely used to assess clinical, communication and practical skills in health professions education, especially in high-stakes contexts. The quality assurance of OSCEs is therefore of critical importance to ensure that such tests are fair, robust and defensible. OSCEs are highly complex to design and implement, and many different aspects have to be scrutinised, but this not always conducted systematically. We have taken a specific approach to quality assuring OSCEs, using what we refer to as the Kane Validity Framework, as this provides a holistic and systematic framework to use when evaluating the quality of an OSCE.

Link

Dr Eric Holmboe

Designing assessment practices to support effective professional development of medical students

Becoming a physician is an intensely developmental process. Learning trajectories will vary by student and abilities. Assessment practices should be designed with this knowledge in mind and co-produced with learners. Effect assessment systems require a multi-faceted, multi-dimensional programmatic approach. At the end of the session, participants will:

1. Understand learning trajectories and their implications in designing programs of assessment;
2. Consider how to leverage and apply co-production principles and practices with learners;
3. Apply programmatic thinking in designing assessment systems

Link

Prof Lambert Schuwirth

Dr Joanna Tai

Prof Margaret Bearman

Harnessing AI ‘for good’ in clinical assessment

Current discourse portrays Artificial Intelligence (AI) as the enemy of academics, a force to be feared rather than harnessed. However, it is an integral part of how students learn, how teachers learn, and how assessors assess. Therefore, it behoves educators to embrace this technology to enhance students’ experience, as well as improve the capacity of educators to facilitate lifelong learning through the generation of critical analytical skills in learners.

Drawing on recent keynote presentations from ANZAHPE, and the published TEQSA report Assessment reform for the age of artificial intelligence Discussion Paper - our panelists introduce an overview of the increasing use of AI in higher education, including an exploration of its use in health professional assessment, and principles of equity and inclusion that need to be considered.

Link

Dr Subha Ramani

Emotional Intelligence Competencies that Enhance Leadership Skills in Health Professions Education

Emotional Intelligence (EI) refers to recognition and effective management of one’s own emotions as well as others. It can be viewed from two distinct perspectives, self-management and relationship management. EI has been studied in many disciplines with findings that indicate increased resilience, internal motivation, and leadership acumen. This session reviewed core attributes of EI, allowed participants to self-reflect on their strengths and areas for improvement in specific EI competencies. Participants were also prompted to  formulate strategies to build their own resilience and leadership skills, and apply effective strategies to enhance both among their learners. After a mini-didactic review of EI definitions and competencies, participants engaged in a self-reflective exercise to review which of the competencies they would like to improve on. Some brief cases were discussed to apply EI skills to overcome specific challenges. The session concluded with sharing of own experiences- successes and challenges, and a commitment to change own practice.

Link

Ms Melissa Oxley

Ms Renee Harvey

Assuring quality of assessment processes

Assuring the competence of medical students through assessment of knowledge, skills and attitude is highly complex. One of the most commonly used assessment formats in medical schools in Australia and New Zealand remains the Objective Structured Clinical Examination (OSCE). The OSCE is a high-stakes clinical assessment, with many variables - often running simultaneously across multiple campuses, involving hundreds of participants (including students, assessors, standardised patients and support staff). Just how do you embed quality assurance processes into these exams to ensure a high-quality exam, with fair and equitable assessment of student competence? Management of the full exam ‘lifecycle’ is critical and in this session we will cover how to embed high quality processes of banking and storing items; exam blueprinting; examiner calibration; participant briefings; results analysis and reporting; and moderation of results.

Link

Associate Professor Helen Wozniak

Exploring the nexus between feedback, action and engagement in workplace-based assessment

Workplace-based assessments (WBA) are intended to promote assessment for learning but realising this promise is not easy. Tensions arise when the richness of being observed is stymied with pressures to tick off requirements, or when opportunities for feedback conversations are clouded by performance-based decision making.

In this presentation Helen will outline the WBA journey at the University of Queensland medical program by sharing lessons learnt from the exploration of the WBA digital footprints collected from 3rd and 4th year medical students. This has included an analysis of the relationship between text-based assessor feedback records and student action plans, and mapping when and where learners engage with WBAs. While managing large WBA data sets presents new challenges, gaining an appreciation of the UQ WBA context informed an ongoing review of the wider assessment system design and opened opportunities to think about how best to foster student agency in the clinical environment. This presentation prompted discussion between participants to consider how best to apply the lessons learnt to their context.

Link

Professor Brian Jolly

Making sense of work-place based assessment

Brian Jolly’s 2012 paper with Jim Crossley (of the same title) was a work that condensed the research on and ideas about WBA at that time around 4 main concepts. First that the right questions be asked; second that they needed to be phrased appropriately, and third, they should be about important and measurable features, and finally they should be directed towards the right people, i.e., those that would know about those measurable features. The paper also raised questions for us about life in general and what to do with it for the best and has been cited between 250 and 300 times.

This interactive workshop will unpack those ideas along with a narrative that describes how the paper nearly did not get written, and when it did some of its ideas became uncomfortable for some clinicians and academics. It raised the issues of interprofessional contribution to assessment, and how the questions should be expressed in the best way (not always agreed upon) to investigate the particular skills or domain of interest.

Participants were given opportunity to outline their challenges in assessment and whether adopting WBA is feasible, and any successes they cherish. The four concepts will be outlined, and their usefulness discussed.

Link

Professor John Norcini

Systems of Assessment in Health Professions Education

To fulfill the various roles of assessment in the educational enterprise, multiple examinations are used. Explicitly or implicitly, they constitute a system of assessment. Historically, this ‘system’ has been composed exclusively of high stakes individual assessments that have been aggregated to make decisions about individual students. However, increasing interest in supporting and creating learning using assessments that have no stakes, as well as the development of the competencies, milestones, and entrustable professional activities (EPAs), has created complexities that make a reductionist approach less effective.

This talk will offer a description of the roles assessments play in the educational enterprise and why they are aided by the application of systems thinking. The elements of a well-functioning system will be detailed as will the factors that support its development.

Link

A/Prof. Lynette Clearihan

Strengthening professionalism in medical education - a new approach at Monash University

If we needed reminding CoVID highlighted the importance of professionalism in healthcare from both a personal practitioner perspective and from a health delivery perspective. However how to teach, assess and remediate professionalism in medical education continues to be riven with challenges and diversity of opinion. These challenges for some can feel overwhelming in the relation to professionalism lapses – identifying what they are and how to manage them. The new approach taken by the Faculty of Medicine, Nursing and Health Sciences at Monash University goes to the heart of this challenge. Its focus is to help students’ use their lived experiences to foster their professionalism knowledge, skills, and attitudes and build their understanding of accountability and self-responsibility. It has involved an extensive process of consultation with staff and students and has been fully operational within the faculty for just over 12months.

This presentation outlines what the new approach is, the process used to develop it and explores some of the barriers and enablers faced along the way.

Link

Prof. Cees van der Vleuten

A life-time in medical education; reflections from an expert

After spending 40 years in academia, Professor Cees van der Vleuten will be retiring at the end of 2022, and the ACCLAiM Consortium had the privilege of hosting a Q&A session that enabled members to learn from this amazing expert.

His current research areas include Assessment for learning; Paradigms in assessment; Programmatic assessment and Examiner Subjectivity. Details can be obtained from his website: www.ceesvandervleuten.com

Link

Dr Helen Rienits

 Incorporating Assessor Expertise to improve WBA efficacy

The failure to fail, and elevated grades in Workplace-Based Assessment (WBA) are well-documented phenomena and assessors play a major role in this process. The resulting WBA appears to be of little real use to any of the stakeholders involved. What do assessors think is the problem and what do they think would improve the situation for them? If we develop and trial WBA designed to suit the assessor and use their expertise, will this result in WBA that is more useful for everyone? Dr Helen Rienits' presentation covers the findings of her PhD thesis which was designed to answer these questions.

Link

Prof. Lambert Schuwirth

Assessment for learning and programmatic assessment

Two interesting developments in assessment of the last decades are assessment for learning and programmatic assessment. Assessment for learning is not the same as formative assessment and programmatic assessment is not the same as merely an assessment program. Every new development in education and assessment has to have a rationale; it has to be a solution to a problem. Prof. Lambert Schuwirth's presentation discusses the principles or programmatic assessment and assessment for learning, the problems it tries to solve and possible avenues for further developments.

Link

Prof. Olle ten Cate

Entrustable professional activities (EPA): Their significance for health profession education

In this presentation, Prof. Olle ten Cate discussed how competency-based medical education aims at assisting graduates in meeting the standards of health care practice. He also explains how EPA is a concept that connects competency frameworks to every day clinical practice.

Link

Prof. Tim Wilkinson

Assessing & fostering professionalism through a systems approach

Professionalism is multifaceted. Its' assessment also needs to be multifaceted. Prof. Tim Wilkinson's presentation covered these facets, some assessment tools and how information can be collated, interpreted and acted on. Difficulties with failure to fail, effective handover/feed forward and remediation were also discussed.

Link

Prof. Elizabeth Molloy

Prof. David Boud

Improving workplace feedback for medical students: Where should we focus?

Prof. Elizabeth Molloy and Prof. David Boud shared their expertise on feedback in medical education. This included the complex issues surrounding: encouraging students to be active in the process, reorientating educators and students to see the feedback process differently, and the current strong coupling of Workplace-Based Assessment with feedback which is seen to influence what learners will share.

Link