Using Gap Analysis to Improve Student Outcomes

By Christine Sproles, MSN, BSN, RN, Assistant Dean of Nursing and Nursing Instructor, Fortis Institute, Pensacola, Florida

A recent article in the Career Education Review entitled “The Path to Success in Certification and Licensure” by LeeAnn Rohmann and Lori Sypher laid a magnificent foundation for the topic of improving student outcomes. Rohmann and Sypher stated in their article “The first step in improving certification and licensure rates is a firm understanding of the current factors leading to an unsatisfactory outcome. There are three components that need to be evaluated: students, tools, and leaders.” In this article, I would like to focus primarily on the tools that assist our students in achieving successful outcomes.

As educators, our number one goal should be student success. As an educator in an associate of science in nursing program, the focus is on students being successful to complete the program.

But even more importantly, the goal is for the graduate nursing student to be successful in passing the state licensure examination – NCLEX-RN!

There are several different types of nursing specialty and comprehensive (or end-of-program) exams – for example ATI and HESI – both of which offer comprehensive or “exit” exams, as well as focused remediation. Our nursing program uses the HESI specialty and exit exams. The specialty exams – Health Assessment, Fundamentals, Pharmacology, Maternity, etc., are used as final exams for those specialty courses. The HESI exit exam is used as a comprehensive evaluation tool prior to students graduating from the nursing program.

Identifying the problem

When I first began teaching at the school where I am currently employed, I had never heard of these specialty exams. At the other institution where I taught, nursing faculty wrote their own final exams, as did I. I had a rude awakening after teaching my first term and administering the Pharmacology HESI exam – 2 out of 12 students in the class passed the exam – and both of these students were LPNs. I was devastated, to say the least! I had failed my students. After a few terms of mediocre scores on the Pharmacology HESI, I went to the Dean of Nursing and discussed with her my dissatisfaction with the low scores. The benchmark for HESI scores is typically 850, which is roughly a conversion score of 78 percent – give or take a few points. Seventy-eight percent is the minimum passing score at our institution for nursing courses and prerequisite sciences. After several terms, the mean score was still only 838 – well below the benchmark. I kept asking myself WHY???? The problem was identified – students were not fully prepared for the rigorous testing of the specialty exams – whether because I failed to cover a topic or because it wasn’t emphasized enough – whatever the problem – it was there. I met with the Dean of Nursing to discuss the problem.

Instructional/gap analysis

From here, the process of gap analysis began. These scores were typical of several nursing courses – not just Pharmacology. All instructors were pretty much doing the same thing. We determined we were going to bring up those test scores! We looked at all the variables that could be contributing to the problem – what and how the material was taught and tested was first on the list. For Pharmacology, I knew it was something I was or was not doing that was keeping my students from being successful. The obvious goal was identified – bringing the scores up to or above the benchmark of 850. This would be a good indicator that students were learning what they needed to know – and they would be better equipped to correctly answer the critical thinking type questions that are included in the HESI specialty exams. So where did we go from here?

Since I was piloting this gap analysis, the first step was to determine the gap. What were the missing components to students being successful on their HESI exams? From the Evolve website, where instructors who use HESI exams have access to test results, I pulled up the results of the Pharmacology exam to review the details. The standard HESI specialty exam has 55 questions, with five questions being pilot questions. The details that are accessible to faculty include the topic (NOT the question), the institution’s results (percentage score), and the national average (also a percentage score). When I first undertook the gap analysis, I noted that my students scored either below 75 percent or more than 5 percent below the national average on 32 out of the 55 questions. That was pretty revealing.

Determine the cause of the gap

The next step, after a breakdown of the data and determining what was missing in content, was to determine the cause of the gap. I began to look at all of the course materials/resources that I was using. This included the textbooks, syllabus, supplemental resources, handouts, homework assignments, and evaluation methods (tests and quizzes). I found that the textbook being used was an excellent text, but it was actually designed more for advanced degree nurses. As a matter of fact, several of my peers who were enrolled in programs for advanced degrees were using that exact text for their Advanced Pharmacology courses. I investigated other texts and found one that fit our needs – one that was from the same publisher. I studied the syllabus to see if there was something missing – was I including all the necessary topics to help my students be successful? Were my lecture materials, PowerPoint slides, and so forth, adequate to cover the material? Were homework assignments beneficial or just “busy work? I had to determine if these resources were helpful or not.

One of the biggest items I needed to evaluate was how I was evaluating my students’ knowledge of the material being taught.

As I began to look at the evaluation methods, I realized several things. First of all, there was no true direction for how my students should study for a test. As you know, course objectives are very few and very broad. They are not specific for drug categories – not specific for content. They cover things like culture, communication, safety, and so forth. Students were always asking for a study guide. As a faculty, we decided we were going to develop specific lesson objectives for each lesson, thereby providing our students with a study guide. These objectives were to be based on Bloom’s taxonomy to ensure higher cognitive levels rather than just knowledge and comprehension. We wanted to focus on the students understanding and being able to apply the information they were learning. As Rohmann and Sypher stated in their article, this is to help students develop critical thinking for use in the clinical setting – not just memorizing facts.

Consider evaluation methods and adjust curriculum

As a result of the gap analysis, it was determined that we needed more structure and focus on our own evaluation methods. We spent time working on understanding how to blueprint our tests based upon the NCLEX-RN test plan and the client needs categories, which are also included in the HESI exam results provided by Elsevier on the Evolve website. We found that we couldn’t just randomly ask questions but that we needed to focus on testing/evaluating content included in the NCLEX-RN test plan. From this analysis, we saw we had low test scores in the area of safety in a number of our nursing courses. We decided that this was an area that needed more focus in our curriculum. As a faculty, we decided to start using a “word” for the term – a focus item. We emphasized safety in our classes, developed an assignment for all nursing courses that included evidence-based research on safety and had the students write a bibliography card on that particular safety issue. For example, in Pharmacology, the students’ assignment was on reducing medication errors. In Fundamentals, the assignment was on how the use of technology affects patient safety.

The faculty determined that an adjustment in the curriculum should be made based upon the common areas of weakness that we see as a result of this instructional analysis. Recently we have identified inconsistent scores across the board in the area of informatics. As a result, we have made our focus for this year informatics. We are incorporating electronic health records assignments in our simulation lab and increasing the use of informatics in our nursing courses and clinical assignments. We have adjusted our safety assignment to now be something that deals with informatics and nursing.

Tracking tool

We developed a tracking tool that includes topics of culture/spiritual, QSEN, NLN, and client needs categories. This is simply a chart with all of these topics broken down and a box where the instructor can record their classes’ overall score in those areas. QSEN includes topics such as communication, informatics, safety, etc. The client needs categories are those specific to the NCLEX-RN exam and include basic care and comfort, health promotion and maintenance, etc. We first started using this tracking tool after identifying the low scores in safety. Over time we saw the low scores in informatics. The reason we didn’t identify informatics sooner is because our scores were so erratic. One time a class would score over 900 in informatics and another time 600. This tracking tool enabled us to identify trends in low HESI scores.

Teaching tool

Another strategy (tool) we began to implement – and require of our faculty – was to incorporate a minimum of 5 NCLEX-style questions in each lesson. Our students were struggling with moving from knowledge and comprehension questions (that they had been accustomed to in their general education courses) to critical thinking – application and analysis questions – in their nursing courses.

It doesn’t matter whether you are a nursing instructor or an instructor in another allied health program.

Providing practice questions is an excellent tool for helping your students understand how to read and answer these higher level questions.

Three huge benefits of using these practice questions in class are exposure to certain topics in a testing format, helping students understand rationales for why the correct answer is correct and the other answers are incorrect, and teaching test taking strategies. For example, looking for key words in the question, eliminating wrong answers, prioritizing, etc. These practice questions really help the students develop critical thinking skills. These questions help prepare the students for HESI and the NCLEX exams.

We provide our faculty with a set of alphabet cards: A, B, C, D. They provide these to their students at the start of class. These lettered cards are different colors, for example, A might be red, B blue, C green, and D yellow. The practice questions are put in a PowerPoint slide to be viewed by the class on an overhead screen. The instructor reads the question and answers aloud. The students are then asked to answer the question by holding up the corresponding card with the letter for the correct answer.

It is important to make this a very nonthreatening learning environment. Let the students know that you want everyone to answer the questions, but not to be concerned with how those around them are answering. I have had classes where every student but one answered a question incorrectly. That one student was not afraid to make their answer known, even though they stood alone. Remind them that this a teaching time – not a testing time. They should not feel embarrassed or singled out. The instructor’s attitude can make a huge difference in the success or failure of this teaching tool.

These questions can be used at the start of class, reviewing content from the previous lesson. They can be used at the end of class, reviewing material just covered. Or they can be used throughout the class to break up the flow and to switch from one topic to another. My students love these! They ask for more!!! These questions can even be used as a pop quiz. If this teaching format is routinely used, the students will be prepared to answer the questions.

Gap analysis findings

So from the instructional analysis, several things were evident: 1) Specific content was not being covered in class. 2) Some items that were significantly important were not being evaluated with either a test or a quiz, so the student didn’t “hang on” to that information. They weren’t retaining it. 3) Testing was done using low levels of Bloom’s taxonomy cognitive domain of mostly knowledge and comprehension – with very few higher level questions at the application and analysis level.

Curriculum revisions

As far as delivery of content was concerned, I had to adjust my style of teaching. One of the first things I did was to make sure that content from the HESI Pharmacology exam was included in my lecture material. I had been using handouts for my students for note-taking purposes that covered each major topic being covered. For example, several pages of handouts were on cardiovascular drugs – one for antihypertensives, one for heart failure, one for the treatment of angina, etc. I combined all of these handouts and made them into a Pharmacology packet – a printable packet of handouts made into booklet format. I revise this packet every year, making additions and adjustments necessary based upon the content covered on the HESI exams and new drugs that are on the market. Many times this information is not found in our Pharmacology textbook. I have to research the information and add it to the packet so that the students are exposed to it. I added the NCLEX questions, as mentioned before. I began discussing difficult concepts with the students using higher levels of questioning based upon Bloom’s taxonomy – more application and analysis questions as opposed to knowledge and comprehension.

The last piece was to add a focused review prior to the specialty exam, to include concepts the students performed poorly on in the previous term(s).

We developed our review materials purely off of the content provided in the HESI report. We try to make sure that we cover topics where students have consistently demonstrated low scores.

We do not allow our students to see the questions and rationales after taking a specialty exam. We have found that this compromises future test results. We developed the review solely on areas of poor performance. So, again, as instructors, we do NOT know the questions being asked. We are not teaching to the test. Rather, we are providing a focused review on topics covered on the HESI specialty exams and on the HESI exit (comprehensive) exam. We also provide specialty reviews for our seniors that we call our “HESI boot camp.” The lead instructor for each specialty nursing course gives a live review of those topics with the seniors during the weeks prior to the exit HESI.

Individualized remediation

As mentioned above, we do not allow our students to see questions and rationales after completion of the HESI exams. However, we do require our students to complete a competency development plan that includes a specific number of hours of remediation time based upon their HESI scores. This specific HESI remediation is individualized for each student in their own Evolve account that is linked to the HESI exam. For example, if a student misses a question or questions on cultural topics, they will have remediation focusing on culture. If they miss dosage calculations questions, they will have dosage calculations remediation. This is a required component of every nursing class and for every student, regardless of their scores on the HESI exams.

The final piece of the gap/instructional analysis was to do a formative evaluation. We performed an evaluation to ensure the revisions we made were contributing to change and improved outcomes. We looked at individual student scores. We looked at group (class) mean scores on the HESI exams. We conducted mid-term and end-of-course evaluations for student feedback. One of the results of this evaluation process was the development of a HESI specialty exam analysis tool. The components of this analysis tool included a topic area where students scored below 75 percent or greater than 5 percent below national average, the institution’s score and the national average, an analysis of the score based upon the topic tested, and a plan for future improvement. For example, the HESI analysis results showed that in the spring of 2013, my students scored 13 percent on a question that dealt with Haldol and extrapyramidal symptoms. At that time, I was not teaching antipsychotic medications. I had a few items listed in a table in the packet that I mentioned earlier, but I really didn’t teach this topic. I have since added this topic to one of the lesson objectives, added this information in the Pharmacology packet, it is included in the class discussion, and I show a video demonstrating patients taking antipsychotic medications, such as Haldol, who exhibit extrapyramidal symptoms. This helps the students understand this concept so much better. I have also included this concept on a unit exam.

Results of gap analysis with curriculum revisions

As I mentioned earlier, there were 32 areas of concern on the Pharmacology specialty exam prior to instituting the instructional analysis and making the curriculum revisions. We identified the need. We evaluated curriculum and evaluation methods. We made a plan. Six months after identifying the gap and doing the analysis and revisions, the Pharmacology HESI scores went from 838 (850 being the benchmark – approximately 78 percent) to 1011 (conversion score over 90 percent). There were only 12 areas of concern (students scoring below 75 percent or greater than 5 percent below national average).

Student satisfaction scores improved by 13 percent. And as most educators know, student satisfaction is directly correlated to success in testing.

The following term, the mean score on Pharmacology HESI was 1123 (96 percent), with a median score of 1151. It was amazing!

Outcomes

After implementing these strategies for a couple of years, our Dean of Nursing moved on to becoming Regional Dean. One of the schools in her region was really struggling with HESI scores, particularly Pharmacology. The Dean asked if I would share these strategies and my Pharmacology packet with this other school, which of course, I did. This was September 2016. After implementing these strategies at their school for just one 12-week period, their HESI results and NCLEX pass rates improved dramatically. These results show HESI scores before and after implementing the gap analysis and instructional revisions.

• Fundamentals 811 to 915
• Pharmacology 746 to 894
• Dosage & Calculations 688 to 1147
• Maternity 80 to 1081
• Medical-Surgical Nursing 898 to 1090
• Pediatrics 923 to 1095

This school finished out the year of 2016 with an NCLEX pass rate of 61.9 percent. For the first quarter of 2017, they had eight first-time testers with eight passing. That is 100 percent NCLEX pass rate after implementing these strategies.

The improvement is remarkable! According to the Dean of Nursing, they still had 2 to test from that cohort, which would give them – worst case scenario – 80 percent pass rate. Far better than 61.9 percent!

It is one thing to see this improvement in our own school, but it was a totally different thing to see such a dramatic improvement in another school who implemented these strategies in just one term.

Our own outcomes in the term immediately after implementing the gap analysis showed increased HESI specialty exam scores, improved student satisfaction, NCLEX pass rates going from 71 percent in 2013 to 89 percent in 2014. At this point in time, our NCLEX pass rate for 2016 was over 90 percent with the first quarter of 2017 being 100 percent. Eighty percent of 2014 graduates were employed as RNs. At the end of 2015, 94 percent of our graduates were employed as RNs, and at the end of 2016, 93 percent were employed. Some graduates have already enrolled in RN to BSN programs immediately after graduation.

Making change is often difficult, but the rewards are amazing! Better student outcomes should be our goal as educators. These are just a few of the ways that we have worked to improve our curriculum, our evaluation process, and our overall student outcomes. I hope you will find something of value to use at your institution to improve student outcomes.

Resources
HESI Review and Testing, Evolve (St. Louis: Elsevier).

Rohmann, LeeAnn and Lori Sypher, (2017). The Path to Success in Certification and Licensure. Career Education Review, March 7, 2017.

CHRISTINE SPROLES is the Assistant Dean of nursing and a nursing instructor at Fortis Institute in Pensacola, Florida. She has been an educator for more than 30 years, teaching courses in medical office administration and nursing.

During her time at Fortis, Christine has developed a means to motivate her students and increase morale along with a Math for Success program to help students progress in math skills throughout their program. Realizing a need to improve performance in her pharmacology course, she performed a gap analysis using HESI specialty exams to identify content areas where students scored low and executed a plan to boost student outcomes. The results were positive with mean scores and student satisfaction increasing significantly.

Christine is an active faculty member at Fortis Institute performing functions such as assisting faculty members with lesson objective development using appropriate cognitive levels and serving on a subcommittee to revise clinical skills requirements. She is chairperson for the Evaluation and Outcomes Committee for the nursing program at Fortis.

Recognized for her commitment to nursing education, she received the Five-Star Faculty award from Fortis Institute in 2014 and the Outstanding Teacher Award from Pensacola Christian College in 2002. Christine received the 2015 Master Teacher Award at the 12th Annual National Conference on Allied Health Education by ABHES and Elsevier.

Christine has been a break-out session speaker at several nursing and allied health conferences nationally, speaking on topics related to student success. She also serves as a reviewer for textbooks, such as McGraw-Hill, Pearson, and Davis.

She earned her Bachelor of Science in Nursing from Pensacola Christian College in Pensacola, Florida. She furthered her education and received a Master of Science in Nursing from the University of Portland in Portland, Oregon.