This month Shannon Dobranski and Susan Belmonte in Pre-Graduate & Pre-Professional Advising (PGPP) describe an assessment project their team worked on over the last year. As a new unit, AY19-20 was their first year completing their own assessment report for the Institute, though they participated as part of the Career Center in the past. Susan Belmonte serves as PGPP’s Assessment committee member for AY20-21. We asked Shannon and Susan about their work.

Tell us about what you were trying to learn from this assessment project:

The three outcomes we were seeking to collect data on were:

• 1. Students who participate in PGPP services will be able to plan their experiences at Georgia Tech with career goals in mind.
• 2. Students who participate in PGPP advising will report a whole-person advising experience.
• 3. Students who participate in at least one virtual advising appointment will report satisfaction with their experience.

Our three outcomes are central to the PGPP mission to deliver highly specialized, personalized advisement to empower students to discern their path and advance their long-term goals. Our outcomes also align with our vision to be known for the quality and effectiveness of our work, the mindfulness and reliability of our team members, and our contributions to the personal growth and professional development of our advisees.

The first outcome is central to our mission and has been an outcome since the formation of PGPP in 2015. Our second outcome is intended to assess the value of the appreciative advising model that informs our practice. Appreciative advising promotes a relationship with students by encouraging them to articulate their goals and to practice the resiliency needed to achieve them. Finally, our third outcome was intended to assess the effectiveness of our expansion into virtual advising. We began offering remote advising opportunities long before COVID-19 and wanted to ensure that the quality of our advising remained high when we met with students by phone or virtual platform.

How did you collect or source the information you needed?

We sent out a survey to our students in fall 2019 and spring 2020 (same survey) and convened a focus group in January 2020.

Were there any challenges you faced with this project? If so, how did you manage them?

Mainly getting students to answer the survey and attend the focus group. For the surveys, we sent out three requests across three weeks. By sending reminders, we were able to get an overall response rate of 20.3%. For the focus group, we did something similar but had very little participation overall (only 5).

How have you been able to use this information to improve your program(s)?

We have sought data for outcome 1 for two years now and have been able to make program and process improvements as a result of what we learned. Appreciative advising uses long-term goals and dreams as the starting point for discussions with our advisees, and we regularly remind ourselves of our mission and use the vocabulary of appreciative advising to support each other in our practice. As a result, we encourage students to think about the long-term, even if they come to us with a more immediate question. Specific steps we have taken to improve our programs include developing tools such as intake forms and Canvas sites to help students remain mindful of their long-term goals.

This year is the first we have sought data for outcomes 2 and 3. Outcome 2 was a result of various discussions about the work we do, the advising philosophy and framework we use, and our experiences working with multiple advisees across several months and even years. Now that we have analyzed the results, we have laid out program and process improvements to strive for in the upcoming year. For example, we now make a point in each individual appointment to ask our advisees how they are doing generally, especially in the context of COVID-19. By doing so, we have found that advisees often open up about what they are feeling and experiencing, which gives us the chance to talk more fully about their well-being. When we ask students about their hobbies and personal interests as part of discussions about applying to graduate school/professional school/fellowships, we use the moment to get them to think beyond the application to how they might continue to pursue that interest in the next stage of their journey.

The move to online advising since March 13 has generated a lot of data for outcome 3--students' satisfaction in virtual advising sessions. PGPP has been able to see that quality of our services, whether virtual or in-person, is high; students expressed roughly the same satisfaction with virtual advisement that they do with face-to-face sessions. When they are dissatisfied, the technology is usually at faulti (e.g. being dropped from a call). In discussing our experiences with on-line advising, we were able to determine that a significant improvement during COVID-19 would be establishing a means of providing private drop-in advisement, which hasn’t always been possible in the “group online advising” we switched to when we began working remotely. As a result of the assessment process and our conversation about our results, we have decided to explore new ways to provide that privacy—at least some of the time—when students drop-in outside of a scheduled appointment. Our conversation also revealed some other benefits of the switch to online services. Moving forward, we will continue the practice of an end-of-year video celebration of our fellowship applicants, and we will convene virtual panels of alumni and other experts who might not be available in person.

If you were to repeat this assessment in the future what would you do differently?

We have used similar assessments in the past and have discovered that the timing of surveys to correspond to reading days along with the repeated request for survey responses yields better results. We have struggled to find a time and incentive that will attract students to our focus groups, so we will continue to experiment with those variables.

Do you have any new areas you are looking into as data or assessment projects this coming year?

Outcomes 2 -3 were new to our assessment this year, so we will continue to assess those and then re-evaluate for the 2021-2022 year. We are also considering ways to assess our contribution to students' "reflective capacity"--their ability to learn and grow from a mindful consideration of their past experiences. We regularly encourage students to keep a journal or to write about experiences that have shaped them. Because professional and graduate schools and fellowship foundations expect students to connect their past experiences to their future ambitions, we spend a lot of our advising time helping students to discover and articulate these connections—both as they are navigating their current experience and as they apply for their next opportunity. This year, we will develop a way to measure our influence in developing this skill, perhaps through a survey question or through a separate exit survey or Canvas module.