HSR's Alumni Engagement Blog

As I mentioned in my last blog post, we do not always obtain a three-factor solution when we analyze our alumni attitude rating items, and when we do obtain a three-factor solution, the same items do not always define the alumni engagement factors we obtain.  Some of these differences result from the fact that we sometimes use slightly different rating items when evaluating the attitudes of any single alumni community.  Schools differ in the academic, campus and alumni outreach experiences that they offer students and alumni, and these differences sometimes warrant that items be added to our survey.  This usually does not result in different factors being identified by our factor analyses.  Usually, the same factors are obtained, but one or two different items might define these factors. 

In other studies, these differences occur because different alumni samples respond to the same items differently.  The same rating items may have been used, but different alumni samples respond to these items differently when describing their attitudes or feelings of engagement.  As such, the factor structures we obtain are slightly different.  Still, the majority of the time, we obtain the same three factors; however, alumni may define these factors using slightly different items or behaviors to define these variables. 

While this discussion may sound somewhat academic to most advancement professionals, it is important because it directly addresses the objective measurement of alumni engagement.  Engagement is a construct that many alumni advancement professionals use to describe the feelings and behaviors of their alumni community.  However, there is little agreement regarding the specific behaviors or attitudes (psychological states) that engaged alumni can be expected to exhibit.  Do engaged alumni attend events?  Is giving the best indicator of alumni engagement?  What role do alumni communications play in maintaining an engaged alumni community?  What causes alumni to become engaged?  Are there any specific strategies that are successful at building more engaged alumni communities?  Understanding what is important to alumni when they talk about their attitudes toward their alma mater, will help us develop a more objective measure of alumni engagement.   

What I want to do now is present the results of the factor analyses we performed on a few of the Alumni Attitude Surveys we have conducted at different schools.  This will begin to give you an idea of the specific Likert-type rating items that we use to measure alumni engagement. 

Table 1 presents a three-factor solution we obtained on an alumni sample.  We identified three factors - quality of education, focus on students and sense of belonging.  However, some items behaved differently in how they related to factors.  For instance, the item - “I feel a strong sense of belonging to the XYZ community”, related more strongly to the first factor - “Quality of Education” than it did to the third factor - “Sense of Belonging”.  However, on the campus of this small liberal arts college, faculty not only played a major role in determining the quality of the education students received, but also were an integral part of campus life.  Alumni wrote about faculty being available for help outside the classroom, the mentoring and one-on-one instruction faculty provided students, the interest faculty showed in the success and well-being of individual students and the friendships students developed with faculty.  Students frequently interacted with faculty outside the classroom.  Faculty were actively involved in campus life and played very important roles in both the classroom and campus community.  They were an integral part of the campus community. This affected how alumni responded to individual rating items and how these items related to the three factors.

The second factor evaluated the school’s “focus on students”.  Some of the items that defined this factor contained content that addressed specific administrative functions or administrators at the College, as opposed to the more generic behavioral statements that we have used on subsequent surveys. This factor, “focus on students”, was defined largely by the quality of the interactions that students had with College administrators and staff. 

 In this study, the third factor – “sense of belonging” was only beginning to be defined.  It appears that alumni who were actively involved in campus life as students felt more a sense of belonging as alumni. 

Table 2 presents a one-factor solution that was obtained on a small, single-gender liberal arts college.  In this study, the purpose of the alumnae survey was to update alumni contact information and Raiser's Edge data records.  The collection of alumni attitude data was not the primary purpose of the survey and fewer items were used.  On this sample, a single-factor solution was obtained for our measure of engagement.  Thirteen items related to this single factor.  When you look at the content of the items that define this factor, you will note that this content describes things alumni can do and say in support of their alma mater, their satisfaction with the quality of the education they received and the sense of belonging alumni feel.  The obtained solution did not include items that evaluated the school’s focus on students, but many of these items had been left off the survey.

Tables 3 and 4 present three-factor solutions that were obtained on two samples from HSR’s National Alumni Engagement Study.  This was a national study in which we used several social media platforms and numerous list serves to collect alumni attitude survey data.  A single survey item asked respondents if they worked in alumni advancement, and we used this item to divide our sample into two groups – individuals who worked in alumni advancement and those who did not.  Table 3 presents the three-factor solution we obtained on alumni who did not work in advancement. 

Table 4 presents the three-factor solution we obtained on alumni who worked in advancement.  For both samples, three factors were obtained – quality of education, focus on students and sense of belonging.  There was some shifting in the ordering of items for the first two factors and minor differences in how many items related to individual factors.  However, the factor structure generally remained the same for these two different samples.  The third factor – sense of belonging, was only partially defined for both samples.  Only two items related to this factor.

Table 5 presents a three-factor solution that we obtained on alumni from a Career University.  This sample was somewhat different from the other samples on which we had obtained alumni attitude survey data.  While the school did offer traditional liberal arts programs and degrees, it was much better known for its culinary and hospitality programs.  More importantly, the University had never made a systematic effort to ask alumni for gifts and the school had no history of a consistent, systematic outreach to alumni.  The University did host large alumni galas at trade shows, however, these events were more focused on highlighting their programs and students to employers than they were directed at engaging alumni.  Still, a three-factor solution was obtained on this sample as well. 

These are the results from just a few of the alumni attitude survey studies we have conducted.  So far, it seems that three variable are important to alumni when then recall their student experience and talk about their attitudes toward their alma mater.  These variables include alumni perceptions of the quality of the education they received, the extent to which the school was focused on student success and well-being and the sense of belonging alumni feel.  However, much more research needs to be performed.  It is quite possible that additional variables relate to alumni attitudes, and additional items need to be included on our measure of alumni engagement.  We are continuously developing new items based on the comments alumni write on our attitude surveys or the discussions among alumni in the focus groups we conduct.

It is possible that the quality of alumni outreach, including alumni communication, alumni events, alumni services and the quality of alumni interactions with administrators and staff does affect alumni attitudes. However, to date, our analyses have not obtained this factor. In response to the open-ended questions we ask, alumni frequently write about how much they like being kept informed about classmates, the campus and upcoming alumni events.  They also write about the way they have been treated as alumni.  Younger alumni frequently write about the need for career services and jobs. 

Alumni use three variables, quality of education, focus on students and sense of belonging, to describe their attitudes toward their alma mater.  However, alumni engagement is also defined by a set of behaviors that all alumni can potentially perform. These behaviors typically involve saying and doing good things in support of their alma mater, such as recommending the school to family and friends, speaking well of their alma mater to others or telling people that they would do it all over again if they had the chance.  An alumnus's willingness to perform these behaviors is an indication of their engagement level.  To date, these items have loaded on the first factor, quality of education.  They have not formed a separate factor.  Perhaps, alumni only recommend and say good things about their alma mater when they feel they received a quality education. 

Identifying the variables and items that alumni use to describe their attitudes and feelings of engagement is a continuous process.  We are constantly evaluating the factor structure of our Alumni Engagement Scale (AES), as well as reviewing the comments alumni write to see if there are any additional items we need to include on our Alumni Attitudes Survey.  This is part of the AES’s ongoing validation process.

As part of this process, Harford Survey Research is currently conducting an Alumni Attitudes Study that is directed at the validation of our Alumni Engagement Scale (AES).  We are answering some of the questions that were raised above.  To encourage participation we are offering schools extraordinary discounts off the cost of our Alumni Attitudes Survey.  If you are considering conducting an Alumni Attitudes Survey in the near future, you should call or email us to find out just how inexpensive your next alumni survey can be.

Comments (0) | Trackbacks (0) | Permalink

Although alumni advancement professionals appreciate the importance of building engaged alumni communities, there is little understanding or agreement about what alumni engagement is or how to best measure this construct.  What do engaged alumni do, say or think about their alma mater that makes them engaged?  What are the behaviors we can expect engaged alumni to perform?  What causes alumni to become engaged?  What types of variables affect alumni attitudes toward their alma mater. How do differences in alumni engagement relate to the decision to give?  How do we objectively measure alumni engagement? 

I am frequently asked questions about alumni engagement - What is it? How is it measured?  What does it mean?  Schools frequently asked about our Alumni Attitudes Survey.  They ask to see the items we use and the questions we ask.  In the next few blog posts, I'll write about our approach to measuring alumni engagement, the variables that are important to alumni, the items we use and some of our findings.  I hope you find it interesting, but first let me talk about how I got started.

More than a decade ago, while serving on my alma mater’s alumni board, I volunteered to conduct my alma mater's very first alumni survey.  They were reviewing bids from vendors and asked my advice.  None looked like what the College needed, and having conducted many employee attitude surveys, I volunteered to perform the survey.  I was looking for something interesting and worthwhile to do as a board member. The alumni survey turned out to be a very worthwhile piece of volunteer work.

At the time, alumni surveys were not that common and Web-based alumni surveys were non-existent. A few schools had published the results of paper alumni surveys online.  We reviewed these items  and were able to gleam a small number of rating items .  There was not a lot of consistency in the types of items schools used. 

To develop additional items, we conducted interviews and focus groups with students, alumni, faculty and staff from the College.  The focus of these methods was to obtain specific behavioral examples of things alumni had experienced which affected their attitudes toward the College.  Our focus was obtaining specific, behavioral examples that related to student and alumni experiences.  Things that alumni had experienced or observed that had affected their attitudes toward the College; that had caused alumni to be more or less engaged.  Flanagan (1954) referred to these behavioral examples as critical incident statements.  We used these critical incident statements to develop additional Likert-type rating items that we would use to evaluate alumni attitudes and engagement. 

Since this initial study, we have continued to conduct interviews and focus groups to obtain specific behavioral examples of student and alumni experiences that affect their attitudes toward their alma mater.  Additionally, on every Alumni Attitudes Survey we conduct, we have used open-ended questions to obtain written feedback from alumni about what their alma mater does well, where improvements can be made, why alumni attend or do not attend events and what affects alumni decisions to give.  This written feedback has also provided us with the critical incident statements we need to continuously expand our pool of alumni attitude items.  When we evaluate alumni attitudes, we use approximately twenty-five Likert-type rating items to objectively measure alumni engagement.  We use factor analytic procedures to identify the variables that define alumni engagement and the individual rating items that relate to each variable.  Typically, three variables are identified and approximately 15 items are needed to define these variables.

The most significant work related to the measurement of engagement has been conducted in the field of industrial-organizational psychology. Macey and Schneider (2007) described employee engagement as a psychological state that is similar to job satisfaction, organizational commitment, psychological empowerment and job involvement, and noted that any measure of employee engagement likely includes some of the same items and evaluate some of the same behaviors found on these other measures. While the field of industrial-organizational psychology may be finding it difficult to differentiate employee engagement from these other measures of employee affect, this construct, engagement, appears to readily describe the types of variables that underlay alumni attitudes.  In developing our Alumni Engagement Scale (AES), we also reviewed these measures of employee affect to identify items that might relate to alumni feelings of engagement.

The relationship between alumni and their alma mater is distinctly different from the employer-employee relationships you find in business and industry, or the volunteer relationships you might find in not-for-profit organizations.  The alumni relationship is unique, but “engagement” appears to offer an accurate description of the variables that underlay alumni  attitudes.  The focus has to be identifying the specific behaviors that define alumni engagement - things that alumni think, do, say, feel or have experienced.

I'll write more about the measurement of alumni attitudes and engagement in my next blog.

Comments (0) | Trackbacks (0) | Permalink