Harford Survey Research
Engage

HSR's Alumni Engagement Blog

Jim Flynn, Ph.D.Engage is HSR’s Alumni Engagement Blog. Dr. Flynn writes about the measurement of alumni attitudes and engagement, best practices for conducting alumni surveys, alumni survey results and strategies for building more engaged alumni communities and increasing alumni giving. If you want to think more strategically about your alumni outreach programs, follow our blog. We want to hear from you, so please feel free to comment on what we write.

Measuring Alumni Engagement – Survey Questions and Items

Wednesday, December 07, 2011

We are often asked by alumni advancement professionals if we can share our Alumni Attitudes Survey. They want to see the individual items we use and the questions we ask.  Today, I will do a little sharing.  I want to talk about our approach to evaluating alumni attitudes and measuring alumni engagement.  I will discuss HSR's Alumni Attitudes Survey, the Alumni Engagement Scale (AES), the questions we ask, the items we use and how we identify the variables that are most important to alumni.

Harford Survey Research’s approach to the measurement of alumni engagement is quite similar to the methods advocated by Macey and Schneider (2007). We use Likert-type rating items to evaluate the behaviors and attitudes (psychological states) that engaged alumni can be expected to exhibit. The rating items evaluate things that engaged alumni can be expected to do, say, think, believe or feel. They evaluate both the behaviors and psychological states that relate to feelings of engagement. The rating items we use are based on items we gleamed from measures of employee engagement (where there are many similarities in how engagement is defined); the behavioral examples we obtained from the interviews and focus groups we have conducted with alumni, faculty and staff and the behavioral examples we have obtained from the comments alumni write in response to the open-ended survey questions we have asked. We have developed a comprehensive pool of Likert-type rating items that evaluate alumni engagement.

Our Alumni Attitudes Survey also includes a set of open-ended questions that obtain written feedback about alumni educational and campus experiences, their attitudes toward the school and their attitudes toward giving. These open-ended questions are a very good source of information for developing additional Likert-type rating items. As we collect survey data on different types of schools, we are continuously analyzing the comments alumni write to insure that our Alumni Attitudes Survey is comprehensive in the behaviors it evaluates. Some of the questions we ask on our Alumni Attitudes Survey include:  

  • What does the school do well? 
  • Where can improvements be made? 
  • Why do alumni attend or not attend events? 
  • Why do alumni make or not make financial gifts to their alma mater?
  • How can your alma mater improve its alumni outreach?

We frequently obtain a considerable amount of information from the comments alumni write in response to these open-ended questions. These comments oftentimes relate to the variables measured by our alumni engagement scale, the AES. However, because academic institutions differ in many important ways, we are continuously reviewing these comments to determine if our rating items are comprehensive in the behaviors and experiences they evaluate. Our open-ended questions are a rich source of information that provides schools with additional insight into alumni attitudes and provide us with the behavioral examples we need to evaluate the validity of the AES. 

The validation of the AES is an ongoing process. We are continuously evaluating its validity and testing the generalizability of its factor structure across different academic settings.  Given the many differences that exist between schools, it is highly unlikely that all alumni will use the same items to describe their attitudes and feelings of engagement. There is a core set of items that are applicable to most schools, but alumni from very different types of schools sometimes focus on behaviors and campus experiences that are different from the majority schools.  For instance, alumni from large public universities, small liberal arts colleges, single-gender schools, HBCUs, the Naval Academy or schools that are affiliated with a specific religious group are likely to have a subset of behaviors or experiences that are specific to their alumni community and affect alumni feelings of engagement.   As differences between schools become greater, it is very likely that there will be a few additional behaviors and experiences that will relate to feelings of engagement.  We are continuously evaluating the validity and comprehensiveness of the rating items we use.

Harford Survey Research has developed a large pool of Likert-type rating items that we use to evaluate alumni attitudes and engagement.  These items generally give us a very accurate picture of alumni attitudes and engagement levels at different schools. When schools are unique or very different from the institutions in which we have already collected data, we will conduct interviews or focus groups to develop new items that are specific to these academic institutions and alumni communities. We make every effort to insure that our Alumni Attitudes Survey evaluates the behaviors and experiences that relate to feelings of engagement at the different schools with whom we work.

Our Alumni Attitudes Survey uses approximately twenty-five Likert-type rating items to evaluate alumni attitudes and engagement. However, when we conduct focus groups at a school, there is always a chance that a few additional items will be  added to our measure of engagement. After collecting our survey data, we use factor analyzes to identify the variables that define alumni engagement for the alumni community we are surveying, as well as identify the rating items that relate to these individual variables. We compute Chronbach Alphas to evaluate the inter-item reliability of our scales, and we use simple multiple regression analyzes to evaluate the strength of the relationships between our measure of alumni engagement and alumni giving.  We obtain from schools the giving data on survey respondents for the last five years.  We compute the percentage of the past five years that an alum made a gift, and we use this participation rate as our outcome measure of alumni giving. The AES is an excellent predictor of alumni giving.

Harford Survey Research is continuously evaluating the validity of the AES.  Very often a three factor solution is obtained  with factor analyzes identifying three variables that affect alumni attitudes - the quality of the education they received, the extent to which the school was focused on students and the “sense of belonging” alumni feel.  Fifteen to twenty Likert-type rating items typically define these three variables. (See Table 3.)

We do not always obtain a three-factor solution on all alumni samples.  Schools can be different in the student experiences they offer alumni, as well as the amount of alumni outreach they are able to perform.  On a few alumni samples we have obtained a one-factor solution for our measure of alumni engagement.  This means that a single variable defines engagement.  However, when we have obtained a one-factor solution, many of the same rating items that were identified by the three-factor solution were also retained by the one-factor solution.  However, fewer items were retained.  Approximately ten items, as opposed to twenty items, are typically retained when we obtain a single-factor measure of alumni engagement.  Here again, HSR is continuously looking at how these factor structures vary across different academic settings.  This is the focus of our current Alumni Attitudes Study.

 

 

 

Table 3 (above) presents the results of the factor analyzes we performed on a set of items that were used to evaluate alumni attitudes at one University. A three-factor solution was obtained. The variables that were identified included quality of education; the extent to which the University was focused on student well-being and success; and the sense of belonging alumni feel. The first factor or variable was defined by six items and had a reliability of.92. (I have highlighted in bold the coefficients in the first column of coefficients that relate to this variable.) The first factor evaluated alumni perceptions of the quality of the education they received, as well as the extent to which they would say and do good things for their alma mater, such as, recommending the school to family and relatives. The second factor or variable, focus on students, was defined by four items and had a reliability of .80. It evaluated alumni perceptions of faculty accessibility, how well students were treated and if they received the guidance and attention they needed to succeed. The third factor evaluated the extent to which alumni feel a sense of belonging toward their alma mater – if they were actively involved in campus life, felt a part of the campus community and look forward to seeing classmates and friends. When alumni talk about their attitudes toward their alma mater, these three variables are the most important to alumni. They represent the lens through which alumni recall their academic and campus experiences. Table 3 provides a sample of the items we use to measure alumni engagement.  For this sample of university alumni (approximately 3,000 alumni from a single school)., these were variables and items that were important to alumni when they talked about their attitudes toward their alma mater. These were also the items we used to score the engagement levels of individual survey respondents.

Harford Survey Research has conducted numerous alumni attitude studies, and as already stated, the items and factor structures reported in Figure 3 are not always obtained. Sometimes different items are included on the survey because of the differences in the academic institutions with whom we are working, and this can result in different items being retained or different factor structures being obtained.  We would likely not obtain the same results, if these data were being obtained on alumni from a school that primarily uses distance learning or computer-aided instruction as its primary method of teaching. Similarly, there may be differences in how alumni from Ivy League schools, HBCUs, single-gender schools and religiously affiliated schools define engagement.  These are all questions being addressed by HSR's systematic program of alumni engagement research. However, although the individual items and factors we obtain may sometimes vary, the variables that are important to alumni have remained relatively constant across the majority of schools we have studied. 

The items listed in figure 3 offer a set of Likert-type rating items that schools can use to measure alumni engagement. We use a 5- or 7-point rating scale to evaluate the degree to which alumni agree or disagree with each statement or item. If you decide to use these items, we recommend that you conduct a few focus groups with alumni to insure that you are measuring all the behaviors and experiences that are important to your alumni. If you can, use the appropriate statistical procedures to identify the variables and items that define engagement for your alumni community.  We validate our scales for every sample of alumni on which we collect survey data. 

Finally, if you are considering conducting an alumni attitudes survey in the near future, you should contact Harford Survey Research.  We are preparing to conduct a national Alumni Attitudes Study directed at the validation of HSR’s Alumni Engagement Scale (the AES). To encourage schools to participate in our Alumni Attitudes Study we are offering extraordinary discounts off the cost of our Alumni Attitudes Survey. If you plan to conduct an alumni attitudes survey, this is an excellent opportunity to have this work done professionally and at a cost that all schools can afford. We have provided a brief description of our Alumni Attitudes Study for schools who would like to participate. Again, this is an extraordinary opportunity for schools to conduct an alumni attitudes survey at a cost all schools can afford.

I will talk more about our alumni engagement research in later blogs.

 


Follow Us!

RSSHSR Alumni Engagement Blog


Recent Posts


Tags


Archive

Need Help Getting Started?

Jim Flynn, Ph.D is available to speak at your school.

Alumni Boards, Advancement Professionals, Administrators, Faculty & Staff

 View More





SIGN UP FOR OUR NEWSLETTER





What Our Clients Say:
I highly recommend HSR’s alumni survey services. Dr. Flynn has extensive expertise in designing surveys, advising on strategies to encourage high participation rates and interpreting the data to advance the strategic objectives of the overall alumni program.

He also is able to clearly articulate the results to a variety of audiences. He has assisted staff in mining data to insure it is used effectively and accurately and in accordance with the program’s strategic objectives. He has also prepared and delivered presentations to the survey base so as to enhance engagement objectives and to university leaders to deepen their understanding of complex and difficult issues. This breadth of expertise has allowed us to not only conduct an accurate survey but also to immediately begin to utilize the data in a manner that has advanced our program.

Patricia McLaughlin, J.D.
Senior Vice President of Institutional Advancement
Johnson & Wales University
Providence, Rhode Island 02903