HSR Alumni Engagement Blog

Measuring Alumni Engagement - Identifying the Variables that are Important to Alumni

Sun, 11 Mar 2012 17:03:00 GMT

As I mentioned in my last blog post, we do not always obtain a three-factor solution when we analyze our alumni attitude rating items, and when we do obtain a three-factor solution, the same items do not always define the alumni engagement factors we obtain. Some of these differences result from the fact that we sometimes use slightly different rating items when evaluating the attitudes of any single alumni community. Schools differ in the academic, campus and alumni outreach experiences that they offer students and alumni, and these differences sometimes warrant that items be added to our survey. This usually does not result in different factors being identified by our factor analyses. Usually, the same factors are obtained, but one or two different items might define these factors.

In other studies, these differences occur because different alumni samples respond to the same items differently. The same rating items may have been used, but different alumni samples respond to these items differently when describing their attitudes or feelings of engagement. As such, the factor structures we obtain are slightly different. Still, the majority of the time, we obtain the same three factors; however, alumni may define these factors using slightly different items or behaviors to define these variables.

While this discussion may sound somewhat academic to most advancement professionals, it is important because it directly addresses the objective measurement of alumni engagement. Engagement is a construct that many alumni advancement professionals use to describe the feelings and behaviors of their alumni community. However, there is little agreement regarding the specific behaviors or attitudes (psychological states) that engaged alumni can be expected to exhibit. Do engaged alumni attend events? Is giving the best indicator of alumni engagement? What role do alumni communications play in maintaining an engaged alumni community? What causes alumni to become engaged? Are there any specific strategies that are successful at building more engaged alumni communities? Understanding what is important to alumni when they talk about their attitudes toward their alma mater, will help us develop a more objective measure of alumni engagement.

What I want to do now is present the results of the factor analyses we performed on a few of the Alumni Attitude Surveys we have conducted at different schools. This will begin to give you an idea of the specific Likert-type rating items that we use to measure alumni engagement.

Table 1 presents a three-factor solution we obtained on an alumni sample. We identified three factors - quality of education, focus on students and sense of belonging. However, some items behaved differently in how they related to factors. For instance, the item - “I feel a strong sense of belonging to the XYZ community”, related more strongly to the first factor - “Quality of Education” than it did to the third factor - “Sense of Belonging”. However, on the campus of this small liberal arts college, faculty not only played a major role in determining the quality of the education students received, but also were an integral part of campus life. Alumni wrote about faculty being available for help outside the classroom, the mentoring and one-on-one instruction faculty provided students, the interest faculty showed in the success and well-being of individual students and the friendships students developed with faculty. Students frequently interacted with faculty outside the classroom. Faculty were actively involved in campus life and played very important roles in both the classroom and campus community. They were an integral part of the campus community. This affected how alumni responded to individual rating items and how these items related to the three factors.

The second factor evaluated the school’s “focus on students”. Some of the items that defined this factor contained content that addressed specific administrative functions or administrators at the College, as opposed to the more generic behavioral statements that we have used on subsequent surveys. This factor, “focus on students”, was defined largely by the quality of the interactions that students had with College administrators and staff.

In this study, the third factor – “sense of belonging” was only beginning to be defined. It appears that alumni who were actively involved in campus life as students felt more a sense of belonging as alumni.

Table 2 presents a one-factor solution that was obtained on a small, single-gender liberal arts college. In this study, the purpose of the alumnae survey was to update alumni contact information and Raiser's Edge data records. The collection of alumni attitude data was not the primary purpose of the survey and fewer items were used. On this sample, a single-factor solution was obtained for our measure of engagement. Thirteen items related to this single factor. When you look at the content of the items that define this factor, you will note that this content describes things alumni can do and say in support of their alma mater, their satisfaction with the quality of the education they received and the sense of belonging alumni feel. The obtained solution did not include items that evaluated the school’s focus on students, but many of these items had been left off the survey.

Tables 3 and 4 present three-factor solutions that were obtained on two samples from HSR’s National Alumni Engagement Study. This was a national study in which we used several social media platforms and numerous list serves to collect alumni attitude survey data. A single survey item asked respondents if they worked in alumni advancement, and we used this item to divide our sample into two groups – individuals who worked in alumni advancement and those who did not. Table 3 presents the three-factor solution we obtained on alumni who did not work in advancement.

Table 4 presents the three-factor solution we obtained on alumni who worked in advancement. For both samples, three factors were obtained – quality of education, focus on students and sense of belonging. There was some shifting in the ordering of items for the first two factors and minor differences in how many items related to individual factors. However, the factor structure generally remained the same for these two different samples. The third factor – sense of belonging, was only partially defined for both samples. Only two items related to this factor.

Table 5 presents a three-factor solution that we obtained on alumni from a Career University. This sample was somewhat different from the other samples on which we had obtained alumni attitude survey data. While the school did offer traditional liberal arts programs and degrees, it was much better known for its culinary and hospitality programs. More importantly, the University had never made a systematic effort to ask alumni for gifts and the school had no history of a consistent, systematic outreach to alumni. The University did host large alumni galas at trade shows, however, these events were more focused on highlighting their programs and students to employers than they were directed at engaging alumni. Still, a three-factor solution was obtained on this sample as well.

These are the results from just a few of the alumni attitude survey studies we have conducted. So far, it seems that three variable are important to alumni when then recall their student experience and talk about their attitudes toward their alma mater. These variables include alumni perceptions of the quality of the education they received, the extent to which the school was focused on student success and well-being and the sense of belonging alumni feel. However, much more research needs to be performed. It is quite possible that additional variables relate to alumni attitudes, and additional items need to be included on our measure of alumni engagement. We are continuously developing new items based on the comments alumni write on our attitude surveys or the discussions among alumni in the focus groups we conduct.

It is possible that the quality of alumni outreach, including alumni communication, alumni events, alumni services and the quality of alumni interactions with administrators and staff does affect alumni attitudes. However, to date, our analyses have not obtained this factor. In response to the open-ended questions we ask, alumni frequently write about how much they like being kept informed about classmates, the campus and upcoming alumni events. They also write about the way they have been treated as alumni. Younger alumni frequently write about the need for career services and jobs.

Alumni use three variables, quality of education, focus on students and sense of belonging, to describe their attitudes toward their alma mater. However, alumni engagement is also defined by a set of behaviors that all alumni can potentially perform. These behaviors typically involve saying and doing good things in support of their alma mater, such as recommending the school to family and friends, speaking well of their alma mater to others or telling people that they would do it all over again if they had the chance. An alumnus's willingness to perform these behaviors is an indication of their engagement level. To date, these items have loaded on the first factor, quality of education. They have not formed a separate factor. Perhaps, alumni only recommend and say good things about their alma mater when they feel they received a quality education.

Identifying the variables and items that alumni use to describe their attitudes and feelings of engagement is a continuous process. We are constantly evaluating the factor structure of our Alumni Engagement Scale (AES), as well as reviewing the comments alumni write to see if there are any additional items we need to include on our Alumni Attitudes Survey. This is part of the AES’s ongoing validation process.

As part of this process, Harford Survey Research is currently conducting an Alumni Attitudes Study that is directed at the validation of our Alumni Engagement Scale (AES). We are answering some of the questions that were raised above. To encourage participation we are offering schools extraordinary discounts off the cost of our Alumni Attitudes Survey. If you are considering conducting an Alumni Attitudes Survey in the near future, you should call or email us to find out just how inexpensive your next alumni survey can be.

Measuring Alumni Engagement – Survey Questions and Items

Sun, 11 Mar 2012 17:06:00 GMT

We are often asked by alumni advancement professionals if we can share our Alumni Attitudes Survey. They want to see the individual items we use and the questions we ask. Today, I will do a little sharing. I want to talk about our approach to evaluating alumni attitudes and measuring alumni engagement. I will discuss HSR's Alumni Attitudes Survey, the Alumni Engagement Scale (AES), the questions we ask, the items we use and how we identify the variables that are most important to alumni.

Harford Survey Research’s approach to the measurement of alumni engagement is quite similar to the methods advocated by Macey and Schneider (2007). We use Likert-type rating items to evaluate the behaviors and attitudes (psychological states) that engaged alumni can be expected to exhibit. The rating items evaluate things that engaged alumni can be expected to do, say, think, believe or feel. They evaluate both the behaviors and psychological states that relate to feelings of engagement. The rating items we use are based on items we gleamed from measures of employee engagement (where there are many similarities in how engagement is defined); the behavioral examples we obtained from the interviews and focus groups we have conducted with alumni, faculty and staff and the behavioral examples we have obtained from the comments alumni write in response to the open-ended survey questions we have asked. We have developed a comprehensive pool of Likert-type rating items that evaluate alumni engagement.

Our Alumni Attitudes Survey also includes a set of open-ended questions that obtain written feedback about alumni educational and campus experiences, their attitudes toward the school and their attitudes toward giving. These open-ended questions are a very good source of information for developing additional Likert-type rating items. As we collect survey data on different types of schools, we are continuously analyzing the comments alumni write to insure that our Alumni Attitudes Survey is comprehensive in the behaviors it evaluates. Some of the questions we ask on our Alumni Attitudes Survey include:

What does the school do well?
Where can improvements be made?
Why do alumni attend or not attend events?
Why do alumni make or not make financial gifts to their alma mater?
How can your alma mater improve its alumni outreach?

We frequently obtain a considerable amount of information from the comments alumni write in response to these open-ended questions. These comments oftentimes relate to the variables measured by our alumni engagement scale, the AES. However, because academic institutions differ in many important ways, we are continuously reviewing these comments to determine if our rating items are comprehensive in the behaviors and experiences they evaluate. Our open-ended questions are a rich source of information that provides schools with additional insight into alumni attitudes and provide us with the behavioral examples we need to evaluate the validity of the AES.

The validation of the AES is an ongoing process. We are continuously evaluating its validity and testing the generalizability of its factor structure across different academic settings. Given the many differences that exist between schools, it is highly unlikely that all alumni will use the same items to describe their attitudes and feelings of engagement. There is a core set of items that are applicable to most schools, but alumni from very different types of schools sometimes focus on behaviors and campus experiences that are different from the majority schools. For instance, alumni from large public universities, small liberal arts colleges, single-gender schools, HBCUs, the Naval Academy or schools that are affiliated with a specific religious group are likely to have a subset of behaviors or experiences that are specific to their alumni community and affect alumni feelings of engagement. As differences between schools become greater, it is very likely that there will be a few additional behaviors and experiences that will relate to feelings of engagement. We are continuously evaluating the validity and comprehensiveness of the rating items we use.

Harford Survey Research has developed a large pool of Likert-type rating items that we use to evaluate alumni attitudes and engagement. These items generally give us a very accurate picture of alumni attitudes and engagement levels at different schools. When schools are unique or very different from the institutions in which we have already collected data, we will conduct interviews or focus groups to develop new items that are specific to these academic institutions and alumni communities. We make every effort to insure that our Alumni Attitudes Survey evaluates the behaviors and experiences that relate to feelings of engagement at the different schools with whom we work.

Our Alumni Attitudes Survey uses approximately twenty-five Likert-type rating items to evaluate alumni attitudes and engagement. However, when we conduct focus groups at a school, there is always a chance that a few additional items will be added to our measure of engagement. After collecting our survey data, we use factor analyzes to identify the variables that define alumni engagement for the alumni community we are surveying, as well as identify the rating items that relate to these individual variables. We compute Chronbach Alphas to evaluate the inter-item reliability of our scales, and we use simple multiple regression analyzes to evaluate the strength of the relationships between our measure of alumni engagement and alumni giving. We obtain from schools the giving data on survey respondents for the last five years. We compute the percentage of the past five years that an alum made a gift, and we use this participation rate as our outcome measure of alumni giving. The AES is an excellent predictor of alumni giving.

Harford Survey Research is continuously evaluating the validity of the AES. Very often a three factor solution is obtained with factor analyzes identifying three variables that affect alumni attitudes - the quality of the education they received, the extent to which the school was focused on students and the “sense of belonging” alumni feel. Fifteen to twenty Likert-type rating items typically define these three variables. (See Table 3.)

We do not always obtain a three-factor solution on all alumni samples. Schools can be different in the student experiences they offer alumni, as well as the amount of alumni outreach they are able to perform. On a few alumni samples we have obtained a one-factor solution for our measure of alumni engagement. This means that a single variable defines engagement. However, when we have obtained a one-factor solution, many of the same rating items that were identified by the three-factor solution were also retained by the one-factor solution. However, fewer items were retained. Approximately ten items, as opposed to twenty items, are typically retained when we obtain a single-factor measure of alumni engagement. Here again, HSR is continuously looking at how these factor structures vary across different academic settings. This is the focus of our current Alumni Attitudes Study.

Table 3 (above) presents the results of the factor analyzes we performed on a set of items that were used to evaluate alumni attitudes at one University. A three-factor solution was obtained. The variables that were identified included quality of education; the extent to which the University was focused on student well-being and success; and the sense of belonging alumni feel. The first factor or variable was defined by six items and had a reliability of.92. (I have highlighted in bold the coefficients in the first column of coefficients that relate to this variable.) The first factor evaluated alumni perceptions of the quality of the education they received, as well as the extent to which they would say and do good things for their alma mater, such as, recommending the school to family and relatives. The second factor or variable, focus on students, was defined by four items and had a reliability of .80. It evaluated alumni perceptions of faculty accessibility, how well students were treated and if they received the guidance and attention they needed to succeed. The third factor evaluated the extent to which alumni feel a sense of belonging toward their alma mater – if they were actively involved in campus life, felt a part of the campus community and look forward to seeing classmates and friends. When alumni talk about their attitudes toward their alma mater, these three variables are the most important to alumni. They represent the lens through which alumni recall their academic and campus experiences. Table 3 provides a sample of the items we use to measure alumni engagement. For this sample of university alumni (approximately 3,000 alumni from a single school)., these were variables and items that were important to alumni when they talked about their attitudes toward their alma mater. These were also the items we used to score the engagement levels of individual survey respondents.

Harford Survey Research has conducted numerous alumni attitude studies, and as already stated, the items and factor structures reported in Figure 3 are not always obtained. Sometimes different items are included on the survey because of the differences in the academic institutions with whom we are working, and this can result in different items being retained or different factor structures being obtained. We would likely not obtain the same results, if these data were being obtained on alumni from a school that primarily uses distance learning or computer-aided instruction as its primary method of teaching. Similarly, there may be differences in how alumni from Ivy League schools, HBCUs, single-gender schools and religiously affiliated schools define engagement. These are all questions being addressed by HSR's systematic program of alumni engagement research. However, although the individual items and factors we obtain may sometimes vary, the variables that are important to alumni have remained relatively constant across the majority of schools we have studied.

The items listed in figure 3 offer a set of Likert-type rating items that schools can use to measure alumni engagement. We use a 5- or 7-point rating scale to evaluate the degree to which alumni agree or disagree with each statement or item. If you decide to use these items, we recommend that you conduct a few focus groups with alumni to insure that you are measuring all the behaviors and experiences that are important to your alumni. If you can, use the appropriate statistical procedures to identify the variables and items that define engagement for your alumni community. We validate our scales for every sample of alumni on which we collect survey data.

Finally, if you are considering conducting an alumni attitudes survey in the near future, you should contact Harford Survey Research. We are preparing to conduct a national Alumni Attitudes Study directed at the validation of HSR’s Alumni Engagement Scale (the AES). To encourage schools to participate in our Alumni Attitudes Study we are offering extraordinary discounts off the cost of our Alumni Attitudes Survey. If you plan to conduct an alumni attitudes survey, this is an excellent opportunity to have this work done professionally and at a cost that all schools can afford. We have provided a brief description of our Alumni Attitudes Study for schools who would like to participate. Again, this is an extraordinary opportunity for schools to conduct an alumni attitudes survey at a cost all schools can afford.

I will talk more about our alumni engagement research in later blogs.

Measuring Alumni Engagement – Models and Theoretical Frameworks

Sun, 11 Mar 2012 17:01:00 GMT

Today, I want to talk about the theoretical models that guide our research related to the measurement of alumni attitudes and engagement. Macey and Schneider (2007) discussed the need for engagement research to objectively measure the psychological states and behaviors implied by this construct. They also underscored the need for engagement research to be able to identify and describe the antecedents, moderators and consequences of engagement. Macey and Schneider were talking about employee engagement, but the implications for the measurement of alumni engagement are pretty clear. Research must be able to describe the things that engaged alumni do, say, think and feel. They must be able to identify the variables that cause alumni to become engaged, as well as identify those variables that moderate alumni feelings of engagement. Any comprehensive model of engagement must be able to objectively describe the antecedents, moderators and consequences of alumni engagement.

So what are the behaviors that engaged alumni perform? We can start building this list by first looking at the things that schools want alumni to do - attend events, visit campus, stay in touch, say good things about their alma mater and do good things for the school. Schools want their alumni to value the education they received, think highly of the degree they received and participate in giving. These are all behaviors that engaged alumni can be expected to perform. They provide a basis for the development of Likert-type rating items that evaluate behaviors which engaged alumni perform.

Feelings of engagement also relate to what alumni experienced as students - the quality of classroom instruction, the availability of faculty, the internships and career services that were offered and how students were treated by administrators, staff and the school. When alumni talk about their student experience, they frequently talk about the friendships they made, the favorite faculty who were both mentors and friends and the student organizations, clubs and athletic teams to which they belonged.. All of these experiences relate to alumni feelings of engagement and provide much content for item development.

Alumni attitudes and feelings of engagement are also affected by what alumni have experienced since graduation - the quality of alumni interactions with the school, their satisfaction with alumni outreach programs and what they perceive to be the value of their degree. Schools will differ in terms of the campus experiences they can provide, the amount of alumni outreach they perform, what they ask alumni to do and how their degree is valued by employers. All of this has an effect on alumni feelings of engagement and play important roles in the measurement of alumni engagement.

Figure 1 (above) offers a graphical representation of some of the behaviors we think are characteristic of many alumni communities. It is not comprehensive in the behaviors it describes, but it does provide a working model to begin talking about alumni engagement and the behaviors that relate to this construct. As depicted in figure 1, most alumni communities have a small group of not satisfied alumni. Sometimes this is because of something the school did; other times, it is just due to individual differences – values, beliefs or personalities that are specific to the individual alum. Many schools have a relatively large number of alumni who hold generally positive attitudes toward their alma mater, but do not attend events or participate in giving. A smaller group might be a bit more involved, and it is typically a much smaller group that regularly attends events and participates in giving.

Schools expend a considerable amount of time and resources encouraging alumni to attend events, participate in programs and make financial gifts to their alma mater. Alumni boards, leaders and volunteers play key roles in building an engaged alumni community. Through visits, events and continuous campus updates, schools strive to engage increasingly larger numbers of alumni. They recruit alumni leaders from their more engaged alumni, and use this team of alumni leaders to accomplish advancement objectives, engage alumni and foster giving. The focus is developing an engaged alumni community that supports its alma mater with gifts.

Figure 1 offers a theoretical framework for talking about the measurement of alumni engagement and designing strategies for building more engaged alumni communities. Any measure of alumni engagement should be sensitive to the behaviors that engaged alumni are likely to perform, the things that engaged alumni are likely to do, say, think or feel about their alma mater. It should include the variables that are important to alumni when they talk about their alma mater. Figure 1 provides a theoretical framework for thinking about what these behaviors might be. These behaviors will likely comprise any objective measure of alumni engagement.

Macey and Schneider (2007) also wrote about the need for any model of engagement to address the antecedents, moderators and consequences of engagement – the engagement process. Figure 2 (above) describes the variables that our research has shown to relate alumni engagement - the antecedents, moderators and consequences of alumni engagement. Student experiences appear to be major determinants of alumni attitudes and engagement. The variables that appear to be most important to alumni when they talk about their student experience include the quality of education alumni received, the extent to which the school was focused on student success and well-being and the degree to which alumni feel a sense of belonging to their alma mater. Institutional values and campus community affect the quality of the student experience, as well as moderate alumni attitudes and feelings of engagement. Alumni outreach programs are designed to engage alumni and enhance the alumni community. Alumni engagement is affected by the student experience, campus and alumni communities, alumni outreach programs and institutional values.

As shown in figure 2, alumni engagement levels have a direct affect on alumni decisions to make a gift . Engaged alumni are much more likely to make gifts to their alma mater than are alumni who are not engaged or have less than positive attitudes toward their alma mater. One of the most basic findings of our research is that satisfied students are much more likely to become engaged alumni, and engaged alumni are much more likely to make financial gifts to their alma mater.

We use the models presented in figures 1 and 2 to talk about alumni engagement, our Alumni Attitudes Survey, survey results, giving and the strategies we use to build more engaged alumni communities and increase giving. They also provide a good theoretical framework for thinking about the measurement of engagement. In my next blog, I will share some of the Likert-type rating items we use to measure engagement, and talk about the methods we use to determine the variables and items that are important to alumni.

Measuring Alumni Engagement - Item Development

Sun, 11 Mar 2012 17:03:00 GMT

Although alumni advancement professionals appreciate the importance of building engaged alumni communities, there is little understanding or agreement about what alumni engagement is or how to best measure this construct. What do engaged alumni do, say or think about their alma mater that makes them engaged? What are the behaviors we can expect engaged alumni to perform? What causes alumni to become engaged? What types of variables affect alumni attitudes toward their alma mater. How do differences in alumni engagement relate to the decision to give? How do we objectively measure alumni engagement?

I am frequently asked questions about alumni engagement - What is it? How is it measured? What does it mean? Schools frequently asked about our Alumni Attitudes Survey. They ask to see the items we use and the questions we ask. In the next few blog posts, I'll write about our approach to measuring alumni engagement, the variables that are important to alumni, the items we use and some of our findings. I hope you find it interesting, but first let me talk about how I got started.

More than a decade ago, while serving on my alma mater’s alumni board, I volunteered to conduct my alma mater's very first alumni survey. They were reviewing bids from vendors and asked my advice. None looked like what the College needed, and having conducted many employee attitude surveys, I volunteered to perform the survey. I was looking for something interesting and worthwhile to do as a board member. The alumni survey turned out to be a very worthwhile piece of volunteer work.

At the time, alumni surveys were not that common and Web-based alumni surveys were non-existent. A few schools had published the results of paper alumni surveys online. We reviewed these items and were able to gleam a small number of rating items . There was not a lot of consistency in the types of items schools used.

To develop additional items, we conducted interviews and focus groups with students, alumni, faculty and staff from the College. The focus of these methods was to obtain specific behavioral examples of things alumni had experienced which affected their attitudes toward the College. Our focus was obtaining specific, behavioral examples that related to student and alumni experiences. Things that alumni had experienced or observed that had affected their attitudes toward the College; that had caused alumni to be more or less engaged. Flanagan (1954) referred to these behavioral examples as critical incident statements. We used these critical incident statements to develop additional Likert-type rating items that we would use to evaluate alumni attitudes and engagement.

Since this initial study, we have continued to conduct interviews and focus groups to obtain specific behavioral examples of student and alumni experiences that affect their attitudes toward their alma mater. Additionally, on every Alumni Attitudes Survey we conduct, we have used open-ended questions to obtain written feedback from alumni about what their alma mater does well, where improvements can be made, why alumni attend or do not attend events and what affects alumni decisions to give. This written feedback has also provided us with the critical incident statements we need to continuously expand our pool of alumni attitude items. When we evaluate alumni attitudes, we use approximately twenty-five Likert-type rating items to objectively measure alumni engagement. We use factor analytic procedures to identify the variables that define alumni engagement and the individual rating items that relate to each variable. Typically, three variables are identified and approximately 15 items are needed to define these variables.

The most significant work related to the measurement of engagement has been conducted in the field of industrial-organizational psychology. Macey and Schneider (2007) described employee engagement as a psychological state that is similar to job satisfaction, organizational commitment, psychological empowerment and job involvement, and noted that any measure of employee engagement likely includes some of the same items and evaluate some of the same behaviors found on these other measures. While the field of industrial-organizational psychology may be finding it difficult to differentiate employee engagement from these other measures of employee affect, this construct, engagement, appears to readily describe the types of variables that underlay alumni attitudes. In developing our Alumni Engagement Scale (AES), we also reviewed these measures of employee affect to identify items that might relate to alumni feelings of engagement.

The relationship between alumni and their alma mater is distinctly different from the employer-employee relationships you find in business and industry, or the volunteer relationships you might find in not-for-profit organizations. The alumni relationship is unique, but “engagement” appears to offer an accurate description of the variables that underlay alumni attitudes. The focus has to be identifying the specific behaviors that define alumni engagement - things that alumni think, do, say, feel or have experienced.

I'll write more about the measurement of alumni attitudes and engagement in my next blog.