Tips: Increasing survey response rates

Getting a high survey response rate is always a concern when administering a survey.  A high response rate will provide an accurate picture of the target population and allow you to make meaningful conclusions.

The response rate is calculated by the total number of complete returned surveys (or 80% or more) divided by the total number of participants you contacted. So if you asked 20 graduating students to complete an exit survey and 15 completed the survey, the response rate is 75%. There are several strategies to increase the response rates:

• Communicate the survey purpose, value, and how results will be used.
   
• Give sufficient time to complete the survey (online survey: 7-14 days)

• Make sure the survey is short, clear, logical, and easy to follow. Pilot-test the survey, so the items and instructions are written in understandable manner to the potential respondents.

• Send out reminders (thank the respondents, show how many responded, include a survey link, and remind respondents about the deadline)

• Use the existing opportunities to gather the target respondents (e.g., administer the survey in class, staff meeting, etc.)

• Offer an incentive (e.g., gift certificate, etc.)  

• Consider the best timing of your survey, so you are not administering the survey when the respondents are busy.  

Other resources: 

• AEA365: "Maximizing response rates" by Jessica Foster
• Division of Instructional Innovation and Assessment, The University of Texas at Austin: “Response Rates.” 
• Teaching and Learning Services, McGill University: Strategies to increase online response rate (for course evaluation)
• Dillman, D. A. (2000). Mail and Internet surveys: The total design method (2nd ed.). New York: Wiley.

Heiland & Rosenthal (Eds.). (2011). Literary Study, Measurement, and the Sublime: Disciplinary Assessment

The Teagle Foundation has announced a free online book on outcomes assessment in literary studies. This edited volume by Heiland and Rosenthal brings together literary scholars, foreign language and English department faculty, and assessment experts to provide disciplinary perspectives to outcomes assessment.

Those of you who are engaged in humanities and liberal arts programs will find the book enlightening and informative. The book, consisting of 19 chapters, responds to questions, such as:

• "How do we accurately depict and assess humanities outcomes that are often perceived as sublime and ineffable?" 
• "How can we localize assessment within the discipline?" 
• "How do we ensure ownership of assessment and make assessment a collaborative and useful process?"

You can access the book for free [here]

In concordance with the publication of the book, a "National Symposium on Assessment in the Humanities" was held at Miami University on February 23rd and 24th, 2011. The papers presented at the symposium is scheduled to be available online soon (according to the website). Stay tuned!

Evaluation event in Japan

February was an exciting month!

One of the FLPEP staff (Yukiko Watanabe) was invited to International Christian University (ICU) in Tokyo, Japan as a visiting scholar for one week to deliver program evaluation workshops and presentations for language programs, writing center, and graduate seminars. Interested in what she presented and what she learned? Read further!

-----------------------------------------
Many ICU faculty and graduate students were enthusiastic to learn about program evaluation and accreditation practices in the United States. The one-week visiting scholar program was a very rewarding experience!

Below is a line up of presentations and workshops I did during my visiting scholar week.

1. Program evaluation 101: Getting started workshop (2 language programs)  
2. Developing and evaluating a writing center
3. Graduate seminar presentations
   - College-driven initiatives on building evaluation culture
   - A nation-wide evaluation needs and capacity survey study
4. One-day open-public program evaluation event
   - A 3-hour workshop on utilization-focused program evaluation
   - Panel presentations 
   - A discussion session 

Since I have so much to say about each event, I will only blog about the evaluation workshop I did for the two language programs today.

The two ICU language programs I separately interacted are both going through major program reform (one in the middle of it and one at the beginning of reform). I often see program innovation and evaluation go hand-in-hand. When there is an agreement to innovate a program, faculty are likely to be open to program changes on the basis of program evaluation. And most likely, if you innovate a program, you want to learn how effective the innovation is/was! Agreement to program change/innovation makes it easier to gain buy-in from program staff to take on an evaluation project when they see that evaluation is use-driven and is based on internal needs. So it was very timely that my visit coincided with their innovation efforts. Since one of the programs was at the beginning stage of program reform, integrating program evaluation from the get-go can provide empirical basis for making decisions on what to change, what to keep, and how to reform the program.

The workshop for these two programs aimed at building understandings on how to conduct situational analysis and how to focus on program evaluation use and questions. Of course, they both had their own unique programmatic issues, but what was interesting was that improving student learning outcomes and faculty collaboration were the two major evaluation themes that emerged out of the discussion. In higher education, student learning outcomes assessment is a common one (especially in the United States), but evaluating organizational collaboration is rare and is a challenge.

Learning from the literature on organizational management, the sine qua non of organizational collaboration is shared vision/purpose and opportunities for co-construction. Some of the evaluative questions on faculty collaboration that immediately come to my mind are: Do faculty have shared understanding of the program goals and student outcomes? Do faculty have sufficient time for fruitful discussion enabled by constructive meaning-making process? Are there any missing voices in decision-making process? How are knowledge shared and stored within and between teams?

Unfortunately, we didn't have time to prioritize evaluation questions during the workshop, but if they decide to evaluate faculty collaboration in the future, it will be very interesting.

One of the challenges for me putting together the workshop materials was to translate evaluation concepts from English to Japanese. Assessment and evaluation are both "hyouka" (評価、ひょうか) in Japanese, and we don't distinguish the two terms. My solution? For student learning outcomes assessment, I translated "assessment" as "調査" (ちょうさ、 "inquiry/study"）in order to avoid the common misconceptions of assessment with testing.

Here is a question to multi-lingual readers of this blog. How do other languages handle the two terms, "assessment" and "evaluation," and how are they translated?

Evaluation quality

"Evaluation quality" was AEA 2010's conference theme. Whenever I engage in program evaluation, I "try to" meta-analyze evaluation quality to improve evaluation practices. Some of the questions I ask in my evaluation context are:

• Was the evaluation design appropriate to the given situation and intended users' needs? 
• Were the evaluation findings put to use as intended? 
• Did the intended users find evaluation findings and processes useful and insightful (and transformative)? In other words, did any learning from evaluation findings and processes happen? 
• Did stakeholders' program theory improve? 
• Was the facilitation of evaluation done in an ethical and responsive manner? 

I also refer back to the professional standards from time to time throughout the evaluation process. The Program Evaluation Standards and Guiding Principles for Evaluators are my friends when I facilitate evaluation.

At the opening plenary at AEA 2010, Eleanor Chelimsky talked about different evaluation quality arguments in three types of evaluation: evaluation for accountability, improvement, and knowledge-building. For example, in the case of accountability-driven evaluation (such as in Request For Proposals), we often see evaluation quality criteria described as "rigorous"(often in RFPs, rigorousness means randomized control trial or quasi-experiment design) and "objective" (i.e., an external evaluator conducting evaluation). In improvement-oriented evaluation, these "rigorousness or objectiveness" may not necessarily be the quality criterion used.

What would then happen, when evaluation is serving competing purposes and interest groups? As an evaluator, how would we balance and negotiate quality criteria to serve different interest groups with different definitions of evaluation quality (as well as our own standard of practice as a professional evaluator)? Is it negotiable at all? Can we get different interest groups to understand with each other?

Underproductive program?

We need convincing evidence to lobby against trimming of foreign language programs and the misperception that language programs are "underproductive." Read the article below to see why we need to showcase value of FL education (with evidence!).    

An article in Chronicle of Higher Ed by Gary A. Olson (Dec 1st, 2010): 
"Many colleges and universities are re-examining their curricula and assessing what should be trimmed and what should be enhanced. Generally, institutions are scrutinizing three key areas of reform: general-education requirements, requirements within majors, and underperforming programs.

The dire financial situation for higher education in most states has accelerated the process of streamlining, especially the elimination of weak programs. The University of Maine created an Academic Program Prioritization Working Group to analyze which programs to cut, and in what order, to meet budget reductions while maintaining high-priority academic programs. The group's final report is appropriately titled "Achieving Sustainability."
The State University of New York at Albany recently announced that it would close programs in the classics, French, Italian, Russian, and theater. Student demand for those programs is ebbing nationally, and many institutions are finding they don't have the budget to keep marginal programs afloat."
Click here to read the whole article in the Chronicle of Higher Ed: 
http://chronicle.com/article/Why-Universities-Are/125556/

Evaluation 2010 Conference: What is the conference like?

This year was my first time attending and presenting at the American Evaluation Association's conference. Let me report you my experience as a first-timer while my memory and excitement are still fresh.

Evaluation 2010, an annual conference hosted by the American Evaluation Association, was held at San Antonio, TX between November 5th and 13th. The conference had over 2500 attendees from various sectors (education, government, health, community programs, etc.). The first 2.5 days were capacity building workshops, followed by a 3.5-day conference. What is unique about this conference is that it's not only academics presenting their work. Evaluators, academics, funders (people from the foundation), policy makers attend this conference. Amazing synergy and dynamic.

There were over 35 concurrent sessions, so you can imagine how hard it was to choose which sessions to attend. Moreover, there is a variety of session types you need to know. For example, demonstration, panel, expert lecture, skill-building workshop, roundtable (presentation + discussion), think-tank (breakout discussion), poster sessions. Be aware that some of the panel and multipaper sessions were put together by the conference organizers, and few of them had an odd combination of papers in one session. There are no abstracts in the conference handbook, so you need to guess what the papers and sessions are about from the title (AEA recommends searching the abstracts online before you attend the conference). I ended up following specific SIG/TIG strands, so I don't need to go though 35 session titles during the break.

My general impression of the conference is that evaluators are good at relating to people and are good listeners. That's what they do, professionally, when they go into programs, right? They listen and try to understand the context. Naturally, evaluators are good at (or trained at) providing constructive comments, sharing their ideas, and creating a collegial network.

Observing how people interact in sessions made me wonder about evaluator traits, because AEA conferencing culture seems slightly different from where I come from (Applied Linguistics/TESOL). One of the session I attended was, in fact, discussing evaluator competencies. What makes an evaluator a good evaluator? Social science research skills? Theoretical and practical knowledge and skills to conduct evaluation via professional development or in graduate schools? Some of the audience in the session agreed that person/communication skills and personality make a difference. I wondered what the result will be, if we administer Myers-Briggs to evaluators.

Conference highlight? The champaign toast with everyone in the auditorium at the closing remark!!

Stay tuned for some of the highlights from the sessions I attended in the next blog.

ACTFL Convention - Evaluation related sessions and events

The ACTFL convention is coming up November 20-22. This is a great opportunity meet and exchange ideas on language program evaluation matters. We encourage you to use this space to announce any evaluation related sessions that you are aware of or connect with others who may be attending the conference.

Here are some interesting sessions we found in the ACTFL convention program related to tertiary language program evaluation. Please feel free to comment or add your own.

Evaluation and Curriculum Development:

Moving Forward: Ongoing Curricular Development and Evaluation
Sunday, November 22, 8:00am to 9:00am
San Diego Convention Center, Room 8

Program Administration Training Evaluation

Language Program Administration Skills and Knowledge: What's Important?
Sunday, November 22, 8:00am to 9:00am
San Diego Convention Center, Room 22

Outcomes Assessment

Creating a College Level Arabic Skills Continuum
Friday, November 20, 6:15pm to 7:15pm
San Diego Convention Center, Room 27B

Assessing Community Service Learning: A
Practical Approach
Friday, November 20, 3:45pm to 4:45pm
San Diego Convention Center, Room 23C