New resouce page: Document analysis & observation

We are happy to announce that we added a new resource page to the FLPEP website on "document analysis and observation techniques in language program evaluation." The page contains a list of how-to books and articles, example studies, and online advice.

Document analysis is often conducted to understand contextual information, such as program or instructional context, official/public policies and plans, program updates, curriculum and syllabus theory/design, and so on. Common types of documents often gathered in language program evaluation include course syllabi, instructor/curriculum handbooks, mission, goals, and outcomes statements, and students' enrollment and achievement records. 

While document analysis can reveal official or stated views of a program, observation techniques can reveal what actually happens in the program. Observation is useful when you are interested in understanding program context, implementation, processes, experiences, and interactions. In language program evaluation, observation is frequently applied to what learners and teachers do in the classroom, but other foci of observation might include language assessment practices, teacher development or induction procedures, learner counseling, and so on. Document analysis and observation can be complementary when identifying gaps between what actually happens in the program and what is formally stated. 

What we've been working on this summer

Every year, summer goes so fast. Here in Hawaii, our new academic year is starting this coming week! Dr. John Norris is back from his one-year sabbatical, and we've been keeping ourselves busy with outreach work and resource building this summer.

Between July 29th--31st, we had a group of enthusiastic Middle Eastern language program (MELP) educators from various universities (not only US-based, but also people from overseas, too!) gather at the University of Texas, Austin to attend our three-day workshop on program evaluation and outcomes assessment (see our previous blog post). We had very energetic discussions on the issues surrounding program evaluation in college programs, including ways to transform evaluation culture, strategies to take advantage of the external evaluation "wave" (we had a series of surfing themes going on, since the facilitators were all from Hawaii!), and so on. Presentation materials and discussion summaries are all posted on our new resource page, so check it out!

John Davis (one of our collaborating staff) created a very useful guide on "Using surveys for understanding and improving foreign language programs". An abbreviated version of the content was delivered at the MELP workshop, and you may find his presentation slides to be useful as well.

Contact us for any updates on evaluation work in your program! We'll be happy to showcase your work in our blog. 

Patton's new book "Essentials of U-FE"

Essentials of Utilization-Focused Evaluation (by Michael Quinn Patton) will be released this month from Sage Publications! 

The new book on U-FE is a concise summary of his popular book, Utilization-Focused Evaluation (4th edition). Patton takes the reader through 17 steps of Utilization-focused evaluation from situational analysis (Step 1) to meta-evaluation of evaluation use (Step 17). Each chapter includes case study examples to show how evaluation theory/approach is applied in various program contexts.

Prog eval workshop for the Middle Eastern Lang educators

This month, we are organizing a three-day program evaluation workshop event (July 29th--31st) at the University of Texas at Austin for the Western Consortium of University Centers of the Middle East. There will be over 40 educators from college Middle East language programs gathering for this event.

The event is a collaboration among the National Middle East Language Resource Center, the Center for Middle Eastern Studies at the University of Texas at Austin, and the University of Hawaii National Foreign Language Resource Center.

We will be offering a variety of practical tools, examples, and evaluation principles at work in college foreign language education contexts. There will also be panel discussions and a breakout discussion session.

Here is a sneak peak of the event:

• A keynote speech "High-value evaluation strategies in foreign language education" by John Norris
  
• A survey development workshop by John Davis
• Four evaluation and outcomes assessment showcase presentations from diverse language program contexts
•  A round-table discussion sessions on (a) externally-mandated program review and (b) data gathering methods for various types of outcomes. 

For details (summaries and schedule), go to: http://www.nflrc.hawaii.edu/evaluation/R_MELP.htm

The presentations and workshops will be videotaped and be available via NMELRC's website.

Claremont's Program Evaluation Professional Dvlpt Workshops

Yes, it's that time of year again! Between August 19th and 22nd, Claremont Graduate University is offering Professional Development Workshops on program evaluation (online and onsite). This year, they feature Michael Scriven's work at the whole-day symposium ("Future of Evaluation") on August 20th.

Registration: http://www.cgu.edu/pages/4729.asp
Workshop titles and descriptions: http://www.cgu.edu/pages/465.asp
The symposium titles: http://www.cgu.edu/pages/465.asp

A one-day workshop costs $50 for a student and $75 for a faculty.

Tips: Increasing survey response rates

Getting a high survey response rate is always a concern when administering a survey.  A high response rate will provide an accurate picture of the target population and allow you to make meaningful conclusions.

The response rate is calculated by the total number of complete returned surveys (or 80% or more) divided by the total number of participants you contacted. So if you asked 20 graduating students to complete an exit survey and 15 completed the survey, the response rate is 75%. There are several strategies to increase the response rates:

• Communicate the survey purpose, value, and how results will be used.
   
• Give sufficient time to complete the survey (online survey: 7-14 days)

• Make sure the survey is short, clear, logical, and easy to follow. Pilot-test the survey, so the items and instructions are written in understandable manner to the potential respondents.

• Send out reminders (thank the respondents, show how many responded, include a survey link, and remind respondents about the deadline)

• Use the existing opportunities to gather the target respondents (e.g., administer the survey in class, staff meeting, etc.)

• Offer an incentive (e.g., gift certificate, etc.)  

• Consider the best timing of your survey, so you are not administering the survey when the respondents are busy.  

Other resources: 

• AEA365: "Maximizing response rates" by Jessica Foster
• Division of Instructional Innovation and Assessment, The University of Texas at Austin: “Response Rates.” 
• Teaching and Learning Services, McGill University: Strategies to increase online response rate (for course evaluation)
• Dillman, D. A. (2000). Mail and Internet surveys: The total design method (2nd ed.). New York: Wiley.

Heiland & Rosenthal (Eds.). (2011). Literary Study, Measurement, and the Sublime: Disciplinary Assessment

The Teagle Foundation has announced a free online book on outcomes assessment in literary studies. This edited volume by Heiland and Rosenthal brings together literary scholars, foreign language and English department faculty, and assessment experts to provide disciplinary perspectives to outcomes assessment.

Those of you who are engaged in humanities and liberal arts programs will find the book enlightening and informative. The book, consisting of 19 chapters, responds to questions, such as:

• "How do we accurately depict and assess humanities outcomes that are often perceived as sublime and ineffable?" 
• "How can we localize assessment within the discipline?" 
• "How do we ensure ownership of assessment and make assessment a collaborative and useful process?"

You can access the book for free [here]

In concordance with the publication of the book, a "National Symposium on Assessment in the Humanities" was held at Miami University on February 23rd and 24th, 2011. The papers presented at the symposium is scheduled to be available online soon (according to the website). Stay tuned!