Mini Project 4
Case Study – Project/Program Evaluation in a Business Setting
LT8000 – Foundations in Instructional Technology
Brenda D. Whitehead
December 8, 2019
Introduction
The CIPP Evaluation Model was developed by Daniel L. Stufflebeam and colleagues in the 1960s. It originated as a guide for evaluating programs launched in connection with Lyndon Johnson’s “War on Poverty”. At that time, it was a contrast to the more widely used evaluation models. The CIPP Model emphasizes continuous evaluation of a program from its beginning to completion, stressing the need for process as well as product evaluations. (Stufflebeam, 1966) It originally was developed to evaluate programs in United States schools and colleges; however, over time, it has been become one of the most influential evaluation models used worldwide.
CIPP is an acronym for the four main components of this model – Context, Input, Process and Product. These components are evaluated to determine a program’s value. It is a decision-focused approach to evaluation. The components cover formative and summative evaluation as follows:
Context – “What needs to be done?” and “Were these needs addressed?”
Input – “How should it be done?” and “Was an effective design strategy utilized?”
Process – “Is it being done?” and “Was the design well executed?”
Product – “Is it succeeding?” and “Did the effort succeed?”
Numerous papers have been written regarding using the CIPP Evaluation Model. One paper is entitled “The evaluation of a blended facility development course using the CIPP Framework” by Dianne Thurab-Nkhosi. According the author, the study sought the impact of a faculty development course using blended learning. Using the CIPP Evaluation Model, the findings indicated that blended learning implementation improved and lecturers completing the course had increased levels of confidence in the use of technology for teaching. It also showed, however, a need to provide continued support and more opportunities for hands-on experience.
Another paper is “Using the Context, Input, Process and Product (CIPP) Model in the Evaluation of Training Programs” by Umam, K. A. and Saripah, I. In this paper, the CIPP model was used to determine a Percentage of Effectiveness (PE) for each of the four components – context, input, process and product. The first three components had scores ranging from 75.04%, 79.44% and 78.75%. The product component for this evaluation had a PE of 82.29%. It was the conclusion of the evaluators that the overall average score of 78.88% (fair) was successful and they would recommend similar programs with some improvements. Likewise, another paper – “Evaluation of Training Courses Applied in Succession Planning in Organizations Using the CIPP Model” by S. Tootian, evaluated training programs. In the evaluation of their courses, it was determined that the context, input, process and product components of the evaluation model did not reveal a favorable level of the courses for succession planning.
One last paper was “Utilizing the CIPP Model as a Means to Develop an Integrated Service-Learning Component in a University Health Course” by Brent Powell and Eric Conrad. The model was used to develop an integrated service-learning component of a university health course. The model showed that the service-learning enhanced health promotion knowledge, enhanced subject matter, and increased student ownership of the curriculum. Their recommendation, therefore, was that service-learning should be considered as a valid pedagogical method in the undergraduate curriculum.
These papers show the effectiveness of the CIPP Evaluation Model in assessing various programs and projects. Sometimes the results may show a successful outcome, sometimes it may show that the outcome is not effective. It is a clear and concise way to evaluate your programs and projects, showing statistical data to corroborate the conclusion
Case Overview
Case Study – Project/Program Evaluation in a Business Setting.
My case study will be evaluating a customer service program for a major company. The company, which is international with multiple branches, has seen tremendous growth. Because of such tremendous growth, the company wants to be sure that they maintain high quality customer service. There must be a collaboration with all branches on the effectiveness on the new customer services initiatives. A consultant was hired to conduct customer service training sessions. The contract was for a substantial amount of money. Now the CEO wants to know, if the training has made a difference in improving customer service. The CEO also wants a cost analysis of the program.
Case Analysis using the CIPP Evaluation Model
CONTEXT EVALUATION – assess needs, problems, assets and opportunities.
Satisfaction Surveys were submitted electronically to customers. Also, informational signs with numbers and e-mail addresses to submit the customer experience were posted in all the stores. Customer responses to the survey showed that customers didn’t feel that store employees were courteous and didn’t make them feel that they were interested when they had concerns. These results are a major red flag to the CEO. As a service-based company, their customers are why they are having tremendous growth. The company’s goal is “to deliver great customer service each time, every time”. Employees were asked to complete engagement surveys. The survey results show that the employees’ engagement level is not aligned with the company’s goal. It was determined that a major customer service training initiative was needed. Poor customer relations would definitely impact their current growth state.
INPUT EVALUATION – assess a program’s strategy, action plan, staffing arrangements and budget for feasibility and potential cost effectiveness to meet targeted needs and achieve goals.
It was decided that consultants would be hired to develop and conduct the customer service training. Because the organization has locations nationally and internationally, face to face training was not the best option. To provide the training to all employees, online classes would be developed and made available. The organization proposes to spend $800,000 to hire the consultants.
The training will be made available to the organization’s employees via all the company’s computer labs, at their desk computer, and tablets. To gauge if learning is taking place, there will be a pre-test and post-test included in the training. There will be job aids posted on the organization’s intranet for employees to access after completing the training, with tips on how to interact with the customers in solving their concerns. On December 1, a marketing plan will be initiated to let all employees know of what is upcoming. On December 15 and 16, informational sessions will be held with the employees to allow time for questions and answers with them. The training will begin on January 2 with an anticipated 3 months for everyone to complete the course. The consultants will monitor the progress through a Learning Management System. Biweekly reports will be discussed with the company to review if learning is taking place. One month after the training has begun, the stores will begin customer satisfaction surveys again to track any progress.
PROCESS EVALUATION – monitor, document, assess and report on the implementation of plans.
As the online customer service training is rolled out to all employees, careful attention will be given to who is attending. Reports will be given periodically to the company leadership on each division’s attendance and completion of the training. Likewise, the customer satisfaction surveys will be consolidated and a report given to the company leadership team monthly.
Technical support staff will be made available by phone or e-mail to render assistance with technical issues, such as computer errors. The consultants will have a team available for questions regarding using the online training program itself. Also, the company will have a team of technical support staff that will be assisting in helping employees utilize the customer service training. This team, along with the company’s training staff will be responsible for managing the online program after the rollout is complete.
PRODUCT EVALUATION – identify and assess costs and outcomes, intended and unintended, short term and long term.
The evaluation of the program’s effectiveness and impact will be determined by reviewing the ongoing customer satisfaction surveys. After a 90-day period, sales data will be reviewed to see if there has been an increase. If so, this could be an indication that the customer service level of the employees has improved and the training has made a difference in sales. At that time a cost analysis can be calculated to see if the training was cost effective and what is the return on the investment. At the end of this period, these reports can also help determine if the outcome is appearing to be long term.
Summary
At the completion of the online training on March 30th, there was a 97% attendance rate. It was not 100% due to a few employees that were out on extended sick leave. The pre-test and post-test evaluation report of the employees shows an average of 5% knowledge gained from the course. The customer satisfaction surveys showed no impact the first month; however, in subsequent months, there was a small percentage increase in satisfaction. This will continue to be reviewed monthly. After the 90-day period after the conclusion of the training, the cost analysis was calculated showing a ½% increase in growth for that period. Modest growth in sales, but a good start. Monitoring will help to continue to see if the program strategy was effective for long term results.
The CIPP Evaluation Model can be rendered ineffective, however. For example, when using this model and it does not address the true needs of the client; or does not convey a clear plan or strategy or provides contradictory data results. It will only be as good as the evaluator(s) that formulates the plan, compiles and reviews the data by following the four components as intended by Daniel Stufflebeam.
References
Powell, B. & Conrad, E. (2015). Utilizing the CIPP Model as a Means to Develop an Integrated Service-Learning Component in a University Health Course. Journal of Health Education Teaching, 6(1), 21-32.
Stufflebeam, D. L. (1966). A depth study of the evaluation requirement. Theory into Practice, 5(3), 121-133.
Stufflebeam, D. L. (1967, June). The use and abuse of evaluation in Tittle III. Theory into Practice 6, 126-133.
Thurab-Nkhosi, D. (2019). The Evaluation of a Blended Faculty Development Course Using the CIPP Framework. International Journal of Education and Development using Information and Communication Technology, V15, n1, 1-10.
Tootian, S. (2019). Evaluation of training courses applied in succession planning in organizations using the CIPP Model. International Journal of Human Capital in Urban Management, 4(2), 111-118.
Uman, K. A. & Saripah, I (2018) Using the Context, Input, Process and Product (CIPP) Model in the Evaluation of Training Programs. International Journal of Pedagogy and Teacher Education, Vol. 2, 183-194.