Assignment 3 Program Evaluation Worksheet

Program Evaluation: Worksheet

           Engage Stakeholders

    Who should be involved?

Leveled Literacy Intervention teachers, learning assistant teachers, consultants, superintendents and classroom teachers should all be involved in this evaluation as they are all stakeholders in the Leveled Literacy Intervention program.  How might they be engaged?

Leveled Literacy Intervention teachers and classroom teachers will be involved with instrument development and testing as well as some data collection while the learning assistant teachers will be involved with data interpretation.  They could also be engaged in delivering the results to their peers.  Consultants and superintendents will be invited to contribute to the data interpretation as well as the presentation of results.       Focus the Evaluation What are you going to evaluate?  Describe program (logic model).

          

I will be evaluating the Fountas & Pinnell Leveled Literacy Intervention program.  Heinemann publishes this program (Copyright 2009).  This program is designed for primary students who have difficulty with reading and writing.  It is structured around the Fountas and Pinnell Benchmark reading assessment system.  Books are leveled according to the reading skills necessary for comprehension.  A gradient is used to show the appropriate level for each grade level from kindergarten to grade eight.  The Leveled Literacy Intervention program uses this same system of leveled books in their intervention program, which is intended to bring students who are below grade level to grade level.  In some cases students may even surpass grade level.  The Leveled Literacy Intervention (LLI) program is designed to provide short-term intensive support.  Students receive 30 minutes of small group instruction five times a week.  Groups are never more than four students, however, it is strongly encouraged to keep them at three students per group.  This intensive support should last on average for 18 weeks but no more than 24.     What is the purpose of the evaluation?

The purpose of this evaluation is to compare the literacy progress of students receiving the LLI program to those only receiving regular classroom instructions. Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Greater Saskatoon Catholic School Division	To determine the value of purchasing the program for more of its primary schools.
LLI Teachers	To determine the best use of the program.
Parents	To decide if they want their children to take part in the program.

What questions will the evaluation seek to answer? -Do the students in the LLI program make significantly more reading gains than students not in the program?

-How are students chosen for the program?

-Did students in the program receive additional supports outside of the program, which the non-program students didn’t receive? -Was the LLI instruction given as per the programs guidelines?  What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
Upon completion of the LLI program do the students in the program make more progress than the students who did not receive the program?	The LLI students would have higher post-test Fountas & Pinnell Benchmark scores than the non-LLI students.
Was the LLI instruction given as per the programs guidelines?	Observation of LLI instruction would indicate that teachers:  -Followed instructional guidelines                  -Used program books and materials                  -Used the additional materials recommended by the program                   -Instruction time met with program standards.
Was the only difference in instruction between the two student groups the LLI program?	Students in both groups will come from the same classrooms and have the same classroom teachers for literacy. Learning Assistance Teachers will not teach either group during the duration of the program.
Were students in both groups at the same reading level prior to the implementation of the program?	Student’s pre-test scores would all be within the same reading level as per the Fountas and Pinnell Benchmark assessment system.

When is the evaluation needed?  The evaluation should be completed during the 2014-15 school year.  What evaluation design will you use?

 

The evaluation design is based on both qualitative and quantitative data.  Qualitative data will be administered through observation and questionnaires to assess that the instruction adheres to the guidelines of the program.  The quantitative data will be in the form of pre and post test comparisons.  Collect the information

           

     What sources of information will you use?

Existing information: Leveled Literacy Intervention Program Guide Personal experience with students having taken the LLI program People: LLI teachers, classroom teachers, students of the LLI program Pictorial records and observations: Observation of LLI classes
What data collection method(s) will you use?

X	Survey		Document review
	Interview		Testimonials
X	Observation		Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
X	Tests		Unobtrusive measures
X	Photos, videos		Other (list)

Instrumentation:  What is needed to record the information? -I will need two questionnaires.  One for the LLI teachers to answer regarding their implementation of the program and selection of students and another for the classroom teachers to complete regarding in-class instruction for both groups.

- A checklist for observation of LLI instruction to ensure program guidelines are being followed will be needed. -Fountas & Pinnell Leveled Benchmark Assessments will be used for pre and post-test reading levels. When will you collect the data for each method you've chosen?

Method	Before program	During program	Immediately after	Later
F&P Benchmark Assessments	X		X
Observational Checklist		X
Teacher Questionnaires		X

Will a sample be used? Yes, a sample of grade two students eligible for the LLI program from three different schools would be used.  Each school would have 18 students who qualify for LLI.  Nine of these students would receive LLI and nine students would not.  Pre and post F&P Benchmark Reading Assessments would be given to each group of students. Pilot testing: when, where, how?

Pilot testing would be used for both the teacher questionnaire and the checklist.  The pilot tests would occur in June of 2014.  An online version of the questionnaires would be given to LLI teachers in schools other than the three selected schools.  Teachers would be requested to complete the questionnaire as a reflection of the previous LLI session.  The checklist pilot would also occur in June of 2014.  These would be done at the same schools involved with completing the pilot questionnaire.  Teachers at each of the three schools would videotape a lesson, which would be forwarded to me for observation and completion of the checklist.  After the completion of both the questionnaires and checklists all three teachers and my self would meet at the most central school to discuss the ease and sensitivity of the questionnaire as well as the accuracy of the information acquired by both the checklist and questionnaire.

         

 Analyze and Interpret

How will the data be analyzed?

Data analysis methods:

The mean, median and mode of the Benchmark reading assessments would be calculated for both the pre and post-tests for all sample groups. The observational data will be done by the use of a checklist to ensure that all elements of the programs lessons are followed.  In addition, a reflective note will be made regarding student engagement and attentiveness.  The checklist data will be calculated to determine a percentage regarding the various elements of the lessons presented and their integrity to the program.  The reflective notes will be summarized. The data from the questionnaire will also be calculated using percentage to establish the level to which teachers maintain the authenticity to the program during the unobserved lessons.  The percentages from the questionnaires and checklists will be correlated to show the increase of the reading levels of each teacher’s sample group.

Who responsible:

Pattie Tastad (the evaluator) along with the LLI instructors from the pilot test schools would be responsible for analyzing the data.  The evaluator will be responsible for housing all data.

How will the information be interpreted—by whom? The learning assistance teachers from the pilot test schools and the evaluator will interpret the information.  These would be different teachers than the LLI teachers.  If they were the same teachers than the classroom teachers from the pilot schools would be utilized.  This way people who were not directly related to program evaluation will be involved in the interpretation of information allowing for a non-biased view.  The information would first be interpreted to see if the LLI instruction was given within the program guidelines.  The percentages on the questionnaires and checklist should be within 90-100% in order to establish that the program was presented as designed.  Pre and post-test reading scores would be interpreted to see the difference between the mean, mode and median of the sample groups.  Two or more reading levels difference would be considered as significant.   The correlation percentages between the questionnaires and checklists to the sample student’s post-test scores would be done to see if teachers with a higher percentage of adherence to the program guidelines also have higher reading gains.  Thereby, showing more program contribution to these reading gains.

               Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
Greater Saskatoon Catholic Schools consultants and superintendents.	Upon completion of the program evaluation a presentation and written summary would be given at the earliest Division Professional Learning Network meeting.
LLI teachers, LATs, and primary classroom teachers	Upon completion of the program evaluation a presentation and written summary would be given at the earliest Division Professional Learning Network meeting.
Parents of students qualifying for the LLI program.	When parents are contacted by the LLI teachers to offer their child a place in the program they would be given a brochure highlighting the results.  If they were contacted by telephone they would be directed to an electronic version of the brochure.

        


 Manage the Evaluation:                         Standards:

	Human subject’s protection		Utility
	Management chart		Feasibility
	Timeline		Propriety
	Responsibilities		Accuracy
	Budget

Manage the Evaluation:

Human Subject's Protection - Students in both sample groups will be assigned a number so that no names will be seen on their pre and post-test scores.  The students who are in the videotapes of the LLI observations will not have their faces shown and a parent consent form will be sent home to give media release permission.  The teachers involved will also sign a media release form even though they too will be assigned a number and no names will be used in the videotapes, checklists and questionnaires.

Management Chart and Timeline – Below is the Gantt chart, which indicates the timeline and Management chart.

Responsibilities –

Development and pilot of questionnaire and checklist are the responsibility of the evaluator and the LLI teachers at the pilot schools.

Choosing students for the LLI program is the responsibility of the classroom teachers.

Administration of the pre-test is the responsibility of the classroom teacher.

Implementation of the LLI program is the responsibility of the LLI teacher.

Observation of the LLI instruction and use of checklist is the responsibility of the evaluator.

Completion of the questionnaire is the responsibility of the LLI and classroom teachers.

Administration of the post-test is the responsibility of the classroom teacher.

Data Analysis is the responsibility of the evaluator and the pilot schools LLI teachers.

Data Interpretation is the responsibility of the evaluator and the Learning Assistance Teachers at the pilot schools.

The evaluator will write the Draft Report and the Final Report.

Budget – Below is the budget.

Standards:

Utility - This program evaluation would be used for the purpose of formulating an independent determination of whether or not the program accomplishes what it claims to.  The purpose would be to assist the Greater Saskatoon Catholic School Division in spending its money in a responsible manner.

Feasibility - The feasibility of this program evaluation would depend on the availability of a team of LLI teachers, classroom teachers and Learning Assistant Teachers who would be willing to assist in the development of the instruments for data collection. In addition, they would also have to be willing to have their LLI instruction observed and to also complete the questionnaires.  The most challenging resource to make this evaluation feasible would be the time allocation for LLI instruction observation.

Propriety - The propriety of this evaluation depends upon the discretion in not revealing the names of the students and teachers involved in the evaluation.  It is important that only the pilot LLI lessons are videotaped and that all other students and teachers are assigned a number.  The use of names is strictly prohibited.  The key to these numbers should be kept away from the public eye.

Accuracy - To maintain accuracy all of the students must partake in the pre and post-tests.  In addition, they should also partake in the LLI instruction as per the design of the program.  The at home reading must be done and students must not miss any instruction.  It is also very important that the questionnaires are clear so that they are all completed in the same spirit and with a consistent understanding.