Assignment 5 Survey Pilot

Fountas & Pinnell Leveled Literacy Intervention Teacher Instruction Survey

Pilot Survey

            My pilot survey was written as part of my final assignment and program evaluation.  My program evaluation plan will be evaluating the reading gains of the Fountas & Pinnell Leveled Literacy Intervention program or LLI.  LLI is an intervention program which has been in use since 2009 and is designed for children who struggle with reading and writing, causing them to fall behind grade level.  This is an intensive intervention program lasting between 18-24 weeks.  Fountas and Pinnell use a scientifically based system of instruction and materials designed specifically for this instruction.  The success of this program is directly linked to its instructional guidelines.

            Therefore, as part of this evaluation I need to assess how closely the Leveled Literacy Intervention teachers are following the instructional guidelines as well as the ease to which teachers are able to use the program.  This survey, therefore, asks teachers for information in six areas:  About You, Supplies, Supports, Instruction, Student Selection, and Final Words.  Follow the link to see the pilot survey.  

 

http://fluidsurveys.com/surveys/pattie-WkV/leveled-literacy-intervention/

 

          After piloting the survey with teachers who are LLI instructors I found that many of my questions were valid.  However, there were still many changes necessary to be made.  Most changes were based on wording.  They were confused by my use of certain phrases such as ‘this past year’.  I was attempting to have them reflect on an entire school year but because they were currently in the middle of a year they were unsure.  By changing the phrasing to be ‘this school year’ the questions were made clearer. 

            Most of the changes made regarding the content of questions were around the use of supplies.  Even though I had read the program guide I misunderstood the use of some of the materials.   When I was asking questions about sharing materials they pointed out that while they had problems at first they were rectified.  This led to changes to two quantitative questions so that more information was acquired.  They also explained to me the importance of the resource CD.  I was unaware of this and that sharing it was very difficult.  This meant additional questions regarding this resource were required. 

            In the area of Supports, again I misunderstood the materials.  I had to combine two questions because I thought I was asking about two different DVDs but they pointed out that this was actually only one DVD.  They also felt compelled to explain why they did not share certain elements with the classroom teacher.  This caused me to add a question asking the teachers to explain their decision not to share this item. 

            In the section on Time, they wanted more clarity about group transitions and instructional time.  This led to a complete rewording of question two.  They also explained to me that entire segments of a lesson could be left out.  Again, I had to

add another question as I thought segments might be cut short but never realized they might be omitted all together.  It was brought to my attention that time was an even bigger issue than I had initially thought.  They felt that when I was asking about scheduling challenges I should ask if the teachers were able to solve them and if so how.  I am a little concerned that this question is now a leading question.  I also had to completely change my variable for question 8, as their answer wasn’t present.

            Finally, it became clear to me that this was not a typical teaching situation.  I asked them how difficult it was for a substitute teacher to step in.  At this, they became very animated and said I had to ask that question because this was a big problem for them and one that needed to be addressed.   We spent an hour going through the survey question by question.  Based on the amount of conversation taking place it would be my estimation that this survey would take about 30 minutes to complete. 

            I was very surprised by the number of changes made to the survey.  I honestly thought there wouldn’t have been so many.  However, I was in agreement with all of the things they saw in need of change.  Without question the revised survey will be easier to answer and give more in-depth and accurate information.  My only regret with the final survey was the difficulty of creating logic paths through branching and looping.  Unfortunately, this became an IT challenge that couldn't be rectified.  Participants can still complete the survey but they will simply have to follow written directions rather than being automatically redirected.  Rather 'old school' but it still works.  By following the link you can see the final survey.

http://fluidsurveys.com/surveys/pattie-WkV/copy-leveled-literacy-intervention/

 

Assignment 4 Fountas & Pinnell Leveled Literacy Intervention Program Logic Model

Program Description Abstract: The Fountas & Pinnell Leveled Literacy Intervention program is a literacy intervention program intended for students in grades k-2.  This program is designed for primary students who have difficulty with reading and writing.  It is structured around the Fountas and Pinnell Benchmark reading assessment system.  Books are leveled according to the reading skills necessary for comprehension.  A gradient is used to show the appropriate level for each grade level from kindergarten to grade eight.  The Leveled Literacy Intervention program uses this same system of leveled books.  The LLI program's goal is to bring students who are below grade level to grade level.  In some cases students may even surpass grade level.   The Leveled Literacy Intervention (LLI) program is designed to provide short-term intensive support.  Students receive 30 minutes of small group instruction five times a week.  Groups are never more than four students, however, it is strongly encouraged to keep them at three students per group.  This intensive support lasts on average for 18 to 24 weeks.  Teachers take part in a three day training session and receive detailed lesson plans.  Each lesson last 30 minutes and is segmented into activities lasting between 5 and 15 minutes.  Teacher materials include Program and Lesson guides as well as prompting guides.  Activities include a wide array of games and strategy practice intended to develop phonemic and phonetic awareness, word study, writing skills, oral language and comprehension.  Student materials include a series of books which are leveled according to the F&P leveling system and correspond with the F&P Benchmark Assessment system. Students receive book/word bags to take home which contain a black-line master of the books read in class as well as word cards to be practiced at home.  Before students are allowed to take part in the program parents are contacted and the intensity of the program is explained.  Students are not to miss a class unless they are extremely ill.  If parents have an upcoming holiday planned and their child will miss several days of the program they will not be admitted.  In addition, LLI classes are never cancelled for field trips, assemblies etc.  Attendance and regular at home practice are important to the integrity of the program. Fountas & Pinnell state that their Leveled Literacy Intervention program will take primary students from below grade level to grade level in 18 weeks if the program is followed according to their guidelines.  They also affirm that students will maintain this grade level literacy standing throughout their elementary school years.

Assignment 3 Program Evaluation Worksheet

Program Evaluation: Worksheet

           Engage Stakeholders

    Who should be involved?

Leveled Literacy Intervention teachers, learning assistant teachers, consultants, superintendents and classroom teachers should all be involved in this evaluation as they are all stakeholders in the Leveled Literacy Intervention program.  How might they be engaged?

Leveled Literacy Intervention teachers and classroom teachers will be involved with instrument development and testing as well as some data collection while the learning assistant teachers will be involved with data interpretation.  They could also be engaged in delivering the results to their peers.  Consultants and superintendents will be invited to contribute to the data interpretation as well as the presentation of results.       Focus the Evaluation What are you going to evaluate?  Describe program (logic model).

          

I will be evaluating the Fountas & Pinnell Leveled Literacy Intervention program.  Heinemann publishes this program (Copyright 2009).  This program is designed for primary students who have difficulty with reading and writing.  It is structured around the Fountas and Pinnell Benchmark reading assessment system.  Books are leveled according to the reading skills necessary for comprehension.  A gradient is used to show the appropriate level for each grade level from kindergarten to grade eight.  The Leveled Literacy Intervention program uses this same system of leveled books in their intervention program, which is intended to bring students who are below grade level to grade level.  In some cases students may even surpass grade level.  The Leveled Literacy Intervention (LLI) program is designed to provide short-term intensive support.  Students receive 30 minutes of small group instruction five times a week.  Groups are never more than four students, however, it is strongly encouraged to keep them at three students per group.  This intensive support should last on average for 18 weeks but no more than 24.     What is the purpose of the evaluation?

The purpose of this evaluation is to compare the literacy progress of students receiving the LLI program to those only receiving regular classroom instructions. Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Greater Saskatoon Catholic School Division	To determine the value of purchasing the program for more of its primary schools.
LLI Teachers	To determine the best use of the program.
Parents	To decide if they want their children to take part in the program.

What questions will the evaluation seek to answer? -Do the students in the LLI program make significantly more reading gains than students not in the program?

-How are students chosen for the program?

-Did students in the program receive additional supports outside of the program, which the non-program students didn’t receive? -Was the LLI instruction given as per the programs guidelines?  What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
Upon completion of the LLI program do the students in the program make more progress than the students who did not receive the program?	The LLI students would have higher post-test Fountas & Pinnell Benchmark scores than the non-LLI students.
Was the LLI instruction given as per the programs guidelines?	Observation of LLI instruction would indicate that teachers:  -Followed instructional guidelines                  -Used program books and materials                  -Used the additional materials recommended by the program                   -Instruction time met with program standards.
Was the only difference in instruction between the two student groups the LLI program?	Students in both groups will come from the same classrooms and have the same classroom teachers for literacy. Learning Assistance Teachers will not teach either group during the duration of the program.
Were students in both groups at the same reading level prior to the implementation of the program?	Student’s pre-test scores would all be within the same reading level as per the Fountas and Pinnell Benchmark assessment system.

When is the evaluation needed?  The evaluation should be completed during the 2014-15 school year.  What evaluation design will you use?

 

The evaluation design is based on both qualitative and quantitative data.  Qualitative data will be administered through observation and questionnaires to assess that the instruction adheres to the guidelines of the program.  The quantitative data will be in the form of pre and post test comparisons.  Collect the information

           

     What sources of information will you use?

Existing information: Leveled Literacy Intervention Program Guide Personal experience with students having taken the LLI program People: LLI teachers, classroom teachers, students of the LLI program Pictorial records and observations: Observation of LLI classes
What data collection method(s) will you use?

X	Survey		Document review
	Interview		Testimonials
X	Observation		Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
X	Tests		Unobtrusive measures
X	Photos, videos		Other (list)

Instrumentation:  What is needed to record the information? -I will need two questionnaires.  One for the LLI teachers to answer regarding their implementation of the program and selection of students and another for the classroom teachers to complete regarding in-class instruction for both groups.

- A checklist for observation of LLI instruction to ensure program guidelines are being followed will be needed. -Fountas & Pinnell Leveled Benchmark Assessments will be used for pre and post-test reading levels. When will you collect the data for each method you've chosen?

Method	Before program	During program	Immediately after	Later
F&P Benchmark Assessments	X		X
Observational Checklist		X
Teacher Questionnaires		X

Will a sample be used? Yes, a sample of grade two students eligible for the LLI program from three different schools would be used.  Each school would have 18 students who qualify for LLI.  Nine of these students would receive LLI and nine students would not.  Pre and post F&P Benchmark Reading Assessments would be given to each group of students. Pilot testing: when, where, how?

Pilot testing would be used for both the teacher questionnaire and the checklist.  The pilot tests would occur in June of 2014.  An online version of the questionnaires would be given to LLI teachers in schools other than the three selected schools.  Teachers would be requested to complete the questionnaire as a reflection of the previous LLI session.  The checklist pilot would also occur in June of 2014.  These would be done at the same schools involved with completing the pilot questionnaire.  Teachers at each of the three schools would videotape a lesson, which would be forwarded to me for observation and completion of the checklist.  After the completion of both the questionnaires and checklists all three teachers and my self would meet at the most central school to discuss the ease and sensitivity of the questionnaire as well as the accuracy of the information acquired by both the checklist and questionnaire.

         

 Analyze and Interpret

How will the data be analyzed?

Data analysis methods:

The mean, median and mode of the Benchmark reading assessments would be calculated for both the pre and post-tests for all sample groups. The observational data will be done by the use of a checklist to ensure that all elements of the programs lessons are followed.  In addition, a reflective note will be made regarding student engagement and attentiveness.  The checklist data will be calculated to determine a percentage regarding the various elements of the lessons presented and their integrity to the program.  The reflective notes will be summarized. The data from the questionnaire will also be calculated using percentage to establish the level to which teachers maintain the authenticity to the program during the unobserved lessons.  The percentages from the questionnaires and checklists will be correlated to show the increase of the reading levels of each teacher’s sample group.

Who responsible:

Pattie Tastad (the evaluator) along with the LLI instructors from the pilot test schools would be responsible for analyzing the data.  The evaluator will be responsible for housing all data.

How will the information be interpreted—by whom? The learning assistance teachers from the pilot test schools and the evaluator will interpret the information.  These would be different teachers than the LLI teachers.  If they were the same teachers than the classroom teachers from the pilot schools would be utilized.  This way people who were not directly related to program evaluation will be involved in the interpretation of information allowing for a non-biased view.  The information would first be interpreted to see if the LLI instruction was given within the program guidelines.  The percentages on the questionnaires and checklist should be within 90-100% in order to establish that the program was presented as designed.  Pre and post-test reading scores would be interpreted to see the difference between the mean, mode and median of the sample groups.  Two or more reading levels difference would be considered as significant.   The correlation percentages between the questionnaires and checklists to the sample student’s post-test scores would be done to see if teachers with a higher percentage of adherence to the program guidelines also have higher reading gains.  Thereby, showing more program contribution to these reading gains.

               Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
Greater Saskatoon Catholic Schools consultants and superintendents.	Upon completion of the program evaluation a presentation and written summary would be given at the earliest Division Professional Learning Network meeting.
LLI teachers, LATs, and primary classroom teachers	Upon completion of the program evaluation a presentation and written summary would be given at the earliest Division Professional Learning Network meeting.
Parents of students qualifying for the LLI program.	When parents are contacted by the LLI teachers to offer their child a place in the program they would be given a brochure highlighting the results.  If they were contacted by telephone they would be directed to an electronic version of the brochure.

        


 Manage the Evaluation:                         Standards:

	Human subject’s protection		Utility
	Management chart		Feasibility
	Timeline		Propriety
	Responsibilities		Accuracy
	Budget

Manage the Evaluation:

Human Subject's Protection - Students in both sample groups will be assigned a number so that no names will be seen on their pre and post-test scores.  The students who are in the videotapes of the LLI observations will not have their faces shown and a parent consent form will be sent home to give media release permission.  The teachers involved will also sign a media release form even though they too will be assigned a number and no names will be used in the videotapes, checklists and questionnaires.

Management Chart and Timeline – Below is the Gantt chart, which indicates the timeline and Management chart.

Responsibilities –

Development and pilot of questionnaire and checklist are the responsibility of the evaluator and the LLI teachers at the pilot schools.

Choosing students for the LLI program is the responsibility of the classroom teachers.

Administration of the pre-test is the responsibility of the classroom teacher.

Implementation of the LLI program is the responsibility of the LLI teacher.

Observation of the LLI instruction and use of checklist is the responsibility of the evaluator.

Completion of the questionnaire is the responsibility of the LLI and classroom teachers.

Administration of the post-test is the responsibility of the classroom teacher.

Data Analysis is the responsibility of the evaluator and the pilot schools LLI teachers.

Data Interpretation is the responsibility of the evaluator and the Learning Assistance Teachers at the pilot schools.

The evaluator will write the Draft Report and the Final Report.

Budget – Below is the budget.

Standards:

Utility - This program evaluation would be used for the purpose of formulating an independent determination of whether or not the program accomplishes what it claims to.  The purpose would be to assist the Greater Saskatoon Catholic School Division in spending its money in a responsible manner.

Feasibility - The feasibility of this program evaluation would depend on the availability of a team of LLI teachers, classroom teachers and Learning Assistant Teachers who would be willing to assist in the development of the instruments for data collection. In addition, they would also have to be willing to have their LLI instruction observed and to also complete the questionnaires.  The most challenging resource to make this evaluation feasible would be the time allocation for LLI instruction observation.

Propriety - The propriety of this evaluation depends upon the discretion in not revealing the names of the students and teachers involved in the evaluation.  It is important that only the pilot LLI lessons are videotaped and that all other students and teachers are assigned a number.  The use of names is strictly prohibited.  The key to these numbers should be kept away from the public eye.

Accuracy - To maintain accuracy all of the students must partake in the pre and post-tests.  In addition, they should also partake in the LLI instruction as per the design of the program.  The at home reading must be done and students must not miss any instruction.  It is also very important that the questionnaires are clear so that they are all completed in the same spirit and with a consistent understanding.