DOCUMENTRESUMEED 029 367EA 002 148By-Peper. John B.Evaluation Designs for Title I of the Elementary and Secondary Education Act for 1968-1969,Philadelphia School District. Pa. Office of Research and Evaluation.Pub Date Feb 69Note- 279p.EDRS Price MF- 1.25 HC-S14.05Descriptors-African American Studies. Bibliographies, Closed Circuit Television. Dramatics. EducationalImprovement. *Educational Research. Evaluation Methods. Instructional Materials Centers. Job SkiLiterature Reviews, Models. Paraprofessional School Personnel. Program Evaluation. Reading Improvement.Research Projec4;. School Community Cooperation. Science Instruction, SpecialistsIdentifiers- *Elementary and Secondary Education Act Title 1. ESEA Title I. PhiladelphiaTo develop an evaluation model appropriate to education in the PhiladelphiaSchool System. six teams of evaluators drawn from a total evaluation staff of 11report on 34 Title 1-ESEA protects processed in the city of Philadelphia during the1968-69 school year. Information in each report Includes a summary of the particularprolect. its problem focus. a literature review, the protect's oblectives. and itsprocedures. Titles of protects reported include: Instructional material centers.learning centers. improving reading skills, salable vocational skills, closed circuittelevision. school community coordinator, kindergarten aides. creative dramatics.counselor aides, art specialist teachers, class for 3-year-old deaf children, parentschool aides. English as a second language. Head Start follow through. andAfro-American history. (JI04


INTRODUCTIONIn September of 1968 an evaluation team was assembled toconduct evaluations of Title I, ESEA, projects for the School DistrictDr. John L. Hayman and his staff had spent theof Philadelphia.previous year recruiting the fourteen professional members. Thesecretarial staff was selected during September and October, 1968.The designs for evaluation of thirty-five of the projects includedin this document represent an initial attempt to explore what evaluation has to add to an important program such as Title I.Although the four evaluation teams and other individualproject evaluators are credited with project evaluation designs, twoother members of this staff have materially contributed to each ofthe project evaluation designs. Mr. James Ayrer, Test ConstructionSpecialist, has provided untold hours of consultation and directassistance in selection, development and processing of tests andother evaluation instruments. Dr. Ying Chuang, Design and AnalysisSpecialist, has given research design consultation to all projectevaluators. He has also made a major contribution to the design ofan evaluation model which will be published soon in a separate document.A brief description is included below in order to provide a synthesisof what might be regarded on the surface as thirty-five separate designs.Before discussing the model for evaluation, it is appropriateto give credit for this document to the four secretaries who have workeddiligently to bring the typed copy through several stages of revision.Mrs. Darliene Christie, Miss Linda Brown, Mrs. Esther Greenberg and MissCarol Desandro have our warmest thanks for their very dedicated effortsA Model For EvaluationA.Statement of the ProblemThe conceptual base underlining educational evaluationneeds to be stated at the onset. Conceptualizations used here may bedivided into two classes: 1. Conceptions of the nature of evaluation;in general, and as related to specific classes of educational programs;and, 2. Conceptions of a model of evaluation systems needed to conducteducational evaluations.Although industrial and military organizations have developedappropriate system models for evaluating success around tangible products,there are few adequate conceptualizations of decision-making processes andtheir associated information requirements in education. At present, thereare at least two approaches to evaluation in education; namely, formativeand summative evaluation. Formative evaluation serves individual projectadministrators and summative evaluation provides comparative informationabout projects for central administration decisions.

The major aim of this paper is to develop an evaluationmodel that is appropriate to education in the Philadelphia School System.A word of caution is in order. When one borrows techniques such assystems evaluation from cther disciplines, he must be sure that thosetechniques are applicable to the problems under consideration. Acknowledging potential shortcomings in a totally mechanistic model, theadaptation recommended here is one which accounts for the specificenvironment of urban education.E.Evaluation DefinedEvaluation means the provision of information throughformal means, such as experimental research and survey research andthe use of judgmental analyses to supply rational bases for makingjudgements which are inherent in decision situations. Several levelsof decision-making are constantly in process where educational projectsoperate. Each requires a different kind of information. Two levelsof decision are most crucial for probable success of a project. Atthe central administration level, information must be provided whichwill permit enlightened funding decisions. At the project operationallevel, information is required which will help field administratorsachieve or alter stated goals and purposes. The evaluation effortshould develop information appropriate to each level.C.Nature of EvaluationIn the evaluation of educational programs containingmany varied projects aimed at a comprehensive approach to major problemsin education, there are at least two major kinds of evaluation proposedin related literature.These two evaluation activities lend themselvesto different purposes.1.Individual Project Evaluation (Formative Evaluation).As projects develop, their administrators require helpin defining behavioral objectives and in measuringthose objectives. The evaluators provide conceptualand technical services within projects at this stage.Also, when projects are clearly formulated and functioning, project administrators require further evidence ofuniformity and disparity of operations within treatments.Evaluators here act as a feedback system in designing andin evaluating process instruments.2.It is always desirable from the central administrationvantage to know which projects are likely to providethe most benefit for dollars expended. This summativeevaluation has three major comparative aspects. Theyare (a) input evaluation, (b) process evaluation, and(c) output evaluation.1 I

(a)Input EvaluationIn conducting an input evaluation, an attempt wasmade to determine the relevance of the project tothe needs of the students, staff, community, andthe innovation in the School District of Philadelphia.This was done by drawing information about theneeds of the School District of Philadelphia fromour pupil contacts with all levels of school personneland by reviewing data in many of the records ofschool personnel by the Federal Evaluation staff.From this a summative input judgemental model wasdeveloped and used to assess 35 individual projects.(b)Process EvaluationThe purpose of process evaluation is to monitor theproject to learn what was done and to provide information to the project manager and his staff so thatthey may improve project quality while it is beingconducted.The process evaluation will be done by employing aninstrument of our own design. It is in the generalform of a check list allowing the evaluator totabulate certain critical aspects of the projects'performance.(c)Output EvaluationOutput evaluation looks for changes in attitudeand/or behavior in four areas: 1) students, 2) staff3) community, 4) innovation.The output evaluation will generally consist of preand post tests. These two tests will be related bya statistical design. The data gathered in the preand post test will come from questionnaires, surveys,interaction analyses, test, etc.One of the potential difficulties in providing information fortwo levels of decision simultaneously is that of maintaining objectivity.To date, the excellent cooperation achieved in providing the two kindsof information has been possible as a direct result of broad commitmentin the School District of Philadelphia to educational improvement.The singular distinguishing feature of an ongoing evaluationeffort is that it develops within itself a mechanism for change. No doubt,by the time final reports are completed there will be many alterationsto account for changes that have occurred. In another year, the modelused here will undergo several revisions. This document is, therefore,to be taken as a cross section of an evaluation effort in a developmentalprocess.John B. Peper

TABLE OF CONTENTSIntroductionProjects By TeamsA.B.C.D.E.F.DAVIDOFF/CLEMSONInstructional Material CentersLearning CentersImproving Reading SkillsSalable Vocational SkillsClosed Circuit TelevisionPhiladelphia Tutorial ProjectOut of School Sequenced Science ol Community CoordinatorKindergarten AidesCreative DramaticsMusic Specialist TeachersCounselor AidesArt Specialist TeachersNew Staffing Patterns For EIP Schools8795105115123135145HOWARD/HARRISClass for Three Year Old Deaf ChildrenGermantown Area SchoolsPrimesiteClass for Mentally Retarded Emotionally DistrubedPennsylvania Advancement SchoolMotivationEducation in World AffairsSpecial Services for Bused StudentsWalnut Street INESpeech Arts TeachersSpeech Therapy ClinicsMulti-Media CentersCultural ExperiencesParent School AidesCounseling ServicesFilm/Media Center for CommunicationAffective EducationEnglish as a Second Language233237241245249257261267271CARULLOHead Start Follow Through277HANKINSONAfro-American History285

1INSTRUCTIONAL MATERIALSCENTERSDirector:Lillian BatchelorEvaluator:Stephen DavidoffAssistant:Barry Clemson

2SUMMARYThe literature supports the notion that individual childrenlearn in different ways, and certain characteristics of the child mustbe considered in designating the most desirable instructional experiences for him.The Instructional Materials Center is a resource facilitycontaining books, audio-visual instructional materials, and the devicesrequired to utilize them. The rich diversity of print and non-printmaterials make the IMC more than just a traditional library. The facilityshould be utilized for enriching classroom instructional units andmaking provisions for individual differences.The primary objectives in the past were:1.To build centers or modify existing facilities (1966-67).2.To stock IMC with hardware and software (1966-67).3.To increase the utilization of the IMC (1967-68).The comprehensive utilization of IMC will continue to be a primary objective in 1968-69, but in addition to this, assessments will bemade to determine the impact of the IMC upon the student.In order to evaluate the impact upon students, the followingareas will be examined:1.Utilization of IMC for instructional purposes by the classroom teacher.2.Materials used by the teacher to supplement the classroomprogram.3.Student use of IMC materials for individual pursuits andinterests other than those directly related to formal classroom assignments.Questionnaires, observational logs and interviews will be usedin conjunction with standardized test scores to assess the degree to whichobjectives are met.PROBLEMLibraries, as a depository of information, have long been reveredas a potential tool in the learning process. The IMC goes beyond thetraditional library since it contains materials for viewing and listeningas well as reading. It is aseumed that the best instruction occurs when

3the pupil is brought into contact with important ideas and experiences insuch a way that learning results and that the student will continue toseek further learning on his own initiative.In the past, books were the unitary means by which ideas wererecorded and communicated. Today, new devices are available to offerunique contributions to the same goal.Before the IMC can be used optimally, a number of questions remainto be answered. Among these are:.What materials and services should the IMC provide for the student?For the teacher?.What reference skills should the students have in order to maximize learning?REVIEW OF LITERATUREA substantial body of research indicates that children may approachlearning tasks in different ways, and that the most effective instructionwill take into account the individual child's characteristics in thisregard. Concepts such as learning strategies, sets, levels, and learningmodes have been prominent in recent literature with respect to individualdifferences in learning.Learning Strategies, Sets, and Levels. The concept of a learning strategy is not new. Dewey's fundamental theory of "learning by doing"fits this paradigm. Furthermore, the observing responses defined byWyckoff (1952), which involve controlling response patterns by manipulating the learner's environment, are also related. In fact, most of thecurrent methods for "learning-how-to-learn" are predicated partly on thestructure of learning strategies (Klausmeir, p. 464).Gagne and Paradise (1961) and Gibson (1965) have written oflearning strategies and have indicated that the strategies which thestudent brings to the learning situation may be more crucial in determiningeducational outcomes than the variables more often treated experimentally.The learning strategy concept includes such things as method of conceptualization and of problem solving; reaction to homework, to flexible andrigid programs, and to different teaching procedures; and performance underdifferent conditions of grouping and class size (Anderson, 1961). Thevast resources of the IMC may aid in concept formation and provide greaterflexibility for instructional programs. Utilization of the IMC allows fora multiplicity of activities within a class and varying group size accordingto instructional task.Harlow (1949) formulated the notion of a "learning set." Hedefined a learning set in terms of knowledge relevant to the learning of

4a distinct task or a distinct class of tasks. It has been defined in aslightly different way as readiness to learn and knowledge of how to learn(Bloom, Davis, and Hess, 1965).Bloom, Davis and Hess (1965) have devoted considerable attentionto learning set in their recent work and have commented as follows:Learning to learn is a far more basic type of learningthan coaching the child on school learning. It includesmotivating the child to find pleasure in learning.Itinvolves developing the child's ability to attend to othersand engage in purposive action. It includes training thechild to delay the gratification of his desires and wishesand to work for rewards and goals which are more distant.It includes developing the child's view of adults as sourcesof information and ideas, and also as sources of approvaland reward (p. 15).Learning level refers to level of task which an individual childis capable of successfully completing. This concept relates to thedevelopmental stages which have been discussed by Piaget and others.Different types of learning or learning tasks may be appropriate forchildren exhibiting different learning levels. Studies by Luchins (1961)and Gagne (1965) distinguished among types of learning differing in complexity from simple signal learning, i.e., Pavlovian conditioning, toprincipal learning and problem solving.Learning Abilities. Students differ, of course, in learningabilities.Stake (1960) and Duncanson (1963) found that task-orientedstudies were useful in identifying those learning abilities which significantly affected the successful completion of a trial. The abilities theydescribed are consistent with those a learner brings to all learning situations -- basic, abilities, relevant knowledge, and prior learning experiences.The abilities identified by Stake and Duncanson are identicalto the intellectual abilities proposed by Bloom (1956). Bloom indicatedthat different abilities are appropriate for different tasks. He stated,for example, ".if we are concerned with the problem of transfer oftraining, by definition we would select intellectual abilities and skillsas having the greatest transfer value (p. 42)." Guilford's (1965) factoranalytic studies also identified and reported the importance of these abilities in predicting achievement.Sensory Mode. Another approach to the problem of learning whichhas shown promise is that developed by Fernald (1921) and used recentlymore specifically for perceptually handicapped children by Frosting, Lefener,and Whitllesey (1961). This approach focuses on the sensory mode which isfavored by a child as he learns. These studies have shown that most children learn by using any of the sensory modes, but some children learn bestwhen presented with visual stimuli, others with oral stimuli, while othersneed to feel and manipulate the stimuli before they can comprehend fully.

5These concerns were also stressed by Snow (1963) in terms of "medialiteracy." He stated that materials should be developed which are specifically designed to organize the knowledge presented by this media.In the final analysis, researchers and curriculum designers musttake the initiative in formulating and utilizing media so that they meettechthe needs of the individual child. Suppes (1964) concurs, ".ology alone cannot produce any fundamental or long-lasting changes in theOnly if its applications are guided by appropriate psycholocu rriculum.gic al principles and subject-matter insight will such (constructive) changesbe b rought about (p. 79) .".Summary. The literature dealing with the relationship betweenthe individual and how he learns is extensive. That which is relevant forthe current project may be summarized as follows:Three clos ely related but conceptually distinguishable learning variablesare learning strategy, learning set, and learning level. Children bringdifferent leve ls of these variables to a given situation and thus presentvarying instruc tional needs.In addition chil dren have different learning abilities, that is, theydiffer in basic ab 'lities, relevant knowledge, and prior learning experi.ences.Children also differ according to the sensory mode which they use mostSome learn best with visual stimuli, others learneffectively in learningbest with oral stimuli, while others need tactual stimulation.A child will learn most ef fectively when the classroom experiences hereceives are designed around his particular characteristics. In otherwords, the most effective ins truction will be that which is manipulated tofit the child, rather than the other way around.The multimedia available in the IMC make enrichment of basicclassroom instructional units and the individualizing of instruction apossibility for increased learning. Furthermore, the IMC may offer children the opportunity to pursue their own interests and special projectswhich could extend and enrich impoverished cultural backgrounds.OBJECTIVESTo provide students with resour ce materials which supplementclassroom instruction as evidenced by:A.(1)(2)(3)increased student use of books an d printed materialsteacher utilization of instructional materials for planning purposesimproved student performance in basic s kills,M,.411.011110.01074,14M-1./11FIVVIONAMWO, M.R.02704(000.10"

6These objectives will be measured by an observation log, teacherquestionnaire and the Iowa Test respectively.To provide instruction in reference skills and in locatinginformation thereby improving student performance as measured by Philadelphia City Tests.B.To develop positive feeling towards using the IMC facilitiesas measured by a student questionnaire.C.To provide materials and services for pursuits and interests(e.g., hobbies, recreational reading) - not directly related to classroominstruction.D.Experimental Design OneThis phase of the evaluation will be conducted using the NationalStudy of Secondary School Evaluation Form for IMC's (see attached instrument). Periodic visits to centers and the filling in of check-lists willlead to a fairly objective rating.The ratings for the several centers will be compared prior toMay, 1969.In addition, circulation figures will be available in fall, andspring to assess student usage. These devices should indicate a measureof use.Evaluation Design 1.b.This will be a pre and post questionnaire (see attached prototype)designed to obtain information on teacher use and teacher perception ofContained are items designed to assess if IMC is providing teachers in areas required by teachers for enrichment of classroom program.HypothesesHH:12'There will be significant increases in teacher use of IMCfrom beginning till end of year as measured by the teacherquestionnaire.There will be significant improvements in the IMC operationsas perceived by the teachers throughout the year as measuredby the teacher questionnaire

STATISTICAL TESTSTo test Hi, the first two items of the questionnaire will beanalyzed to see if ther4 is a significant change in the proportion of teachers selecting a given response (.05 level).1.To test H2, items three through ten will be analyzed to seeif there has been a significant change in the proportion of teacher'sresponses between pre and post questionnaires.2.ModelITEM SELECTIONA BC DPrePostb - c *4(see Guilford p. 186-188)In addition to the above, an inter-correlation of items may bebetween teacher use and servicerun to determine if there is a relationshipprovided by IMC.Evaluation Design 1.c.What impact does IMC have on student performance in basic skills?

8Statistical Hypothes,s:H:0There will be no significant (p .05) difference on the IowaTest of Basic Skills (composit) between classes who frequently or infrequently attend 1MC's.To test the overall statistical hypotheses a multiple classification ANOVA will be used as follows:ANOVA(Grades 4, 5, 6)STUDENT GAINClass AttendanceLowAveracreHi hPer Month0 - 2IOWAGAINSCORESv----3 - 51 NBASICSKILLSV26 - 8V39 or moreV4123

9Sample for design 1.c.GRADE 4, 5, 6DISTRICTSCHOOLLOCATIONLIBRARY SUPERVISOR1PowellLeaMcMichael36th and Powelton47th and Locust36th and FairmountSylvia Marder2LandrethMcDanielF. Douglas23rd and Federal21st and Moore22nd and NorrisHarold Jones3TaggertKearney4th and Porter6th and FairmountEleanore Serinsky5Elverson13th and SusquehannaGloria Roseman6FitlerSeymore and KnoxJudy Marcus7WebsterHedge and UnityLea Pinson

MONTHLY IMC ATTENDANCE FORMDirections: The purpose of this form is to improve the services of theIMC.Please fill out monthly as follows:(1) List information requiredin items 1 through 4.(2) List the teacher's name, grade taught, eachvisit, average time of stay and purpose of visit. If the visits wererelated to a class project they should be counted as academic. If theywere used for free-reading, hobbies, etc., they should be categorized asnon-academic. It is realized that there is a fine line between thecategories -- but the distinction can be made by answering the following:"Was the purpose of the visit directly related to a classroom instructionalunit?"1. School4.Date:FromTo2. Location3. PrincipalPLEASE PRINTTeacher(Full Name)GRADEDATE OFVISITAVERAGE TIMEPER VISITPURPOSE OF VISITACADEMICNON-ACADEMIC

11Evaluation Design 2.What role should the Library fulfill in IMC operations?Statistical Hypothesis.There is no significant (p .05) difference on Iowa Subtest ofReference Skills among students who receive instruction from interns andstudents whose schools do not have library interns.(Grade 5 and 6)ANOVAROLE OF INTERNSchools without internsSchools with internson Iowa SubtestScore- KnowlI dge & Use ofReference Skills1Cell data would include individual gain scores on the ReferenceSkills Subtest at the Iowa.Instruments to be used.1) Iowa Test of Basic Skills

12TEACHER QUESTIONNAIRETITLE I EVALUATIONLibrary - Instructional Material CenterThe following questionnaire is designed to gain information that willhelp improve the services offered by the IMC.needed information may be obtained.Through your cooperation,Please take a few moments from yourschedule and fill in the required information.As in all research, individualreplies will be confidential.DIRECTIONS:1.USE A NO. 2 PENCIL.2.Fill in your Name; Form of Test (A); Semester (Fall); Grade;in the Boxes provided.Then blacken the letter box belowwhich matches each letter of your name, etc.It will not benecessary to fill in any information other than what is calledfor in this direction and direction #3.3.Print your school's name in the space provided.4.Select only one response for each question.5.DO NOT USE SPACE E on the Answer Sheet.6.Return the Questionnaire and Answer Sheet in a sealedenvelope to your building Principal.1.2.3.How many times per month do you take your class to the Library - IMC?A.0C.3 or 4B.1 or 2D.5 or moreAbout how long will each visit last?A.Less than 15 minutesC.31 to 45 minutesB.16 to 30 minutesD.More than 45 minutesHow familiar are you with the contents of the IMC?A.Very familiarC.Slightly familiarB.Moderately familiarD.Not at all familiar

2In the next series of questions (#4 to #10) you are asked to select theresponse which best describes your school's IMC and the staff who are responsiblefor servicing it.Select:AInfrequentBOccasionallyCFrequentlyDVery frequentlyTHOSE IN CHARGE OF THE IMC IN MY SCHOOL: me when new equipment and materials come in.A.InfrequentC.FrequentB.OccasionallyD.Very frequentAssist me in selecting materials and A.V. aids for the y frequentInform me of areas of pupil interest they have observedA.InfrequentC.FrequentB.OccasionallyD.Very frequentHelp me develop reading lists for special unitsA.InfrequentC.FrequentB.OccasionallyD.Very frequentProvide ideas and materiais for bulletin boardsA.InfrequentC.FrequentB.OccasionallyD.Very frequentProvide facilities and assistants in the production of sionallyD.Very frequent

14310.11.Train and schedule projectionists for A. V. y frequentThe source of instructional materials that I draw most frequently, uponfor classroom instruction is:A.Public and University librariesB.Ideas from other teachers and professional publicationsC.University method coursesD.Instructional Material CenterQuestions 12-15Lists statements concerning the MRC.Select the one responsefor each statement which best describes your feelings about the statement.12.13.I would prefer to take my class to the MMC more frequentlyA.Strongly agreeC. Moderately disagreeB.Moderately agreeD. Strongly disagreeI can send an individual or committees to work in the MMC as often asI wish.14.A.Strongly agreeC.Moderately disagreeB.Moderately agreeD.Strongly disagreeThe MNC is NOT available for student use prior to 9 a.m., during lunch,and after school.15.A.Strongly agreeC.Moderately disagreeB.Moderately agreeD.Strongly disagreeI have difficulty in obtaining audio-visual devices (e.g., slide projector,charts, etc,) from the MMC for classroom use.A.Strongly agreeB.Moderately agree4C.Moderately disagreeD.Strongly disagree

15Questions 16-30-asks you to rate aspects of the INC on a Scale fromexcellent to poor.16.How ndequate is the number of instructional materials personnel tomeet your needs?17.A.ExcellentC.FairB.GoodD.PoorHaw adequately do members of the instructional materials staff aidteachers in the effective use of instructional materials and equipment?18.A.ExcellentC.FairB.GoodD.PoorTo what extent do members of the EMC Staff help you in the productionof appropriate instructional materials?19.A.ExcellentC.FairB.GoodD.PoorTo what extent do members of the MMC Staff help students make effectiveuse of,instructional materials?20.A.ExcellentC.FairB.GoodD.PoorAs a resource center to be used for leisure activities the studentsconsider the IIIMC to be21.A.ExcellentC.FairB.GoodD.PoorTo what extent does the MC Staff involve teachers in the selection ofmaterials?22.A.ExcellentC.FairB.GoodD.PoorHow adequately are books organized for effective use?A.ExcellentC.FairB.GoodD.Poor

1623.How adequately are periodicals, pamphlets and similar materials organizedfor effective PoorHow adequately are the audio-visual materials organized for effective use?A.ExcellentC.FairB.GoodD.PoorHow accurate and up to date are the card catalogue and shelf-list files?A.ExcellentC.FairB.GoodD.PoorHow accessible are instructional materials?A.ExcellentC.FairB.GoodD.PoorHow adequate are the periodicals to meet student needs and interests?A.ExcellentC.FairB.GoodD.PoorHow adequate are the periodicals to meet student needs and interests?A.ExcellentC.FairB.GoodD.PoorHow adequate are the periodicals to meet faculty needs?A.ExcellentC.FairB.GoodD.PoorHow good is the condition of audio-visual equipment?A.ExcellentC.FairB.GoodD.Poor.

17LEARNING CENTERSDirector:Lore RasmussenEvaluator:Barry Clemson

18SUMMARYThe Learning Centers Project is an attempt to develop and todemonstrate new and different models of teaching and teacher behavior.The project hopes that these models will have an impact upon the children in the program, u

to give credit for this document to the four secretaries who have worked diligently to bring the typed copy through several stages of. revision. Mrs. Darliene Christie, Miss Linda Brown, Mrs. Esther Greenberg and Miss Carol Desandro have our warmest thanks for their very dedicated efforts. A Model For Evaluation. A. Statement of the Problem