Table. Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers Florida Grade Language Arts Number of Assessment Items - 45 Standards Level by Objective Hits Cat. Goals Objs # of objs by % w/in std by Title Level Mean S.D. Concurr. # # Level Level 9 A - Reading 5.5.7 YES D - Language E - Literature 6 0 5 6 0 0 NO 5 50 5 50 0.7 YES Total 6 75 7 5 45.5 0.5
Table. Depth-of-Knowledge Consistency Between Standards and Assessment as Rated by Six Reviewers Florida Grade Language Arts Number of Assessment Items - 45 Level of Item w.r.t. Standard Standards Hits % Under % At % Above DOK Consistency Title Goals # Objs # M S.D. M S.D. M S.D. M S.D. A - Reading 5.5.7 0 4 70 YES D - Language 6 0 0 0 0 0 0 0 0 NO E - Literature 0 0.7 6 YES Total 6 45.5 0.5 5 74 7
Table. Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers Florida Grade Language Arts Number of Assessment Items - 45 Range of Objectives Balance Index Standards Hits # Objs Hit % of Total Rng. of Know. % Hits in Std/Ttl Hits Index Bal. of Represent. Title Goals # Objs # MeanS.D. MeanS.D. Mean S.D. Mean S.D. Mean S.D. A - Reading 5.5.7 4. 0.69 40 6 WEAK 7 6 0.6 0.04 WEAK D - Language 6 0 0 0 0 0 0 NO 0 0 0 0 NO E - Literature 0 0.7.67 0.47 7 5 NO 6 0.7 0.07 YES Total 6 45.5 0.5.5. 9 9 0.50 0.4
Table.4 Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria as Rated by Six Reviewers Florida Grade Language Arts Number of Assessment Items - 45 Standards A - Reading D - Language E - Literature Categorical Concurrence Alignment Criteria Depth-of-Knowledge Consistency Range of Knowledge Balance of Representation YES YES WEAK WEAK NO NO NO NO YES YES NO YES 4
Table.5 Source-of-Challenge Issues by Reviewer Florida Grade Language Arts Item Number Comments by Reviewer The stem is misleading. Ackwardly worded sentence, misleading stem Wording of item is confusing 5
Table.6 Depth-of-Knowledge Levels by Item and Reviewers Intraclass Correlation Florida Grade Language Arts Item Rater Rater Rater Rater 4 Rater 5 Rater 6 4 5 6 7 9 0 4 5 6 7 9 0 4 5 6 7 9 0 4 5 6 7 9 40 6
Table.6 Depth-of-Knowledge Levels by Item and Reviewers Intraclass Correlation Florida Grade Language Arts Item Rater Rater Rater Rater 4 Rater 5 Rater 6 4 4 4 44 45 46 47 4 49 50 5 Intraclass Correlation: 0.7544 Pairwise Comparison: 0.6504 7
Table.7 Notes by Reviewer Florida Grade Language Arts Item Number Comments by Reviewer If "performances" is grade-appropriate word. "Performance" is probably above third-grade level. More than one problem listed-details. 4 Main idea 6 Character points of view 7 Distractors are too easy Inference-very literal (word relationship) Chart says in ocean water but answer says creek. Two of the distractors are too close. 4 Distractors contribute to difficulty. Ackwardly worded Distractors are tough. Very strong distractors 6 Distractors increase item difficulty If "chirped" is grade-appropriate word. 0 If "snuggle" is grade-appropriate word. 0 I assumed that "snuggle" was grade appropriate. If not, then 4 G and H are too close Strong distractors increase difficulty.
Table. DOK Levels and Objectives Coded by Each Reviewer Florida Grade Language Arts Item DOK 0 PObj 0 A..... 4. 5 A... 6 A... 7 E... E... 9 E... 0.. A.... 4. 5. 6. 7 A.... 9. SOb j0. A...... A.... E... A... E..... A..... E... E... A..... A...... A... A... E.... E..... A....... A... E.... A...... A.... E.... E..... A..... E... E... A..... 4 4 A...... A... E... E... A... E..... A..... E... E... A..... SOb j4 A... 5 5 A...... A..... 7. E..... A...... E... A... E.... SOb j5 9
Table. DOK Levels and Objectives Coded by Each Reviewer Florida Grade Language Arts Item DOK 0 PObj 0 0 A...... 4. 7 5. 6. 7 E... A... 9. 0 A... E.... A... 4. 5. 6. 7.. SOb j0 A... E.... E... E... E... E... E... A... E... A.... 7. A...... E.... A... E.... E... E... E..... A.... A... E.... A...... E.... A... E... A... E... E... E... A.... A.... A... E.... A... A..... A.... 4 4 A... E.... E... E.... E... E... A.... A... E.... A... A.... E..... SOb j4 5 5 A..... E.... 7 E... E... E... A.... A.... 7 E... A...... E.... SOb j5. 0
Table. DOK Levels and Objectives Coded by Each Reviewer Florida Grade Language Arts Item DOK 0 PObj 0 9 A... 40. 4. 4. 4. 44. 45 A... 46 47 4 49 50 5 SOb j0 A....... E... A... A........ A... A....... E... A... 4 4 A....... 7 E... A... SOb j4 5 5 A....... E... A... SOb j5
Table.9 Objectives Coded to Each Item by Reviewers Florida Grade Language Arts Low (L) Low-Medium Medium Medium-High High 0 >0 - <5.594 5.594 >5.594 - <7 7 A... A.. A.. A.. A.. A.. A.. 4 5 A.. A.. A.. A.. A.. A.. 6 A.. A.. E... 7.7 E.. E.. E.. E.. E.. A.. A.. E.. 9 E... E... E... E.. E.. E.. 0...... A.. A.. A.. A.. A.. A........ 4..... 5 E.. E.. E.. 6 E.. E.. E.. E.. 7 A.. A.. A.. A.. A.. A.... E.. E.. 9......
Table.9 Objectives Coded to Each Item by Reviewers Florida Grade Language Arts 0 A.. A.. A.. A.. A.. A.. E... E... E... E.. A...... E... E... E... E.. E.. 4.7.7 E.. E.. E.. E.. 5 E.. E.. E.. E.. 6 A.. E... E... E.. 7 E.. E.. E.. E.. A.. A.. A.. A.. A.. A.. 9 E... 0 A.. A.. A.. A.. A.. A...7.7 E.. E.. E.. E.. E.. A.. A.. A.. A.. A.. A.. 4 A.. A.. 5 6 E.. 7 A.. E.. E.. E.. 9 A.. A.. A.. A.. A.. A.. 40
Table.9 Objectives Coded to Each Item by Reviewers Florida Grade Language Arts 4 4 4 44 45 46 47 4 49 50 5 A.. A... E.. A... E.. A.....7. E.. E.. A.. A.. 4
Table.0 Items Coded by Reviewers to Each Objective Florida Grade Language Arts Low Low-Medium Medium Medium-High High (L) 0 >0 - <7. 7.4 >7.4 - <0 0 A A.. A.. A... [LM: ] A.. [MH: 69] A...4 [H: 0]. [M: 0] 4 5 5 5 5 5 5 6 6 7 7 7 7 7 7 0 0 0 0 0 0 6 0 0 0 0 0 0 4 4 7 9 9 9 9 9 9 45 45 45 45 45 45 4 4 4 4 4 4 6 6 6 6 4 5 5 5 6 6 5 5 6 6 7 7 9 9 9 9 9 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 7 7 40 40 40 40 40 40 4 4 4 4 4 4 4 4 4 4 4 4 44 44 0 0 0 0 0 0 9 9 9 9 9 9 4 4 4.4.5.6.7 [LM: 6]. [M: 4] D D.. D.. D... D.. D.. D... D.. D...5 7 4 4 4 4 4 4 4 4 4 5
Table.0 Items Coded by Reviewers to Each Objective Florida Grade Language Arts E E.. E.. E... [M: ] 6 9 9 9 6 6 9 E.. [M: 7] 7 7 7 7 7 6 4 4 4 4 E...4 E...5 E.. E.. 9 9 9 5 5 5 6 6 6 5 5 5 5 6 7 7 [M: 0] 7 7 6 7 7 7 44 44 44 44 E... E.. E...4 E...5 6
Table. Number of Reviewers Coding an Item by Objective Florida Grade Language Arts Low Low- Medium Medium- High (L) Medium High > - < > - <6 6 (Item Number: Number of Reviewers) A A.. A.. A... A.. A...4..4.5.6.7. D : (L) :6 5:6 6: 7: 9:6 45:6 (L) : :6 :6 4:6 6:4 5: 6: 7: 9:5 4:4 44: 0:6 9:6 :4 4:4 7: 4: : 4: (L) (L) :6 4:5 : 4: (L) :6 : :5 7:6 :6 4:4 0:6 4: (L) 5:6 : (L) 5: 6:5 6: (L) 6: 7: :6 : :6 0:6 : 40:6 :6 4: : (L) 4: : 4:6 7
Table. Number of Reviewers Coding an Item by Objective Florida Grade Language Arts D.. D.. D... D.. D.. D... D.. D...5 E E.. E.. E... E.. E...4 E...5 E.. E.. E... E.. E...4 E...5 6: 9: : : 6: 9: (L) (L) 7:5 6: : : 4:4 :4 (L) (L) : 9: 5: 6: : : 5:4 6: 7:4 : 6: 7: 44:4 (L) (L) (L) (L) (L) (L)
Table. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade Language Arts Low DOK Low- Matched Matched DOK Matched- High High DOK (L) DOK DOK > - < > - < A []: A.. []: A.. []: A... []: A.. []: A...4 []: A.. []: A.. []: A... []: A.. []: : [] :6 [.] :6 [.67] 5:6 [.5] 6: [.5] : [.5] 9:6 [.] :6 [.] 45:6 [.] 4: 7: [.5] [] :6 [] (L) : [] :6 [] : [.5] 4:6 [.] 5: [.5] 6:4 [.5] 6: [.5] : [.] 7: [.5] 7:6 [.] :6 [] 9:5 [.] :6 40:6 4: 4:6 4:4 44: [] [.7] [.5] [] [] (L) [] 0:6 9:6 :4 4:4 [] [] [] [.5] 0:6 [] 4: [] :5 [.] : [] 5: [.67] 4:4 [.5] 6: [] 6: [] 5:6 [.7] :6 [.] : [.5] 6:5 [.6] 0:6 [.5] : [] 7: [] (L) 9
Table. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade Language Arts A...4 []: A...5 []: A...6 []: A...7 []: A... []: D []: D.. []: D.. []: D... []: D.. []: D.. []: D... []: D.. []: D...5 []: E []: 7: 4: : 4: [] [] [.5] [] :6 4:5 : 4: [] [.4] [.5] [] (L) 0
Table. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade Language Arts E.. []: E.. []: E... []: E.. []: E...4 []: E...5 []: E.. []: E.. []: E... []: E.. []: E...4 []: E...5 []: 6: 9: : : 6: 9: [] [] [] [] [] [] (L) 7:5 6: : : 4:4 :4 [] [] [] [] [] [.5] (L) : 9: 5: 6: : : 5:4 6: 7:4 : 6: 7: 44:4 [] [] [.67] [] [] [] [.75] [] [.75] [] [] [] []
Level Description DOK A Reading A.. The student uses the reading process effectively. A.. Uses a table of contents, index, headings, captions, illustrations, and major words to anticipate or predict content and purpose of a reading selection A... Selects from a variety of simple strategies, including the use of phonics, word structure, context clues, self-questioning, confirming simple predictions, retelling, and using visual cues to identify words and construct meaning from various texts, illustrations, graphics, and charts A.. Uses simple strategies to determine meaning and increase vocabulary for reading, including the use of prefixes, suffixes, root words, multiple meanings, antonyms, synonyms, and word relationships A...4 Clarifies understanding by rereading, self-correction, summarizing, checking other sources, and class or group discussion The student constructs meaning from a wide range of texts. Reads text and determines the main idea or essential message, identifies relevant supporting details and facts, and arranges events in chronological order. Identifies the author s purpose in a simple text Recognizes when a text is primarily intended to persuade.4 Identifies specific personal preferences relative to fiction and nonfiction reading.5 Reads and organizes information for a variety of purposes, including making a report, conducting interviews, taking a test, and performing an authentic task.6 Recognizes the difference between fact and opinion presented in a text.7 Recognizes the use of comparison and contrast in a text. Selects and uses a variety of appropriate reference materials, including multiple representations of information such as maps, charts, and photos, to gather information for research projects D Language D.. The student understands the nature of language. D.. Understands that there are patterns and rules in the syntactic structure, symbols, sounds, and meanings conveyed through the English language D... Understands that language formality varies according to situations and audiences D.. The student understands the power of language. D.. Understands that word choices can shape reactions, perceptions, and beliefs D... Identifies and refers to symbol, theme, simile, alliteration, and assonance in oral and written texts D.. Recognizes different techniques used in media messages and their purposes D...5 Understands that a variety of messages can be conveyed through mass media E Literature E.. The student understands the common features of a variety of literary forms. E.. Identifies the distinguishing features among fiction, drama, and poetry and identifies the major characteristics of nonfiction E... Understands the development of plot and how conflicts are resolved in a story E.. Knows the similarities and differences among the characters, settings, and events
Level Description DOK presented in various texts E...4 Knows that the attitudes and values that exist in a time period affect the works that are written during that time period E...5 Identifies and uses literary terminology appropriate to the grade level, including symbol, theme, simile, alliteration, and assonance E.. The student responds critically to fiction, nonfiction, poetry, and drama. E.. Recognizes cause-and-effect relationships in literary texts E... Recognizes and explains the effects of language, such as sensory words, rhymes, and choice of vocabulary and story structure, such as patterns, used in children s texts E.. Responds to a work of literature by explaining how the motives of the characters or the causes of events compare with those in his or her own life E...4 Identifies the major theme in a story or nonfiction text E...5 Forms his or her own ideas about what has been read in a literary text and uses specific information from the text to support these ideas