![]() To get the reliability, Sena checked if the graphs were scored consistency by the different raters. In non-science language, fourteen raters scored the same five graphs and gave a total score to each graph (calculated as a percent of total points possible, minus any items that were not applicable). In academic terms, she looked at inter-rater reliability (IRR) through Intraclass Correlation (ICC) using a two-way average measures for consistency with mixed effects (k-14, n=5). Sena also tested the checklist as a part of her dissertation to see if it had any reliability as an instrument – that is, whether the checklist does its job and can be trusted as a reliable tool. The training slides on this website are drawn from those guidelines. With input from Stephanie, Sena developed guidelines for raters’ to help address common areas of confusion and/or ambiguity in the checklist. The interviews also highlighted parts of the checklist that were confusing or interpreted differently by different raters. what do people think when they use it to rate graphs) and found raters’ understanding and use of the checklist aligned with the intended purpose of the tool. Sena (Pierce) Sanjines checked the validity of the modified checklist using cognitive interviews (i.e. In 2016, they launched a slightly modified checklist based off of feedback and implementation tests. They used the checklist with many clients and in hundreds of workshops over the years. In 2014, Stephanie Evergreen and Ann Emery developed the checklist based on Stephanie’s extensive review of relevant research and the practical experience of both Stephanie and Ann as data designers.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |