At the Elizabethtown Area School District, we use an application called Kodiak to manage our student information. Kodiak is a database driven suite of utilities that allow teachers and administration to view demographic data, schedules, attendance, discipline, and most importantly, assessment data for all pre and post test. This program was developed and also supported by the technology department at EASD. At present, Elizabethtown is the only school district to use Kodiak as its SIS.
This presents a problem with newly hired teachers. Because Kodiak is only used at Elizabethtown, new teachers to the district do not know how to use the assessment tools. Few have toyed with the application and have a general idea of what the tools can do, but interviews show that all of the new teachers are not using the tools to their maximum capability. Our goal is to have all newly hired teachers become proficient at using these tools.
This goal is paramount because without using these tools, teachers are not fully analyzing student assessment data. The analysis of this data is necessary because it allows teachers to determine which state standards are not being met by which students. With that information, teachers can adjust the curriculum accordingly to assist those students in passing state assessments. The passing of these assessments will allow the school district as a whole to attain AYP status.
This course (tentatively titled Kodiak: Assessment Tools), is needed so that all new staff can utilize the data that Elizabethtown Area School District has been collecting for the last four years. Although the final design of this course has not been finalized, it is apparent that proper instruction will allow new teachers and administrators to maximize instruction and meet goals set forth by EASD and the state of Pennsylvania.
Click on the links below to view the documents (PDFs)
Task Analysis
Skills Hierarchy
Main Objectives- Given a computer and the latest build of Kodiak, the staff will be able to search for a student or group of students, create a group out of those students, and view assessment data for either a single student or a group of students. The staff will be able to perform this task in under 2 minutes by recalling the procedure outlined in the handouts.
Affective Objective- Given a computer and the latest build of Kodiak, 80% of teachers who completed this course will demonstrate that they value this tool by viewing assessment data at the beginning of each term.
Enabling Objectives- Given a computer and the latest version of Kodiak, the staff will be able to launch Kodiak and login within 20 seconds.
- From the main Kodiak screen, the staff will be able to search and select a student within 30 seconds following the procedure illustrated in the handout.(Prereqisite)
- From the main Kodiak screen, the staff will be able to search and select a group of students (class section, students with IEPs, etc) within 60 seconds following the procedure illustrated in the handouts.(Prerequisite)
- After selecting a group of students, the staff will be able to create a group within 30 seconds following the procedure illustrated in the handouts.(Prerequisite)
- From the main Kodiak screen, the staff will be able to view previously created groups and select one group within 20 seconds following the procedure illustrated in the handouts.
- Given a group or single student, the staff will be able to view the tests available within 20 seconds following the procedure illustrated in the handouts.
- Given a list of assessments, the staff will be able to select a test within 20 seconds following the procedure illustrated in the handouts.
- Given a test, the staff will be able to select a graph type and view the assessment data within 20 seconds following the procedure illustrated in the handouts.
The majority of assessment in this session will consist of the student demonstrating that they have mastered a topic. The session will contain a number of examples, and the students will prove they know the material by executing the steps necessary but with different conditions. The steps will be the same, but the variables will be different. This assessment will be performed using a computer and the latest build of Kodiak, and the students will be observed using Apple Remote Desktop.
This assessment will be authentic because in order to display mastery, the students must perform the tasks that they will be using every day in their own classrooms. To be an effective teacher, all relevant data must be analyzed. This course will teach that, and the teachers taking this course must be able to demonstrate it.
This is an example of one assessment:
Using Kodiak and the supplied documentation, perform a search and select all 3rd grade males with IEPs from East High Elementary. You must complete this task within 60 seconds to demonstrate mastery of this objective.
In order to fully utilize the data present, groups must be created to display trends that exist in the student body. Although the teaching strategy is yet to be determined, the students will display that they have mastered a topic by completing the specified task in compliance with the course objectives. Any performance less than the stated objectives proves that the student has not mastered the topic.
The only problems that may arise are technical in nature. All teachers are supplied laptops by the district, so equipment supply should not be an issue. The working condition of the equipment is where problems may arise. That being said, it may be difficult for the teachers to demonstrate mastery if their computers are not working properly.
I believe that this assessment is valid because all of the students in this course have had access to identical materials. The tasks at hand are authentic in that they will be used frequently. Most of the tasks fall into the cognitive domain, and this assessment uses said domain in proving mastery.
Situation will arise, however, when a course attendee will not grasp all of the concepts. Should this situation become evident, additional training may be scheduled. This training will take place during teacher contract time and no additional credits (Act 48, 80) will be awarded.
This session will be different than most because it will not involve a known audience. As new staff are hired to Elizabethtown Area School District, they will be taught how to view assessment data. Unfortunately, the past experiences of the students will not be known. Acquiring the data necessary will be a round about task that will rely heavily on college catalogs.
As all teachers are college graduates, it is a safe assumption that they all will be able to comprehend written texts. As a result of that, the teachers attending this session will be given written materials (procedures, flowcharts) pertaining to these tasks. Since the teachers will be demonstrating the tasks from the written materials, it shouldn’t be necessary to change that portion of the assessment.
As far as general abilities, the new staff will have already received their laptops and gone through a training session that showed them the basics of the applications we use. With that in mind, the teachers should be able to follow the basic operation of the program. Therefore, they shouldn’t need any additional training with computer basics. As a result, we will not deviate from the designed practice and assessment.
Finally, it is a safe bet that since the teachers will be teaching in a Pennsylvania school, they are fluent in English. Since there are no language barriers, all of the practice and assessment can occur in their present form.
The practice portion of this course will be based directly on the course objectives. While sitting at a computer with the latest build of Kodiak, the teachers will practice the objectives by following the procedures illustrated in the handouts. The instructor will view the desktops of all of the students with Apple Remote Desktop in order to view problems.
After instructing the teachers on how to perform the steps necessary to view assessment data, the instructor will change the conditions and instruct the teachers to perform the previous steps again. For example, the instructor will show the teachers how to create a group using Kodiak. The instructor will then tell the teachers a condition (find all students with IEPs) and have them create the groups again. As the teachers perfect the procedure, the instructor will assign them the same exercise but with more difficult conditions (find all male students with IEPs in 3rd grade at Rheems Elementary.)
The instructor will observe each of the teachers using ARD as they perform the practice. Should the teachers have any questions, they are encouraged to ask the instructor. Also, the instructor will call attention to any incorrect steps that have been witnesses using the monitoring software.
The practice is identical to the procedure shown by the instructor, and more importantly, is identical to the assessments that the students will perform after practice.
This course will be a hybrid of multimedia, procedures that will be available in both online and printed form, and instruction that uses a projector to display the information to be learned.
The beginning of the lesson will show a brief keynote on the importance of assessment data. The presentation will show examples of test data and how those data can be used to attain AYP status. This keynote will be a sort of activating strategy in that it will grasp the teachers’ interest and convince them that knowing how to view assessment data is important. An introduction like this is cost effective because it is easy to assemble and has the potential for a high degree of impact.
Documentation will also be distributed to the teachers in this session. This documentation will be presented in two forms: online HTML documents and printed documents. The printed documents will merely be hard copies of the online docs, but this way the teachers will have materials in their hands during the session. The online documentation will be available in the case where a teacher loses the printed forms or needs help at a different location where the printed materials are not available.
Finally, the demonstrations of the procedures will take place on a projector screen. The instructor will have a copy of Kodiak and will perform the tasks for the students on the screen. This will allow the teacher to answer questions immediately while also being able to show the students the correct procedure. Spoken directions may not be enough for some teachers, and that’s why a projector will be used.
The materials for this course will be reviewed on three different levels by various people. Each person will play a different role in the document creation. These people have been chosen based on their knowledge of Kodiak and technical proficiency.
The first person that will see this documentation is the author of the program. He will check the documents for factual correctness. He will also check if all procedures are followed correctly. Being the author, he has discussed requested functions of the program with school administrators and understands the reasoning behind all of the design decisions. This will help with the creation of the course documents greatly, as he understands the flow of logic that was initially used to create the different functions in Kodiak. He also has a background in education, and will check the documents for spelling and grammar.
The second person that will examine the documents is our K-12 math coordinator. She is an avid Kodiak user, and is technologically proficient. Her main role in reviewing the documents is to determine if the order of the steps is indicative of how to perform the task. She will follow the steps and note if there are any blatant discrepancies. She will also provide another level of grammar checking. She represents the end-user who is familiar with the topic, but forgets a step or two.
The final user to review the documents is one of our 5th grade elementary teachers. Technologically speaking, she is not that proficient and she will be the first to tell you that. Her job is to follow procedures she has never seen before and note if she has any difficulties. She represents the end-user that is new to the material. It's a safe bet that if she can understand the procedure and complete the task, that other individuals will be able to complete the task as well.
Instructor QualificationDuring the implementation phase of this process, an instructor must be chosen to teach the class. Ideally, the person chosen will be a teacher who has become an expert at the concepts in viewing assessment data. Most likely, the teacher of the course will be an expert who is filling the role of instructor.
That being said, whoever is leading the class should possess a vast amount of patience. Most students, especially adult learners, are intimidated by technology and will no doubt get flusterd at one time or another. The instructor should be able to notice when a student is becoming frustrated and differentiate instruction to that student in order to bring him/her back on task.
The instructor should also possess the ability to view problems from different perspectives. Because he/she is a Kodiak expert, he/she will be able to perform the various tasks intuitively. A student learning the system will not possess the skills necessary to perform the tasks automatically. The instructor must realize this and understand what the student sees on the screen, not just what he/she sees as an instructor.
Finally, the instructor must be able to handle a number of tasks at the same time. Invariably, some students will not be on the same task as others. The instructor must be able to analyze the progress of all students and help each one with their current tasks.
The instructor will be qualified to teach the course if he/she:
- Has come highly recommended by another faculty or staff member
Can perform all course tasks and objectives without the use of documentation or other aides- Demonstrates, through direct observation, that he/she is capable of differentiating instruction to a group of adult learners
- Has demonstrated proficiency with Apple Remote Desktop
Pilot CourseCreating a pilot group for this class is a difficult task. As a result of the small numbers of students taking this course, it will not be easy to find a pilot group that doesn't already have some experience with the tools.
Optimally, I would like to pilot this course with teachers who have limited knowledge with Kodiak, but experience tells me that they will most likely not be willing to participate. This is because the teachers who value the assessment tools are already using them. Those who don't value the tools will not be interested in being part of a pilot group.
The effectiveness of this training course will be assessed through a number of evaluations. The first of which is a questionnaire that will be distributed to the students at the conclusion of the course. Questionnaires of this type are common in technical training courses because they give the instructor some baseline knowledge that he/she can use to improve the course. The questionnaire that will be used for this course is linked below.
This course will also employ a summative evaluation in the form of a final assessment for the course. Throughout the course, the students will take part in formative assessments that will measure how well they have learned each objective. In order to assess whether or not all of the knowledge has been absorbed, the students will be given an final assessment that ties in to the main course objective. No percentage grade will be given on this assessment, as it is a mastery test. A passing grade will be awarded when the student can perform the final assessment to completion.
The students’ use of their newly acquired skills will be assessed at a later time. Most likely, a survey will be distributed that asks the students various questions relating to their use of these skills. Server logs will also be examined. The students are asked to understand that through the inspection of these server logs, administration is not spying on them, but rather gathering data in order to make better decisions with regard to the training course as a whole. Students of the course will also be interviewed in order to determine if the skills they learned have been retained, and also to gather their opinions on how to increase the retention rate of the material.
Level 1 Questionnaire
The ISD model has one major flaw that doesn’t allow it to be used in all situations. The flaw is that it requires too much time and too many people. I am employed by a school district and we don’t have the time allowances or the budgets to follow a model like this to the letter. The following is a brief redesign of the model that will fit the needs of the Elizabethtown Area School District.
In my district, technology training occurs in two situations. The first is when a session is requested by the district office, and the second occurs on an optional basis in which teachers can earn Act 48 credits. As a result of these facts, an adapted version of the needs analysis stage of the model can be used. Instead of using the needs analysis to determine why the training is to take place, this needs analysis will be used to better incorporate the session’s topic into other aspects of the teachers technology use.
The task analysis phase will be eliminated from this model. In the Seels and Glasgow model, the task analysis is used to map out the steps to a procedure. At EASD, our proprietary software allows the users to perform a given task a number of ways. As a result, we will skip directly to forming the objectives for our sessions so that those objectives can be attacked from a number of perspectives. In this manner of thinking, the task analysis is redundant. For example, if you are looking to perform task X, you must perform steps Y and Z. However, the objective is still to merely perform task X. The steps taken to perform the task will be covered when documentation is developed. As a result of this, the instructional analysis phase will also be eliminated.
In order to qualify for Act 48 credits, the teachers must attend training classes. For this reason, the step in which a delivery system is to be chosen can also be eliminated. All of our session take part in a classroom environment, and there is no room for variation of this fact.
The final aspect of the model that will be changed is that of evaluations. Our training sessions will not use a summative evaluation, since said evaluation will be covered through formative evaluations. The technology that we teach is linear. By that, it nearly impossible to proceed to step two unless step one is mastered. For that reason, formative evaluations will be used, but summative evaluations will not. Also supporting this is fact that all teachers will have the documentation available to them should they experience difficulty with the task.
Graphical Representation of Redesigned Process
Well what can be said of the ISD model? I think that in general, it is a great tool to guide the creation of any program that teaches a concept step by step. When all of the steps are followed, the resulting course is strong and provides the outline necessary for the students to learn the given topic. Teaching in and of itself is an art, but the ISD model uses a formulaic approach to not teach the course, but to to create the course as a whole.
This is beneficial in that it takes the course creator through the process step by step. That's where the first problem with the model rears its ugly head. There are too many steps that require too much time. I'll grant the model the benefit of the doubt on this, as all of the steps are necessary in a general form, but they do take inordinate amounts of time.
From the beginning of this process, I've always felt as though the ISD model is most effective in an environment where creating a course is the main objective. The model works when the overall assignment is to create a course to solve a given problem. By this, I believe that the ISD model is best used when there is some degree of separation. For example, organization X may be looking to solve a problem. That organization in and of itself may have a difficult time using the model. An outside entity Y, however, would have little problem using the model as a basis for creating a course for organization X. There is a degree of separation present that allows entity Y to have an objective view of organization X. Organization X would have a tough time being objective in the matter because the gap they are looking to close is from within.
The biggest problem with the ISD model that I see within a school district is its reliance on people. In an ideal situation, the model requires management, subject area experts, multimedia experts, focus groups, and many other such entities. In a corporate setting where course creation is the goal, there are individuals that feed these needs by way of their employment. It's their job to be an SME. In education, everyone already has their job. Involvement in a project such as creating a training course involves volunteering time, an asset that most educators don't have much to spare. Also coupled with this is that people are hesitant to take on responsibilities that go above and beyond their current job descriptions. Extra work usually ends up becoming a responsibility. The ISD model does not fit well into that paradigm.
Conversely, the ISD model is an asset to any company that's looking to design and sell training courses. Those are the types of organizations that have the assets to create a course based on the ISD model. The final course that is derived from the model is extremely marketable, and closes identified gaps with ease (assuming that the needs analysis was done correctly).
The biggest difficulty that I've had with this model is that I know I will not be able to use it verbatim in my duties at work. The main problem is that I don't know what steps can be eliminated. I have a good idea on what could go, but only time will ultimately tell what adaptations I can make to the model.
All in all, the ISD/ADDIE model will be useful in my endeavors to create training courses. I may not use all of the steps, especially when designing courses with severe time constraints, but I feel that I can use pieces of the model to construct courses that are effective.