Questionmark assessment preparation, delivery, and reporting process

Authoring or preparing questions

1. Place questions in the Questionmark Word Template and PROOF before contacting ETC that they are ready for assessment 8 working days prior to the assessment date for the DDS program. 5 working days prior to sending paper backup copies to the offsite testing centers.
2. If you have numerous media files (pictures, graphs, video clips) please add an extra day.
3. Please use care in naming convention coronary, circulation, chemic, heart, disease Question files using the template can be created by course director, course faculty, or the admin staff.
4. Provide feedback so students know WHY they missed the Avoid giving correct response by itself.

Authoring or preparing assessment

1. We will prepare the exam, using the questions you
2. We will put the draft exam on the QM site for you to
3. The file will consist of the question stem, all the answer choices, correct answer choice, and point value for every question in the exam.
4. Please review each aspect of the assessment and make final edits, if any, using the review feature of
5. We will make the final changes and place this version on the QM site in preparation for delivery to The above steps can be repeated until we get everything right

Assessment Delivery

1. Paper version of the assessment for backup purposes is put under DistributionFiles Admin staff makes copies of paper exam (Dental 10 paper copies and Hygiene 5 copies). Faculty takes copies to the lecture hall where the assessment is to be administered.
2. Official scrap sheet is created by ETC and is put under DistributionFiles folder. This is customizable per department's needs. Admin staff makes copies for every student prior to the assessment day, faculty takes copies of scrap sheet to the lecture hall, and distributes to the students.
3. One of the IT staff will be available during the assessment to assist in any technical problems the students might
4. VERY IMPORTANT: IT staff will NOT PROCTOR any assessments at any
5. The assessment link is on Students logon to Blackboard, and go to the respective course to take their assessment.
6. VERY IMPORTANT: Please DO NOT change the name of a QM exam in your gradebook or make any modifications to the way the exam is configured under the Assessment Folder…this could cause reporting problems with grades or compromise your exam. If you have questions, contact ETC.
7. Students will be provided feedback (including score and the questions that they got wrong) as soon as they complete

Assessment Reporting

1. Within a couple of hours (depending on when the assessment is completed), ETC will provide you with standard reports namely Assessment Overview, Item Statistics, Question Statistics, and Excel spreadsheet that contains every student and every question presented in the exam.
2. If you have to make adjustments to the grades please contact ETC for

VERY IMPORTANT: IT staff will NOT COMMUNICATE WITH STUDENTS regarding grades of any assessments at any time. All grade inquiries have to go through the faculty

Authoring or preparing questions

Folder structure

All the folders reside on \\dentalfs05\QM. Access to this folder is on username and password basis. Also, each faculty will be given privileges to only the course folder that they are a part of Map network drive to access the above folder. You don't have to do this every time depending on where you are working. If you are working from home, then you should use VPN connection and map network drive every time. If you are working from your office, you do this one time. If you need VPN please contact Dental School IT Helpdesk 410 706 2084 or dshelp@dental.umaryland.edu.

Once you are on \\dentalfs05\QM go to your department's folder, your division if any, your course, the semester and year being offered, the exam/test/quiz number, and then you will see the different folders where you can put assessment content, e.g. Raw (for QM template files), Media (for media associated with specific questions). Please follow this organization structure to prevent confusion for all the parties involved. If you don't see the folder structure, please let us know and we will have it ready for you in a few minutes.

Initially only the course director, course faculty (for DDS), the ETC staff, department admin staff will have access to these folders. If you don't have access to your course folder, please let us know and we will take care of it. If you would like other faculty members to have access to these folders then you must let us know and we will give them access.

QM Word template files with questions should be located in this folder DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\RawFiles. The filename should be the name of the major topic area. The topic can be as granular or as global as you would like. We will be using the filename to create the topic on the Questionmark server. Please follow the naming convention mentioned below.

If you have any images/pictures or movie files (without audio) then you would place them in this folder DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\Media. Please follow the naming convention mentioned below.

After the assessment is completed by all the students we provide the course director with some standard reports which will be located in this folder DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\Reports. We give Item reports on Analysis, Question Statistics, and an Overall Assessment View . This will assist the course director in determining the quality of the questions and, if necessary, facilitate making modifications to these questions in the future. If the grades need to be adjusted, please let us know and we will work with you and guide you in the process. Adjusting grades is a manual process.

The draft paper version of the exam which has the question, all the answer choices, the right answer, and the point value for every question in an assessment will be available in this folder DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\DistributionFiles for you to review and make final corrections. This is also the folder where we put the official scrap sheet and paper version of the exam that will serve as the backup for an exam. The course director, course faculty, or the admin staff can make copies of exams from this location.

General folder structure

DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\RawFiles DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\Media

DeptName\DivisionName\SubjectCourseNumber-title\SemesterYear\ExamNumber\DistributionFiles DeptName\DivisionName\SubjectCourseNumber-Title\SemesterYear\ExamNumber\Reports

 

DDS Example:

EPOD\Foun518-FoundationsClinicalDentistry\Fall2007\Exam1\RawFiles EPOD\Foun518-FoundationsClinicalDentistry\Fall2007\Exam1\Media EPOD\Foun518-FoundationsClinicalDentistry\Fall2007\Exam1\DistributionFiles EPOD\Foun518-FoundationsClinicalDentistry\Fall2007\Exam1\Reports

DH Example:

HPP\DentalHygiene\DHYG311-PreventionAndControl1\Fall2007\Exam1\RawFiles HPP\DentalHygiene\DHYG311-PreventionAndControl1\Fall2007\Exam1\Media HPP\DentalHygiene\DHYG311-PreventionAndControl1\Fall2007\Exam1\DistributionFiles HPP\DentalHygiene\DHYG311-PreventionAndControl1\Fall2007\Exam1\Reports

Filename for the question(s) file(s)

The filename for the question file should be the name of the topic. These files will be put under the RawFiles folder. Every topic will have a question file appropriately named because this name is what we will use to create the folder structure for the repository. Even if you have only a couple of questions per topic, create a separate file. Please do not put more than 50 questions per file.

Example:       Amalgam.doc

AlternativeDevices.doc HistologyEpithelialTissues.doc

Question name or question description

The question description is for every question in a question file(s). Please pay close attention to the naming convention. The maximum length of the question name or description is 200 characters.

keyword(s), keyword(s), keyword(s), Faculty initials+0001 Example:  amalgam, matrices, RB0001

alternative devices, home care instruction, orthodontic appliance, EP001

Feedback for incorrect answer choice

Our Academic Dean has mandated that feedback be given to students immediately after the students have finished their assessment. To make the assessment a useful educational tool, we encourage providing students with information so they know WHY they missed the question. Avoid just giving the student the correct response as this just encourages memorizing the correct information rather than learning WHY their answer was incorrect

Filename for the media file(s)

The filename for the media should identify the "QM question filename + the question number".

The media filename should only contain letters, numbers, and a hyphen - or underscore _. Questionmark does not like special characters like # or $ or % or & or spaces between words. You can have more than one media file for a question. Please follow the naming convention.

Example:       AmalgamRJB001.jpg (picture file)

AmalgamRJB001_2.jpg (second picture file for question 1) PhysiologyHistologyEpithelialTissuesRBB007.avi (movie file for question 7) PhysiologyHistologyEpithelialTissuesRBB007-2.avi (second movie file for question 7) AlternativeDevicesEP001.jpg (picture file)

AlternativeDevicesEP004.mov (movie file)

Preparing to Use the QM Word Template:

Before you start working with the QM word template you will need to perform some settings so that the template and the word program will work efficiently.

Find out the version of Microsoft word you have Word 2003 or Word 2007.

Disable the smart quotes in Word 2003 or Word 2007 to prevent formatting characters being carried over to the QM repository delaying the process of creating the assessment.

Install and enable the Macros in Word 2003 or Word 2007 so that your QM word template will work properly.

Follow the filename conventions mentioned above for the question filename. You can get the Blank QM Word Template by clicking on this link. First, save the file on your computer and then proceed to Using QM Template and introduction to the QM toolbar

You can create the basic question types using the QM word template including multiple choice, yes/no, true/false, and likert scale (survey type) questions. Media files cannot be attached using the QM template and will be imported into the questions by the ETC staff. They can be incorporated by visiting the ETC (8102) if you wish to learn how to do this.

Our Academic Dean has mandated that all the questions and answer choices be scrambled. In a multiple choice question avoid “all of the above” or “none of the above”.

If you have less than 5 answer choices for a multiple choice or a likert scale question DO NOT DELETE the information in the angle brackets <…>).

You can have more than 5 answer choices for a multiple choice or a likert scale question. In order to add more choices click on Add Choice in the QM tool bar of the QM template. Again, refer to Using QM Template

Our Academic Dean has also required feedback for the all the incorrect answer choices in all the questions. Please see above for more details.

VERY IMPORTANT - - NEVER EMAIL EXAM CONTENT TO ANYONE---use the QM site to

refer others to your questions for review or exam development. If you are working from home, please use VPN (talk to Desktop support for more details).

VERY IMPORTANT - - Please proof your exam question stem, all the answer choices, correct answer choice, point value for every question in the exam and the feedback before emailing the ETC staff to begin the preparation of your exam

You can email us, your staff or the course faculty about the status of the assessment but DO NOT INCLUDE COURSE CONTENT. Instead, refer the interested party to the QM site.

Authoring or preparing assessments

We will prepare the assessment using the questions you select. If we have any questions or concerns we will contact you.

We will put the draft assessment file under DistributionFiles folder. This draft exam will include the question, all the answer choices, the right answer, and the point value for you to review and make final corrections. This file will also include the official scrap sheet and the version of the exam that will be used to create the backup paper copies for the exam. The course director, course faculty, or the department assistant can make backup copies of the exam and the official scrap sheet from these files.. Prior to the exam paper copies (10 for dental and 5 for dental hygiene) need to be made along with enough scrap sheets for the entire class.

Extra credit questions

You can intersperse extra credit questions within the assessment, so the student will be unable to distinguish between extra credit from the other exam questions. The total score of the assessment will include points for all the questions. Since, the students don't know which an extra credit question is and which is not, they will be answering all the questions.

If you want to keep the extra credit and the regular questions separate, or give students an option of whether they want to attempt the extra credit questions then two blocks within the assessment can be created.

Assessment Delivery

ETC arrives in advance with 4 backup laptops, power cords, and many network cables. These are only available for those student's equipment malfunctions (laptop and/or power cord). Students who do not bring the required equipment must go the Dental School help desk and check it out.

Exams begin promptly at the designated time and all students MUST logon to the exam within a 15 minute “grace” period or take a paper version (graded by hand). Depending on the ETC's schedule, those arriving late maybe able take a makeup QM exam (only if the instructor schedules a room, picks a date, time, student names). ETC provides technical support during the exam and makes sure all grades were processed electronically by having students check with an ETC member once they have submitted the exam.

Course Directors are responsible for bringing backup exam copies and scrap sheets to the assessment. They proctor and obtain additional proctors in instances where assessments are delivered in multiple rooms. The Associate Dean for Academic Affairs prohibits all ETC staff from serving as proctors. If proctors are not available, the assessment will need to be re-scheduled. Course directors collect the scrap sheets once students have checked out with the ETC staff. They also administer written exams for students with technical issues or that arrive late (after the 15 minute grace period). Course directors are responsible for reminding students to “keep their eyes” on their own work and enforcing issues related to violations of the Judicial Policy.

Assessment Reporting

Upon completion of an assessment, including all make ups given using QM, reports will be placed within the Reports folder for each assessment including the Assessment Overview, Item Analysis, Question Statistics and an excel file containing details of every student. Contact ETC for any grade adjustments.

Frequently Asked Questions

Installing Questionmark Word Template

Create a folder called something like "QMWordTemplate" and save this attachment. After you start entering the questions, use the File->Save As and save the file with a different name (may be use the topic name) to make sure you have a copy of this blank template.

Open any word document. Go to Tools menu, click on Options, click on Security Tab, then click on Macro Security ... button, make sure you are in the Secuity Level Tab and select Medium. Now click on the Trusted Publishers Tab and make sure that both check boxes are checked. This completes the installation. You will have to close the word document and open the template. Everytime you open the template you will get a Security Warning dialog box with the option to Disable Macros, Enable Macros, and More info. Always click on Enable Macros. And now you are ready to put your questions into the tool.

Different Question Types available in Questionmark Authoring Manager

In addition to the traditional True/False, Multiple Choice, Yes/No, and Likert scale questions, the following are also available upon request using Questionmark Authoring Manager. Keep in mind that images, graphs, and video clips may be added to most question types.

Fill in the blank(s):

• Text is entered into the blank space(s), with each space scored separately. Case sensitivity can be set.
• May have one or more words per blank.
• Words have to be spelled correctly by the participants to obtain points. You may add variations of the spelling (e.g. for Annapolis, may allow annapolis, anapolis, annapollis) and the word order (e.g. Annapolis Maryland or Maryland Annapolis), but unless one of these variations is selected, NO credit will be given.
• This question type requires recall of the information and correct spelling (or variations) unlike a multiple choice question where students can eliminate choices or identify a response they have seen before. You may have partial or "all or none" grading.

Text Match/Short Answer:

• Similar to fill in the blank but has only one blank. May have one or more words.
• Words have to be spelled correctly by the participants to obtain points. You may add variations of the spelling (e.g. for Annapolis, may allow annapolis, anapolis, annapollis) and the word order (e.g. Annapolis Maryland or Maryland Annapolis), but unless one of these variations is selected, NO credit will be given.
•  This question type requires recall of the information and correct spelling (or variations) unlike a multiple choice question where students can eliminate choices or identify a response they have seen before.
• Item may be "unscored" and used to collect information or scored manually.

Hotspot question type

• Need an image where an area needs to be identified.
• Need a marker image to point to the area that needs to be identified. Only one hot area can be set in this question.
• Placement of the marker may be set by the mouse to determine the area that will be used to grade the item. For multiple hot areas use the “Drag and Drop” question type.

Drag and Drop question type

• Similar to hot spot question.

More than one hot area may be used.

• A different marker is needed to identify each hot area.
• You may have partial or "all or none" grading.

Knowledge matrix

• Several multiple choice questions presented together. Questions are graded separately.
• Similar to Pull Down.
• Good choice where all questions pertain to topic, e.g. viewing an image for multiple characteristics or an abstract.

Pull down

•  Similar to knowledge matrix allowing multiple questions pertaining to information to be presented, in this case using a video clip.
•  Answers appear in drop down for each question.

Select a blank

• Similar to pull down.
• Used to present single question and different format for a multiple choice question.

Matching

• Similar to knowledge matrix or pull down allowing multiple questions pertaining to information to be presented.

Case Based

• Requires students to apply diagnostic ability based on patient history, chart, radiographs, intra- or extra-oral photos.
• Ask ETC staff for examples of this question type.

How to use QM Word Tool/Template

To add a question – click on the down arrow next to Select question type and choose the question type you want to add. This will provide you with a template for the question. You can type or copy & paste text into the specified area. The text does not have to be within the angle brackets <>. You can only create true Multiple choice, True/False, Yes/No, and likert scale type questions with this tool/template. For any other question type you will need QM Authoring Manager.

To delete a question – click on the Delete question button, enter the question number and confirm the delete.

To add a choice to the question – click on Add Choice button and you will be provided with the template to enter your answer choice. You can only add choices to Multiple Choice question type and Likert Scale question type and not to True/False or Yes/No question types.

Publishing to Questionmark – this step will be done by the Instructional Technology Specialists. After you are done with entering the questions in the tool/template, send the file (or the first few times you bring the file to us) with details as to where the questions should be put in the folder structure. And we will take care of the rest with your input.

What does a Blocked Assessment in Questionmark mean?

You have already seen a blocked assessment in QM.  The first block is usually a set of questions and the second block is usually your Judicial Policy.

You can go back and forth within a block, change answers, flag a question, etc.  To move to the next block you have to click on the submit button.  Once you click on the submit button you cannot go back to a previous block.

Some assessments might have more than just 2 blocks.  These blocks might have the basic multiple choice, true/false, yes/no types of question or they might have a different style.  One of them provides you with content and based on that content you will be asked to answer questions. In this case, you can go back and forth between the written material, e.g. case or abstract, and the questions pertaining to the case or abstract.

In ALL cases, clicking SUBMIT means YOU ARE FINISHED WITH A BLOCK OF QUSTIONS AND CANNOT GO BACK TO THE BLOCK YOU HAVE SUBMITTED.  So do not submit a block of questions unless you are finished reviewing your questions and making your final changes.

NOTE:  Except for the Judicial Policy, all the blocks are included in the timed assessment.

Being Prepared for Taking Questionmark Exams – Hardware and Software

Prior to taking QM assessments, students are responsible for checking their own technology (both hardware and software) to make sure everything is working.  This includes: laptop, network cables and power supplies.  Additionally, it is the student’s responsibility to go to the following sites on Blackboard to make sure they have the latest version of Questionmark Secure or to load it and take the Mock Exam to make sure all technology is working properly before arriving for assessments. 

ORG - Educational Technology for DDS Students

ORG - Educational Technology for Dental Hygiene Students

If a student is aware of technical issues prior to taking the assessment, these issues need to be resolved though the IT Help desk, room G-426 or DSHelp@umaryland.edu  or by calling 6-2084.  DO NOT WAIT UNTIL THE DAY OF THE ASSESSMENT TO RESOLVE HARDWARE AND SOFTWARE ISSUES.  A limited amount of replacement equipment may be checked out to use during the assessment if a student’s technology is not working properly.

ETC will have a limited supply of loaner equipment available during assessments to “swap out” equipment for those students who bring their laptop, internet cable and power supply to assessment but end up having equipment failure during this time.

Students presenting at an assessment without their laptop, network cable or power supply will be required to use assessment time to go to the Dental School HELP desk to obtain loaner equipment.  For these students, the exam will end at the end of the class time regardless of the amount of time showing on the QM assessment.

• Exam Instructions PPT - a PowerPoint to assist students with navigating through the exam and dealing with error messages.

• Scrap Sheet - You will be provided with an electronic copy of the Scrap Sheet file tailored to your exam by ETC. You must print out enough copies as needed for your entire class. At the discretion of the instructor, this sheet may being used to take attendance at your exam. Each student MUST sign the scrap sheet and return it at the end of the exam.

• Video Surveillance may be used during examinations