QTI version 2.1 final released.

Version 2.1 of the IMS Question and Test Interoperability (QTI) specification was released on 25 January 2003. It's taken a long time to get QTI 2.1 completed, however now that is has finally released hopefully there will be a significant uptake. It has been a long process, with a very large number of people involved over the years. QTI 2.1 is really completion of QTI 2.0, so the process of developing it has been continuous since the start of the 2.0 working group several years ago. QTI 2.1 adds support for tests, as well as enhancing the support for individual question types.

Further work will be needed to develop a range of profiles that specify subsets of the full specification that fit the requirements of specific groups of users. An entry-level profile already exists which duplicates functionality of the Common Cartridge profile of QTI 1.2, but is much simpler than the older profile as benefits from QTI 2's response processing templates which removes the need to individually specify duplicate response processing rules in each question where a single marking algorithm is to be used. This entry-level profile should be easy to import into almost all existing electronic assessment systems, however it is a very small subset of QTI 2.1, and does not provide any functionality that is not already widely available. Further profiles that address the needs of higher education are still needed.

As we came towards the end of the development process, we also realised that there were many things that we would have liked to include given unlimited time and resources. Hopefully we will be able to develop some of these ideas as extensions which can later be merged into a future iteration of the specification. One which I would like to see is the ability to define graphs generated from templates, and taking input in the form of lines or points being added. In the time since we started QTI 2.0 web technology has moved on, and with HTML 5 this type of interaction should be very feasible. I am glad however that we made many of the decisions about the specification before HTML 5 arrived, because the XHTML base provides all of the flexibility needed, whilst being simple enough that it does not create a great obstacle to the specification being used with delivery systems other than a web browser.

Niall

CETL-MSOR Conference / New Partners

Following on from CAA, we put in an appearance at the CETL-MSOR Conference in Sheffield. Although we were not presenting, it was useful to join the meeting and hear about delegates' experiences in assessing mathematics, statistics and operational research.

A recurring theme was the need for assessment resources in statistics in service teaching across a wide variety of disciplines and at all levels. A selection of resources which would begin to address this need will be rebuilt from the CALMAT question collection for a Glasgow University module this autumn.

We were able to add another informal partner to QTI-PET following a lunch-time discussion during the conference. So during the week we welcomed 5 new partners on board, bringing us to a total of 12 QTI-PET partners who will help us to pilot the tools and feed back to us about their experiences. We are arranging a series of training sessions to get everyone up to speed with the tools in time to get some material in front of students this coming semester.

QTI @ CAA 2012 / new Uniqurate release

Last week's participation in the 2012 International Computer Assisted Assessment (CAA) Conference was a great success for all of the QTI projects. QTI was clearly positioned front and centre throughout the conference, with stream A on Wednesday almost exclusively devoted to it. The closing speaker, Paul Bailey of JISC, went out of his way to stress the importance of QTI and open standards within the field of electronic assessment, and to talk about JISC's commitment to this area of research.

The conference was preceded by a QTI Workshop on the Monday, which was most productive, particularly from the perspective of Uniqurate in that it allowed us to get some new faces using the application - and exposed some new bugs! Consequently, a new version of Uniqurate has been released today that addresses the following bugs:

• Adding an image into a question causes invalid XML to be generated
• Using IE causes invalid XML to be generated (regardless of images!)
• Inconsistent behaviour of watermarked input fields - sometimes the watermark remained when you tried to type into the field, meaning that what you got was a mismash of what you wanted and the watermark text

The version of UQ released today addresses these issues and represents quite a significant change under the hood, which should hopefully resolve any remaining inconsistencies across browsers in the XML that gets produced. As a bonus, while I had the image insertion code open, I also added the ability to not only add an image from a URL on the public internet, but also to push an image from your local hard disk and have it bundled as part of the eventual question content package. (This itself meant a big change under the hood with respect to the way the app handled content package manifests, but you end users don't care about that!)


Upcoming work will probably be to add a component that gets the student to place/drag items into a correct order - this seemed to get some appreciative nods for a "next up" suggestion, but if you have any other ideas, particularly if they're cross disciplinary in nature, fire them at me!

Enhanced item rendering in QTIWorks

After another flurry of activity, I've just finished and released the 8th development snapshot of QTIWorks for folk to play around with:

https://www2.ph.ed.ac.uk/qtiworks

This iteration of work has focused mainly on the delivery of single assessment items to candidates, joining up various bits of pipework that control this process and adding in some new rendering features and options that we'll need later for test rendering (and are quite useful in their own right too).

I'm afraid you'll still have to make do with playing around with the sample items that have already been loaded into the system for the time being. However, you'll probably be pleased to hear that the next work iteration will add in the first round of functionality for getting your own assessments into the system and doing fun things with them. (At last!) To make the wait a bit easier, I have gone through the existing samples and selected a reasonable set of options for how they should be delivered, which makes them much more fun to play with, as well as more illuminating to people who are less familiar with QTI.

Key changes in this release (1.0-DEV8)

• Assessment item rendering now incorporates the following new features:

1. Display of a model solution (via the QTI <correctResponse>).
2. The ability to reset a session back to the state it was in immediately after the last run of template processing, which effectively clears all candidate input back to the original state, but leaves randomly-chosen things intact. Existing "reinit" functionality has been made a bit clearer.
3. An explicit "closed" state, which is either entered explicitly by the candidate, or when an adaptive item becomes complete, or when the number of attempts hits your chosen value of maxAttempts. I think this will need a bit more work, as it was never implemented in QTIEngine or MathAssessEngine.
4. A new "playback" feature lets the candidate step through every interaction they made with the item, which is a possible way of implementing the allowReview concept from test rendering.

• A number of "knobs" are now available for controlling how you want to render a single item, including:

• maxAttempts (as seen in tests)
• author mode on/off (controls whether authoring hints are shown)
• a simple prompt to be shown at the top of the question (similar to rubrics in tests, but simpler)
• restrictions on what the candidate can do, including:
• close a session explicitly when interacting
• play a session back when closed
• reinitialize a session when interacting or when closed
• reset a session when interacting or when closed
• see a model solution when interacting or when closed
• see the result XML
• see the item XML source
• I have gone through all of the existing samples and set the above "knobs" to values that show each in their best light. For example, some suit a very formative "try as many times as you like" approach with rich feedback, others are more austere so are better displayed in a more rigid way. This should make the samples much more useful for people trying them out.
• The authoring debugging information has been improved, and now shows bad responses (e.g. string submitted to a float), and invalid responses (e.g. wrong number of choices made).
• There's now a rich database structure underpinning all of this, which records everything the candidate does and the changing QTI state during this process. This is currently used to implement the "playback" functionality, and will prove invaluable for analysing result data when the system delivers real tests.
• The HTTP calls that control the typical delivery of a "candidate session" via a browser are now as RESTful as you would pragmatically expect in this type of scenario. A more formal RESTful web service API will be trivial to do from this, but I'm going to hold off until anyone actually needs it.
• HTTP responses have been improved so that they always include the Content-Length headers. Responses that don't change also send cache-friendly HTTP headers such as ETag and Cache-Control.

Short term development roadmap

• Next up is adding in the web-facing functionality for creating your own assignments into the system, uploading content into them, validating them and trying them out (using the various "knobs" I listed above).
• After that it's finally time to implement the entry and exit points for candidates who get sent to QTIWorks to do an assessment. This is where LTI will come in.
• Then... tests. At last!

And relax.

Showcasing QTI 2.1 in QTI Works

As well as being a tool for managing, trying and delivering your own QTI assessments, QTI Works will also act as a showcase for QTI and the things it can do, much as the existing QTIEngine and MathAssessEngine already do.

With this in mind, I have started to feed Graham Smith's excellent collection of QTI examples into QTI Works, and plan to continue this process over the next few months while it takes shape.

These examples will be bundled into development snapshots of QTI Works for you to play around. So far, I have assimilated 3 sets of examples:

• The IMS standard examples (and a few extra bits)
• Examples demonstrating the MathAssess QTI extensions
• A small set of example items from language testing (German & Russian)

You can play around with these at: https://www2.ph.ed.ac.uk/qtiworks

(Yes, it does look very spartan at the moment, but don't worry about that!)

This exercise is actually very useful for this project for a number of other reasons:

1. It provides me with some sample data for running automated "integration tests" on the JQTI+ / QTI Works software. (For example, I feed them all of the code performing the various bits of QTI logic, such as reading in XML, validating the data models, writing out XML, running template processing, running response processing etc. This is invaluable for finding and fixing bugs, and making sure we can handle "real world" examples properly.) 
2. As well as being useful for integration testing, they also help with so-called "regression testing", which helps make sure that I don't break things inadvertently during the development process.
3. These examples have been around for a few years now, so this process is a good way of doing some QA on them and making sure they're right up to date with the QTI specification.

Enjoy!

Announcing QTI Works

Development work during the first few months of the QTIDI project has been progressing successfully, though you would certainly be forgiven for doubting this! As well as me being notoriously bad at finding time to write blog posts, the first development iterations on QTIDI have focused on "refactoring" all of the QTI-related software components we have been building and using on recent JISC projects so that they make a much better foundation for the technical goals of this project. (Rubbish analogy time: I've taken lots of random things, laid them on a carpet, and hit them with hammers until they break into little pieces. I'm now joining these pieces up to make something nice. Hopefully. The analogy is rubbish as I can't think of any examples where I've made something nice out of random bits on a carpet. Luckily, I'm better at doing this with software, so probably shouldn't have used this analogy in the first place.)

One of the key bits of work done so far has been a redesign of JQTI, which is the Java software library for "doing" QTI stuff that Southampton developed a few years ago. I've previously blogged about why I thought this is necessary (see http://davemckain.blogspot.co.uk/2011/06/refactoring-jqti-jqti.html) and the result of this is coming together under the not-very-original name of JQTI+. This refactoring work is now almost complete, with the exception of QTI tests, which JQTI never quite implemented fully and will be revisited in a few months.

On top of JQTI+, I'm building the replacement for MathAssessEngine that will become the main technical deliverable of this project. MathAssessEngine, which was based on the original QTIEngine, is also going to be torn apart and redesigned so that it can do all of the things it now needs to do, and do them all really well.
To reflect the scope of the work we're doing, we've decided to give the replacement for MathAssessEngine a completely new name and, after a couple of months of riotously bad naming attempts, we've decided to call it QTI Works.

I will deploy regular public development snapshots of QTI Works while it takes shape over the next few months, which you will be able to find at:

http://www2.ph.ed.ac.uk/qtiworks

If you remember to wear a hard hat, you can go in and try the first development snapshot now. This showcases the brand new (and vastly improved) QTI 2.1 validation functionality in JQTI+, as well as demonstrating the newly-refactored rendering and delivery of QTI 2.1 assessment items. (You'll have to make do with a selection of sample items for the time being... the functionality that will allow you to upload, validate and try out your own items is still written on the back of some envelopes. This will turn into real code that you can play with during the next development iteration. Hopefully!)

Demos at the CETIS Conference

QTIDI and its sister project Uniqurate were represented at the CETIS Conference in Nottingham in February 2012. We demonstrated

• The Learning Tools Interoperability (LTI) Connector running a Common Cartridge test;


• The Uniqurate question authoring tool, which also has facilities for creating a contant package containing a question and its associated media and/or stylesheets;


• Interoperability between our tools and those of colleagues from Germany, France and Korea as well as the UK, using the latest version of MathAssessEngine, which is currently morphing into QTI Works.