QTI version 2.1 final released.

Version 2.1 of the IMS Question and Test Interoperability (QTI) specification was released on 25 January 2003. It's taken a long time to get QTI 2.1 completed, however now that is has finally released hopefully there will be a significant uptake. It has been a long process, with a very large number of people involved over the years. QTI 2.1 is really completion of QTI 2.0, so the process of developing it has been continuous since the start of the 2.0 working group several years ago. QTI 2.1 adds support for tests, as well as enhancing the support for individual question types.

Further work will be needed to develop a range of profiles that specify subsets of the full specification that fit the requirements of specific groups of users. An entry-level profile already exists which duplicates functionality of the Common Cartridge profile of QTI 1.2, but is much simpler than the older profile as benefits from QTI 2's response processing templates which removes the need to individually specify duplicate response processing rules in each question where a single marking algorithm is to be used. This entry-level profile should be easy to import into almost all existing electronic assessment systems, however it is a very small subset of QTI 2.1, and does not provide any functionality that is not already widely available. Further profiles that address the needs of higher education are still needed.

As we came towards the end of the development process, we also realised that there were many things that we would have liked to include given unlimited time and resources. Hopefully we will be able to develop some of these ideas as extensions which can later be merged into a future iteration of the specification. One which I would like to see is the ability to define graphs generated from templates, and taking input in the form of lines or points being added. In the time since we started QTI 2.0 web technology has moved on, and with HTML 5 this type of interaction should be very feasible. I am glad however that we made many of the decisions about the specification before HTML 5 arrived, because the XHTML base provides all of the flexibility needed, whilst being simple enough that it does not create a great obstacle to the specification being used with delivery systems other than a web browser.

Niall

A significant date, and some reflections on the Uniqurate project

Hi all,

I just wanted to post to mark a significant date - today represents my last officially contracted day on Uniqurate, and thus the formal end of the development phase of the project. I am still working on the technical documentation, and I'm sure I'll also contribute to the final reporting process, but as far as actually being employed to work on Uniqurate, to coin a phrase: that's all she wrote!

I wanted to take the opportunity to say what a pleasure it's been to work with the project team and the wider community. It's always a joy to work with the usual QTI-heads, but what really made this project for me was the wider involvement from not only our official partner institutions, but also the extended community that's begun to grow as a result of the efforts on QTI-PET. I think these fresh perspectives really have helped further the cause of open standards for e-assessment, and for QTI, and we've seen a greater maturity in the final application as a result.

So in this post I wanted to take time to reflect on Uniqurate - not the application itself, which I covered in my last blog entry; this time, I want to talk about the project itself - particularly the role of Agile methodologies.

There have been some interesting lessons learned from my perspective, particularly speaking as someone who is usually an advocate of Agile for managing software development projects. I have taught Agile approaches to students on Software Engineering courses, and used them to excellent effect in the past both in software development and for general project and time management. I am a fan of Agile. I agree with the Agile Manifesto and its principles. However, during Uniqurate something became clear to me: 

Agile is not always the best option for a software development project in HE.

Back in May 2011, when the original call for projects went out from JISC, the Technology Transfer strand specified that projects should proceed according to a "rapid, open or agile methodology". Previous QTI projects on which I had worked had used Agile approaches and their eventual software artefacts distributed under open source licensing, so the established team members were very much in their comfort zone, including me. As the person who took point on writing the response to the call for Uniqurate, naturally it detailed a process that was very much focused around iterative development cycles, with an ongoing, continual process of feedback from end-users and response from the software development team - the response from the latter coming in the form of changes to the application.

We got underway with a flurry of activity and enthusiasm from all concerned. Once the teaching semester started, though, the realities of peoples' day jobs intruded. The level of interaction was inevitably reduced particularly among project members were confronted with teaching commitments and the need to support their students. With the project extending over a comparatively long period, it became relatively for a demographic who could reasonably be described as "harassed, overworked academics" to put it on the back burner.

In DSDM Atern, there is a pre-project questionnaire designed to ascertain the suitability of DSDM. Although one would need to translate the various DSDM-specific terms used, this document applies just as well to Agile approaches in general. Among other things, it stipulates that easy access to key representatives of the user community must be available to the developers throughout. It also stipulates that key members of the team "buy in" to the Agile approach.

Although I believe we had a successful development process, I am not sure that it was truly Agile; neither am I sure that had we undertaken DSDM Atern's pre-project questionnaire at the outset that we would have been truly able to green-light an Agile approach. Past projects had involved people enthusiastic about pushing the boundaries of e-assessment and open standards, many of whom were not directly involved in teaching or only had limited teaching commitments. On this project, we were actively seeking involvement from those whose "day job" was teaching. Thus, there were long periods during which we had little or no engagement from the user community. This was nobody's fault, let me state that clearly for the record; it was simply the reality of an academic role in British HE. Naturally, when the semester breaks occurred and people emerged for air, suddenly the enthusiasm and engagement reappeared and the lines of communication reestablished.

In hindsight, I also wonder whether or not our project team truly understood the Agile process - another tickbox we perhaps would have failed to tick on the DSDM paperwork. This was revealed to me during some of the correspondance leading up to the recent Programme Meeting. One of the team revealed that she'd not felt comfortable exposing academic members of staff to the application in the early stages, as it was limited in nature and scope at this point, and she did not want to put them off. Belatedly, I realised this had also been a contributory factor in perhaps not receiving the constant user feedback we'd have liked. I had perhaps been remiss in explaining one of the crucial tenets of Agile - that user involvement and feedback occurs throughout, particularly in the early stages of development - - think of it like a child's formative years!

Ultimately, we might say we did a two phased "big design up front" project instead of Agile. The initial enthusiasm and available of project members meant that when direct feedback dried up, I was able to extrapolate what I had into something approximating an overall specification for the application. I ended up developing this over two cycles, one leading up to CAA, then another leading up to when I actively finished development, around September (i.e. when my own teaching commitments started to intrude!).

I do wonder, however, what would have been if we'd actually set out to do this. Given the early availability of project members we could have created a formal specification as the very first task. In the actual event my extrapolations worked, but there were times when I felt I was making it up as I went along.

Armed with a good spec, we could then send academics back to the trenches while the software developers did their thing - perhaps having people resurface in the period between semesters and perhaps again at the start of the summer to nudge things back on track if they'd gone awry. Some might actually say this would actually still be Agile - it is still iterative development, after all, although I'd submit the sprints/timeboxes would be far too long to truly fit that description. 

Whatever moniker one would choose to attach to such approach, I think it has merit for future JISC projects that predominantly involve software development. I can't help but wonder what kind of application Uniqurate would have been in that alternate universe where JISC hadn't specified thou shalt be Agile, and we formally set out with a big design up front approach.

Tl;dr* - we need to be more agile about whether we choose Agile for managing software development on e-learning and academic-led projects.

------------

For those who don't speak meme: Tl;dr - "too long; didn't read", or "the short version"

Uniqurate: new release... the last big push of development

Today's release of Uniqurate represents what will most likely be the last big push of software development - at least development that will take place under the auspices of the Uniqurate project itself. Doubtlessly there will be bug fixes still to come, plus some little bits to come in QTI-PET down the line, and I'm sure there will be more development in the future on other projects and by others in the community. However with the start of the teaching semester next week and development officially ending in November, it's the end of the big stuff for now.

Without further ado, here's what's new:

• Multiple choice component:
• No feedback is now a valid scenario. A response will turn green if both feedback and distractor (answer) text has been filled in, orange if the distractor only is filled in (but which is now valid), and red when nothing's been filled in.
• There is now an option to copy feedbacks. This means that if you don't want to have individual feedbacks for all your "wrong" answers, you can fill in one of them and use the Quick Feedback button.
• There is a checkbox that will present a multiple choice component as a pull down list when delivered to the student. This might be useful for questions with lots of distractors!

• Tests
• You can now click on the edit icon next to a question edit that question in situ within the test.

• General
• There are now options to copy a component, and to move/drag a component into a different position within the question. Dragging is also generally improved (e.g. it will scroll properly if you drag a component towards the button or top of the browser window)
• The much requested option to put feedback immediately after the component (rather than at the end of the question) is now in place. The slider at the top of the component pane toggles between Feedback shown with components and Feedback shown at end of question. The first option means that feedback will appear immediately underneath the component to which it applies. The second option will display all feedback at the bottom of the question.
• Far too many bug fixes to list - but includes workaround for the Firefox issue where scroll bars weren't appearing on the QTIWorks preview.
• Probably many interesting and exciting new bugs introduced :-)

It has to be said that the recent demo/workshops have been invaluable in providing me with a distillation of end-users' needs and issues. A special shout out to those who participated in the QTI-PET session in Oxford - much of what's gone into today's release came out of that session. Bug tracking and fixing in particular, while not a glamourous activity, absolutely depends on end users using the software and feeding back to the developer(s). Being physically there and part of the demo session gave that process an immediacy and dynamism that is lacking when reduced to emails bouncing back and forth.

So, with the bulk of software development done, and if you'll indulge me, I just want to take a brief moment to reflect on Uniqurate. 

Back at the end of last year, I had a very different vision for this application. The original intention was to create a loosely coupled set of mini-editors - i.e. one for a multiple choice, one for a a text entry, and so on. I also bandied about the term "de-maths-ing QTI". The intent there was that any questions likely to involve anything vaguely arithmetic would be generalised, the edges hammered off, and used as a baseline to create yet another mini-editor. We spoke about identifying questions with cross-disciplinary utility, so as to avoid those proverbial edges being too sharp and thus not needing that much hammering in the first place!

The real quantum leap for Uniqurate came just after Christmas. A discussion with Sue M early in the development of what came to be known as "friendly mode" saw the mini-editors become "components". The crucial difference was the concept of having multiple components per question. 

Laying the foundations to support this took some time, and was frustrating from my perspective - early releases of UQ looked nice but didn't do very much!  However, contrast what we now have with the original vision. You can create a question in UQ that tests the student's knowledge of a subject by having them engage in a variety of ways. The custom maths component which offers the potential for some very rich feedback scenarios.

The change of approach meant that we took a bit of time to get rolling but in the end, I think we have a much better application for it. 

I am really pleased with Uniqurate, even if I do say so myself. Certainly it goes light-years beyond what I envisaged at the start of the project. The short version of the Uniqurate project proposal was that Aqurate was too simple and Mathqurate too complex; our challenge was to create something in between. I think we've succeeded in that aim, but along the way something unexpected has happened: Uniqurate is in some respects more complex than Mathqurate, at least in terms of the content that one can produce. I think that's a good, no, a great thing. It means that those new to QTI can dive in and create some "clever" content without ever seeing a scrap of XML. In the long run, it can only help to encourage the adoption of QTI outside of the existing community.

Testing times on Uniqurate

No, we're not having problems on the Uniqurate project, quite the opposite in fact - today's release adds test authoring. Sorry, I couldn't resist the headline :)

You will find that the button on the main menu to create a test is now enabled, and when used, you'll observe that the edit option changes to reflect that you have a test rather than a single question in memory.

Tests are divided into sections, and each section contains a number of questions. For the simplest use case, you can just leave all the default settings alone, click the Add question to section button a few times and select some questions. Give it a title, possibly fill in a bit of explanatory text that the student will see, and then save the test. Your work should then work in any delivery system compatible with QTI 2.1 tests.

However, Uniqurate does go a little further than that. You can have multiple sections in a test, and select random questions from the section so that no two deliveries of the test are the same. You can repeat a question within a section - useful if you want students to try several times with different randomised values (of course, this assumes that the question has randomised values!). You can even deliver different sections to the student depending on how they've done on the previous sections.

The test screen is a little busier than the question authoring screen, so when time permits we'll try and put together some documentation that talks people through the various options. In the meantime, however, the various little icons scattered around the screen will pop up help about the option nearby.

Unfortunately, QTIWorks does not yet have tests up and running. JAssess should run tests fine, however, and as long as you avoid the maths components of UQ, the venerable MathAssessEngine should also run them too (although it does have a bit of a problem displaying the question feedback).

Meanwhile, people, as its author I hereby declare our old test authoring tool Spectatus officially obsolete! Awww. Please join me in a minute's silence for that worthy warhorse :-)

Uniqurate - the final thrust (ooer)

Following the recent RSC workshop, there was a release of Uniqurate that included quite a few bug fixes and a few feature requests. Most of the bugs should have been fixed (and I've probably introduced a whole bunch of new ones, but hey :-) and I've added some of the new features. However, there were a number of ideas for new components that emerged.

We've been fortunate in that around the CAA conference onwards I've been available to work on UQ more or less full time, and as a result we've made a lot of progress in July and August. However, the bad news is that my involvement in Uniqurate ends in November, and along with the fact that once we get into September and teaching commitments rear their ugly head, this quick burn development is going to be curtailed. Short version: there is probably enough time remaining to get test authoring done and perhaps one other component.

So I'm going to put the possible other component to a vote. Feedback from the RSC workshop, plus components that have been mooted in the past, leads us to the following list - numbered for reference, but not in any particular order:

1. An "Excel" component - essentially a grid of (random) numbers, some of which will be blank, and the student has to fill in the blanks. The blank values would be derived from the non-blank values.
2. A Medicine Component for questions where you have a list of medicines and a list of "strengths" for each - the question would choose randomly a drug and a strength to set up a calculation (but I'd need a little more explanation here :)
3. "Fill in the blanks" within text - "blanks" could be either a text field into which the student types, or a pull down menu of choices
4. Diagram labelling - drag labels onto the right place of a diagram. Potentially, drag bits of a diagram into the right place of a diagram.

These aren't components, but feature requests, any one of which would probably use up the non-test development time:

5. Option to place feedback alongside the input it refers to
6. Confidence-based marking
7. A random wrapper - i.e. you define a group of components, and at question run time it chooses a random component within this wrapper to show. Thus you could set up a variety of different components but with related content - e.g. several maths components with slightly different variations of a concept, MCQs with different but related distractors, etc.

To help you decide, here's some thoughts about these from my perspective:

3 and 4 are eminently doable. Although questions with graphical interactions seem to get good responses from students, the scope of the sister QTIDI project means that we're probably stuck with the legacy Java applet-based graphics for now at the renderer end of things. So if it came down to 3 or 4 my vote would be 3, to concentrate on the component that would result in the best experience when it was finally rendered.

I understand the utility of 5, but it would mean a fundamental change in how the "friendly" mode composes its content, so it would be very complex to do and would burn up a lot of development time for something that is basically aesthetic.

1 and 2 would need to be fleshed out a lot more, and I think there are others with greater cross disciplinary utility - but I'm game if people vote them highest!

6 is often brought up as a popular choice - again, game if people vote it highest.

However, my own vote would be 7. I think it would be relatively easy to implement both from a UI and QTI perspective, yet add a new dimension of flexibility to questions. You could specify a whole raft of similar but subtly different components around a given subject, and really put a student through their paces (although, arguably, one could do this at test level).

Over to you, guys - what do you think?

Custom maths component for Uniqurate

I am very keen to see QTI used in other disciplines than maths, and I believe it's an important aspect of the current projects - I've specifically used the term "de-mathsing"in the past.

However, the reality is that even in non-maths disciplines people still need e-assessment resources that can handle some element of maths. We may not be talking about the complex, computer algebra system-enabled level seen in some of the MathAssess resources - ultimately, my feeling is that creating such rich content will most likely always be the purview of Expert Mode and those individuals who can craft QTI resources within it. Nevertheless, the ability to specify a simple expression and build a question around it spans many, if not most subject areas, and it is not unreasonable on the part of the user community to expect their e-assessment tool to support this.

In the early days of Uniqurate I'd expected much of this need to be served by individual quick maths components that we would identify across a range of disciplines - much as was the case with the quick maths triangle. However, most of the feedback emerging of late was more generic - a need for "more maths" was being articulated without very much in the way of detail. I wanted to scratch this itch but avoid travelling down the same path as Mathqurate. Thus, a general "simple maths" component started to formulate in the back of my mind, and I'm pleased to announce that the release of Uniqurate that I deployed on Friday introduces this new functionality.

You will observe that the components are now divided into three tabs, Text, Maths and Other (as the list of components on the left side was beginning to disappear off of the bottom on my MacBook screen :-) In the Maths tab alongside the existing "triangle" component is our new maths component. When dragged onto a question canvas this results in a fairly simple component that just shows Question Text and Answer, but clicking on one of those fields will pull up a new window that looks like that below:

The screenshot above shows a good example of how this component may be used. Here, we have a question around the old chestnut pi r². It might be argued that this is actually a trick question - we're specifying our radius in centimeters but actually want the area in millimeters! This lets us showcase the different feedback that can be set up. Note that there is a correct and "non-correct" answer - the latter being what I might refer to as wrong, despite others out there hating such a pejorative term :-) However, you can also add custom feedbacks which will be triggered on alternative answers. Our example above shows only one, but you can have as many as you like.

The example shown involves only one variable (pi doesn't count, it's a constant, along with e) but you can have as many variables as you like - the table underneath the answer field will expand to accommodate them all. Our example is also using explicit values in the min and max fields - but you can use variables or even expressions there. The constraint field also gives you further control, by specifying conditions that must be met when the question runs - you might use this to avoid zero values in variables, for example, or to ensure that a variable value makes sense in the context of the value of another variable.

Even if I do so say so myself, I am very pleased with how this component came out. Initially, I was not expecting to be able to achieve the level of flexibility available in the final version - certainly the use of expressions in the min and max fields was not in my mind at the beginning, and even the use of variables in those fields I considered problematic. The whole thing was opened up courtesy of a chunk of code I created a couple of weeks or so ago that converts a sensible, human-readable maths expression into QTI. So, if one pipes

(pi*r^2)*10

into this function, one gets


<product>
    <product>
        <mathConstant name="pi"/>
        <power>
            <variable identifier="r"/>
            <baseValue baseType="integer">2</baseValue>
        </power>
    </product>
    <baseValue baseType="integer">10</baseValue>
</product>

straight back. Regardless of the complexity of the expression - whether it is just a single, explicit value or a six line, bracket-tastic epic, what you will get is a QTI excerpt that can be injected into a question wherever an expression is expected.

I won't go into more detail; this is perhaps a little more techie-oriented than is the norm on this blog (or certainly in posts I write here) but I wanted to mention it for the benefit of other developers who might be confronted with similar needs. There may be applications for this code not only for editor apps but for those who are converting resources in other standards into QTI. Uniqurate (like all of our projects to date) is open source and the code is available on SourceForge. Those of you so inclined, feel free to grab and hack - and be sure to let me know if you do (I'm looking at you, Mr. Pierce :-)

Meanwhile, for the rest of you - please go off and use it, break it, and (inevitably!) come screaming back to me. To test your questions you'll need to use JAssess for now, as support within QTIWorks for the QTI features this component depends on is still being developed. As ever, you'll find JAssess on the QTI Support site http://qti-support.gla.ac.uk/ along with a wealth of other useful resources. At the moment, you will also need to get in touch with Graham Smith to be able to run your own questions in JAssess, but you'll find his email address prominent on the initial JAssess landing page.

QTI @ CAA 2012 / new Uniqurate release

Last week's participation in the 2012 International Computer Assisted Assessment (CAA) Conference was a great success for all of the QTI projects. QTI was clearly positioned front and centre throughout the conference, with stream A on Wednesday almost exclusively devoted to it. The closing speaker, Paul Bailey of JISC, went out of his way to stress the importance of QTI and open standards within the field of electronic assessment, and to talk about JISC's commitment to this area of research.

The conference was preceded by a QTI Workshop on the Monday, which was most productive, particularly from the perspective of Uniqurate in that it allowed us to get some new faces using the application - and exposed some new bugs! Consequently, a new version of Uniqurate has been released today that addresses the following bugs:

• Adding an image into a question causes invalid XML to be generated
• Using IE causes invalid XML to be generated (regardless of images!)
• Inconsistent behaviour of watermarked input fields - sometimes the watermark remained when you tried to type into the field, meaning that what you got was a mismash of what you wanted and the watermark text

The version of UQ released today addresses these issues and represents quite a significant change under the hood, which should hopefully resolve any remaining inconsistencies across browsers in the XML that gets produced. As a bonus, while I had the image insertion code open, I also added the ability to not only add an image from a URL on the public internet, but also to push an image from your local hard disk and have it bundled as part of the eventual question content package. (This itself meant a big change under the hood with respect to the way the app handled content package manifests, but you end users don't care about that!)


Upcoming work will probably be to add a component that gets the student to place/drag items into a correct order - this seemed to get some appreciative nods for a "next up" suggestion, but if you have any other ideas, particularly if they're cross disciplinary in nature, fire them at me!

Enhanced item rendering in QTIWorks

After another flurry of activity, I've just finished and released the 8th development snapshot of QTIWorks for folk to play around with:

https://www2.ph.ed.ac.uk/qtiworks

This iteration of work has focused mainly on the delivery of single assessment items to candidates, joining up various bits of pipework that control this process and adding in some new rendering features and options that we'll need later for test rendering (and are quite useful in their own right too).

I'm afraid you'll still have to make do with playing around with the sample items that have already been loaded into the system for the time being. However, you'll probably be pleased to hear that the next work iteration will add in the first round of functionality for getting your own assessments into the system and doing fun things with them. (At last!) To make the wait a bit easier, I have gone through the existing samples and selected a reasonable set of options for how they should be delivered, which makes them much more fun to play with, as well as more illuminating to people who are less familiar with QTI.

Key changes in this release (1.0-DEV8)

• Assessment item rendering now incorporates the following new features:

1. Display of a model solution (via the QTI <correctResponse>).
2. The ability to reset a session back to the state it was in immediately after the last run of template processing, which effectively clears all candidate input back to the original state, but leaves randomly-chosen things intact. Existing "reinit" functionality has been made a bit clearer.
3. An explicit "closed" state, which is either entered explicitly by the candidate, or when an adaptive item becomes complete, or when the number of attempts hits your chosen value of maxAttempts. I think this will need a bit more work, as it was never implemented in QTIEngine or MathAssessEngine.
4. A new "playback" feature lets the candidate step through every interaction they made with the item, which is a possible way of implementing the allowReview concept from test rendering.

• A number of "knobs" are now available for controlling how you want to render a single item, including:

• maxAttempts (as seen in tests)
• author mode on/off (controls whether authoring hints are shown)
• a simple prompt to be shown at the top of the question (similar to rubrics in tests, but simpler)
• restrictions on what the candidate can do, including:
• close a session explicitly when interacting
• play a session back when closed
• reinitialize a session when interacting or when closed
• reset a session when interacting or when closed
• see a model solution when interacting or when closed
• see the result XML
• see the item XML source
• I have gone through all of the existing samples and set the above "knobs" to values that show each in their best light. For example, some suit a very formative "try as many times as you like" approach with rich feedback, others are more austere so are better displayed in a more rigid way. This should make the samples much more useful for people trying them out.
• The authoring debugging information has been improved, and now shows bad responses (e.g. string submitted to a float), and invalid responses (e.g. wrong number of choices made).
• There's now a rich database structure underpinning all of this, which records everything the candidate does and the changing QTI state during this process. This is currently used to implement the "playback" functionality, and will prove invaluable for analysing result data when the system delivers real tests.
• The HTTP calls that control the typical delivery of a "candidate session" via a browser are now as RESTful as you would pragmatically expect in this type of scenario. A more formal RESTful web service API will be trivial to do from this, but I'm going to hold off until anyone actually needs it.
• HTTP responses have been improved so that they always include the Content-Length headers. Responses that don't change also send cache-friendly HTTP headers such as ETag and Cache-Control.

Short term development roadmap

• Next up is adding in the web-facing functionality for creating your own assignments into the system, uploading content into them, validating them and trying them out (using the various "knobs" I listed above).
• After that it's finally time to implement the entry and exit points for candidates who get sent to QTIWorks to do an assessment. This is where LTI will come in.
• Then... tests. At last!

And relax.

Showcasing QTI 2.1 in QTI Works

As well as being a tool for managing, trying and delivering your own QTI assessments, QTI Works will also act as a showcase for QTI and the things it can do, much as the existing QTIEngine and MathAssessEngine already do.

With this in mind, I have started to feed Graham Smith's excellent collection of QTI examples into QTI Works, and plan to continue this process over the next few months while it takes shape.

These examples will be bundled into development snapshots of QTI Works for you to play around. So far, I have assimilated 3 sets of examples:

• The IMS standard examples (and a few extra bits)
• Examples demonstrating the MathAssess QTI extensions
• A small set of example items from language testing (German & Russian)

You can play around with these at: https://www2.ph.ed.ac.uk/qtiworks

(Yes, it does look very spartan at the moment, but don't worry about that!)

This exercise is actually very useful for this project for a number of other reasons:

1. It provides me with some sample data for running automated "integration tests" on the JQTI+ / QTI Works software. (For example, I feed them all of the code performing the various bits of QTI logic, such as reading in XML, validating the data models, writing out XML, running template processing, running response processing etc. This is invaluable for finding and fixing bugs, and making sure we can handle "real world" examples properly.) 
2. As well as being useful for integration testing, they also help with so-called "regression testing", which helps make sure that I don't break things inadvertently during the development process.
3. These examples have been around for a few years now, so this process is a good way of doing some QA on them and making sure they're right up to date with the QTI specification.

Enjoy!

Announcing QTI Works

Development work during the first few months of the QTIDI project has been progressing successfully, though you would certainly be forgiven for doubting this! As well as me being notoriously bad at finding time to write blog posts, the first development iterations on QTIDI have focused on "refactoring" all of the QTI-related software components we have been building and using on recent JISC projects so that they make a much better foundation for the technical goals of this project. (Rubbish analogy time: I've taken lots of random things, laid them on a carpet, and hit them with hammers until they break into little pieces. I'm now joining these pieces up to make something nice. Hopefully. The analogy is rubbish as I can't think of any examples where I've made something nice out of random bits on a carpet. Luckily, I'm better at doing this with software, so probably shouldn't have used this analogy in the first place.)

One of the key bits of work done so far has been a redesign of JQTI, which is the Java software library for "doing" QTI stuff that Southampton developed a few years ago. I've previously blogged about why I thought this is necessary (see http://davemckain.blogspot.co.uk/2011/06/refactoring-jqti-jqti.html) and the result of this is coming together under the not-very-original name of JQTI+. This refactoring work is now almost complete, with the exception of QTI tests, which JQTI never quite implemented fully and will be revisited in a few months.

On top of JQTI+, I'm building the replacement for MathAssessEngine that will become the main technical deliverable of this project. MathAssessEngine, which was based on the original QTIEngine, is also going to be torn apart and redesigned so that it can do all of the things it now needs to do, and do them all really well.
To reflect the scope of the work we're doing, we've decided to give the replacement for MathAssessEngine a completely new name and, after a couple of months of riotously bad naming attempts, we've decided to call it QTI Works.

I will deploy regular public development snapshots of QTI Works while it takes shape over the next few months, which you will be able to find at:

http://www2.ph.ed.ac.uk/qtiworks

If you remember to wear a hard hat, you can go in and try the first development snapshot now. This showcases the brand new (and vastly improved) QTI 2.1 validation functionality in JQTI+, as well as demonstrating the newly-refactored rendering and delivery of QTI 2.1 assessment items. (You'll have to make do with a selection of sample items for the time being... the functionality that will allow you to upload, validate and try out your own items is still written on the back of some envelopes. This will turn into real code that you can play with during the next development iteration. Hopefully!)

QTIDI - Connecting QTI Assessment to VLEs

The QTIDI project, funded by JISC under the Assessment and Feedback programme, is preparing a package of software and documentation for transferring the QTI Works rendering and responding engine - a direct descendant of MathAssessEngine - and software to link it to popular VLEs to institutions in the HE and FE sector. The project will provide a documented and packaged version of the tools for distribution to Kingston University, Harper Adams University College and the University of Strathclyde. Should other institutions wish to join in the project as receiving partners at a later date, they will be made welcome. Alongside, we have the Uniqurate project, which is building on the Aqurate and Mathqurate editors to create a more intuitive, user-friendly authoring too for QTIv2.1 questions. Eventually it will also incorporate the functionality of Spectatus to provide a one-stop editor for questions and tests.

Here's a quick roundup from Niall and David about what’s going on in QTIDI:

Niall: “I have been working on getting a good IMS LTI example working in Java. (The IMS example software is written in PHP.) A basic LTI 1.0 test with consumer (LMS) and tool components is almost complete. I'll be writing some developer documentation for LTI containing this code as well as C# and PHP example code. I have also (with help from Dave) set up a MathAssessEngine NetBeans project on my Laptop. I have also been taking part in the IMS QTI working group meetings.”

David: “Most of the work so far has been continued refactoring of JQTI to make it more sensible, and an initial webapp framework for SonOf(QTI|MA)Engine. So, all this first iteration is going to do is provide a "validation service". This will let you upload a standalone item XML or item/test Content Package, which will then be fully validated and will generate a summary report. Once that's in place, next iteration will be refactoring the rendering of standalone items, as well as a first cut of the REST API for that, which will be the way in for LTI.”

Note that the original MathAssessEngine will be replaced during this project, so the appearance and the way things get uploaded will change. The test mode is needing improvements, so there will be some changes there too. This means, of course, that feedback from clients will be more useful at the moment in pointing out "would like to see" suggestions (and the reasons for them), which can then be matched up with the specification to make sure things work the way they should, rather than as a debugging exercise.

Updates on Spectatus

Spectatus has been updated to work with the new IMS Question and Test Interoperability (QTI) version 2.1 schema, and has been given some new functionality. Submission mode for a test can be set to either individual or simultaneous, and navigation mode may also be controlled.

Question handling has been improved: any errors encountered on import into a test are displayed in a new message area at the bottom of the Spectatus interface (so there is no longer any need to count your questions to check that an import worked), and it is now possible to preview a single question. The issue with trying to edit a question in Windows 7 has been resolved.

Filling in of metadata is now easier for test authors, in that many of the fields (such as author and institution) default to their values from the previous editing session. The main taxonomy has been standardised, while a user-defined secondary one may be specified. We would be interested to learn what additional metadata test authors would like to be able to maintain.

A download link and project information can be found here.

Initial Activities and Dates for your Diary

The QTI-IPS team had its first face-to-face meeting in Glasgow on 06/04/2011. We were able to spend some time on the Project Plan, particularly the workpackages, with the aim of ensuring that all the tasks we have said we will do are covered by the plan. We shall have plenty to do, but we can draw on resources already in existence from previous projects and our ongoing activities. In particular we have:

• 3 different renderers for QTI: MathAssessEngine, JAssess and SToMPII


• The Spectatus test authoring application, which will be extended for packaging items and tests


• QTICat for profile building


• QTITalk for documentation


• A growing collection of items and tests, demonstrating the specification


• Contact with and support from colleagues in the wider QTI community and beyond.

We’ll be using a Virtual Machine (VM) to bundle all the tools, examples and other facilities, prior to deployment on the Amazon Cloud. The renderers are currently being brushed up ready for use in the VM and the other tools are under construction.
Meanwhile, work on the profiling aspect of the project is under way, with a list of elements and attributes for the profile in preparation, along with some examples to demonstrate the profile.

We have already had some support contacts through the public lists, and of course we expect that this aspect of the project will increase as time goes on. There will be two QTI Roadshow events in the early summer – one at Strathclyde on June 2nd and another at Kingston around the same time. These events will give users an overview of the capabilities of QTIv2.1 and the renderers and other tools and will enable participants to talk to the team about adopting QTI in their teaching.