MCQ-writing: a bibliography in no particular order (Part 1)

So, having asked lots of folk for their suggestions around MCQ-writing, I have had some great suggestions of papers and books. Here is the list (part one – in the hope that there will be a part two 🙂 ) in particular order at all:

  • Developing and Validating Multiple-choice Test Items by Thomas M. Haladyna (Routledge; 3 edition (2 Jun. 2004))
  • The College Instructor’s Guide to Writing Test Items: Measuring Student Learning Outcomes in the College Classroom by Michael Rodriguiz (Routledge (8 Jun. 2017))
  • A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment (2010) by Thomas M. Haladyna , Steven M. Downing & Michael C. Rodriguez
  • Writing Multiple-Choice Test Items That Promote and Measure Critical Thinking (2001) by Susan Morrison & Kathleen Walsh
  • Education techniques for lifelong learning: writing multiple-choice questions for continuing medical education activities and self-assessment modules (2006) by Jannette Collins

  • Writing Multiple Choice Questions by Ruth-Marie Fincher & Ropbert Nesbit. Chapter 16 of: Handbook on Medical Student Evaluation and Assessment (Gegensatz Press; 1 edition (August 6, 2015)) [Note: I can only find this as an e-book or on American Amazon]
  • Writing Multiple-Choice Questions by Boland, Robert J, MD; Lester, Natalie A, MD; Williams, Eric, MD. Academic Psychiatry34.4 (Jul/Aug 2010): 310-6. 

  • E‐assessment by design: using multiple‐choice tests to good effect (2007) by David Nicol
  • Improving the fairness of multiple-choice questions: a literature review (2004) Paul McCoubrie
  • Faculty development on item writing substantially improves item quality (2011) Naghma Naeem, Cees van der Vleuten & Eiad Abdelmohsen Alfaris (Adv in Health Sci Educ (2012) 17:369–376)
  • Faculty development programs improve the quality of Multiple Choice Questions items’ writing (2015) by Hamza Mohammad Abdulghani et al
  • How-to-guide for writing multiple choice questions for the pharmacy instructor (2016) by Kamila A. Dell & Gwendolyn A. Wantuch

I also had some suggestions of authors whose work I could/should look at. They are:

  • Joanna Bull and anything she wrote on question authoring
  • Sally Jordan  of the Open University
  • Helen Ashton

I will continue to collect ideas and suggestions and next week I will publish any more I receive. Many thanks to those of you who have been in touch. You are brilliant!


Multiple-choice question writing: cry for help!

Morning all!

I am hoping to put together an annotated bibliography, which I’ll put online here, about multiple choice questions writing in HE. And I need your help. Do you have any suggestions, or thoughts, about useful/helpful/informative/interesting papers I should include? Have you written something you would like me to add..?

Please do let me know here (in the comments section) or via Twitter (@amandaksykes).

Thank you thank you thank you 🙂

The publishing dilemma for a new(ish) academic

In the way of things that makes coincidence sometimes seem an inadequate rationalisation, I finally submitted a paper to a journal on the same day that Steve Wheeler published the first of two blog-posts about publishing journal articles. I have a real issue with having to publish in a peer reviewed, behind a paywall, journal. This not because I dislike peer-review, on the contrary, my experience to date has been positive, but I dislike the secrecy and the lack of conversation that goes with the process and I struggle with the concept that publically funded work (I am paid by a UK University) is not freely available to everyone.

I suppose I could have insisted that my article be submitted to an open-reviewing-not-behind-a-pay-wall journal. But, as Steve points out, particularly for those of us at the beginning of our academic career, this is problematic due to the pressure on us to publish in high impact journals. I have not submitted to a high impact journal, I have submitted to a journal that I thought might be interested in my work but still has blind reviewing and a paywall.

So, you may ask, why send the article there and not to a journal that I know has open reviewing and no paywall, the type of journal I believe in and want to support? Because I was advised to submit it where I did by someone with more experience and a huge publication record. And because submitting to a recognised journal was an objective for my annual performance review and therefore, ultimately, because I want to keep my job. This, I guess, is one of the oldest dilemmas in the book: employment and conformity to a (broken?) system or potential employment issues but sticking to one’s guns.

And even acknowledging that I acquiesced to employment+conformity, I am left in a quandary. I think that some of the work that has gone into the paper is worth talking about; I feel quite excited about it. I’m surprised by this because I was rather cynical about writing it up; it felt like ticking a box or jumping through a hoop rather than something meaningful. But I think there is something meaningful in what I wrote. I want to talk about the work with people. I want to ask them what they think. I want their feedback and their input to make the work better; more robust. But I don’t know if I am allowed. I don’t know how much of the work I can talk about on this blog for example? Can I publish my model here and ask people what they think, or do I need to wait until it’s published behind a pay-wall and then hope that maybe someone will bother to read it and engage with me about it? Once it is published can I make the PDF available here , or will I get into trouble? If so, how do I make it available?

Which brings me back to the beginning. I don’t like articles behind paywalls. I want everyone to be able to read my work if they think it is worth their time. I want people to give me feedback and feed-forward. I’d rather they were constructive so that I can improve the work, but I’d take negative, at least it might be the opener for a conversation. As it is, behind a paywall probably means little if any readership and so not much comment at all. And I find that frustrating!

Flipping the classroom

So, I am about to write a paper on the Flipped Classroom with a colleague and I need a place to list some sources so that I don’t forget them; I’ve chosen here. Maybe I’ll be a little more discerning later once I know more about what I’m doing, but until then here’s a run-down of what I’ve looked at so far:

1. This contains a really useful quote about leaner-centred educators, which is what flipping the classroom is all about.

2. This is a really useful physics paper about the effect if flipping the classroom using a control group (the ethics of this are interesting). On Google Scholar this has been cited 314 times, so I’m guessing this is an important piece of work to consider. I should probably also look at the 314 that cite it, but thankfully I am not undertaking a review of every single piece of literature, so we’re all good!

3. The Flipped Learning Network has a section on research, reports and studies, including some big review papers of 40+ pages…

4. This is an article about a chap who teaches Biology in the USA from the Seattle Times.

5. Tom Driscoll, whose 2012 thesis was on flipped learning, has a blog site that pulls together all sorts of useful stuff

6. Tom’s graduate thesis is here:

#BYOD4L Day three: Curating

This is the topic I’ve been looking forward to the most: curating.

As I said in a previous post, I struggle with the myriad ways to curate information and with filtering information too. And if I do, then I figure undergraduates must too.

In response to my concerns about ‘ways’, a colleague of mine told me that the best thing to do is find a tool I like and stick with it – make everything I curate open to the people I want to share it with and stop worrying about what else is out there. I think this is GREAT advice, and the joy of BYOD4L is that I’m getting the opportunity to look at the curating tools that are out there.

And, in the spirit of trying tools that are new to me I joined the #BYOD4L Google hangout today. And then I had to leave, suddenly, with no explanation because I needed to go home. Which started me thinking about scenario two (bear with me, there is a link…). The lecturer describes providing all sorts of extra resources, but that her students don’t use them, they only do the things they have to do for assessment. I guess this is a good example that the old (oft misquoted) adage, ‘build it and they will come’ [it’s actually ‘build it and he will come’ from the movie Field of Dreams] is not really true.

For me this is an issue about design. Students are crazy busy people so often have to be strategic; they need to concentrate on the things that will make a difference to them. And every now and again they are likely to have to suddenly leave too, just like I did today (see I told you there was a link) and so the design of learning activities and resources needs to be such that  there is a reason for students to use them, or revisit them if life gets in the way (the video of the Hangout is a great resource). Either that or we need to think about how we introduce students to the culture of Higher Ed and show them the value of reading around a subject so that they can see the value of taking the time to immerse themselves in the resources we provide, rather than in something else.

If engaging with additional resources will help students many of them will engage, so I’d suggest that the design of teaching and assessment probably needs a different emphasis. Instead of teaching stuff, reward process. So, for example, begin a repository of relevant articles, then require the students to add to this repository and then discuss the articles in a seminar or interactive lecture, then the links have worth. This could be done in Pinterest or or on a wiki in the University supported learning platform.

Another idea I particularly like is to award some of the assessment grade to students when they show how they researched their essay/project/assignment. You can do this by getting them to draw a ‘map’ of their research journey (have a look at this paper for an example) and it helps to avoid plagiarism too.

Ultimately I believe that the skills of filtering and curating useful information are the skills we need to be learning, for ourselves and with our students, because to be able to do this successfully is becoming more and more important in an age where ‘stuff’ (content/information) is easily and freely available.


#BYOD4L Day two: communicating

I haven’t been able to spend my day thinking about the scenarios today (note to self to watch them first-thing tomorrow) as I’ve been a bit busy, and I’m kinda tired too. As a result I guess my thoughts are probably a little less formed than they might otherwise be.

Anyhoo, I decided to think about scenario two. As someone who works in our Learning and Teaching Centre my first thought is about showing him the LTC website and the CPD sessions and the L&T contact for his department so that he can start having conversations with others. And then encourage him onto the Post grad Cert in Academic Practice.

But this is all face-to-face and quite how that fits with BYOD is beyond me really. I am struggling with this (and feeling a little concerned to post!)

Perhaps I’d also suggest he take a course like this, or another free online course. And then maybe that he join online communities like the PeerWise Community to ask questions and see what other people are doing. I’d also want to know what, in particular he is interested in. For example, I have a colleague who is flipping his classrooms and I’d put them in touch to have a chat, and I’d probably introduce them via email or Twitter (dependent on this guys preference).

None of this feels very inspired, nor is it embedded in or supported by, tech, but I think ultimately for me this is about personal interaction and often face-to-face personal interaction…

#BYOD4L Day one: Connecting

So, I’m beginning #BYOD4L and I’m not sure where to start. I’ve introduced myself (ILO1) and said hello to a few people, and I’m going to play in the Twitter conversation later. I have Google+ and Twitter and Facebook open on my computer and have (and will continue) to comment on them.

And I’m reflecting on what I do here (ILO3) but I’m a little concerned about creating ‘a digital artefact on [my] own or with others that capture[s my] learning based on discoveries linked to one of the scenarios’. It is interesting to be in this position again. To be a student, unclear of exactly what it is that is expected of me. (I don’t want to be asking ‘what is it they want me to do?’ and yet I find myself thinking this.  :-))

So, for now I am going to think about the scenarios instead, and specifically scenario two: a lecturer who doesn’t ‘do’ technology. She has a mobile and an iPad, but doesn’t really know what to do with either, or why to do it and thinks its all for nerds or a waste of time. So, her question is: can she do her job (lectures, seminars, practicals) without using it all?

My simple answer is yes! I work with a number of very successful academics who use very little tech and are supersuccessful in what they do.

But here’s the thing… For me, tech is here and it’s coming and it will continue to come. And increasingly it is being demanded by students, or used by students, to approach their learning. And sometimes there are brilliant solutions that tech can provide that will enhance her ability to teach. For example, Aropä allows students to answer a question, mark a number of their peers answers and then get feedback on their own, with very little administration from the teaching staff, so freeing time to have conversations about learning instead.

So, should she be forced to use Apps – no. Could someone sit down with her and discuss the things she does and the things she’d like to do with her students and see if there’s a way that tech can help her to engage her students in learning – yes, absolutely!

For my practice: I have to remember how it is to be a student (I do try) and that finding a tech solution to student learning and engagement conundrum needs to be approached with care…

Oh, and I just learned how to do umlauts on a mac!

New tools I’ve discovered in BYOD4L so far

As I wrote in a previous post, I’ve tried a whole bunch of solutions to the ‘problem’ of information overload.

Whist having a wander around the #BYOD4L course site today I have come across some more tools.

So, here’s the list:

1. Padlet: This seems to be an online bulletin board where people can post comments and links. As a tool it could become a collection point for information for a course, or an individual. It doesn’t seem to be hugely pretty though (although I do appreciate that it can be customised) and I wonder if Pinterest does the same (or similar) thing?

2. Edshelf: I’d love to talk about what this is, but right now I can’t get into it! On the Google-search-page-preview it says it does ‘Reviews and recommendations of tools for education’ which sounds interesting. I’ve contacted them on Twitter, so we shall see what happens. Maybe their server is down or something..?

3. I like this I think. The How Scoop.It Works video on YouTube is really great. Maybe this could be one option for curating information. And it seems to let you publish updates to Twitter and other social media platforms. Good stuff, possibly…

Not bad for an hour or so!

Oh, and in an entirely unrelated way (and a little later, this is an edit to the rest of the post above), I just discovered Etherpad (open source, real-time, collaborative document editing) through a blog I follow called Music for Deckchairs… It’s a brilliant blog mind you and this post is no exception…

Too much information, and too many ways to store it

Some days I feel like I’m drowning in electronic ‘stuff’; information and articles and papers that I don’t have time to read but that I’d really like to keep, just in case I happen upon some time when I’m cleaning behind the filing cabinet.

And so I try all sorts of new/interesting ways to creatively managing this ‘stuff’, and then I end up drowning in (mostly) electronic solutions instead, forgetting i have accounts and not using them to their potential at all!

I use Evernote, only I’d forgotten that I use Evernote and I’d been not using it and storing things in Word documents in DropBox instead. So, I use DropBox too, clearly, and iCloud (although I don’t know what this is really). And then there’s Facebook and Twitter and Google+ (I have an account but I don’t really know what this does either) and Pocket and Penultimate and Skitch (although I think this only does pictures and I only have it because it links to Evernote) and real physical beautiful Moleskin notebooks (in five different colours), some of which talk to Evernote. Then there’s my three (or is it four) email accounts. I have a 3-Minute Journal, an Outlook calendar, a Google calendar and an iCloud-mail-something calendar too. Then there’s Pinterest which seems like a good idea for ‘storing’ things, but it is public, so I’m not sure if that would be appropriate and what about Tumblr?

And if I keep thinking about it I can keep naming ‘solutions’!

Which makes me wonder how it is to be a student? As an undergraduate, how would I navigate all of this ‘stuff’ when I have NO idea which bits of it are important, and which aren’t? And who would teach me, or would I just have to figure it out on my own?

And so then I wonder about our responsibilities as academics? What should we teach our students about all of this? The assumption that they already know is unhelpful. We need to show them how to manage all that information, to think critically about it and decide what is useful to them and how to store it in a meaningful and retrievable way for future use.

Which makes me wonder, what would be a good tool for that, and can anyone explain Reddit to me?

Blogging takes longer than I thought

This blog-writing malarkey takes much longer than I thought it would; writing and editing and reading and rereading. I’m not sure if this is because I’m not really a writer (although like most people I know, I’ve always wanted to write a book – when I win the Lottery I guess!) or because I’m not exactly sure about what I want to say or perhaps because I’m still not sure if this is a good idea…

And then I wonder how this relates to my professional practice? Are these the reasons why I choose to spend time writing a blog-post when I should be rewriting a course handbook or is my reluctance to tackle the handbook simply procrastination about a job which isn’t hugely creative, even if it is essential.

The important thing for me though, is to learn from this,  to try to work out how this experience can influence and drive that which I find less stimulating or exciting or creative or collaborative, whilst still doing a good job in the job I love.