In Mechanical Turk, how do you limit to one HIT per worker

Question

I know from communication with Mechanical Turk workers that there is a way to limit the number of HITs a specific worker can complete, but I cant figure out how to do it. Any help would be greatly appreciated!

Answer 1

I've developed a script that mostly solves this problem. The main idea is to check the worker ID against a database and then hide the HIT if the worker has already completed a related HIT.

So that you don't need to host your own database server, I've made my script available as a (free) service at: http://uniqueturker.myleott.com. Please let me know if you have any trouble using the script, or if you have any questions or suggestions.

I'm also including the script here, in case you wish to use it with your own URL/database. If you go that route, you'll need to set up a web interface to your DB that takes a worker ID and returns "1" if the worker is allowed to work on the HIT and "0" otherwise. Then you'll just replace "YOUR_URL" below to point to that web interface:

<script type="text/javascript">
 (function() {
  var assignmentId = turkGetParam('assignmentId', '');
  if (assignmentId != '' && assignmentId != 'ASSIGNMENT_ID_NOT_AVAILABLE') {
   var workerId = turkGetParam('workerId', '');
   var url = 'http://YOUR_URL/?workerId='+workerId;
   var request = new XMLHttpRequest();
   request.open('GET', url, false);
   request.send();
   if (request.responseText != '1') {
    document.getElementById('mturk_form').style.display = 'none';
    document.getElementsByTagName('body')[0].innerHTML = "You have already completed the maximum number of HITs allowed by this requester. Please click 'Return HIT' to avoid any impact on your approval rating.";
   }
  }
 })();
</script>

Answer 2

Create a hit that really is a single HIT but use javascript to dynamically change the HIT every time it is viewed. Then when posting the HIT set "Number of assignments per HIT" to the number of participants you want. In this way you will only get unique participants.

Depending on the the type of HIT you want to run this is a technique that might work for you. I have used it to randomize the stimuli shown to the participants.

Answer 3

You can also do this with external questions. I run psychology experiments on mechanical turk, so I need unique participants. In addition to requesting that workers only perform one HIT, I use a python script to verify uniqueness. My HITs all run a cgi script to produce the question. The script consults a log file, and if the worker has previously accepted a related job, it politely informs them that because I need unique participants, the HIT won't be available.

I used to do this with qualifications, but found that it really limited participation.

Answer 4

If you want x unique users, make the number of assignments per HIT equal to x in the "Design" section. Then, when loading your csv file, only put one HIT in the file.

Answer 5

See

http://docs.amazonwebservices.com/AWSMechanicalTurkRequester/2008-08-02/

You can set two types of limits:

• The maximum number of assignments any Worker can accept for a specific HIT type you've created. This value is undefined until you set it.
• The maximum number of assignments any Worker can accept for all your HITs that don't otherwise have a HIT-type-specific limit already assigned. The initial default value is 10.

Initially, all your HITs are grouped together with an overall limit (default of 10) applying to the group, regardless of HIT type.

Note that this refers to the number of assignments that a worker can have currently accepted. Once the worker has submitted an assignment, they can accept another assignment.

You probably shouldn't care how many HITs a worker completes overall, but there might be a reason why you want to change the number a worker can have currently accepted from the default 10. Of course, a worker can only accept one assignment from a HIT with multiple assignments.

If you really, really want to limit the number of HITs a worker can actually do, you're going to need to either specify that you're not going to accept more than a certain number per worker, explicitly stating that you're going to reject any submissions once the limit has been reached OR you could monkey with qualification types to do that (but that could be a lot of work)!

As an example of the latter, if you want to limit someone to doing N total assignments, you could a qualification type for each HIT and grant no more than N types to any one worker.

Answer 6

Just to clarify if you want only unique workers to complete a single hit all you have to do is to set the max assignments to however many unique workers you want and the Mechanical Turk by default will assign only unique workers to that hit.

Now if you want unique workers across multiple hits then you have to have to get fancy and use an external question coupled with a script that logs worker id's ext...

Answer 7

psiTurk (MTurk behavioral research app) automatically prevents workers from repeating HITs using an approach similar to @david-l. Disclosure: I'm one of the developers on the project.