For now, you can accept at most HITs of this kind (more later!). We use the UniqueTurker script to enforce this. Sorry for any inconvenience.

How (not) to name an object.

We asked various workers to "name the object in the bounding box", for a large number of images.
For our research, we need you to evaluate the names we got: Is each name adequate (see definition below)? If not, what's wrong? Also, were different workers perhaps naming different objects in the image?

In this HIT you will get images, with a total of names. This HIT contains quality control items (see also detailed instructions below).

Read the instructions carefully, and hover over words and buttons in the task for additional examples. Please contact if anything remains unclear.

1. Approval, rejection, bonus, warnings

►    The quality of your responses is crucial for our research! This HIT contains quality control items.

►    Mistakes can happen, but if there are several then a friendly warning may appear when you click SUBMIT. Don't worry if you get this warning only once in a while! But if it happens more regularly, please make sure you understood the task before doing any more HITs of this kind, and/or try to work more carefully.

►    If you make many mistakes then a more serious warning (but still friendly) will appear when you click SUBMIT, advising you to make sure you understand the task, double-check your answers before submitting, and only proceed with HITs of this kind at your own risk.

►    Regardless of the warning, you can still SUBMIT your results. Your submission is not automatically rejected. We do not reject workers lightly, not only based on automatic quality controls, and with human consideration.

►    We reward a bonus if you answered all quality controls correctly (bonus amount per HIT specified in title).

2. When is a name "adequate"? Use the definition!

When judging if a name is "adequate", you must use the following definition:

DEFINITION: A name is "adequate" if there is an object in the image, whose visible parts are tightly circumscribed by the red bounding box, that one could reasonably call by that name.
Pay attention:
1. If there are several objects that tightly fit in the bounding box (as tightly as its rectangular shape and orientation allow), then names for any of them are adequate.
2. The bounding box for an object needs to cover only the parts of the object that are visible; parts of an object that are not visible (for instance, that are behind some other object or outside the picture frame) can be outside the bounding box.

3. Examples

Workers entered "pan" / "food":   only "pan" is adequate!

In this case the bounding box tightly fits only around the pan (as tightly as an upright rectangle can fit a circle); but it is a little bit too big to tightly fit the food. Therefore, only "pan" is adequate; the worker who entered "food" was naming the wrong object.

Remember: a name must be for an object that is tightly surrounded by the bounding box.
"Legs" / "person":   both are adequate names, but for different objects.

The bounding box fits tightly around the legs, and also around all visible parts of the person (as required by the definition), so both names are adequate. However, these names are intended to name two different objects: a person is not the same object as the person's legs -- one is merely a part of the other.

Remember 1: two names can both be adequate and yet name different objects.
Remember 2: only the visible parts of an object need to fit in the bounding box (as for "person").
"Burger" / "hotdog":   only "hotdog" is adequate, but both are intended for the same object!

Even though a hotdog is not normally considered a burger, the two workers clearly intended to name the same object: the food item in the bounding box.

Remember: two different names, one adequate and the other inadequate, can still be intended for the same object.
We got the name "ground":   what was this worker thinking?!

The name may seem totally random, but the ground is in fact showing in the image -- it's just that it is outside the bounding box. In general, for inadequate names, you have to look very closely if there is anything in the picture which the worker could have been naming.

Remember: named objects can be in the background, outside the bounding box, out of focus, etc. -- so pay attention!
"Helicopter" / "airplane" / "truck" / "boat" / ...:   only one name is truly adequate, but they are all intended to name the same object!

You have to look closely, but the object seems to be a helicopter. Other workers have misinterpreted the picture, but all were at least intending to name the same object (after all, there are no other airplane-like or boat-like objects in the image).

Remember 1: many different names, even if some are inadequate, can still all be intended for the same object.
Remember 2: to know for which object a name was intended, you have to look closely also at all other objects in the image -- could it be any of those?
For more instructions and examples, just hover over terms and icons in the task below.

► Are the following names
Definition: A name is "adequate" if there is an object in the picture, whose visible parts are tightly circumscribed by the red bounding box, that you could reasonably call by that name.

Green button -- adequate names:
- If multiple objects fit tightly in the bounding box, then names for different objects are adequate -- for instance, for a bounding box tightly fitting a bed and a blanket over it, both "bed" and "blanket" are adequate names.
- "person" can be an adequate name if the bounding box fits only their head, in the case that the rest of the person is not visible (for instance, the rest is outside the picture frame).
- "sandwich" is adequate for something you would not call a sandwich, but you know that other people sometimes do. Don't be a definition warrior.

Yellow button -- possible/minor inadequacy:
- "horsse" for a horse, containing a small typo (minor inadequacy).
- "office" for a room that could in principle be a living room, but the image is not clear (possible inadequacy).
- "skier" for someone who is actually snowboarding, but you have to look quite closely to see this (minor inadequacy).
- "tree" if there is a tree in the bounding box, but the bounding box does not fit the tree tightly (minor inadequacy).

Red button -- severe inadequacy:
- "woman" for a person who is obviously, clearly a man, you don't even need to look closely.
- "shoe" if there is nothing remotely resembling a shoe in the image.
- "house" if the house is clearly, far outside the bounding box.
for the picture on the right? If not,
what's wrong
Clicking a yellow or red button will make other buttons appear, to choose the type of inadequacy.
Hover your mouse over the different icons to understand their meaning.

If a single name has multiple errors (e.g., "horsse" for a cow), choose the error you think is most severe; typos count as minor errors.

- The name "horsse" for a horse: click possible/minor (yellow) inadequacy, and the typo icon to indicate a linguistic mistake.
- The name "roof" when in fact the entire house fits in the bounding box: click severe (red) inadequacy of the bounding box type ('named object not tightly in bounding box').
- The name "horsse" for what is obviously a cow: click severe (red) inadequacy of the 'named object mistaken for something it's not' type -- although there is also a small typo, misinterpreting the object in the image is more severe.
- The name "shoe" if there is nothing in the image resembling a shoe: click severe (red) inadequacy with the "?!" icon, because the name seems completely random.
- The names "man" and "woman" for a person whose gender is really not clear: for both names click possible/minor (yellow) inadequacy of the 'named object mistaken for something it's not' type.
? (←hover for definition & additional examples)

Name given: Adequate? If not, what is (or could be) inadequate about the name, primarily?

► If you had to guess, which names were likely intended for
naming the same object
Simple visual example:

"House" and "building" get the same color (in this case green; but it could also be red, blue, or purple -- doesn't matter as long as it's the same!), because they are intended to name the same object. The other names get different colors. Beware that the logo is a different object from the truck -- hence different colors -- even though the logo is on the truck.

Positive examples (assign the same color):
- "blanket" and "blaknet" were likely intended to name the same object, so should get the same color, even though one has a typo.
- "donkey" and "horse", given an image showing a single animal, were likely intended to name that animal, even though at least one worker must have misidentified it -- so these names should get the same color.
- "dinner" and "lunch", given an image showing a meal, were likely intended to name the same object, even though the workers conceived of it differently -- same color!
- "vehicle" and "car" were likely intended to name the same central, prominent car that's in the bounding box, even if there are other, distant vehicles in the background.

Negative examples (assign different colors):
- "house" and "roof" are not intended to name the same object, so should get different colors, because the roof is only a part of the house -- even if only the house's roof is visible in the picture.
- "person" and "clothes" are not intended to name the same object, even if they have the exact same bounding box; we don't normally confuse people with clothes. So in this case the two names should get different colors.
? (← hover for more instructions/examples; can be tricky!)

Please remember, from the instructions:
▹   two names can both be adequate and yet name different objects (see the "legs"/"person" example at the top, with only legs visible): in such cases assign a different color to each of these names.
▹   two different names, where one is adequate and the other inadequate, can still be intended for the same object (see the "helicopter"/"boat" and "burger"/"hotdog" examples at the top!): in such cases assign the same color to each of these names.

Names: Assign the same color to names (only) if they were likely intended for the same object:

► Any comments? [Optional]