Wowzer! If you were one of the 3344 visitors who cast 410,089 evaluations for Click!, you know what a commitment it really was. I can’t say this enough: Thank You!
This was a very dedicated group of people who gave a lot of time to the process. On average photographs were viewed 22 seconds prior to submitting an evaluation and that’s an eternity in web-time. If you calculate the average time on a photo with the 389 images, we are talking a gargantuan effort and this is especially true when you consider that our evaluation tool was almost universally disliked.
We didn’t set out to make life difficult, but we were concerned that the evaluation be as fair as we could possibly make it and that meant minimizing influence. There were some encouraging comments here and there about the tool, but for the most part, people seemed to dislike it and they were going to make sure I knew it. “Frustrating,” “tedious,” “tiring,” and “time-consuming” were just some of the words that came into my inbox. There was my personal favorite: “can you fix the website so i can VOTE !!!??! ! it is nearly IMPOSSIBLE to navigate thru your website and see all the pictures quickly and easily. can you fix that ASAP! !?” …and you know it’s really bad when you get a call from you father at 10 p.m. and the conversation goes something like this:
Dad: Shell, you know you can’t skip around in there…?
Me: Yes, Dad, I know…it’s because of all these reasons (blah, blah, blah and frustration starts to mount)
Dad: I know, but there are close to 400 photographs in there!
So, let’s recap. The evaluation tool was designed to be fair. Images were randomized, attribution was withheld, comments (all 3098 that were left during the evaluation period) were withheld, linking was prevented and our algorithm ensured that all images would be seen an equal amount of times (approx 1054 times per image). Sure, we could have provided clickable thumbnails and ways to skip around, but that would have easily skewed results toward images that looked good in thumbnail format and there would have been no way to assure that each image would get equal time. In addition, an interface like that would have made it really easy for people to go vote for a particular work and skew the results — something we were specifically trying to avoid.
While most feedback indicated frustration, I will share one very interesting comment that I think really indicates some success with our methodology: “I have fresh empathy for curators who have to decide what’s in or out. It must be a grueling process“. When we first started working on Click! almost every curator who had been a part of an open call or a juried process had much the same story to tell, that it is often a tiring and tedious process. The feedback from the crowd was a decent indicator that the process really was reflecting some of the realities of a juried process.
Thank you for hanging in there. This really was an amazing effort and your dedication and support will shape this exhibition. In the coming weeks we will be discussing more issues and release more data as we go along, so stay tuned and let me know your thoughts in the comments area.
Author RSS 
RSS 
Eastern Parkway/Brooklyn Museum
In the end i came to the conclusion that this was truly democratic and fair …. some amazing work – there is no shortage of talent out there
My only negative thought about the voting tool was the use of the slider for ranking. I found that giving that much granularity to the choice was more confusing and not a real benefit over a simpler five-point likert scale or similar. That may have accounted somewhat for the longer times spent at each image.
I do think it made a lot of sense to break away from yay/nay Digg-style voting as that feels too binary for making a comment about artwork.
I’m fascinated by this project, and can’t wait to see the results. As a photographer, the vagueness of the assignment (if you will) really challenged me.
I also can only imagine the work that went into this project, and can understand why it has made people so uncomfortable.
Hey Shelley, your commitment to fairness was well-placed, and I think anybody, under pressure, would admit that you made some really smart & effective decisions there.
Have you seen kittenwar.com? Goofy little site, but in some ways the direct comparison model (I like this one better than that one) might be a quicker & easier evaluation process.
I tended to evaluate Click! images on an imaginary curve (trying to leave the slider in the middle for “average” images), which is hard to do without seeing them all. I couldn’t think of what the absolute scale would be, otherwise.
If you take the kittenwar approach, and randomly set each image against many others, it might be easier for the evaluators to decide quickly, and over time you’d get an overall comparison (check out kittenwar’s all-time winners & losers pages).
–Matt
Hi Matt,
The issue that kittenwar presents is interesting. We wanted evaluators to come to images individually w/o any kind of influence. The second images are directly compared, the results would drastically change based on which works were being shown. Randomization and a larger data set might have helped that, but not entirely and not enough for me. By the very nature of the comparison, you’d essentially be influencing to a degree. This is one reason why we didn’t use tags and allow folks to browse or jump in by subject matter – too much influence.
I’m also not in love with the X vs. Y, good/bad, yes/no when we are comparing art, something we all know to have many complexities. This goes to what Davin brings up. We went with the slider because we felt a 1-5, 1-10, starred ranking (or anything else like yes/no) would assign a numbered value (or be too black and white) to something we personally believe is more complex. The slider allowed for this concern and I’m going to talk a lot more on the mechanics of it in a post to come.
While the process of viewing nearly 400 photos was tedious, the ability to view and vote, leave the site and then come back to it at another point in time helped a lot – I broke it down into 3 or 4 sessions over a few days. The most frustrating part for me was viewing photos that were completely off-topic and where the artist statement made no attempt to explain why the photo represented “The Changing Face of Brooklyn”. I think that one of the reasons why the average viewing time before voting was so long is that the voter took the time to not only view the image but also to read the statement and take both into account when voting. I know that I did.
This was a great idea for a project that involved so many members of the community and their friends and family – I visited many Brooklyn neighborhoods that I probably never would have seen because of this project and was very happy with the opportunity to contribute my photo. I also have a new appreciation for the time involved in any kind of juried/curatorial work. Good luck to everyone that entered and I can’t wait to see the exhibit!
Shelley you did an admirable job and the process was very fair. Thanks to all at the Brooklyn Museum for allowing the general public to participate in such a fine event. I look forward to viewing the exhibition at the museum.
OK, I slogged through all of the photos and admittedly moved on when I got bored with the process by leaving the slide scale in the middle and moving on to the next image as a poet and an artist currently focused on computer generated images my approach was an organic solution to the meaningless tedium of judging other peoples work – there are no judges, agents or museum directors in the creative process – the only critique that matters for all work good or bad can only be transcribed from the artist’s interior dialog – in the case of photography we are talking about a fleeting moment in time that no one is capable of judging within the brief moment the image is captured (happy accidents seem to be the rule of thumb)
the alchemy of poetics allowed me to converse with these images on an intermittent basis
manipulated images (i.e. photoshop) provide a window of opportunity to express the artist’s creative impulses into a series of images – a new genre on the cutting edge of critical comprehension and individual exploitation
that said I relished the small part I played (all three days) in this mind expanding collaboration
I actually like the evaluation tool. I liked the freedom the slider gave. I wouldn’t agree that this system of evaluating works was completely fair to the artists, however. I say that based on knowledge of my own choices and how they were made. When I began evaluating, everything looked fresh and blew me away as a great image. I rated high often. After seeing a few score images I began to find myself saying.. gosh, it’s a nicely shot image.. but I’ve seen this before. At that point I started rating the nicely shot images of the same old kind of scene more toward the middle and reserved high marks for the more innovative, more unusual, more emotive, more whatever.. but the require had become that it HAD to ‘grab’ me immediately. I did not attempt to go backward and rejudge the first 30 to 40 minute’s worth of work, so in some ways those photos were judged on a more generous scale and benefited from that. If I could have done the evaluation any way I wanted, I’d have wanted a stack of 8×11 prints. I’d have made piles.. things like… Coney Island Boardwalk, kids, black & white images. Then I’d have worked each pile out separately, spreading them out and ordering them from most to least appealing to assign a value. I’d have loved to have had a photographic expert at my elbow to help me identify technical aspects I probably missed.
While I glanced at some of the photographer’s comments I didn’t read them all and tried not to be too influenced by them. I believe a piece of visual art should be strong enough to stand on it’s own and be appreciated without artist’s commentary. The commentary can be a nice extra, it can also limit what the viewer sees. Perhaps the viewer would find meaning and depth the artist didn’t plan for consiously… too much artist commentary can distract the viewer from that depth by focusing them on what the artist thinks he wants his viewer to see. Very interesting exercise. Thanks for the opportunity!
It’s shocking to hear that the evaluation tool was so disliked. I found it simple, intuitive and very satisfying to use. When judging images, particularly such a large number, the subtle details matter. The slider interface gracefully captured that. The evaluation would have been frustrating if we had been limited to something like a 5 point scale. The before/after thumbnails were the right size: large enough to provide a sense of context but small enough to prevent me from processing the image. I rated every image and there’s no way that would’ve happened without this interface. I’ve pointed several people to it as a great example of elegant design.
If others were mostly frustrated about the methodology, I sympathize. Nonetheless, the rigor makes this more than just some marginal online poll or social networking experiment. Evaluating stacks of images can be exhausting. It’s worth it because you never know what great shot awaits.
I’m very curious how the ratings of different groups (experts, etc) compare. Will the data be published as part of the exhibition?
I really enjoyed participating. Thanks.
Hi Mary,
You bring up an interesting point here because our interface design didn’t provide a way for you to go back and revisit works to change an evaluation. It’s funny, our first instinct was to provide this functionality, but in a meeting with two of our consultants (James Surowiecki and Derek Powazek) they cautioned us against providing it. The idea was the “crowd” was curating and in this, the “crowd” would self-correct over time. As long as we could get enough people to participate, there would be enough sets of eyes seeing photos at differing times (owing to the randomization) to allow for the crowd’s self-correction. They explained this very idea was the difference between an individual curating and a crowd curating and it really sold me as being right for the nature of the show. Looking back, I’m really glad that conversation took place. As an aside, however, I did receive a ton of feedback that participants really wanted this functionality.
Hi Chad,
So much data is going to be published we are drowning in programming tasks to bring it to you! There’s going to be a killer advanced search (daunting too) where you can visually see how different groups rated different sets of images according to what you query. That said, we are being sensitive to the artists and the works, so you won’t see things like lists or scores, but you will be able to get a really good idea generally based on relative size of images and some pretty cool graphs on each work. More on this closer to launch!
Thanks for your thoughts, everyone! It’s nice to keep the conversation in the blog and out of my inbox
Another click-related post to come tomorrow or the next day.
Sounds good… also can we split this up between portrayals of changing Brooklyn between in-towners and out-of-towners like me? That would make an interesting study.
I was very excited to participate in Click, and I admit I had trouble capturing this quickly changing borough. I didn’t submit a work in time, but was looking forward to judging the other contributions.
Like others, I was frustrated with the number of photographs, but more importantly I was frustrated with the criteria for judging. The theme was “The Changing face(s) of Brooklyn,” and a photo was judged purely on how well it met that theme.
I saw amazing photos that met the theme, but I wasn’t judging it on how an amazing photo it was, or how creative, or even how much i liked it.
The idea of crowd opinion compared to professional curator opinion is great, but in the end I was turned off from the voting process because it felt like I was just a moving slider indicating the effectiveness of the photo with the theme.
I judged the function and not the form.
I do, however, guess that most people subconsciously voted higher for pictures they thought were of a higher quality, but they weren’t really “supposed to.”
Maybe an added a slider for “quality” would have made me feel better.
i wish i could still do it online. this sounds like a great art project. maybe we can start one for queens???
peace out! :p
Hey Matt V, good to hear from you and thanks for eval’ing. When considering a work we asked people to consider two things (both of these were mentioned on the first page when you started and on the “consider this” area) – 1. how well did it fit the theme and 2. was this an exceptional image – consider the techniques used and the aesthetics. True, we did only provide one sliderbar, but we expected people to think about both issues and then give it an overall rating based on those two considerations. We got a lot of feedback that people would have rather had two distinct sliderbars, rather than having to aggregate two issues on one scale. On the other hand, our curators have to think about many issues when selecting images, so this seemed like an OK implementation. I guess the frustration at least mirrored the curatorial process somewhat
I can’t wait for the opening night of CLICK. I was so excited that I was able to participate.
It would be nice if we could meet ‘the photographers’
Patty
hi, i was wondering-will the exhibition have a opening “night” or will the museum be closing @ 5pm on the 27th? thank you!
Hi Robert, Patty,
Thanks for writing – there is no reception or party for Click! The exhibition will be open as of this Friday – normal hours for us, so that’s 10AM – 5PM.
Hey Shelley,
You got me on that one, I missed the part about considering aesthetics. That slider just took all my attention!
Looking forward to seeing the exhibit. My girlfriend Eudie did a nice write up for it in this week’s Village Voice. She said it got a big photo at the beginning of the Voice Choice section. It hits the streets tomorrow.
Maybe I’ll see you there this weekend!
Sweet, Matt, I can’t wait to see the Voice – sounds awesome and the press peeps around here will love the headsup! Would love to see you Saturday