Don’t get me wrong, we still care about use rate and want as many visitors as possible to use ASK, but we are no longer consumed by that metric. We have kept the tactics that work best. For example, our ASK Ambassador program is going strong and still makes the greatest difference: there is a direct correlation between Ambassador staffing and use rate. We’ve seen this repeatedly since launching the Ambassador program in 2017. We’ve also continued to play with engagement via ASK, particularly in relation to major special exhibitions, which—let’s face it—is why a majority of visitors come to the Museum. For the David Bowie is exhibition, we created a special trivia activity that was so popular we could barely keep up. Jessica will share more about that in a future post. She’ll also share about what we are currently doing in relation to Frida Kahlo: Appearances Can Be Deceiving, on view until May 12.

The ASK Ambassadors wear branded t-shirts and hats (hats optional).

In addition to these initiatives, I’m delighted to say we’ve added a Pratt fellow to our team this year who has been focusing on the data. Sydney Stewart is a second year graduate student in the Museums and Digital Culture program in the School of Information at Pratt Institute. (Full-disclosure, I teach for that program and Sydney is one of my stellar students). It’s amazing what we have been able to learn by having one person focus on the data, and I’ve invited Sydney to share her research and results here in the coming weeks.  

When it comes to truly getting into the ASK data, I find we’re constantly bumping up against what happens when you build minimal viable product (MVP) as part of an agile process: short-sightedness. We purposefully weren’t thinking long-term when developing the dashboard (the interface the team uses to answer questions and process chats), only what we needed in the moment. Because we were building MVP,  we didn’t plan for or build ways to access the larger data set. We only created tools for our initial needs, which were basic metrics and ways to share conversations with curators for fact-checking purposes. Now that we are trying to get a better handle on the scope and possibilities of the data, we are having to look into building tools to access it. For now, the Tech team runs reports for us when we know what to ask for. I suppose you could say we’re having to get agile once again by using Sydney’s research path as a way to help us understand what we actually want and need to know from the data set. Unfortunately, that makes it a little difficult for her as there is a delay between her determination of needed data and our ability to give her that data. Fortunately, she’s been more than up to the task, and she’ll share some of her creative workarounds and what she’s been able to do with existing metrics.

Our period of radio silence is over, so stay tuned!

Five star app reviews fueled our increased use rate desire. With all these great reviews, surely more visitors would use the app.

Turns out, not that many more people are using it. After a great deal of testing and improved marketing efforts with insights from an outside evaluator, we have managed to double our use rate from 1% to just over 2% pretty consistently. That’s with a lot of effort on our part: incentives and contests, staff hired specifically to promote the app, and marketing materials including palm cards and object labels. When the stars align and our team is really on fire, we’ve seen over 3%, but that is likely the best we are going to get with what we’ve got. And you know what? Maybe that’s OK.

I’ve been doing a lot of soul searching about this project, about what success should look like, and about the original goals for ASK. And to me, the raw numbers are becoming less important. We created ASK in order to facilitate engagement with art. Do I wish more people used it? Sure. But those that do get something really special out of it. That might just be enough. Interestingly, when you compare ASK to other engagement strategies we’ve offered in the past, it fits within a norm. For example, our audio guides (offered until 2012) and our In Conversation kiosks also netted about 2% use rate.

So where do we go from here?

It’s time to shift our focus away from “more” and towards one of the most unique aspects of ASK: the data. We have over 11,000 chats just waiting to be examined. We have metrics not only on app use, but also most-asked-about artworks, information related to where and how many places people ask us questions, and more. We can determine the kinds of questions people have about works of art, how they think about or look at artworks, how they experience the museum, and more. All of this data would be of interest to researchers of all kinds: educators, art historians, and technologists. It’s certainly of interest to us.

The ASK dashboard tracks certain metrics already, but we have yet to delve into the 11,000+ chats we’ve had with users.

That is where we are headed. ASK is no longer about “more,” but instead about identifying the unique insights we can gain about how visitors view and understand works of art. As Shelley stated in an early post about measuring success: “Three components help us determine the health of the ASK: engagement goals, use rates, and (eventually) institutional knowledge gained from the incoming data.” We’ve nailed the engagement goals. Use rate is the best we can get with the tools at our disposal. It’s time to shift to the final measurement: institutional knowledge.

The next step is to reach out to our colleagues across various departments—curatorial, education, visitor services—and determine what questions they have. We’ll then add those to our list of questions and begin to narrow down a research focus. As we delve into the data, we will be sharing insights along with way. I can’t wait to see what we learn.

The Brooklyn Museum has a large audience of school-age visitors and their accompanying adults. Scavenger hunts have turned out to be an easy and dynamic way to engage them.

Last June, our colleagues in the Education Division invited ASK to participate in the annual “Bring the Cool” Family Festival, a day-long event organized in collaboration with local non-profit Cool Culture. The festival includes art-making activities and creative play for young children and their families, and it’s always a lot of fun, so we were happy to join in.

Since this year’s festival theme was “Color My World,” we put together a scavenger hunt with eight stops around our American Art galleries. We wrote a set of eight simple color-themed clues for eight varied works in the collection, from a Coclé gold disk embossed with a face to a nineteenth-century Brooklyn landscape painting. When users downloaded the app at the hunt’s starting point, we could guide them exclusively through our ASK exchange in either English or Spanish. We invited them to send photos of the works and to share personal answers to related questions.

The kids who tried the scavenger hunt seemed to enjoy it so much that we thought it was something we should try again. Meanwhile, as we moved into the summer and schools let out for vacation, our ASK Ambassadors reported an increasing number of museum visitors asking for “something to do with children” during their visit.  Responding quickly to this seasonal shift in attendance, the ASK team invited younger children to try the American hunt but also started compiling clues for favorite objects around the rest of the Museum.

Attendance was high in “Georgia O’Keeffe: Modern Living” during the summer and the ASK team came up with rhyming clues to interest young visitors in the show.

These family- and child-oriented chats turned out to be really popular. We chatted frequently with young museumgoers in the special exhibitions “Georgia O’Keeffe: Living Modern” and “We Wanted a Revolution: Black Radical Woman 1965-85” as well as the permanent galleries for Ancient Egyptian Art, Decorative Arts, and more.

As the opening date for the fall exhibition “Soulful Creatures: Animal Mummies in Ancient Egypt” approached, we were hearing more and more conversation around the Museum about ways to engage families and children with this show. We offered to create a hunt itinerary specifically for “Soulful Creatures,” focusing even more on providing interesting educational facts as follow-ups to the clues.

Several children of staff members took time from a summer afternoon to test an early version of our “Soulful Creatures” hunt in the Egyptian galleries.

Since “Soulful Creatures” wasn’t installed yet, we decided to “beta-test” our script in the permanent Egyptian galleries, with the valuable assistance of several staff members’ children. These young volunteers did a run-through of the hunt and chatted with us afterwards. They gave extremely helpful feedback about the instructions they’d received, the difficulty level of the questions, and the choice and spacing of the objects. They also had some great questions about ASK in general, and we took notes for any future project involving younger visitors.

Each object has a clue as well as facts to share once the user locates it.

When “Soulful Creatures” opened on September 29, we were ready to go. The ASK team had selected ten objects in the show and written a script with two sets of clues, one for beginner readers (about ages 4-7) and one for more advanced learners (ages 8-11), as well as entertaining facts to share about each object once the user had located it. Our ASK Ambassadors were prepared to pitch the hunt to visitors entering the show and to provide assistance with downloads and getting started.

So far, we’ve guided young “mummy-hunters” ranging in age from four through twelve years old, and almost half the hunts have included two or more children together. Some users completed all ten clues, while others (depending on available time or attention span) were satisfied after finding four or five works. Our Visitor Services department is also offering a family packet for this exhibition, so various options are available for kids—we’ve just asked our ASK Ambassadors to pitch the ASK hunt only to families who haven’t already taken advantage of the packet.

We’re often happily surprised when our young users include themselves in their  “I found it!” photos.

Like all our work, this process of shaping and expanding ASK scavenger hunts has been a team project, and it’s turned out to be a team favorite as well as a popular option with visitors. We’ll be thinking about new hunt ideas for the new year as we continue to connect with some of our youngest museum visitors.

Luce gives open access to some 2,000 of the many thousands of American objects held in storage. Visitors can use iPads mounted throughout the space to search works by accession number.

Luce has five iPad kiosks for visitors to search the collection online since none of the works have any information beyond an accession number. We loaded ASK on 4 of the 5 iPads (the fifth was being used for a survey) and tried it for a week. During that time we had 14 chats come into the dashboard and learned some important things in the process:

• We need to identify the kiosk function. On the iPad, the app is already open so users don’t have the benefit of seeing the prompts. We need a sign letting people know the iPad is how they can find out information about the works in Luce.
• We need to respond very quickly. Because users are stationary, it feels like a really long time before an answer comes our prompt to put down the phone and look at art doesn’t apply here. We have to make sure to send a response fast, even if it’s a partial answer just to start. This was an initial concern about kiosk-ifying ASK and we were right to be concerned.
• We need to turn off the camera feature. Because the iPads are mounted, the user can’t easily access it anyway. One iPad got stuck on the camera feature somehow and since the iPad case covers the camera button, we had to restart the kiosk.
• We need a way to refresh the conversation. The app stays open, which means that people can read the running conversation from the entire day. We watched several people do just this and it’s a behavior we’ve seen before in other iterations of Q&A kiosks. However, we need to be able to start each day with a fresh conversation, otherwise it goes on too long.
• We need to remove access to emojis. Quite a few silly incoming messaging were in the form on nonsensical emoji streams. Since we tested this right before school started, we think it was probably bored kids. The team’s dashboard can’t read or send emojis anyway, so this is a moot feature.
• We need to be able to hide selected messages from appearing on the iPads. If someone does send nonsense, we want to be able to remove this from view so others aren’t tempted to do the same and so that those who want to read the conversation thread can do so without interruption. Plus it’s simply annoying for the team.

While some visitors using the Luce iPads had genuine questions, like a regular app exchange, others were compelled to send us silly messages. The team dealt with them quite politely, but eventually started ignoring them.

We are working with HFC, our contract developers, to create a special version of the app that has the features we need to be able to install it on a stationary kiosk. We will give it another go. Fingers crossed!

“Georgia O’Keeffe: Living Modern” was immediately well-attended, but not many visitors were using ASK in the show.

When we realized that we weren’t reaching as many visitors to Georgia O’Keeffe: Living Modern as we’d like to, we thought about ways to pique their interest about the app. One solution? A contest.

We tested this concept in early May and began running it on a weekly basis after our recent pilot testing.

We promoted our app contest with these cards featuring the cover of the exhibition publication.

The ASK Ambassadors have been handing out dedicated flyers explaining the contest: visitors are encouraged to make their own “creative connections” within the exhibition, by choosing pairs of objects that share an idea or a visual motif. They send photos of the two objects, with a brief explanation for their choice (and their email address, in case they win and we need to contact them!). The prize is a free copy of the Georgia O’Keeffe: Living Modern publication, an incentive that seems to entice visitors to give the contest a try.

Behind the scenes, the ASK Team compiles the entries and chooses a weekly winner. To guide the process, we’ve established a set of criteria. A winning entry should do several of the following things:

1. Connect works in different media (e.g., a photograph and a garment)
2. Connect works from different galleries
3. Connect works from different decades or eras in O’Keeffe’s life
4. Go beyond the pairs already staged in the exhibition by our curators
5. Offer a thoughtful or even humorous explanation.

Some contest entries were straightforward visual comparisons, like this early fashion illustration by O’Keeffe paired with a late photograph of the artist.

So far, we’ve had an average of 25 entrants per week. Even though we’re very familiar with the exhibition at this point, some of the object pairs have truly surprised us! Entrants have made comparisons that are formal, biographical, or thematic—or more than one of the above.

This contest entry made a more subtle analogy between two objects created by O’Keeffe, the ruffled decoration on a blouse and the forms of a painted flower: “The pin tucks in her blouse and the tonal shadings so elegantly reflect the painting.”

We’ve enjoyed reading everyone’s clever and enthusiastic entries, and it’s always fun to notify a person that they’ve been chosen as the week’s top pick. So far, our winners have come from California, New Jersey, Vermont, Manhattan, and our own borough of Brooklyn.

Meanwhile, the first week of the contest doubled our app traffic in this exhibition, and as the contest has continued, there are weeks when more than half our total chats are coming from the O’Keeffe galleries.

This visitor wrote, “These two both have a living flower next to something *not* living that O’Keeffe could see from her window.”

Even in everyday app chats, we like hearing what our users have to say about art, and this contest has offered a more focused situation for them to share their ideas—while once again proving the appeal of contests, games, and creative challenges.

Our texting palm card included the phone number as well as “helpful hints” on what kinds of things to text. Unlike the app (which is geo-fenced), functionally you can text us anytime, but we won’t answer outside of Museum hours. The system will autofire the same “out of office” notification” app users also receive.

Our developers did some backend magic so that the Twilio messages would push to the dashboard, which allows us to keep that single source for incoming messages. Unlike the app, the SMS messages do not utilize location aware, which meant the team was flying somewhat blind in the dashboard. Normally, when a visitor sends us a message, the nearest beacon responds and the dashboard populates with the artworks on view in that gallery. Each artwork has the associated metadata from our collection online as well as “snippets” (question and answer pairs) from previous conversations that have been tagged to the work. SMS messages provided few of these tools to the team, who had to manually search the collection online (or good, old Google) if they didn’t know the work already. However, this isn’t the first time we’ve dealt with this challenge, so going into this pilot we felt pretty confident we could handle it. As expected, response time suffered a bit, but overall the team did really well and lack of location data didn’t hinder them much at all. One happy discovery was that the image recognition the developers put into place last year occasionally worked with MMS message, so if a user sent us an image (many do), there was a chance the dashboard would find it and pull the metadata as well.

I have to say, out of all the pilots, I thought this would be the most successful, and I was right. Visitors really responded well to the idea of texting, and we ended up having to reorder palm cards twice. Despite this enthusiasm, use rate was not what I would like to have seen; we averaged just above 2%. Interestingly, not all of that traffic was from texting. We still had some iOS and Android users, only 19 of whom were repeat users, so some folks were finding and using the app despite our promotion of texting (and likely not all international visitors). Out of curiosity, I wanted to compare this most successful pilot to our most successful two non-pilot weeks. Turns out the average use rate of those two weeks is higher than any of the pilots. So alas, while this pilot had the best use rate of the three, it did not hint at a “solution” to our use rate plateau and was not better than our best two (non-pilot) weeks combined.

By way of quick recap: pilot 1 (provided devices) proved charging for devices doesn’t work, while free devices didn’t give us more app traffic; pilot 2 (ASK on Demand) showed that while people liked the idea of an in-person chat, few took us up on it; and pilot 3 (texting) did not show better numbers than really successful non-pilot weeks.

What’s next? I’m not 100% sure, but we’re kicking around a few ideas. We just had a big team meeting with the ASK team and Ambassadors to discuss these metrics and share observations, which I’m still chewing over. Some things we are exploring include providing devices for scheduled groups and keeping the texting service as an alternative for those that just don’t want to download.

It’s going to take a little while to figure out, but I promise more to come after vacation (yay!) and time to think. 

Over the course of ASK, we have seen distinct patterns in the ways people engage with the app and the ASK team: they ask us questions, seek more information, and share their opinions. Visitors have also responded very well to opportunities to meet the ASK team in person during  pop-up tours, Art History Happy Hour events, or when team members are stationed in the galleries, which they occasionally are. Users also enjoyed the chance to meet the ASK team when their office space was in a public space, though not enough to keep them there for good. Knowing all this, we began to wonder, what if we updated the user experience of ASK to reflect these aspects that visitors enjoy and provide visitors clearer choice within their experience?

While still providing the opportunity to chat via text, the core of the ASK experience, could we also offer FAQs, most-asked about objects, or a “surprise me” feature that provides bite-sized content? This would help address the reported pressure some visitors feel about having to ask a question. In addition to texting and FAQs, could we offer the ultimate personal experience by offering visitors the opportunity to have conversation on-demand, in-person with one of the ASK team members? Functioning something like an airplane “call button,” this option would allow visitors who prefer an in-person conversation or are just really enjoying their text conversation to request the ASK team member to join them in the gallery. Should this concept work, we could play with incentives like only surfacing the “call button” after a certain number of exchanges or galleries visited or we could offer the option faster for repeat users. There are lots of possibilities to explore here.

This kind of approach would require changes to the app functionality and design, which we’re prepared to tackle, but only if we could prove some of the basic concepts valid. In particular, I wanted to confirm people would take us up on the in-person request component. We created a dedicated palm card for this pilot highlighting the various ways people might engage with us and the ASK Ambassadors emphasized the in-person option in the pitches.

Dedicated palm cards highlight the in-person concept. The “helpful hints” simultaneously provide instructions as well as concrete suggestions on to how to use the app based on needs and interests.

I’m somewhat surprised to report that over the course of the two weeks, our app traffic was right within the normal range and only six people took us up on the offer of an in-person appearance. One of the things I was curious about was how many people would bypass texting altogether and just request in-person time. Timing of the request really varied. Two of the five requested a team member immediately, one (a family group) ended their very engaged time via the app with an in-person request, one refused to download the app and the ASK Ambassador requested a team member on their behalf, and two took the ASK team member up on her offer to join them one or two messages into the conversation.

Our ASK Ambassadors managed to snap two quick photos of visitor interactions during this pilot. On the right, Isabella speaks with a visitor in the O’Keeffe exhibition. On the left, Rachel speaks with visitors in “We Wanted a Revolution.”

I will say that I think one reason for the limited uptake on the in-person interaction was our over-zealous guarding of the ASK team’s time. We were so worried that the team member would be unable to extract herself from a very interested visitor (this has happened often on tours) or be expected to present some manner of tour, that we asked the Ambassadors to really stress that it was an opportunity to “say hello.” I understand from the Ambassadors, who did a great job executing exactly what I asked, would often preface the opportunity with some kind of explanation that the team might be busy answering questions via the app and would only be free for a minute. In retrospect, I’m sure this made the in-person request feel like a total imposition. I know if I were a visitor, I at least would have hesitated before requesting someone if it were presented to me in such a way.

Again, I come away from this pilot without any true conclusions except that we might want to revisit it, but with less protective language around the team’s time and a more simple invitation to have someone join them in the gallery for a bit. August might be our month to take the learnings from running these pilots try them again. So far, the only thing we’ve been able to definitively say based on the pilots is that charging visitors for iPods loaded with the app won’t work. I suppose that’s something!

We are running a series of three pilot projects that will help us determine the direction of the next two years of the program, which we’re calling ASK 2.0. This is the first of (at least) three posts on the results.

We began our pilots with the low-hanging fruit: provided devices. If downloading is a barrier, don’t make people download anything. Providing devices would also eliminate the related “excuses” we hear from visitors: not having enough storage, data, or battery life. Of course, it’s not quite as simple as just handing someone a device. We had to make sure we’d get it back, which means there had to be some kind of check out process. This is its own kind of barrier, albeit a familiar one for anyone who has ever rented an audio guide. Would a check-out process prevent people from using it? I also wondered if people would be willing to potentially juggle two devices—our iPod Touch and their phone—because despite some focus group and survey participants proclaiming that they don’t like to use their phones during a museum visit, 76% of those same survey respondents admitted that they use their phones at some point during their visit (to take photos, text, use social media, Google something, etc.). Finally, I wondered about the perception of value. What would happen if we put a dollar amount on the experience and charged for the devices? Would it turn people away? Increase download rates? Encourage people to chat more to get their money’s worth? I had to know.

Seems fitting to return to iPod touches for this pilot since that’s what we used for our 2014 pilot that ultimately led us to build ASK. Here we set them up with lanyards and protective cases. We enabled “guided access mode” to lock them down on the app only.

We ran the test for two weeks and included the A/B test around value. For the first week, the device (iPod touches) were available to check-out for free. For the second week, the iPods cost $5. I have to say, the results surprised me a bit.

For the first (free) week, we checked out out 42 devices, with 30 chats coming from those devices. So not everyone who checked out a device used it (not the surprise). The ASK Ambassadors asked visitors returning devices what they thought about their experience and for most people that didn’t use it, they admitted they didn’t “need” it because the museum was well-curated and explained. I suppose that’s hard to argue with! One surprising result we saw were that the iPods averaged more than double the messages of all users that week. The average number of exchanges for the week was 11.9, but for the iPods specifically it was 24.3! Now this could partially be explained by the fact that several kids used the iPods, and kids tends to send a flurry of photos and comments, so there is a lot of back-and-forth in a short time, but it’s something to think about. Another surprising (and disappointing) thing was that providing devices didn’t net us any more chats than our usual weekly average. So while 30 people used it, that wasn’t 30 more people than usually use it. Hmm.

Alex and Kahlah help a visitor check out an iPod during the free week. We used the Info Desk in the lobby as home base for storing visitor IDs collected as collateral and for storing and charging the iPods.

The second week of the test ($5 rental fee) didn’t work at all. We only had one visitor rent an iPod, and it was for her kid. The engagement numbers don’t mean much with one sample (though it was really high at 36 exchanges). This was also a slow week for us, under the average number of chats for the week. This may have nothing to do with the iPods and perhaps more to do with a few of our ASK Ambassadors being out sick, but hard to know. We did see an uptick in download rate, however, and the ASK Ambassadors told me that it was an easy “sell” to get people to download for free as opposed to renting an iPod. Unfortunately, those downloads did not translate to app use.

So what does this all mean? Honestly, I think it means we need to revisit providing devices for free again. We’ll work this into the plan for August, unless one of our remaining two pilots is wildly successful. So far the second pilot, which I’ve named “ASK on Demand,” is showing some promise. More to come next week!

Here’s where we stand: the ASK Ambassadors, our dedicated guerrilla marketers, have been on the floor since mid-February. We’ve had a few successes thanks to their hard work.  One unexpected boon is with the ASK Ambassadors doing their pitch, for the first time, users pretty much universally understand that they are chatting with a human (as opposed to a bot). One of the ways this confusion manifested was people sending us keywords instead of phrases, sentences, or questions and then often apologizing later once they figured out it was a human. ERm’s evaluation confirmed this confusion. (Side story: one focus group participant wondered aloud if he was chatting with a real person and when someone else pointed out the photo of the team, he replied that it was like a United Colors of Benetton ad; no way it was the real team and had to have been staged. Yikes.) Now, however, users are ready to jump into conversation and we’re getting a lot more people replying directly to our auto-fire system message that let’s them know we’ll get to them and to feel free to put down their phone. “OK,” they tell us. A different kind of confusion perhaps, but preferable.

ASK Ambassador floor captain, Alex, (left) loves pitching the app to folks waiting on the elevator. Kahlah (right) talks to people waiting in line at admissions.

I’m happy to report a noticeable uptick in use rate thanks to ASK Ambassador mojo. We’re regularly above our previous 1% average use rate and have even seen some weeks above 2%, which is great. But it’s not enough.

We are at the point in this project where it’s time to take stock. We’ve paved the way for what it means to engage visitors with art via text message. We know people love ASK; our app store ratings continue to be stellar. So it’s not an engagement problem. We’ve done lots of testing around messaging to highlight what’s unique about ASK, and that seems to be working pretty well. There’s more clarity for users around the experience than ever before. We’ve run contests and incentives with varying degrees of success (one in the O’Keeffe show, which we’ll blog about later). Still we struggle with use rate. It’s time to admit fully and completely that the thing that stands in our way is the fact that ASK is an app. ERm’s evaluation told us this and our own experience echoes this truth. We’re seeing across the industry that people don’t want to download apps. Colleen Dilenschneider has pulled together some great research on this. It’s somewhat heartening to know that we’re not alone in the struggle, but frustrating if you, like us, have built an app!

So now, what? As I hinted at the beginning of this post, we’re running a series of pilot tests to figure out our next move. Two of the pilots focus on removing the app format as a barrier and the third focuses on heightening the motivation and value-add factor of downloading. The pilots will take a total of six weeks, two weeks for each, and we’ll know a lot more at the end of this journey.

For the first pilot, which finished last week, we offered visitors iPods loaded with the app. They were free for check-out the first week, available for a rental fee of $5 the second week. I must say, I was surprised by the results…which I’ll share next week in the first of our series of posts about the pilots. Stay tuned.