How Google's art app matches your selfies with museum paintings

Been near a phone or computer screen in the past few days? Odds are you have seen a Google "art selfie" — the photo of a friend alongside a painting of someone else who looks vaguely similar.

This bit of software cleverness is the viral sensation of the moment, as users had matched more than 30 million selfies with paintings by Wednesday afternoon.

Are you the modern-day incarnation of landed gentry from 18th-century England? Or do you share the pale features of a 19th-century Norwegian lass, gazing wistfully from a balcony? Perhaps the streetwise savvy of a giant Los Angeles mural is more your thing.

It is all done with facial-recognition software, which has been around for years — used in such applications as Facebook tagging, passport and driver's license screening, and even identifying ticket holders for admission to concerts.

So Google's innovation did not consist of inventing a technology, but applying it in a novel fashion, said Jeff Salvage, a computer science professor at Drexel University.

"As I tell my students, with the internet, a computer, and the ability to do research — if you have a great idea, you can just build it," he said.

The selfie app matches the user's face with its closest counterpart from among thousands of portraits in museums, galleries, and murals around the world — in many cases works by lesser-known artists. The app was launched in December, an addition to the two-year-old Google Arts & Culture platform, but it took off only in the past week.

"This took us by surprise," company spokesman Patrick Lenihan said. "We're thrilled that people are using it to explore artwork and some artists they might not have found otherwise."

The main goal of the Arts & Culture platform is to help museums and nonprofits share their digitized collections with a broader audience, with Google lending the technical know-how as needed. The Philadelphia Museum of Art and the city's Mural Arts Program are among more than 1,500 participants worldwide.

As the selfie app racked up more users, some critics have warned that a mass collection of facial images would pose a threat to privacy.

Not in this case, said Google's Lenihan. The images are not saved, but are discarded immediately after being matched with artwork.

Google shared few details about its particular version of facial-recognition technology. But in the simplest terms, this type of software would employ a two-step process, Drexel's Salvage said.

The software must first identify a face within an image, detecting where the person ends and the background begins — similar to the function built in to most digital cameras. Then it must extract features and see how they match up with others.

That might mean calculating the relative proportions and placement of the eyes, nose, and mouth, or perhaps directing the software to find hidden mathematical relationships between other facial dimensions — the sort of imperceptible patterns that would emerge only through "machine learning."

"What you're trying to do is give it some rules, but also in general trying to compute information from the data without telling it what to do," Salvage said. "You don't know what you're looking for at the beginning."

The app can be downloaded at the Google Play store and at Apple's app store. Once installed, the user must scroll down past the home screen to the message "Is your portrait in a museum?"

The app allows the user to take either a selfie or a photo facing the other direction aimed at anyone else's face, as we have done with several Philadelphia notables. Shown an image of his art twin, 18th-century merchant John Wright Stanly, Mayor Kenney said he could see some resemblance.

"I don't wear a wig," he said. "But the nose is probably pretty close."