Mark noticed something amiss together with his toddler. His son’s penis looked swollen and was hurting him. Mark, a stay-at-home dad in San Francisco, grabbed his Android smartphone and took photos to document the issue so he could track its progression.
It was a Friday night in February 2021. His wife called an advice nurse at their health care provider to schedule an emergency consultation for the following morning, by video since it was a Saturday and there was a pandemic happening. The nurse said to send photos so the doctor could review them prematurely.
Mark’s wife grabbed her husband’s phone and texted a couple of high-quality close-ups of their son’s groin area to her iPhone so she could upload them to the health care provider’s messaging system. In a single, Mark’s hand was visible, helping to higher display the swelling. Mark and his wife gave no thought to the tech giants that made this quick capture and exchange of digital data possible, or what those giants might consider the pictures.
With help from the photos, the doctor diagnosed the problem and prescribed antibiotics, which quickly cleared it up. However the episode left Mark with a much larger problem, one that might cost him greater than a decade of contacts, emails and photos, and make him the goal of a police investigation. Mark, who asked to be identified only by his first name for fear of potential reputational harm, had been caught in an algorithmic net designed to snare people exchanging child sexual abuse material.
Because technology corporations routinely capture a lot data, they’ve been pressured to act as sentinels, examining what passes through their servers to detect and forestall criminal behavior. Child advocates say the businesses’ cooperation is important to combat the rampant online spread of sexual abuse imagery. But it might probably entail peering into private archives, equivalent to digital photo albums — an intrusion users may not expect — that has forged innocent behavior in a sinister light in a minimum of two cases The Times has unearthed.
Jon Callas, a technologist on the Electronic Frontier Foundation, a digital civil liberties organization, called the cases canaries “on this particular coal mine.”
“There could possibly be tens, lots of, hundreds more of those,” he said.
Given the toxic nature of the accusations, Mr. Callas speculated that almost all people wrongfully flagged wouldn’t publicize what had happened.
“I knew that these corporations were watching and that privacy shouldn’t be what we might hope it to be,” Mark said. “But I haven’t done anything flawed.”
The police agreed. Google didn’t.
‘A Severe Violation’
After establishing a Gmail account within the mid-aughts, Mark, who’s in his 40s, got here to rely heavily on Google. He synced appointments together with his wife on Google Calendar. His Android smartphone camera backed up his photos and videos to the Google cloud. He even had a phone plan with Google Fi.
Two days after taking the photos of his son, Mark’s phone made a blooping notification noise: His account had been disabled due to “harmful content” that was “a severe violation of Google’s policies and is perhaps illegal.” A “learn more” link led to a list of possible reasons, including “child sexual abuse & exploitation.”
Mark was confused at first but then remembered his son’s infection. “Oh, God, Google probably thinks that was child porn,” he thought.
In an unusual twist, Mark had worked as a software engineer on a big technology company’s automated tool for taking down video content flagged by users as problematic. He knew such systems often have a human within the loop to be certain that computers don’t make a mistake, and he assumed his case could be cleared up as soon because it reached that person.
He filled out a form requesting a review of Google’s decision, explaining his son’s infection. At the identical time, he discovered the domino effect of Google’s rejection. Not only did he lose emails, contact information for friends and former colleagues, and documentation of his son’s first years of life, his Google Fi account shut down, meaning he needed to get a latest phone number with one other carrier. Without access to his old phone number and email address, he couldn’t get the safety codes he needed to check in to other web accounts, locking him out of much of his digital life.
“The more eggs you may have in a single basket, the more likely the basket is to interrupt,” he said.
In an announcement, Google said, “Child sexual abuse material is abhorrent and we’re committed to stopping the spread of it on our platforms.”
A number of days after Mark filed the appeal, Google responded that it will not reinstate the account, with no further explanation.
Mark didn’t understand it, but Google’s review team had also flagged a video he made and the San Francisco Police Department had already began to analyze him.
How Google Flags Images
The day after Mark’s troubles began, the identical scenario was playing out in Texas. A toddler in Houston had an infection in his “intimal parts,” wrote his father in a web based post that I stumbled upon while reporting out Mark’s story. On the pediatrician’s request, Cassio, who also asked to be identified only by his first name, used an Android to take photos, which were backed up routinely to Google Photos. He then sent them to his wife via Google’s chat service.
Cassio was in the course of buying a house, and signing countless digital documents, when his Gmail account was disabled. He asked his mortgage broker to change his email address, which made the broker suspicious until Cassio’s real estate agent vouched for him.
“It was a headache,” Cassio said.
Images of kids being exploited or sexually abused are flagged by technology giants thousands and thousands of times annually. In 2021, Google alone filed over 600,000 reports of kid abuse material and disabled the accounts of over 270,000 users in consequence. Mark’s and Cassio’s experiences were drops in a giant bucket.
The tech industry’s first tool to significantly disrupt the vast online exchange of so-called child pornography was PhotoDNA, a database of known images of abuse, converted into unique digital codes, or hashes; it could possibly be used to quickly comb through large numbers of images to detect a match even when a photograph had been altered in small ways. After Microsoft released PhotoDNA in 2009, Facebook and other tech corporations used it to root out users circulating illegal and harmful imagery.
“It’s a terrific tool,” the president of the National Center for Missing and Exploited Children said on the time.
An even bigger breakthrough got here along almost a decade later, in 2018, when Google developed an artificially intelligent tool that might recognize never-before-seen exploitative images of kids. That meant finding not only known images of abused children but images of unknown victims who could potentially be rescued by the authorities. Google made its technology available to other corporations, including Facebook.
When Mark’s and Cassio’s photos were routinely uploaded from their phones to Google’s servers, this technology flagged them. Jon Callas of the E.F.F. called the scanning intrusive, saying a family photo album on someone’s personal device needs to be a “private sphere.” (A Google spokeswoman said the corporate scans only when an “affirmative motion” is taken by a user; that features when the user’s phone backs up photos to the corporate’s cloud.)
“That is precisely the nightmare that we’re all concerned about,” Mr. Callas said. “They’re going to scan my family album, after which I’m going to get into trouble.”
A human content moderator for Google would have reviewed the photos after they were flagged by the unreal intelligence to verify they met the federal definition of kid sexual abuse material. When Google makes such a discovery, it locks the user’s account, searches for other exploitative material and, as required by federal law, makes a report back to the CyberTipline on the National Center for Missing and Exploited Children.
The nonprofit organization has grow to be the clearinghouse for abuse material; it received 29.3 million reports last yr, or about 80,000 reports a day. Fallon McNulty, who manages the CyberTipline, said most of those are previously reported images, which remain in regular circulation on the web. So her staff of 40 analysts focuses on potential latest victims, so that they can prioritize those cases for law enforcement.
“Generally, if NCMEC staff review a CyberTipline report and it includes exploitative material that hasn’t been seen before, they may escalate,” Ms. McNulty said. “Which may be a toddler who hasn’t yet been identified or safeguarded and isn’t out of harm’s way.”
Ms. McNulty said Google’s astonishing ability to identify these images so her organization could report them to police for further investigation was “an example of the system working because it should.”
CyberTipline staff members add any latest abusive images to the hashed database that’s shared with technology corporations for scanning purposes. When Mark’s wife learned this, she deleted the photos Mark had taken of their son from her iPhone, for fear Apple might flag her account. Apple announced plans last yr to scan the iCloud for known sexually abusive depictions of kids, however the rollout was delayed indefinitely after resistance from privacy groups.
In 2021, the CyberTipline reported that it had alerted authorities to “over 4,260 potential latest child victims.” The sons of Mark and Cassio were counted amongst them.
‘No Crime Occurred’
In December 2021, Mark received a manila envelope within the mail from the San Francisco Police Department. It contained a letter informing him that he had been investigated in addition to copies of the search warrants served on Google and his web service provider. An investigator, whose contact information was provided, had asked for every part in Mark’s Google account: his web searches, his location history, his messages and any document, photo and video he’d stored with the corporate.
The search, related to “child exploitation videos,” had taken place in February, inside every week of his taking the photos of his son.
Mark called the investigator, Nicholas Hillard, who said the case was closed. Mr. Hillard had tried to get in contact with Mark but his phone number and email address hadn’t worked.
“I made up my mind that the incident didn’t meet the weather of against the law and that no crime occurred,” Mr. Hillard wrote in his report. The police had access to all the data Google had on Mark and decided it didn’t constitute child abuse or exploitation.
Mark asked if Mr. Hillard could tell Google that he was innocent so he could get his account back.
“You may have to consult with Google,” Mr. Hillard said, based on Mark. “There’s nothing I can do.”
Mark appealed his case to Google again, providing the police report, but to no avail. After getting a notice two months ago that his account was being permanently deleted, Mark spoke with a lawyer about suing Google and the way much it may cost a little.
“I made a decision it was probably not value $7,000,” he said.
Kate Klonick, a law professor at St. John’s University who has written about online content moderation, said it might probably be difficult to “account for things which can be invisible in a photograph, just like the behavior of the people sharing a picture or the intentions of the person taking it.” False positives, where individuals are erroneously flagged, are inevitable given the billions of images being scanned. While most individuals would probably consider that trade-off worthwhile, given the advantage of identifying abused children, Ms. Klonick said corporations need a “robust process” for clearing and reinstating innocent people who find themselves mistakenly flagged.
“This is able to be problematic if it were only a case of content moderation and censorship,” Ms. Klonick said. “But that is doubly dangerous in that it also leads to someone being reported to law enforcement.”
It might have been worse, she said, with a parent potentially losing custody of a toddler. “You possibly can imagine how this might escalate,” Ms. Klonick said.
Cassio was also investigated by the police. A detective from the Houston Police department called in the autumn of 2021, asking him to come back into the station.
After Cassio showed the detective his communications with the pediatrician, he was quickly cleared. But he, too, was unable to get his decade-old Google account back, despite being a paying user of Google’s web services. He now uses a Hotmail address for email, which individuals mock him for, and makes multiple backups of his data.
You Don’t Necessarily Know It When You See It
Not all photos of naked children are pornographic, exploitative or abusive. Carissa Byrne Hessick, a law professor on the University of North Carolina who writes about child pornography crimes, said that legally defining what constitutes sexually abusive imagery may be complicated.
But Ms. Hessick said she agreed with the police that medical images didn’t qualify. “There’s no abuse of the kid,” she said. “It’s taken for nonsexual reasons.”
In machine learning, a pc program is trained by being fed “right” and “flawed” information until it might probably distinguish between the 2. To avoid flagging photos of babies in the tub or children running unclothed through sprinklers, Google’s A.I. for recognizing abuse was trained each with images of probably illegal material found by Google in user accounts prior to now and with images that weren’t indicative of abuse, to offer it a more precise understanding of what to flag.
I actually have seen the photos that Mark took of his son. The choice to flag them was comprehensible: They’re explicit photos of a toddler’s genitalia. However the context matters: They were taken by a parent fearful a few sick child.
“We do recognize that in an age of telemedicine and particularly Covid, it has been mandatory for folks to take photos of their children in an effort to get a diagnosis,” said Claire Lilley, Google’s head of kid safety operations. The corporate has consulted pediatricians, she said, in order that its human reviewers understand possible conditions that may appear in photographs taken for medical reasons.
Dr. Suzanne Haney, chair of the American Academy of Pediatrics’ Council on Child Abuse and Neglect, advised parents against taking photos of their children’s genitals, even when directed by a health care provider.
“The final thing you wish is for a toddler to get comfortable with someone photographing their genitalia,” Dr. Haney said. “In the event you absolutely need to, avoid uploading to the cloud and delete them immediately.”
She said most physicians were probably unaware of the risks in asking parents to take such photos.
“I applaud Google for what they’re doing,” Dr. Haney said of the corporate’s efforts to combat abuse. “We do have a horrible problem. Unfortunately, it got tied up with parents attempting to do right by their kids.”
Cassio was told by a customer support representative earlier this yr that sending the images to his wife using Google Hangouts violated the chat service’s terms of service. “Don’t use Hangouts in any way that exploits children,” the terms read. “Google has a zero-tolerance policy against this content.”
As for Mark, Ms. Lilley, at Google, said that reviewers had not detected a rash or redness within the photos he took and that the following review of his account turned up a video from six months earlier that Google also considered problematic, of a young child lying in bed with an unclothed woman.
Mark didn’t remember this video and not had access to it, but he said it appeared like a personal moment he would have been inspired to capture, not realizing it will ever be viewed or judged by anyone else.
“I can imagine it. We woke up one morning. It was an attractive day with my wife and son and I desired to record the moment,” Mark said. “If only we slept with pajamas on, this all might have been avoided.”
A Google spokeswoman said the corporate stands by its decisions, regardless that law enforcement cleared the 2 men.
Guilty by Default
Ms. Hessick, the law professor, said the cooperation the technology corporations provide to law enforcement to deal with and root out child sexual abuse is “incredibly necessary,” but she thought it should allow for corrections.
“From Google’s perspective, it’s easier to simply deny these people using their services,” she speculated. Otherwise, the corporate would need to resolve tougher questions on “what’s appropriate behavior with kids after which what’s appropriate to photograph or not.”
Mark still has hope that he can get his information back. The San Francisco police have the contents of his Google account preserved on a thumb drive. Mark is now attempting to get a duplicate. A police spokesman said the department is desperate to help him.
Nico Grant contributed reporting. Susan Beachy contributed research.