Geoffrey Cain on Big Data, AI, and the Specter of “Security”
if you want to be scared about the future a well written article about China’s state security apparatus and AI will do the job. It almost makes one nostalgic for the likes of Ceau?escu…
Irfan was a technology worker in his thirties who came from Urumqi, the regional capital of Xinjjiang and commercial hub in northern Xinjiang, where he implemented mass surveillance projects until he left his job in 2015. He fled Xinjiang in 2018 and settled in Turkey.
Irfan had originally experienced trouble finding a job in the poor Urumqi region. Many of the top jobs went to Han Chinese migrants who sidelined the local Uyghurs.
“But I had a friend who had some access to the mayor of Urumqi,” Irfan explained. “He put in a call for me. And then the telecom company contacted me in 2007 with an offer. They needed an IT manager who could help set up one of the first surveillance systems there.
In his new job, Irfan was granted access to two key locations: the local public security office, where he helped manage the company surveillance network, and the telecom company’s network itself.
“Our mandate was to scour the city and plant cameras everywhere we could. My supervisors said the government wanted us to fight crime. I accepted that. I thought it was an honorable mission.”
With a small team, Irfan searched street corners, alleyways, streets where cars tended to speed, and pockets known for robberies and purse snatchings, as documented in municipal data. Then his team would connect cameras to optic cables that went back to the public security buildings, where police operatives would survey the city from a control room.
After installing the latest in what quickly became a vast network of cameras, Irfan would return to his control room in a nondescript concrete building, where he and other information technology workers sat in front of large wall-mounted video screens.
Throughout 2010 and 2011, Irfan and his colleagues became increasingly aware of how they could train AI algorithms to recognize faces and behaviors, match them up with a national database of citizens, and help police find perpetrators.
“We had the hardware, we had the cameras, we had almost everything we needed to make this work,” he said. “But we realized that we were missing the key ingredient: we needed more data.
“Otherwise, the facial recognition technology was useless. The AI needed data, in the form of either facial images, social media, criminal records, credit card swipes, or whatever other data resulted from some kind of activity or transaction. Then the system could plough through all the information we fed it and find correlations that humans couldn’t, in a fraction of the time.”
“Why was there so little data?” I asked him.
“State secrecy,” Irfan replied. “The government didn’t have good information on its own country and its people. And so we didn’t have the quality data that we needed to feed into the AI software. Without a good database on all our citizens, we couldn’t match up people’s faces or criminal records that easily. We couldn’t use AI to catch criminals. It was a terrible system.”
Irfan’s team scoured other company offices and the government for data. They came up empty-handed. “The solution didn’t come from the government,” Irfan confirmed. “It came from the corporations.”
For Chinese companies, the ability to control massive digital payment platforms—Chinese consumers didn’t use credit cards, preferring to pay with their mobile apps—naturally led to the urge to expand into credit rankings, making use of all that data on hundreds of millions of people who were making payments every day. What if they could additionally be ranked in categories like “trustworthiness,” based on their online shopping and payment activity? It was like a credit score but more all-encompassing.
In rapidly developing countries like China, millions of people had never had access to traditional credit and hadn’t established a credit score. By gathering mass data on everyone, China could leapfrog the credit hurdle and empower people across the country, giving them access to loans.
But social credit was sinister too. Li Yingyun, an executive at one credit service, Sesame Credit, told the Chinese magazine Caixin that “someone who plays video games for 10 hours a day, for example, would be considered an idle person, and someone who frequently buys diapers would be considered as probably a parent, who on balance is more likely to have a sense of responsibility.”
People all over China with better social credit scores would qualify for harmless benefits: VIP bookings at hotels and car rentals, and more prominent profiles on dating websites. Those who fell too low on the system could be denied bank loans and apartment rentals.
But China’s social credit system was far from centralized into an Orwellian panopticon that documented everything. China’s impenetrable bureaucracies and office politics stood in the way. A Tencent employee admitted to me that “each division in the company was reluctant to share the data being gathered from each platform we owned like QQ and WeChat. The affiliates and the offices inside those companies all competed with each other.”
What if consumers could be ranked in categories like “trustworthiness,” based on their online shopping and payment activity?
