AI helps scour video archives for evidence of human-rights abuses

THANKS ESPECIALLY to ubiquitous camera-phones, today’s wars have been filmed more than any in history. Consider the growing archives of Mnemonic, a Berlin charity that preserves video that purports to document war crimes and other violations of human rights. If played nonstop, Mnemonic’s collection of video from Syria’s decade-long war would run until 2061. Mnemonic also holds seemingly bottomless archives of video from conflicts in Sudan and Yemen. Even greater amounts of potentially relevant additional footage await review online.

Outfits that, like Mnemonic, scan video for evidence of rights abuses note that the task is a slog. Some trim costs by recruiting volunteer reviewers. Not everyone, however, is cut out for the tedium and, especially, periodic dreadfulness involved. That is true even for paid staff. Karim Khan, who leads a United Nations team in Baghdad investigating Islamic State (IS) atrocities, says viewing the graphic cruelty causes enough “secondary trauma” for turnover to be high. The UN project, called UNITAD, is sifting through documentation that includes more than a year’s worth of video, most of it found online or on the phones and computers of captured or killed IS members.

Now, however, reviewing such video is becoming much easier. Technologists are developing a type of artificial-intelligence (AI) software that uses “machine vision” to rapidly scour video for imagery that suggests an abuse of human rights has been recorded. It’s early days, but the software is promising. A number of organisations, including Mnemonic and UNITAD, have begun to operate such programs.

This year UNITAD began to run one dubbed Zeteo. It performs well, says David Hasman, one of its operators. Zeteo can be instructed to find—and, if the image resolution is decent, typically does find—bits of video showing things like explosions, beheadings, firing into a crowd and grave-digging. Zeteo can also spot footage of a known person’s face, as well as scenes as precise as a woman walking in uniform, a boy holding a gun in twilight, and people sitting on a rug with an IS flag in view. Searches can encompass metadata that reveals when, where and on what devices clips were filmed.

Zeteo was developed for the UN, with input from its investigators, by Microsoft. The American software giant has built a few such programs as part of a project it calls AI for Humanitarian Action. The goal is to accelerate prosecutions, says Justin Spelhaug, Microsoft’s head of “technology for social impact”. Half a dozen organisations are now using specially developed Microsoft software to comb video for potential evidence of war crimes and the like. Microsoft provides the technology at little or no cost.

A recent achievement hints at how such capabilities can help. The Atlantic Council, an American think-tank that sees great promise in its tests of machine vision, sought to identify a man who had been photographed in Syria, his face blurred out, holding chopped-off heads. The outfit’s Digital Forensic Research Lab ingeniously studied how squiggly patterns in the man’s camouflage met at seams. After scouring the web for imagery of people in similar fatigues, the researchers found non-blurred images of a man wearing those very fatigues. The researchers have identified the man and his affiliation with the Wagner Group, a Russian mercenary firm.

Developing software that spots certain objects or actions in video is often straightforward. It involves feeding algorithms for object recognition with masses of imagery of whatever is to be found. This means it is relatively easy to train software to recognise leaping cats or other things that abound online. But footage showing a violation of human rights is rarer. This makes it hard to assemble a collection of visual examples that is big and diverse enough to teach software to find similar fare. But there is a creative workaround.

Banned cluster munitions have been dropped on civilians in Syria, and Mnemonic wants to pull together video clips that show that the bombardment has been systematic. To help with that, a programmer in Berlin, Adam Harvey, is developing software called VFRAME. Training it requires at least 2,000 distinct images of each type of cluster munition, and five times as many would be better. Finding that would take ages. Mr Harvey therefore produces the imagery himself.

With funding from Germany’s government and other sources, Mr Harvey 3D-prints replicas of prohibited bomblets such as the AO-2.5RT, a Russian-made submunition dropped in Syria. He adds markings and, for some of the replicas, rust, scuffs and other damage. The replicas are then photographed, from many angles and in different lighting, amid rubble, rocks, leaves, mud and sand. For the realism of greater chromatic variation, a handful of old and new camera-phones, as well as multiple lens settings, are used.

The approach is paying off. In tests on portions of Mnemonic’s Syria and Yemen archives, VFRAME catches roughly 65% of clips that show one of the handful of types of cluster munitions modelled. Mr Harvey expects the detection rate to reach 80% by mid-June. VFRAME will then be unleashed on Mnemonic’s full archives. As for scanning the “firehose” of video posted on social media, Mnemonic’s Dia Kayyali says testing with VFRAME has begun.

Mnemonic sends video to legal bodies. So far, these have included a Belgian court, war-crimes investigation units in France, Germany and Sweden, and several UN legal teams. But Daanish Masood of the UN’s Department of Political and Peacebuilding Affairs also envisages a use for software that scours online video for violence and its aftermath as a source of operational intelligence, requiring less than legal certainty. He hopes VFRAME will eventually help UN peacekeepers track marauding armed groups.

Similar software seems to have already been put to another intelligence use. The Intelligence Advanced Research Projects Activity (IARPA), an R&D body for America’s spooks, gave Carnegie Mellon University in Pittsburgh $9m to develop machine-vision software called E-LAMP. How that software has been put to use by intelligence agencies is unknown, but it probably includes finding terrorist videos for training and propaganda. Alexander Hauptmann, one of E-LAMP’s creators at Carnegie Mellon, says the software had to be able to spot, among other things, cake-making and cellphone repair, likely proxies for mixing explosives and building detonators.

E-LAMP finds those activities, he says. Even so, such programs can be fiddly. E-LAMP was also given to a Washington, DC non-profit, the Syria Justice and Accountability Centre (SJAC), that seeks video evidence of war crimes. But SJAC stopped using the software after two years. It required too much processing power and maintenance, says its director, Mohammad Al Abdallah. Accuracy was patchy, too. SJAC’s queries for video of small missiles positioned for launch would typically also find electricity poles.

A new trick promises greater accuracy. SJAC is adopting a program called JusticeAI that seeks matches between a video’s audio and a sound library. JusticeAI recognises things such as a missile’s hiss, the popping of cluster munitions, a siren near gunfire and protest chants that turn to screams. Its users include Mnemonic and the UN’s Office of the High Commissioner for Human Rights. The software was developed by a Silicon Valley charity called Benetech with funding from America’s government. Microsoft contributed code and $300,000.

Heady stuff, to be sure. But efforts to archive as much video of abuses as possible, be it for prosecutors or historians, face an additional hurdle. Facebook, YouTube and other big online platforms also use such software as a more efficient way of spotting and removing unsavoury images that some regulators claim could inspire copycats. Compared to those companies, human-rights groups are mere “poor cousin” users of such technology, laments Sam Gregory of Witness, a Brooklyn charity that helps people film abuses.

It adds up to a paradox. The software advances that are now helping human-rights groups document atrocities are also making it easier for social-media platforms to suppress potential evidence. Mr Gregory argues for the creation of “evidence lockers”—repositories that would keep grisly video out of the public eye but available for authorised viewing. The proposal seems sensible. Momentum, however, has yet to build, even though the matter has become more urgent. On April 29th the European Parliament approved a rule that threatens online platforms with eye-watering fines for not removing, within an hour, content a member state deems terrorist. As a result, automated deletions are up.