New York City’s AI task force stalls

In May 2018, Mayor Bill de Blasio announced the formation of the Automated Decision Systems Task Force, a cross-disciplinary group of city officials and experts in artificial intelligence (AI), ethics, privacy, and law. Established by Local Law 49, the goal of the ADS Task Force is to develop a process for reviewing algorithms the city uses—such as those for determining public school assignments, predicting which buildings should be inspected, and fighting tenant harassment—through the lens of equity, fairness, and accountability.

But nearly one year later, little progress has been made, casting doubt that the task force will fulfill its mandate: issuing a report of policy recommendations by fall 2019.

“My major concern is the task force has been on a trajectory of nothing. A lot of time has been wasted,” says Rashida Richardson, director of policy research at AI Now, a research institute at NYU that focuses on the social implications of artificial intelligence. (AI Now co-founder Meredith Whittaker is a member of the task force.) “Squandering almost a year’s worth of time makes me concerned about the value and robustness of the final product.”

Automated decision systems have been in use in city government for many years. Because of their opaque nature (they’re often off-the-shelf products from private companies) and the fact that there’s little knowledge of what systems are actually in use, there has been little governmental oversight and accountability.

Meanwhile, many of these systems are biased and flawed. The risk assessment algorithm used by Broward County, Florida, to predict future criminals was the subject of a ProPublica expose on racially biased software. After an algorithm in use by the Arkansas Department of Public Health began dramatically reducing benefits for Medicaid recipients, the state was sued. A judge ordered the state to stop using the automated system for determining home health care hours.

And in the 1970s, a flawed algorithm informing FDNY station closures left broad swathes of the city susceptible to fire, disproportionately affecting predominantly low-income black and Latino neighborhoods. Matching algorithms used by NYC public schools have favored white students and disadvantaged students of color.

Local Law 49 was praised as a significant step toward achieving equity and fairness in New York City. But there were clear challenges from the very beginning: The law was broad, sweeping, and ambitious. It requires a level of transparency that many agencies—like the NYPD, which frequently does not disclose information publicly, citing interference with public safety—and the tech companies that develop these products are not accustomed to.

At an April 4 hearing before the City Council Committee on Technology, task force co-chair Jeff Thamkittikasem, director of the Mayor’s Office of Operations, testified that the group has not reached consensus about what constitutes an automated decision system, despite meeting about 20 times over the past year.

“The task force has spent time looking at what falls under an agency ADS; it’s taken more time than we thought it would,” Thamkittikasem said, adding that because the law’s definition of ADS is broad, members flagged a vast array of computer models along the spectrum, including sophisticated machine learning models, as well as “calculators and advanced Excel spreadsheets.”

Thamkittikasem also told the council that the task force does not know what automated decision systems are in use, does not plan to create or disclose a list of systems the city uses, and has not held any public meetings.

At the hearing, members of the task force, along with data experts and privacy advocates expressed frustration with the lack of progress and reluctance to disclose what automated systems are in use.

In prepared remarks, Janet Haven, executive director of Data & Society, a New York-based research group focused on the social and cultural issues surrounding AI and data-centric technology, said, “We have seen little evidence that the task force is living up to its potential. New York has a tremendous opportunity to lead the country in defining these new public safeguards, but time is growing short to deliver on the promise of this body.”

During his testimony to City Council, Albert Fox Cahn, a privacy advocate who departed the group in December, voiced alarm about mismanagement and disempowering of the task force. One issue was the use of the Jain Family Foundation, a non-profit research institute that the city hired (they worked pro bono) to help provide project management and research support. It was never an official member of the task force, yet its scope increased as time went on from providing background research to authoring proposed language and policy documents for the task force to ratify.

“Increasingly, the foundation was writing a first draft of the task force’s report,” Cahn told the City Council during the hearing. “The foundation’s role drew complaints from numerous task force members, so it was eventually phased out, but it’s a telling example of how the role of task force members themselves was circumscribed as part of this process.”

The Jain Family Foundation’s work included attempts to define an ADS. They presented options for the group to vote on, but since the definitions did not reflect the views of the task force, they did not reach consensus. The Jain Family Foundation stopped its work in December.

“Everything about the task force report was ambiguous and up to the task force to decide, except the definition of an automated decision system,” Cahn later told Curbed. “That was the one clear thing presented by the City Council [in the Local Law] and it was unfortunate that the task force hasn’t operated from the baseline understanding as defined by the Council … I believe it was the Mayor’s Office that raised fears that [was an overly expansive definition. During the hearing [the task force chairs] talked about not wanting every Excel document scrutinized. Something important to understand in this discussion is some of the most powerful and sweeping tools can be run on relatively simple platforms.”

Task force members Julia Stoyanovich, a data science, computer science, and engineering professor at NYU, and Solon Barocas, a Cornell professor focusing on the ethics of machine learning, submitted joint testimony to the City Council that expressed particular concern over the lack of information made available to them, stressing the importance of knowing about actual systems in use. Without real-life data sets and case studies, the recommendations would be generic and ineffective for New York City’s needs, and could have been completed using existing academic research.

“A report based on hypothetical examples, rather than on actual NYC systems, will remain abstract and inapplicable in practice,” they wrote. “The task force cannot issue actionable and credible recommendations without some knowledge of the systems to which they are intended to apply … The apparent lack of commitment to transparency on the part of task force leadership casts doubt on the City’s intentions to seriously consider or enact the report’s recommendations—recommendations largely about transparency.”

City officials are also growing impatient. In a March 26 letter to Thamkittikasem, Comptroller Scott Stringer emphasized the importance of algorithmic accountability and expressed disappointment in the task force’s work to date, particularly that disclosure of automated decision systems has not occurred. He requested a list of all algorithms that inform public services or placement in a public facility—like school selection, homeless shelter placement, bail determinations, domestic violence interventions, and child protective services—by May 26, as well as information about how each is used and how they were developed.

“Algorithms should be subject to the same scrutiny with which we treat any regulation, standard, rule, or protocol. It is essential that they are highly vetted, transparent, accurate and do not generate injurious, unintended consequences,” Stringer wrote. “Without such oversight, misguided or outright inaccurate algorithms can fester and lead to increasingly problematic outcomes for city residents, employees, and contractors.”

This lack of progress to date reflects the overall difficulty of regulating technology, a field that’s coming under increased scrutiny at federal, state, and local levels. This month, the House and Senate introduced the Algorithmic Accountability Act, which, if passed, would require the FTC to create rules for assessing the impact of automated decision systems. HUD recently sued Facebook for housing discrimination in its ads, the New York Civil Liberties Union is suing ICE for its immigrant risk assessment algorithm, and a Connecticut judge recently ruled that tenant screening companies that use algorithmic risk assessments must comply with fair housing rules.

Five months after New York City announced the ADS Task Force, Vermont announced a statewide Artificial Intelligence Task Force, which had similar directives as New York City’s: to make recommendations on oversight and regulation of algorithmic systems in use. It’s held multiple public meetings and is due to release its report in June, showing that with determination and proper support from government institutions, this type of work, while difficult and uncharted, is possible in a timely manner.

To help improve transparency, AI Now compiled a list of all the automated decision systems it knows the city uses, which is far from an exhaustive list. The ADS Task Force is due to host its first public forum on April 30 at New York Law School. More details here.