Unlocking the FCC data at Hack for Western Mass

A team of civic-minded hackers in Western Massachusetts have built a tool to make it easier to access documents about the FCC proceeding on regulating the prison telephone industry.

by Peter Wagner, June 3, 2013

Challenge representative Peter Wagner of the Prison Policy Initiative explaining how the FCC organizes data by separate “proceedings”. (Photo: Stephen Brewer)

Sondra Morin, Gyepi Sam, and Al Nutile exploring the name space of the FCC’s database structure. (Photo: Stephen Brewer)

Al Nutile (center) sharing an idea with Gyepi Sam (left) and Peter Wagner (right). (Photo: Stephen Brewer)

John Tobey and daughter weigh in on the database structure. (Photo: Stephen Brewer)

Jake Mitchell reviewing the project plan. (Photo: Stephen Brewer)

The team listening to the ideas of Aaron Smith (rear, left) about his discoveries on how the FCC was organizing their data. (Photo: Molly McLeod)

Gyepi Sam explains the team’s code to the Hack for Western Mass group. (Photo: Molly McLeod)

A team of civic-minded hackers in Western Massachusetts have built a tool to make it easier for advocates, policymakers and journalists to access documents and weigh in on telecommunications debates currently before the Federal Communications Commission.

Critical debates about telecommunications policy are archived in plain sight on the FCC’s website, but they are organized in a way that’s extremely difficult to use. Focusing on just one issue currently before the FCC — the possible creation of price caps to reduce the $1/minute cost of phone calls from prisons and jails — the team created an alternative interface for the FCC’s data. This dataset was particularly challenging because it includes the comments of about 100,000 people in approximately 7,000 pdf files, and the content ranged from well-formatted pdf files to bad scans of handwritten letters from incarcerated people.

On the first morning of Hack for Western Mass, I made a short pitch about the prison phone industry and the need to make the data more accessible. A team was formed to make this data searchable and to add some basic tag support so that journalists and others could more easily find relevant filings.

The software written by the team scrapes the FCC website for new comments, downloads the pdf files, extracts or OCRs any text, and stores that text in a database so that it can be searched. When someone finds a document of interest we then link back to the original PDF on the FCC’s website.

Our software also imports all of the meta data currently available on the FCC website (submitter, date, address etc.) and contains a tagging system so that documents can have metadata added that is useful to the people interested in a specific proceeding. For example, the FCC scans in all letters that come in to proceeding in batches and labels them all with the author “numerous”. To fix this, our system allows users to manually tag each page with two pieces of meta data: the state of origin and one of a number of different content types: whether the document or page is from telephone companies, correctional system administrators, state regulators, incarcerated people or the people who aren’t incarcerated and have to pay the high rates required by the monopolistic contracts. With these tags, users can search for keywords and filter by the existing tags. In this way, a journalist or a member of Congress could quickly find the comments of people from their particular area.

Current progress

The basic structure of the site and tool is done, but the data is still being processed. The scraper was written in Python, and a series of Python scripts then convert PDFs to the portable network map format, clean the images using OnPaper, and perform the optical character recognition using GOCR. Rails manages the database and ActiveAdmin GEM provides an interface to manage content for authenticated users. Solr is setup as well for fast search results. The scripts to download and process PDFs from the FCC website work, and thousands of documents are currently being processed. Tagging has already begun in a spreadsheet, and when the data is fully imported, we’ll set it up to add new FCC entries nightly and we’ll make an interface so that the tagging can be crowdsourced.

Future work and adaptability of the code

This project was conceived to empower people concerned about prison telephone regulation to have better access to this data. But it could be readily adapted to other issues of concern at the FCC. (And could, perhaps, inspire some improvements in how the FCC manages and publishes public comments.)

One idea we didn’t get to — and can’t until the text importation is complete — is to use our database of the text of the filings to perform various kinds of statistical analyses about who is making which arguments.

A final website with the data will be unveiled soon after the data processing is complete. The team’s code is available now on Github.

Many thanks to this weekend’s team:

• Jennie D’Ambroise
• Sam Duncan
• Jonathan Hills
• Jake Mitchell
• Sondra Morin
• Alfred Nutile
• Gyepi Sam
• Aaron Smith
• John Tobey

Stay tuned for the public launch of the FCC tool!

Thoughts on processing fees in the jail telephone industry

Some additional observations about prison phone fees, based on discussions we had with the phone companies after the report’s release.

by Peter Wagner, May 29, 2013

In our report, Please Deposit All of Your Money: Kickbacks, Rates, and Hidden Fees in the Jail Phone Industry, we catalog the many fees customers pay at every step of the way, including fees for making deposits, keeping accounts open, and getting refunds. Our report puts these fees in a larger context, and discussions we’ve had since the report’s release lead us to some additional observations about fees. In our report, we wrote:

To be sure, businesses in many industries incur some processing costs by accepting credit or debit cards in person, via the internet, or over the telephone.⁶⁵ Businesses usually respond by setting minimum purchase levels for a take out food order, charging a slightly higher rate per gallon of gasoline, or by simply writing it off as the cost of doing business. But this section of the report suggests that prison telephone companies may be approaching the question from the other end: providing telephone services in order to make money by charging extra fees. Indeed, because the commission system reduces the potential for corporate profit from the telephone calls, fees that should be no more than supplemental income are turned into a central source of profit.

Our report leads two phone companies to clarify & improve policies

As a result of the report, at least two companies, Turnkey and NCIC, made several clarifications and improvements to their fee policies.

by Peter Wagner, May 28, 2013

Several phone companies have been keeping busy in the couple of weeks since we released our new report exposing the hidden fees in the prison phone industry, mostly for the better.

I’m proud to report that, as a result of the report, at least two companies, Turnkey and NCIC, made several clarifications and improvements to their fee policies:

Massachusetts residents say NO to prison visitor dog sniffing policy

The Massachusetts Executive Office of Public Safety and Security claim that “the department has received no public feedback about the plan” to begin using dog sniff searches to screen prison visitors is, at best, incorrect

by Leah Sakala, May 23, 2013

We were concerned as soon as we heard about the Massachusetts Department of Corrections’ plans to start using dogs to screen families and friends visiting loved ones in prison. After all, encouraging in-person visits is key to allowing incarcerated people to maintain the family and community ties that are central to success after release. Using dogs to search family members, friends, clergy, volunteers, and other visitors is deeply invasive and degrading, and can turn essential family visits into potentially traumatizing experiences.

We (along with many other concerned Massachusetts residents) told the Department of Corrections, the Executive Secretary of Public Safety, and the Governor’s Office that the dog sniffing policy is a terrible idea. And on April 10th, the Boston Globe Editorial Board warned the Department of Corrections about the grave consequences the policy could have.

So, we were very surprised to read in a recent Valley Advocate article that Terrel Harris, spokesman for the Executive Office of Public Safety and Security, claims that “the department has received no public feedback about the plan.” It’s clear that this statement is, at best, incorrect, as we’ve clarified in a Letter to the Editor printed in this week’s issue.

The message is being sent loud and clear: the public knows that this policy will unnecessarily deter family visits and thereby make it harder for people released from custody to successfully rejoin their communities. The dog sniffing policy has already been put on hold; it should be canceled outright.

Please Deposit All of Your Money: Kickbacks, Rates, and Hidden Fees in the Jail Phone Industry

The newest Prison Policy Initiative report exposes the prison phone industry's hidden fees, which saddle the families of incarcerated people with staggeringly high phone bills.

by Leah Sakala, May 8, 2013

The newest Prison Policy Initiative report exposes the prison phone industry’s hidden fees, which saddle the families of incarcerated people with staggeringly high phone bills.

Prison Policy Initiative on North Country Public Radio

by Leah Sakala, April 17, 2013

North Country Public Radio’s Brian Mann talks with Peter Wagner about an encouraging trend towards declining prison populations.

Letter to MA Dept. of Corrections: Families shouldn’t be screened by dogs

by Leah Sakala, March 25, 2013

The Massachusetts Department of Corrections plans to begin using drug sniffing dogs to screen families and friends who come to visit loved ones in prison. We submitted a letter urging Department of Corrections Commissioner Louis Spencer to halt this harmful and degrading policy in its tracks.

Letter urging the Massachusetts Department of Corrections to not subject prison visitors to dog searches

The Massachusetts Department of Corrections should immediately cancel the plan to use dogs to screen visitors to correctional facilities

by Peter Wagner, March 21, 2013

March 21, 2013
Commissioner Luis S. Spencer
Massachusetts Department of Corrections
50 Maple Street, Suite 3
Milford, MA 01757-3698

Dear Commissioner Spencer,

I am writing as the Executive Director of the Prison Policy Initiative based in Easthampton, MA to strongly urge the Massachusetts Department of Corrections to immediately cancel the plan to use dogs to screen visitors to Department of Corrections facilities. This policy will unnecessarily deter family visits and thereby make it harder for people released from custody to successfully rejoin their communities.

Social science research has repeatedly found that allowing incarcerated people to preserve family and community ties is essential for facilitating reentry and reducing recidivism. In-person visits allow family members to stay in touch and maintain their relationships through periods of incarceration.[1] Using dogs to search family members, friends, clergy, volunteers, and other visitors is deeply invasive and degrading, and can turn essential family visits into potentially traumatizing experiences. For example, a child who is deeply afraid of dogs will resist visiting an incarcerated parent if he or she must undergo canine screening. Subjecting visitors to dog screening not only harmful, but also unnecessary, as the vast majority of correctional systems around the country maintain secure facilities without resorting to such extreme measures.

Additionally, the nation’s leading correctional professional organizations all agree that correctional facilities should encourage, not discourage, community members who wish to visit their loved ones in prison. The American Correctional Association, for example, urges correctional facilities to support “successful family and community reunification.”[2] The Re-Entry Policy Council, a joint project of the U.S. Department of Justice, the U.S. Department of Labor, and the U.S. Department of Health and Human Services, created to facilitate incarcerated individuals’ reintegration in the community, recommends that correctional institutions “help to re-establish, expand, and strengthen relationships between prisoners and their families.” [3]

The dog sniffing visitor screening policy runs in entirely the wrong direction. Rather than encourage, it will discourage crucial family contact, impeding reentry and contributing to increased recidivism. It should be canceled immediately. Thank you for your consideration of this important matter.

Sincerely,

Peter Wagner

CC: Governor Deval Patrick and Chief of Staff Brendon Ryan; Andrea Cabral, Executive Secretary of Public Safety.

Footnotes:

[1] See, for example, Minnesota Department of Corrections, Effects of Prison Visitation on Offender Recidivism, November 2011. Available at: https://mn.gov/doc/data-publications/research/publications/?id=1089-272991

[2]American Correctional Association, “Public Correctional Policy on Reentry of Offenders,” resolution 2001-3, passed August 15, 2001, reviewed and amended Aug. 16, 2006 and August 6, 2011. Published in American Correctional Association, Public Correctional Policies 2012, 79. Available at: https://www.aca.org/government/policyresolution/PDFs/Public_Correctional_Policies.pdf

[3] Council of State Governments, Reentry Policy Council, Policy Statement 13 in Report of the Re-Entry Policy Council: Charting the Safe and Successful Return of Prisoners to the Community (New York: Council of State Governments, January 2005). Available at: https://csgjusticecenter.org/wp-content/uploads/2020/02/report-of-the-re-entry-policy-council-charting-the-safe-and-successful-return-of-prisoners-to-the-community.pdf

Sign-on letter to Census Bureau: it’s time to end prison gerrymandering

by Leah Sakala, January 28, 2013

Can your organization sign on to a letter calling on the Census Bureau to complete the research necessary to end prison gerrymandering in 2020? The next census isn’t for seven more years, but the research and planning is taking place right now and the Bureau >needs to hear from your organization. Thank you!

Movement victory: FCC proposes to regulate prison phone rates

At a Nov. 15th phone justice rally in Washington D.C., we, along with our partners at SumOfUs, delivered signatures from 36,690 people across the country calling on the FCC to rein in predatory prison phone rates. And it looks like the FCC will act soon -- Commissioner Mignon Clyburn announced that the FCC Chairman has circulated a prison phone regulatory proposal that the commission will now vote on.

by Leah Sakala, November 15, 2012

At a Nov. 15th phone justice rally in Washington D.C., we, along with our partners at SumOfUs, delivered signatures from 36,690 people across the country calling on the FCC to rein in predatory prison phone rates. And it looks like the FCC will act soon — Commissioner Mignon Clyburn announced that the FCC Chairman has circulated a prison phone regulatory proposal that the commission will now vote on.