Lessons Learned from the Census API Debacle

Over the past few months we've learned a lot from the public data made available on the US Census Bureau's website. We learned how tenants are apathetic about community involvement. We learned about the number of immigrants working in apartment maintenance. We found a list of the most and least common apartment types in Chicago. And of course, every RentConfident apartment safety report contains data from the US Census so that renters can get an idea of what to expect from their new neighborhoods. But the census bureau is not infallible. In fact, on Wednesday a simple error in an email from the census bureau this week caused major ripples throughout the open data community. From this event we can learn quite a bit about how readers interpret digital information, and apply that to rental situations.

Those of you who don't do data work might not know what an API is. Basically it's a way for one website to retrieve information from another website automatically without logins or passwords. When a website has a "share on Facebook" button, they are using Facebook's API to submit their content to Facebook. RentConfident uses APIs to regularly pull updated information into our reports from about 30 different sites. (The rest of our data is added in by hand.) We pull code violations from the city of Chicago, map data from Cook County, and of course population data from the US Census Bureau's API.

This week I and many others within the open data community received the following email from the US Census Bureau:

Note that there is a PDF file attachment. I read the email, read the attachment, and chuckled to myself. Then I went to check Twitter. Sure enough on Twitter I saw messages like these:

Surely there is a better way to complying with the HTTPS requirement than completely terminating the Census API? cc: @OliverSherouse [2/2]

— Brett J. Gall (@TheBrettGall) May 31, 2017

To improve security and privacy, and by federal government mandate, the U.S. Census Bureau kills API calls to https://t.co/UpUgeUbaRU pic.twitter.com/S5zlhrefc4

— william fleurant (@wfleurant) June 1, 2017

US Census Bureau is shutting down their API at https://t.co/L1thlNVvbn 🙁 https://t.co/Fa7b0C34gr

— Matt Baggott ?? (@mattbagg) May 31, 2017

People were claiming that the API was shutting down permanently, which would break a lot of websites. People were blaming the government and crying "censorship," but they were mistaken. The cover email was flawed and didn't state the whole story. The PDF attachment - which that last tweet even linked to directly - provided more extensive information stating that the address used to make API calls is going to be moving to a new location (from "HTTP" to "HTTPS"), and that therefore any links to the old server would stop working. Unfortunately, recipients just read the cover email and immediately sent out false information on Twitter. It was a simple courtesy message that went horribly wrong.

The authors of these tweets are very intelligent people. I'm sure most of them are tech folks who know how to debug and read code for those tiny errors that can make a computer program break. I'm sure most of them have college educations. But they still missed the whole point of the email and then spread their misconceptions around.

Upon being notified of the problem, the US Census Bureau issued a clarifying tweet immediately, and sent out a follow up email the next morning. Pretty quick response for a government agency. On Thursday a lot of top open data Twitter users had to issue retractions and apologies for jumping the gun yesterday.

We can learn a lot from this incident.

Attachments are Dead. In this mobile era you cannot expect people to open PDF attachments. Senders may assume that everyone will read the full content of an email, but in reality people skim and may not be able to open attachments for several hours. This is something landlords and agents in particular need to keep in mind when emailing tenants.

Corporate Emails Have a Lot of Authors. Corporate communications pass through many hands before getting sent out. In the case of the Census email I would guess that the IT folks assigned the drafting of the email to an intern, who wrote it and sent it to the legal department, who then passed it to the public relations department, totally changing the nature of its message by the time they sent it to everyone who uses the API. So tenants, when you get a message from property management saying something totally bizarre, (like "all tenants must like us on Facebook or get evicted") make sure you check with a decision maker in the office to make sure you are interpreting it correctly.

Sometimes it isn't the government's fault. If something goes wrong involving people, agencies and the government, laypersons are going to immediately blame the government, no matter who is really at fault. I saw a number of tweets blaming the Trump administration for shutting down the API, when a) the government mandate that forced all government sites to switch to "HTTPS" was issued during the Obama administration nearly two years ago, b) it wasn't even closing down in the first place, and c) data encryption is actually a good thing that harms no one.

This also has echoes in the landlord-tenant arena. Both sides are quick to blame their problems on the laws that govern rentals rather than their own inability to communicate well and simple human error. Tenants are quick to blame their landlords for errors made by their staff. I'm all for accountability but one needs to focus one's anger in the right directions.

Consumers are suspicious. Consumers of social media have come to assume malicious motives and conspiracies lurk behind every message received from authority figures. While in some cases this may be true, in most cases authority figures - be they government agencies, software developers, your parents or your landlord - are not making decisions specifically to cause you harm. They may act carelessly or selfishly, but they generally are not outright malicious.

Trust but verify. Twitter - and social media in general - consists of a lot of wannabe pundits all trying to scoop each other, especially on big changes. I'm sure those who tweeted that the API was closing down were trying to be helpful to their friends or to boost their reputations as informed members of the open data community. If you see something on Twitter that looks like big breaking news, make sure to verify the information before you spread it around. Even respectable news establishments can be misled based on early information, flawed eyewitness accounts and tweets. Renters, when you are reading about a landlord on a site like Yelp, make sure you consider the source as well as the content.

Read the fine print. Finally, when you receive something in writing from an authority figure you must read the fine print. This includes things like leases and appliance instruction manuals from your landlord. Failing to read the find print caused a lot of embarrassment among the Twitter open data community this week. I hope it didn't cost any data engineers their jobs.

Was there ever a time when you got called out for jumping to conclusions? Have you ever stuck your foot in it on Twitter based on gossip? Let us know what happened in the comments, and we'll see you next week!