indiscreet acumen

dan newman

April 25, 2016
by dnewms

Creating a Choropleth Using Census Data

I recently put together an interactive choropleth of Michigan using American Community Survey (ACS) data from the U.S. Census. It looks like this (click on it or here for the actual interactive version):
Here are the steps to create a similar map. Please let me know if there are easier ways:

  1. Download shapefiles from the Census.
    You can get any granularity down to ZCTA (the Census’ version of ZIP codes), but beware, the larger the area you request, and the more detail, the greater the file size. The ZCTA file of the entire U.S. is more than a gigabyte. (I learned this step from here).

    For this map, I chose ZIP Code Tabulation Areas for the state of Michigan.

  2. Get some data for your map.
    You can do that in the American FactFinder. After finding the data you want, you need to narrow it down to match your geography selection, and then download that .csv file.

    For this map, I headed to Advanced Search in FactFinder, searched for Median Rent (B25058), selected a geography of 5-Digit ZIP Code Tabulation Area, narrowed that to those ZCTAs within the state of Michigan, and refined the search. Under the table view of this data, I selected to download it with the data and annotations in a separate file.

  3. Join your shapefile and data.

    To do this, you can download free GIS software, QGIS, which allows you to open up a shapefile and attach the data. There are other, more code-based ways of doing this that I tried, but failed to achieve.

    For this map, in QGIS, I created a new project, and added a vector layer, selecting the file ending in “zcta510.shp”. This gave me the ZCTA outlines. Then I added in the ACS median rent data by adding another vector layer, and selecting the file “ACS_14_5YR_B25058.csv”. To join the layers, I right clicked on the ZCTA layer and selected properties. On the Joins tab, I clicked the green plus. The ACS data was already selected as the join layer, and for the join field I chose “GEO.id2”. This GEO.id2 matched the ZCTAs in our shapefile. The target field I then chose was “ZCTA5CE10″– which made sure the right areas matched with the right rents. I clicked okay, and the data and map were joined.

    Now, still within the properties of the ZCTA layer, I selected style, and selected Categorized from the dropdown at the top. I could then select the data on which to stylize the map – for this, it was “ACS_…HD01_V01”, which corresponded to the column in the ACS .csv of median rent. I choose some nice colors for the color ramp, and hit classify, which automatically filled in categories of rents to shade your map. I hit OK, and the map took on colors by median rent.

  4. Export your data for web presentation
    While you can see your map with properly shaded ZCTAs by rent in your GIS software, you need to get the map on a webpage. To do this, you need to export your map to a new shapefile, and convert the shapefile to a friendlier format for the web, like geoJSON or the much smaller file format of topoJSON.

    For this map, in QGIS, I right-clicked on the ZCTA layer, and saved as an ESRI shapefile. I then ran commands to convert this shapefile to geoJSON using gdal and ogr2ogr, and further converted the geoJSON to the more manageable topoJSON (instructions here).

    If you don’t know how to run commands, or have much terminal experience, there are a few web tools available. First, you can take your shapefile and upload to – make sure to include your .shp, .dbf, .prj, and .shx, or you’ll lose the rent data. Mapshaper will show you the map, without data (don’t worry!), and then you can select export, to geoJSON. Mapshaper says it can export to topoJSON, but I had trouble with using the resulting file (to get the final .json file size down from MB to KB, I had to use the command line conversion tools above).

    You can then take this geoJSON to Mapstarter, and play around with how you want the map to look. The choropleth option is available in the colors tab. Mapstarter then allows you to export your map with all the code you need for a pretty web-based, interactive version built on d3.js, which you can further customize.

June 24, 2015
by dnewms

Seattle Schools Data Graphing

Navigating Seattle Public Schools data can be tough. While its recently updated website is great, all the data is separated into single .pdfs, with the only aggregated data in incredibly broad District Scorecards.

Digging deeper, I found a District Summary from December 2012. Focusing on elementary schools’ free and reduced lunch rates, and converting this data into a .csv, I used d3.js and borrowed heavily from this example to plot it all on a map.

I’m still working on getting the tooltip to display the school name and free and reduced lunch percentage – so if you know how to achieve that, let me know. Eventually, I aim to create a catalog of all the data locked up in the .pdfs to be easily navigated through dropdown selections, say of 3rd grade achievement scores, and displayed neatly on a map.

This ties in with an after-school tutoring program I hope to launch this fall in collaboration with Rotary and Rotaract clubs around the area. It will run twice a week at Sand Point Elementary’s library. If interested in that, also let me know! The Sand Point Squirrels will thank you.

March 20, 2015
by dnewms

Shi Shi Beach

In the upper-left corner of the contiguous United States, beaches are carved out by the relentless tide. One such beach, Shi Shi, is located in the Olympic Wilderness. However, it takes a few more permits than the typical trip to uninhabited land.


Most of that upper-left corner is part of the Makah Reservation. To enter certain parts of reservation, like the trailhead out to Shi Shi, you need to pick up a recreation pass. These are for sale in several places around Neah Bay, like the main grocery store, and run $10. Then, you can drive out to the trailhead — if you are only staying for the day.

If you are planning to camp on Shi Shi, you’ll need to park at one of the few houses on the way to the trailhead. These also run about $10 per day — though the day starts at midnight, so one night equals two days. Make sure to have the correct amount of cash and your own envelope to stuff into the dropbox. Then you can start walking to the trailhead.

The trail itself is known for its mud and soggy nature. There are plenty of side trails to avoid the mud, but it’s better just to wear galoshes and trek straight through. Through shady woods, you’ll come upon a wider road, and it’s another half-hour to the end of the reservation land and the beginning of the wilderness. There’ll be a box to fill out a slip for the Park Service, but the rain will have disintegrated most slips. This will also be the last time that you’re told bear cans are required. Make your way down to the beach carefully, and use the strung up ropes as you need.

Make sure to know when the tides are in or out, as the tide pools are a main feature. Starfish as purple as plums, or orange as… oranges. Crabs. Barnacles. Washed-up styrofoam.

Freshwater comes from a few streams that run perpendicular to the beach, although the water itself is brown from all the organic matter it carries. Even after boiling or treating it, you may have to close your eyes. Wood that’s dry enough for fire can be scarce. And it can get quite windy, so make sure your tent is secure. While readjusting tent lines in the middle of the night gives you a view of the stars, no one really wants to leave a sleeping bag.

On the way back to civilization, in Clallam Bay, stop at Sunsets West Co-op. The British (co)owners will make you a strong cup of coffee with a bun-less chiliburger (or, as one menu item says, “any breakfast you want, just ask”). And then you can listen to the banter between the motorcyclist visiting someone at the nearby prison and the local retiree, who inevitably talk about just how warm it’s getting and where have all the bees gone.