Census-CSV-Excel

It has been a very busy few weeks for ChattaData, though you wouldn't know it from reading the blog. I bought a laptop to dedicate to my datasets and programs and am still fine tuning it. I got Tableau to use for exploration, analysis and presentation so more graphics and interactive elements will be coming from there. (There might be some workarounds for Safari users who can't view tableau workbooks, more on that later.) Installed Microsoft's Pivot program, spent a few minutes with it so far...not sure how useful it is yet.

If you haven't looked at our Census progress yet, they are posting new files daily at the Take 10 Map site. As of this moment the Mail Participation Rate for Tennessee is 54%, Hamilton County 54%, and Chattanooga 51%. I have been downloading the daily files on these, and recently they added the filter where you can download by state. I took the CSV label literally and tried to open it directly into Tableau as a comma separated value file. It wasn't happy since it is actually a pipe delimited file. A trivial thing to fix, but a good reminder of the fact that I haven't come across much data that was ready to go. In that respect, Excel is one of my new best friends. I have used it for years for pretty basic things, but as I have been populating some mysql databases lately it has been a workhorse. I use some other spreadsheet programs as well, but have several actions in Excel that make it indispensable.

So there. I have purchased a pc laptop and sung praise to Excel, all in one week, all in one post.

Le Sacre du Printemps

My first paying 'job' was volunteering for the National Park Service for the hawk watch at Signal Point. The job was simple. I had to help spot, identify,count and record migrating birds of prey. Doing this meant that I spent my weekends in spring and fall with the Chattanooga birding crowd. When it comes to citizen data collection, you will be hard pressed to find a group more dedicated than the birders. Open Data? Crowdsourcing? These folks have been doing it for a long time. Below are two examples with data from point counts* and a volunteer effort to save a huge dataset.

You can go straight to the USGS Point Count Database to look at some numbers or start at the Migratory Bird Data Center which has other data and background info. The North American Bird Phenology Program is well worth looking at as well. In short, they house an historic collection of bird migration data collected on observation cards. Six Million of them. They are working with a small army of volunteers to scan and enter them all into a database so that they can be analyzed. (There was an article on Wired about the project last year that does a good job of explaining the importance of this data.) Having all of these migration records available will deepen our understanding cycles of bird populations. Mashing this data with other sets on development, climate and environment will tell us a lot about the impact that habitat loss has on specific species of migratory birds.

* A point count is basically what it sounds like. A specific point is chosen and marked, often by nailing a small coin sized marker to a tree. The birder counting that route visits it at regular times and records all bird seen and heard, mostly the latter.

** Original photo credit: R. Bruce Wilkey - 1980 - yes, that is me and yes that jacket is awesome.... have to see if I still have it.

TBL at TED 2010

Tim Berners-Lee at this year's TED conference discussing The Year Open Data Went Worldwide. Excellent follow up to his talk last year with some incredible visualizations of Open Street Map edits around the world with a focus on post-quake Haiti.

Pie eating contest

I went to a community meeting to hear someone attempt to persuade the other attendants to agree with them on some thing or other. The Persuader got points instantly for having handouts of the charts and graphs used in the presentation. Oh, and they brought enough so everyone could see. I arrived a few minutes late, so everyone was on the second or third pie chart. I thought I had missed The Pitch and was only going to get the chart parade. Next pie chart, and the next, every time I am having to call in a whiskey tango foxtrot to the first page to see if the quantities represented as proportions on the charts were the ones in the table on page one. Nope. Then, I felt a subtle wave go across the room and seize someone who blurted out, "What is your point!?". Civility broke down at that point and took a while to restore itself. Sort of.

I was a witness to a chart fail.

Lessons learned from a data presentation standpoint.

1. If presenting a table of numbers accompanied by a bar chart of the exact same thing, just do the chart.

2. When trying to illustrate percentages of something compared to the percentages of several other things, don't use a series of pie charts (on different pages and of greatly varying sizes). Use another simple bar chart or better yet, a segmented bar chart.

3. Don't show percentage without reference to the totals. Please.

4. Finally, don't mention in the presentation that there is another set of data that conflicts with the one you are presenting and not have it available for comparison as well.

This is just a summary of the chart fail that occurred for The Persuader. Eight pages of information without context or source reference, and seven of those were pie charts without totals represented. After what seemed like 10 minutes, The Persuader lost the room because the point was not being communicated and the pie charts only agitated everyone.

What I love about this experience is that it showed data at work, albeit poorly, at the community level. This is nothing new, not at all, but the presentation of data can always be better. The Persuader went to a source, gathered data, charted it and brought handouts for a meeting that could have been 10 or 100 people. So when I wonder what Open Data/Open Gov means to us as individuals and communities, this is one example.

Chart Note: You too can add serious or sarcastic charts to your site with the Google Chart Tools API.

Never go full fish-eye.

Here is a fish-eye map of troop positions in Chattanooga in 1863. I find it to be a strange presentation of information. Perhaps I am just old fashioned in liking my troop position maps to be aerial views, even a bird's-eye view, but I never considered the fish-eye view. When looking at it as a whole, several trees are the dominant objects. Lookout Mountain is placed at 8 o'clock, so the fish's perspective was one from Missionary Ridge. Why this map adheres to the standard alignment of the cardinal directions, I have no idea. It seems that if you are getting this fanciful, might as well put North somewhere near 3 - 4 o'clock to place Lookout Mountain and Chattanooga as prominent features to those reading the map with human eyes.

All fun aside, it is an interesting graphic. I am not sure if it was a particular style of the time used for eye candy in publications, but expect so. You can get a high quality image of it from the Library of Congress' American Memory site to zoom in and explore. If you haven't spent time on the site, you should. Plenty of maps and photos of Chattanooga and the region. High quality (as in 160Mb) TIFFs are available of the photographs in the collection as well as the high quality map files.