Week 40

Published on Author malmLeave a comment

Self-referential data visualisation II: ggplot and nvd3

[avatar user=”malm” size=”small” align=”left” link=”file” /]

Last week I sketched out how to build a simple data visualisation pipeline to extract and clean some statistics from my blog posts before displaying them in a graph.  I used a library called Vincent to generate the visual representation.  This week I’ll look at another couple of other visualisation options, namely ggplot and nvd3.

ggplot

ggplot is at attempt to accommodate its widely-used R equivalent in Python with the aim of providing those coming from R a straightforward migration path:

“a plotting system for Python based on R’s ggplot2 and the Grammar of Graphics.  It is built for making professional looking, plots quickly with minimal code.”

ggplot is relatively easy to use and integrates well with Python’s pandas data analysis library.  However it is limited in terms of wow factor and not particularly Pythonic given its R heritage.  Using the same pandas dataframe from last week, we can generate a multi-line graph .png output shown below with this code:

from ggplot import *
wks = df['week'].astype(int).tolist()
counts = range(0,100001,20000)
gg = ggplot(aes(x='week'),data=df) +\
  geom_line(aes(y='htmlsize', colour='red')) +\
  geom_line(aes(y='wordcount', colour='blue')) +\
  labs(title="ggplot",x="Weeks",y="Count") +\
  scale_x_discrete(breaks=wks,labels=wks) +\
  scale_y_discrete(breaks=counts,labels=counts) +\
  xlim(1,len(wks)) + \
  ylim(0,100000) + \
theme_seaborn(style='whitegrid')
ggsave(plot, 'blog_ggplot.png')

blog_ggplot

nvd3

nvd3.js is a library that builds on d3.js with a specific focus on interactive charting being:

“an attempt to build re-usable charts and chart components for d3.js without taking away the power that d3.js gives you.”

The library is maintained by a commercial outfit, Novus Partners, and comes with a helpful Python library nvd3 which can be pip installed then used as follows to generate the nice interactive JavaScript-based HTML visual representation shown below.  Click here or on the image to get a closer look at the graph and generated code.  Note how clicking on the key circles nicely reshapes the graph:

from nvd3 import lineChart
with open('blog_nvd3.html','w') as html:
    chart = lineChart(name="lineChart", use_interactive_guideline=True, margin_bottom=60, margin_top=60, margin_left=90, margin_right=90, x_is_date=False,     x_axis_format='d',y_axis_format='d', height=450, width=1000)
    weeks = df['week'].astype(int).tolist()
    hs = df['htmlsize'].astype(int).tolist()
    ws = df['wordcount'].astype(int).tolist()
    chart.add_serie(y=hs, x=weeks, name='htmlsize')
    chart.add_serie(y=ws, x=weeks, name='wordcount')
    chart.buildhtml()
    html.write(chart.htmlcontent)

blog_nvd3

This visual is a good starting point and almost there but it is missing a few key features – the y-axis doesn’t start from zero, labels aren’t named and x-axis tick-values don’t show all weeks.  From closer examination of nvd3, fixing those problems would require changes to the library:

  • Specifying the y-axis range requires setting a forceY element in the JavaScript which the nvd3 add_chart_extras function doesn’t seem to be able to do.
  • It isn’t possible to pass an axisLabel into the nvd3 linechart initialisation.
  • It also isn’t possible for tickValues to be set up at  initialisation of an nvd3 linechart.

In order to verify fixes for these problems I edited the previously edited nv.d3.js JavaScript with the corresponding elements modified in place then wrote a function to generate just that code. The key adjusted JavaScript is presented below along with its graphical output.   As with the previous version, click here or on the image below to get a closer look at how it all works:

nv.addGraph(function() {
    var chart = nv.models.lineChart();
    chart.useInteractiveGuideline(true);
    chart.margin({top: 60, right: 90, bottom: 60, left: 90});
    chart.xAxis
      .axisLabel("Weeks")
      .tickFormat(d3.format(',d'))
      .tickValues(weeks)
      .scale(weeks)
      .orient("bottom");
    chart.yAxis
      .axisLabel("Counts")
      .tickFormat(d3.format(',d'));
    chart.forceY(0);
    chart.showLegend(true);

    d3.select('#linechart svg')
      .datum(datum)
      .transition().duration(500)
      .attr('width', %d)
      .attr('height', %d)
      .call(chart);
    });

blog_nvd3raw

There was an inevitability at the outset that this exercise would end up in hacking hand-crafted JavaScript.  Prior experience with Python visualisation libraries yielded graphs that weren’t quite right and required a descent into lower abstraction layers to sort out.  nvd3 arguably produces the closest thing to output suitable for the sort of visualisation pipeline highlighted last week.  However, even there, as demonstrated above, there are pixels the library cannot reach.  The only way to control them, short of modding nvd3, is by dropping into the lowest abstraction layer, namely nv.d3.js itself. It’s another inadvertent example of Atwood’s Law in action:

any application that can be written in JavaScript, will eventually be written in JavaScript.

If you’re interested in trying this out for themselves, I have released a new source file called blog3.py.  This script contains my latest self-referential data visualisation pipeline which includes full working code for generating the vincent, ggplot, nvd3 and raw nv.d3.js representations covered in the last couple of weeks.

Manufacturers and Devices

The 19.5-centimeter (7.7-inch), 390 gram (0.9-pound) Robohon sports a 2-inch touchscreen, a quad-core CPU, and can connect to 3G, LTE, and wifi networks – but who cares! It’s a freakin’ robot that makes phone calls, shoots pictures and videos, and sends emails!

In a nutshell, it’s a stripped back device that lets you call, text, set a reminder or set an alarm. That really is about it.

Punkt01front

Google and Android

Come November 18th, Google will have to change the requirements that it places on its hardware partners, specifically the ones that the Russian regulator feels are unfairly restricting apps that aren’t created by Google.

Apps and Services

  • Peeple was widely covered in the news for all the wrong reasons this week. The Verge referred to it as “YeIp for People”, a questionable proposition that lets users rate people as if they were Amazon purchases.   The negative feedback generated by initial impression prompted its CEO to take to LinkedIn to defend the product as a “positivity app for positive people” without really explaining why some see it as a cynical growth hacking experiment that ‘non-peeple’ can’t opt out of:

there’s currently no way for users to opt out of Peeple. Anyone can sign up anyone else if they have their cell number, and although only positive reviews are shown on the profiles of people who haven’t signed up, members of the public can’t see their reviews unless they join. It’s also not clear whether negative reviews are judged to be so based only on the star rating or whether the actual content is also taken into account. If just the former, it means that users could give people extremely negative reviews but a good star rating, with the targets of these write-ups never knowing about them unless they signed up. Call it a growth hack.

  • Interesting analysis from a long-form digital magazine called The Atavist on why they’re ditching their native mobile apps and going full web:

Now, after nearly five years and 51 stories in The Atavist Magazine—plus tens of thousands of publishers and individuals making their own on the Atavist platform—we’re discontinuing our native mobile apps to place all of our focus on the web.

With iOS 9, it finally seems possible to realize some of the major trends identified last year, like “invisible apps” that recede into the operating system and notifications as a primary interface.

From left to right, that’s what Apple calls Today, Notifications Center, and Spotlight Search / Siri Suggestions.

Hardware

a portable FM transmitter the size of a shoebox that starts working as soon as it’s connected to a small antenna, a power source, and an audio signal. Pocket FM resembles a radio receiver more than a transmitter, and a single device can air radio programs over a radius of about six kilometers.

Pocket-courtesy-MiCT

UHA Phase 12, photo Greg Beron

Security

  • Yet another Snowden revelation on the formidable extent of control security services can exercise over individual phone users.  Their armoury apparently includes ‘silent exploits’ operating at a lower level than the likes of Stagefright which allow them to take control of smartphones remotely without the user being aware of what’s going on:

“a specially crafted message that’s texted to your number like any other text message but when it arrives at your phone it’s hidden from you. It doesn’t display.”

  • Cue a picture of the GCHQ panopticon:

GCHQ

Military Artificial Intelligence

  • Annie Jacobsen’s “The Pentagon’s Brain” is a Google talk that provides an inside take on DARPA’s military research projects and objectives.   She’s clearly a fan but taken in the round, the direction she outlines is rather alarming and right in line with the popular Hollywood trope that the US military will be the first organisation to create Artificial General Intelligence (AGI) and that it will be embedded in a war-machine:

“Our weapons are heading to autonomous robotics”

  • It does all rather make you wonder whether the agenda here isn’t being driven by Hollywood plotlines.  To wit, a US Army spokesman talking about Tony Stark iron man exoskeletons:

  • DARPA are also investing heavily in neurotechnology which today is admittedly more science-fiction than fact as described in this review of the current state of the art.  However, that was also once the case with atomic weapons and we all know how dramatically their arrival impacted the globe which should serve as a salutary warning:

It seems the possibility of weaponization might not lie in some distant future—and there is ample precedent for the rapid transition of technology from basic science to disruptive, global menace. After all, just 13 years elapsed between the discovery of the neutron and the atomic blasts in the skies over Hiroshima and Nagasaki.

Cloud and Digital Transformation

  • Digital Transformation is now a Board and C-level issue for many companies according to this study from PwC which also highlights an interesting divergence in expectation between the CEO and rest of the C-suite:

Digital transformation becoming a board-room issue, PwC study says

Among enterprise users, iOS accounts for nearly two-thirds of the market. More than any other factor, that embrace accounts for the realignment

“As Slack ramps up development and companies spend increasing amounts of time within the service, it’s going to subsume some of those integrations and launch them as first-party features so you never leave the tool. … With teams that are all-in for using Slack, there’s no emailing links or attachments of documents, no futile time wasting trying to invite everyone to a group call. In the future, all office software will be in one place”

Wearables and The Internet of Things

  • AWS and the Internet of Beer sounds goofy but it’s actually a really great extended walk-through a real-world E2E IoT architecture.  A Raspberry Pi with GrovePi+ board and sensors connected to a beer keg serves as a proxy for an IoT device.  The rest is a standard web scale AWS setup.

It is amazing how fast this space is evolving. Hardware really is becoming more like software. In just a little more than a year, it has become far easier to go from an IoT idea to something that looks and works great. … The use of the AWS cloud for heavy lifting combined with the decreasing size and cost of IoT devices is a powerful combination.

Software Development

  • Elvis has left the building.   But not before ensuring proper code hygiene:

gitCommit

Blogging and Tweeting

one service has quietly emerged as the go-to alternative to Facebook and other big social media sites: the email newsletter service TinyLetter.  … Your list of email addresses is yours in a way few things online ever are.  … Sending a newsletter is as simple as posting a status update to Facebook.  … Anyone with an email address which is pretty much everyone, can subscribe to your newsletter. Journalists and techies have flocked to TinyLetter in recent years, slowly building its reputation as the hippest place to publish online.

  • This analysis of the language employed by Twitter users and what they choose to tweet on suggests a correlation with their earnings and social status:

People who are perceived as less anxious and religiously unaffiliated seem to have higher earnings, while those who have higher income tweet more anger and fear while posting more objective content. Lower income users swear more, and emotions including sadness, surprise, and disgust are more frequently associated with them than higher income users. Additionally, people who earn more tweet more about politics and NGOs.

Work and Startups

  • Most employees have no idea whether they are fairly paid or not according to a study by PayScale.  Two thirds being paid at market rate think they’re underpaid.  Seems like that only those who genuinely are underpaid relative to market rate know it:

W150921_SMITH_WHATWE

  • Controversial but arresting take from an entrepreneur on “why I’m seed funded and you’re probably not” calls out these four areas of focus in this order.  All too often 4. is elevated above the others:
    • The Team
    • The Market
    • The Business Model
    • The product or service you are planning to build/offer
  • Carole Cadwalladr of the Guardian published a colourful ride through the Silicon Valley tech startup VC scene.   She asks the question whether the dotcom bubble is about to burst again without going so far as explicit prediction.  However a distinct tone of unsustainability is clear in the repeated references to disruption for disruption’s sake and talk of ‘decacorns’ and ‘unicorpses’:

How many unicorns can you fit through the eye of a needle? Anyway, unicorns are over. It’s all about decacorns now. Companies that are worth tens of billions of dollars. … Companies in Silicon Valley think differently from the rest of us. Their ambition is of a different order. Their rate of growth is like nothing we’ve ever seen before. And it’s in San Francisco, where technology meets the real world, that we’re starting to see the beginnings of what this clash of civilisations will look like.

  • FirstRound post from Mike Arauz, co-founder of a startup called August that adopted holacracy as a total transparent methodology for organisational self-management.  It’s famously an approach used by Zappos, a much cited canonical example of the “exponential organisation”.  They’ve since dialled it down a notch but remain committed to the general approach.  Arauz suggests the following five fundamental principles for running your company in a radically transparent way:
    • Nail a memorable common purpose.
    • Digitize your org to tap your network and blow up the hierarchy.
    • Distribute authority and embrace total self-management.
    • Never stop iterating.

    • Work in public.

Culture and Society

Leave a Reply