Week 41

Published on Author malmLeave a comment

Self-referential data visualisation III: bokeh, dimple and Qlik

[avatar user=”malm” size=”small” align=”left” link=”file” /]

Over the last couple of weeks I’ve outlined a few different approaches for using code to generate a multi-line visualisation of size statistics for my blog.   I covered using ggplot, vincent and nvd3 purely from Python and also how you can descend into JavaScript with nv.d3.js.  This week I’m going to risk inducing visualisation fatigue amongst faithful readers by trying out three more approaches.  First up a popular Python library called bokeh followed by a JavaScript library that builds on D3 called dimple.js.  Finally I will try out a widely-used shrink-wrapped commercial product, Qlik Sense.

bokeh

bokeh is a library that’s often cited as a useful resource for rapidly creating graphs:

Bokeh is a Python interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of novel graphics in the style of D3.js, but also deliver this capability with high-performance interactivity over very large or streaming datasets. Bokeh can help anyone who would like to quickly and easily create interactive plots, dashboards, and data applications.

The library exposes a large API set and although the documentation is comprehensive and the design pretty Pythonic, it proved to be somewhat difficult to work out exactly how to style the output let alone explore interactivity options.  The code below worked with the same pandas dataframe used before to yield the results shown below:

import bokeh
from bokeh.charts import Line, output_file, show, vplot, hplot
from bokeh.models import FixedTicker

bokeh.charts.defaults.width = 1000
bokeh.charts.defaults.height = 600
line0 = Line(df,
  x='week',
  y=['htmlsize', 'wordcount'],
  color=['blue','red'],
  title="malm.teqy.net blog stats",
  xlabel='Weeks',
  ylabel='Count',
  legend=True)
wks = df['week'].astype(int).tolist()
line0.make_grid(0,FixedTicker(ticks=wks))
line0.make_grid(1,FixedTicker(ticks=range(0,100000,10000)))
output_file("blog_bokeh.html", title="")
show(vplot(hplot(line0)))

bokehPlot

dimple.js

dimple.js is yet another charting library that builds on D3.js:

The aim of dimple is to open up the power and flexibility of d3 to analysts. It aims to give a gentle learning curve and minimal code to achieve something productive. It also exposes the d3 objects so you can pick them up and run to create some really cool stuff.

If you’re unfamiliar with the territory, there are a bewildering variety of JavaScript charting libraries that build on D3.js including C3, nvd3 and Dimple.js.  This helpful post helps explain some of the background reasons why; not that it makes it any easier. Dimple is arguably the best one I found.  The following JavaScript template code was used as a Python string to generate the graph below it.  I was able to produce two separate y-axes to allow the lines to be nicely overlaid which better illustrates the correlation between HTML size and word count in blog articles.  Either click here or on the graph to get a closer look at both the interactive visualisation and generated code:

wks = df['week'].astype(int).tolist()
width = 1000
height = 600
htmlsizes = df['htmlsize'].astype(int).tolist()
wordcounts = df['wordcount'].astype(int).tolist()
data = []
for i,wk in enumerate(wks):
   data.append({"week":wk,"htmlsize":htmlsizes[i],
                "wordcount":wordcounts[i]})
   title = "malm.teqy.net html and word counts"
s = '''<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
       <script src="http://dimplejs.org/dist/dimple.v2.1.6.min.js"></script>
       <div id="chartContainer">''' % (width, height, json.dumps(data), title) 
return s</div>

dimplePlot

Qlik Sense

Qlik Sense Desktop is a popular commercial chart creation utility.  It occupies the same niche as Tableau in offering users a way of working with and visualising data without resorting to coding.  In order to do that it provides a clear and simple to use interface that can work with a wide variety of data sources and render and style a range of output graphs as illustrated here:

QlikSense

In order to set up Qlik Sense to process the information in the pandas dataframe, it is necessary to convert it to csv format first. This is very simple to do with pandas and the resulting output when fed into Qlik Sense and styled with an average line added yields the graph shown below:

df.to_csv("blog_summary.csv")

qlik

If you’re interested in following up on any of this, I have released a new source file called blog4.py.  This script contains full working code for generating the vincent, ggplot, nvd3, raw nv.d3.js, bokeh and dimple.js representations covered over the last three weeks.  Phew.

Devices and Manufacturers

  • The barrier to entry into the smartphone market has been getting lower over the last few years with a broad and deep software and hardware supply chain in China allowing entrants with ambition and money to build the ultimate brand accessory. Still, it’s something of a “jump the shark” moment to hear that Pepsi have also joined the club:

PepsiCo Inc said on Monday it is working with a licensing partner to market a line of mobile phones and accessories in China in the next few months.

“ClearForce enables OEMs to differentiate smartphones by providing customers with new dimensions in user interfaces such as speed scrolling, zoom, gaming, and text or photo editing by applying variable force with a finger or stylus. Synaptics® has been working closely with leading global OEMs and LCMs to deliver this new dimension in touch with force-enabled smartphones expected to ship in early 2016.”

One customer commands a huge share of the industry. You either do business with that customer, or fall behind competitors who do business with them.

Microsoft, it seems, is becoming a bit more like the old Apple, even as Apple becomes more like the old Microsoft.

  • Recode profile a photography startup called Light which is blending the images from multiple smartphone camera sensors . They’ve announced their first camera product, the L16.

20151007 Light L16 front

  • The Granite cipherphone which was announced at MWC has now been officially launched and is available to buy online.   Although the hardware looks like a commodity Archos offering, the software seems significantly differentiated from stock Android majoring on encryption features delivered by the underlying Granite OS developed by Brazilian outfit, Sikur.  They’re banking on customers paying a premium price for secure software, targeting 150k unit sales in 2015 and double that in 2016 for Granite:

The phone, conceived by IT security specialists from Sikur, and made by Archos, can now be ordered from its dedicated website for the hefty sum of $849. … The hardware you get for that is not unlike the existing Archos 50 Diamond.

Google and Android

FUMconclusion

  • It’s a timely intervention with security updates top of mind for many working in the Android ecosystem following the disclosure of the Stagefright exploit.   Some activist customers are taking to social media and blogs demanding updates.  This open letter to Motorola management includes an interesting suggestion namely shipping their products with a guaranteed certainty around the level of post-launch updates based on Gold, Silver and Bronze support ratings.  At least that way, the discerning customer would know what to expect.
  • TechRadar suggests that the refusal of Stagefright to go away is a vivid expression of Google’s fundamental lack of control over the Android ecosystem.  The scare is proving to be a nightmare for both Google and Android OEMs many of whom will have struggled to patch the first set of vulnerabilities only to find that Zimperium uncovered a second set.

The vulnerability that the fix will eventually fix evolves so that the fix doesn’t fix it any more.

  • Monday Note on Google’s Accelerated Mobile Pages (AMP) initiative designed to speed up mobile web access by delivering only the core HTML content and stripping out “javascripts, iframes, embeds, large chunks of the CSS etc.“.  The technology has huge potential to transform the experience of accessing content on mobile devices:

Depending on network conditions and the speed of your smartphone’s processor, the loading time is reduced by up to 80%-90%. I’ve been testing amp-pages for a couple of months now, it works spectacularly well.

389_amp_guardian

 

Apps and Services

It means you can — for the first time — experience an event unfolding in both real time and after it’s finished as they’re captured for experiencing it as it unfolded at any time.

  • It’s US only for now but the potential for the service is not lost on sharper analysts.   Stratechery wrote a great analysis pointing out that what Twitter have effectively done with Moments is reinvent the newspaper around the concept of story fragments – or ‘micronews’ if you like – which positions it in the same curated news contender space as Apple and Facebook:

“if the Internet broke down newspapers to their component stories, Twitter breaks down stories to their component moments, and those moments are chronicled not only by normal people on the ground but by the best news-gatherers on the planet.”

Moments screenshot 1

  • Ethereum has been having a moment of its own over the last couple of weeks being hit with the ‘next generation disruptive technology’ stick.  Billed as a “social operating system“, it’s a complex proposition to explain.   Founder Stephan Tual describes it as follows:

Ethereum is a platform that makes it possible for any developer to write and distribute next-generation decentralized applications. Borrowing the concept of distributed consensus and cryptographic proof that makes cryptocurrencies such as Bitcoin so effective in trustless payments, Ethereum extends the use of these technologies to trustless agreements.

Cloud and Security

  • InfoQ on Amazon’s recently published “AWS Well Architected Framework Guide” which surveys the four key pillars of any AWS install namely Security, Reliability, Efficiency and Cost.
  • Amazon have also formally jumped onto the IoT bandwagon with the announcement of a new platform called AWS IoT aimed at an increasingly crowded space.   In their case it makes a lot of sense – many IoT initiatives today aren’t really possible.  AWS IoT is built on a combination of legacy tech and the Amazon cloud:

The IoT platform is one of several to use a decades-old lightweight messaging protocol called MQTT. That technology has been outstripped in some use cases, but works well with devices because it can handle intermittent connectivity particularly well,

  • Fascinating engineering blog post from the founder of customer analytics startup Segment.com which details how they rebuilt their architecture around Docker and other CI/CD (continuous integration/continuous delivery) tech.

Artificial Intelligence

20151007-code-mobile-2015-

“The real risk with AI isn’t malice but competence.  A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble.”

  • With so many remarkable advances in artificial intelligence announced almost every day, HBR asks what it takes to stay human in the Robot Age.  As befits HBR, their primary focus is the role of management in the transition.  This observation doesn’t fill one with confidence:

this is the first great wave of technological evolution that is being driven by managers in the grip of the ideology of shareholder primacy.

Cars

  • Autonomous cars are coming:

Mars

Digital Innovation

  • This important research post looking at the growing gap between the most productive firms may be explained by differential abilities to conduct and absorb digital transformation.  It also provides an excuse to reference William Gibson’s famous quote:

“the gap between the globally most productive firms and the rest has been increasing over time, especially in the services sector. Some firms clearly “get it” and others don’t, and the divide between the two groups is growing over time. … If the future is here just not evenly distributed then increasing the speed of diffusion holds the promise of a highly productive catch-up period in which the average advances to the frontier.”

productivity

  • The Digital Catapult is a UK government-sponsored centre created to “rapidly advance the UK’s best digital ideas” with a view that intervention will lead to “new products, services, jobs and value for the UK economy“.   A key part of its remit is an explicit focus on data:

“Our focus is on data and metadata – on the data value chain. This is how we describe the ways in which people and organisations create, collect, transport and analyse data.”

  • Here’s a link to their Creative Content Showcase showcasing some of the talent they are supporting including Andrew Fentem whose product FlickPixels (“wallpaper for the IoT”) was recently covered in the blog:

Product Management

  • InfoQ just published a presentation by Sherif Mansour, Principle Product Manager at Atlassian responsible for their enterprise collaboration tool Confluence.  It contains some interesting insights into why building simple products is hard and can’t be achieved through “a beating with the simple stick“.   Mansour outlines how Atlassian uses personas, customer interviews and journey maps to surface underlying requirements.   Ultimately it’s all about how good you are at covering these three bases:

  • This related post outlining “how to design better products in 5 days“ covers broadly similar territory suggesting the following five steps to drive agile sprint activity:
    • Understand (clarify the problem you’re solving). 
    • Diverge (brainstorm what’s possible).
    • Converge (rank solutions, pick one).
    • Prototype (create a minimum viable concept).
    • Test (observe what’s effective for users).

Blogging

What matters instead is influence, and one way to build it is by guiding audiences through the chaos of so much content. Today’s there’s no better way to do that – and demonstrate influence — than producing an email people will actually open. … It’s about the problem of discovery in an overwhelming age.

Culture and Society

at GUADEC 2009, of approximately 170 attendees, I believe I was one of only eight women. Of the software developers working on the entire GNOME project at the time, I was one of only three.

W150918_SCHROEDER_TALKTOME

Leave a Reply