Week 39

Published on Author malmLeave a comment

Self-referential data visualisation I: Vincent

[avatar user=”malm” size=”small” align=”left” link=”file” /]

In January I published a post on some analysis I’d done on all my 2014 weekly roundups.  As we’re now around 75% of the way through 2015 I thought I’d revisit my blog stats.  Last time round I used a Python script to scrape the blog posts and then create the raw stats.  Excel was used to generate the graphs from these stats. This time around I wanted to try out the data visualisation pipeline described by Kyran Dale in his illuminating talk at PyCon UK in which he suggested using the Python pandas data analysis library for data preparation and switching to JavaScript for visualisation, specifically D3.js:

I took a few shortcuts in order to get something working in limited time.  In particular avoiding MongoDB as an intermediate JSON data store and using a visualisation grammar called Vega that simplifies access to D3.  A pure Python library called Vincent (see what they did there?) allows you to generate Vega directly from a pandas dataframe.  I’ve released an updated version of the original blog.py script imaginatively called blog2.py that partially implements the workflow Kyran outlined with these limitations built-in.  Running the blog2.py script output yields the following terminal output:

$ python blog2.py
A] Retrieve JSON stats from cached stats
B] Load JSON into pandas dataframe and clean it up
 week htmlsize wordcount
0 1 66331 3036
1 2 77970 4019
2 3 62670 2570
3 4 67475 2832
...
36 37 71646 3537
37 38 70583 3440
38 39 76900 3709
Sum of htmlsize: 2787803
Sum of wordcount: 132451
Average wordcount: 3396
---- EXECUTION TIME: 0.920894 secs ----

Plus the all-important graphic itself:

blogStats

Which is created in a file called ‘blog.html’ using Vincent direct from a reduced scope pandas dataframe rdf thus:

lines = vincent.Line(rdf)
lines.scales[0].type = 'ordinal'
lines.axis_titles(x='Weeks', y='Count')
lines.legend(title='Title Here')
createVincentHtml('blog.html','N',lines.to_json())

The data shows a constant average total 3.4k words per post per week and ongoing strong correlation between word count and HTML page size.  The next time I return to this topic, I’ll look to fill the gaps and explicitly switch to D3.js in the hope I can tidy up the graph.

Devices and Manufacturers

  • Who killed Nokia?  They did it to themselves they did according to research conducted by two Professors of Strategy at top European business school INSEAD.  The write-up is a pretty brutal indictment of a dysfunctional organisational culture which is a topic that returns below in respect of the VW emissions scandal:

Nokia lost the smartphone battle because of divergent shared fears among the company’s middle and top managers led to company-wide inertia that left it powerless to respond to Apple’s game changing device.

In a recent paper, we dug deeper into why such fear was so prevalent. Based on the findings of an in-depth investigation and 76 interviews with top and middle managers, engineers and external experts, we find that this organisational fear was grounded in a culture of temperamental leaders and frightened middle managers, scared of telling the truth.

Apple is a company that builds greatness because they built a process to make their customers better. They do it without permission and they do it without offense. But when the buyer perceives the change they feel in debt to the object. Getting payments on that debt of gratitude is how Apple is rewarded.

  • Blackberry’s first Android effort is revealed as the curiously-named Priv smartphone.   Positioned as a device that “combines the best of BlackBerry security and productivity with the expansive mobile application ecosystem available on the Android platform” it’s clearly aimed at the growing cipherphone opportunity.

  • Vertu launched a new flagship, the Signature Touch that “runs on Android 5.1 and packs an octa-core Snapdragon 810 chipset with 4GB of RAM, along with 64GB of internal storage plus another 64GB on the included microSD card“.

  • 5 million Raspberry Pi’s have now been manufactured by Sony UK at their Pencoed factory in Wales.  It’s a great, albeit all too rare, success story for high-tech UK manufacturing:

Google and Android

“Nexus 6P is the first all-metal-body Nexus phone. Built in collaboration with Huawei, this 5.7” phone is crafted from aeronautical-grade aluminum, with a USB Type-C port for fast charging, a powerful 64-bit processor, and a 12.3 MP camera sensor with massive 1.55µm pixels (hello, better photos!).”

The practice of bundling products and services together may violate antitrust laws if a company dominates the market for a product that customers need, and then forces them to buy a complementary product or service

Apps and Services

  • Quartz on the evidence that if you are a US smartphone user “you only really use three apps on your phone” and one of them is Facebook:

The big winner, unsurprisingly, is Facebook. It’s a top-three app for almost 80% of users, according to comScore, and number one for almost half of them. (With 126 million US unique visitors in June, it’s the most popular app in the US by a solid margin. It’s also on almost half of users’ smartphone homescreens, according to the report.) And as Facebook goes deeper into mobile video, its share of time spent seems likely to increase.

TimeInApps

  • This presentation from Facebook engineer Simon Whitaker is a picaresque and at times jaw-dropping ride through Facebook’s codebase and tools initiatives. Presumably they have their reasons for embracing NIH with gusto and the deck provides reasonable justifications for each case but, man, are they building a substantial legacy for themselves.    They’ll need to keep hiring software engineers to look after all this and more:

http://www.slideshare.net/quellish/simon-whitaker

SiliCon Valley

“there is a trend in Silicon Valley to focus only on solving the problems of the rich, whereas entrepreneurs in the rest of the world tend to focus more on the problems of ordinary people – and maybe it is that second sort of innovation that will prove the better long term bet.”

It’s really about love. And sacrifice. And hard work. And putting in the daily things that it takes to achieve great things. And how in the daily routine of being yourself, committing to goals and just living life, you realize that goals were easier to obtain than you had imagined. It’s a love story of sorts.

Dieselgate

  • Good HBR take on the VW emissions omnishambles outlines three vital lessons the company failed to absorb and now has to act upon really quickly if it is to recover:
    1. Being clean and green has real, bottom-line value.
    2. Protecting the environment builds trust, and trust is precious capital.
    3. Trust comes from transparency, and transparency is the norm today.
  • The key takeaway:

“being clean and green is important, but only if it’s authentic. And remember that everyone is watching you.”

SEPT15_23_454561263

  • Travis Bradberry in this thoughtful LinkedIn post goes further suggesting that what lies at the heart of the issue is nothing less than the integrity of VW’s leadership, which has this week changed.  He outlines some key leadership antipatterns including micromanagement, toleration of poor performance, cult of personality and the tyranny of the urgent:

The tyranny of the urgent is what happens when leaders spend their days putting out small fires. They take care of what’s dancing around in front of their faces and lose focus of what’s truly important—their people. Your integrity as a leader hinges upon your ability to avoid distractions that prevent you from putting your people first.

This sustained and sophisticated software scam suggests truly pathological levels of managerial desperation and contempt: desperation around a failed promise of clean diesel technology and unsubtle contempt for unsuspecting regulators and customers alike. Precisely how gullible did these wizards of Wolfsburg think people would be?

If Volkswagen knew that every customer who buys a vehicle would have a right to read the source code of all the software in the vehicle, they would never even consider the cheat, because the certainty of getting caught would terrify them

https://youtu.be/ktyMmzZ6WeM

Software on TV

Artificial Intelligence

We define AI-powered business applications as software in which the user experience is driven by AI. Today, that takes the form of voice (Apple Siri), text-based interfaces (x.ai’s Amy) or intelligent notifications (Google Now). Other examples of AI-driven UX like Google Search’s autofill differ in that they do not automate a step within a workflow.

ai-chart-final-option2

Cloud, Infrastructure and DevOps

  • Hashicorp are one of the leading players in the DevOps commercial scene.  Founded just 5 years ago, the company is already behind key tools like Vagrant and Packer.  Now they’ve announced a meta-orchestration tool called Otto that seems to be positioned as a competitor to Docker Compose.
http { 
    js_set $hello_world " 
        var str = 'Hello World!'; 
        // JavaScript  
        str; 
    "; 
    server { 
        ... 
        location /{ 
            return 200 $hello_world; 
        } 
    }

By giving developers a way to express relationships between data rather than fixating on the data itself, graph databases offer a powerful new way to tame the growing complexity of big data.

Digital Enterprise

The rise of the citizen developer will be a common theme over the next few years, and enterprises that embrace the potential of Shadow IT will be best positioned to innovate and thrive.

Any significantly sized business will generate a huge amount of data from across its various operations, and the task of finding actionable intelligence hidden inside it all is not to be underestimated.

The evolution of wearable technology points to devices and chips which will be embedded by 2025, rather than worn explicitly. If a single chip is embedded within the human body, or encoded within cells, then in theory every human would become a hub or a node, addressable and traceable within the larger IOT ecosystem and every owned device assigned to that Human IP address. Your ‘social graph’ will become more valuable to the Government than your Social Security or NIS number and it’ll supercede these eventually. … In other words, humans could well become the ultimate ‘thing‘ within The Internet Of Things.

Wearables and Insideables

HUAWEI Honor Band Zero IP68 Water Proof Pedometer Sleep Monitor SMS Reminder Call ID for Android / IOS Smartphone E-475268

DARPA responded to a request from Fusion that “brain-neural interfaces” have not yet been implanted in soldiers, though test devices have been implanted in the brains of volunteers already undergoing brain surgery. We’ve changed the headline to reflect that implantation of chips in soldiers’ brains has not happened yet.

Space and Science

The researchers believe that an Earth-sized planet with an atmosphere similar to Earth’s would indeed spin as it rotates around small orange dwarf and some red dwarf stars.

Recurring slope lineae

The Martian paints what I believe is a much truer picture of our nation’s space program, one that requires the collaborative efforts of all teams in order to successfully explore whatever it is that’s potentially “out there.”

The Martian

Startups, Work and Management

EEengineering

  • FirstRound profile piece on GiveForward highlights the central role that a unique and authentic culture plays within the organisation and how it colours their approach to fundraising, hiring and retention:

Weird job descriptions weed out the boring, stuffy people and bring us the weird, creative and fun folks we want to work with.

“They build trust by taking the opportunity to demonstrate their ability as they do their daily work, by asking knowledgeable questions and offering insightful suggestions. .. They build a team by using problems and crises in the daily work to remind members of the team’s purpose and what it values most. … They build a network by taking opportunities afforded by routine activities to build and maintain relationships with colleagues outside their group.”

Internal emails and interviews with people who’ve worked on that system show that as CEO Travis Kalanick and operations executives pressed engineers to build new features for the business, it came at the expense of stability in the back-end systems that power them.

  • And also provides this rather interesting insight:

Culture and Society

One start toward reclaiming conversation is to reclaim solitude. Some of the most crucial conversations you will ever have will be with yourself. Slow down sufficiently to make this possible. And make a practice of doing one thing at a time. Think of unitasking as the next big thing. In every domain of life, it will increase performance and decrease stress.

  • A tremendous Quora post on historical counterfactuals (“what if” scenarios) asks “what is the smallest amount of believable change in historic facts that would have had the greatest impact on the world as we know it today“.  One respondent suggests the haunting possibility of Czar Nicholas’ son being born without haemophilia and all that entailed:

If you want to know how major world events can hinge on one event?  That’s pretty much it… a baby born (Alexei) born with Hemophilia with a pair of loving and worried parents…. whose love and concern would throw the world into a century of cataclysm and catastrophe.

  • Staying on the theme of Russia, a recently-published book called The Red Web outlines how the country has essentially adapted the intimidatory methods and tactics used the old Soviet era to the digital age co-opting ISPs to host surveillance hardware and support their deep packet inspection goals.  The end goal being to establish “digital sovereignty” over Russian citizens. One wonders what privacy activist Edward Snowden makes of it all:

  • It’s not usual for a tech blog to review a pop record but then again, Chvrches are not an ordinary band.  Their first album, The Mother We Share, was widely feted not least for its emotional content.  Their second (traditionally difficult) album, Every Open Eye, is also a triumph according to the Verge.  The Atlantic review the band’s progress and the influence of Depeche Mode on their music – quite literally in the case of Clearest Blue:

“In today’s musical climate of disposable pop with little to say versus deliberately obscure electro where all hints of a topline are buried beneath layers of effects and fear of seeming mainstream, Depeche Mode still stand alone, unafraid to foreground melody and imbue music with emotion in a way few other songwriters can.”

Leave a Reply