- Self-referential data visualisation I: Vincent
- Devices and Manufacturers
- Google and Android
- Apps and Services
- SiliCon Valley
- Software on TV
- Artificial Intelligence
- Cloud, Infrastructure and DevOps
- Digital Enterprise
- Wearables and Insideables
- Space and Science
- Startups, Work and Management
- Culture and Society
- Share this:
Self-referential data visualisation I: Vincent
I took a few shortcuts in order to get something working in limited time. In particular avoiding MongoDB as an intermediate JSON data store and using a visualisation grammar called Vega that simplifies access to D3. A pure Python library called Vincent (see what they did there?) allows you to generate Vega directly from a pandas dataframe. I’ve released an updated version of the original blog.py script imaginatively called blog2.py that partially implements the workflow Kyran outlined with these limitations built-in. Running the blog2.py script output yields the following terminal output:
$ python blog2.py A] Retrieve JSON stats from cached stats B] Load JSON into pandas dataframe and clean it up week htmlsize wordcount 0 1 66331 3036 1 2 77970 4019 2 3 62670 2570 3 4 67475 2832 ... 36 37 71646 3537 37 38 70583 3440 38 39 76900 3709 Sum of htmlsize: 2787803 Sum of wordcount: 132451 Average wordcount: 3396 ---- EXECUTION TIME: 0.920894 secs ----
Plus the all-important graphic itself:
Which is created in a file called ‘blog.html’ using Vincent direct from a reduced scope pandas dataframe rdf thus:
lines = vincent.Line(rdf) lines.scales.type = 'ordinal' lines.axis_titles(x='Weeks', y='Count') lines.legend(title='Title Here') createVincentHtml('blog.html','N',lines.to_json())
The data shows a constant average total 3.4k words per post per week and ongoing strong correlation between word count and HTML page size. The next time I return to this topic, I’ll look to fill the gaps and explicitly switch to D3.js in the hope I can tidy up the graph.
Devices and Manufacturers
- Who killed Nokia? They did it to themselves they did according to research conducted by two Professors of Strategy at top European business school INSEAD. The write-up is a pretty brutal indictment of a dysfunctional organisational culture which is a topic that returns below in respect of the VW emissions scandal:
Nokia lost the smartphone battle because of divergent shared fears among the company’s middle and top managers led to company-wide inertia that left it powerless to respond to Apple’s game changing device.
In a recent paper, we dug deeper into why such fear was so prevalent. Based on the findings of an in-depth investigation and 76 interviews with top and middle managers, engineers and external experts, we find that this organisational fear was grounded in a culture of temperamental leaders and frightened middle managers, scared of telling the truth.
- John Gruber reviews the iPhone 6s and examines the host of subtle changes that taken together suggest there is some validity behind Apple Marketing’s assertion that “The Only Thing That’s Changed Is Everything”. Meanwhile, one long-term Android user who moved to iOS for the first time with an iPhone 6s wrote up his thoughts on the switch on Reddit both positive and negative. Overall he’s pretty upbeat about the product features and quality. Commentators have their own views about what the killer features are. For John Biggs at TechCrunch it is 3D Touch. For Asymco, the incremental improvement of an already great package defines what it means to be a great product company:
Apple is a company that builds greatness because they built a process to make their customers better. They do it without permission and they do it without offense. But when the buyer perceives the change they feel in debt to the object. Getting payments on that debt of gratitude is how Apple is rewarded.
- Blackberry’s first Android effort is revealed as the curiously-named Priv smartphone. Positioned as a device that “combines the best of BlackBerry security and productivity with the expansive mobile application ecosystem available on the Android platform” it’s clearly aimed at the growing cipherphone opportunity.
- Vertu launched a new flagship, the Signature Touch that “runs on Android 5.1 and packs an octa-core Snapdragon 810 chipset with 4GB of RAM, along with 64GB of internal storage plus another 64GB on the included microSD card“.
- 5 million Raspberry Pi’s have now been manufactured by Sony UK at their Pencoed factory in Wales. It’s a great, albeit all too rare, success story for high-tech UK manufacturing:
— Philip Colligan (@philipcolligan) September 23, 2015
Google and Android
- Google have announced two new Nexus phones, the Huawei 6P and the LG 5X both running the Marshmallow release. The former will launch as the new Android flagship:
“Nexus 6P is the first all-metal-body Nexus phone. Built in collaboration with Huawei, this 5.7” phone is crafted from aeronautical-grade aluminum, with a USB Type-C port for fast charging, a powerful 64-bit processor, and a 12.3 MP camera sensor with massive 1.55µm pixels (hello, better photos!).”
- Bloomberg are claiming that Google is under US antitrust scrutiny over Android apparently over a variation of full line forcing and product tying. Or as Bloomberg put it:
The practice of bundling products and services together may violate antitrust laws if a company dominates the market for a product that customers need, and then forces them to buy a complementary product or service
- The EU had similar concerns about Google in the past as highlighted in the blog a year ago.
Apps and Services
- Quartz on the evidence that if you are a US smartphone user “you only really use three apps on your phone” and one of them is Facebook:
The big winner, unsurprisingly, is Facebook. It’s a top-three app for almost 80% of users, according to comScore, and number one for almost half of them. (With 126 million US unique visitors in June, it’s the most popular app in the US by a solid margin. It’s also on almost half of users’ smartphone homescreens, according to the report.) And as Facebook goes deeper into mobile video, its share of time spent seems likely to increase.
- This presentation from Facebook engineer Simon Whitaker is a picaresque and at times jaw-dropping ride through Facebook’s codebase and tools initiatives. Presumably they have their reasons for embracing NIH with gusto and the deck provides reasonable justifications for each case but, man, are they building a substantial legacy for themselves. They’ll need to keep hiring software engineers to look after all this and more:
- Seven reasons why Silicon Valley’s dominance may be over. The short answer being that entrepreneurs there are increasingly focussed on first-world problems unlike innovators in other parts of the world:
“there is a trend in Silicon Valley to focus only on solving the problems of the rich, whereas entrepreneurs in the rest of the world tend to focus more on the problems of ordinary people – and maybe it is that second sort of innovation that will prove the better long term bet.”
- Underlying this trend, NYT report on how India is now the “next big frontier” for US tech companies. It’s a completely different environment for doing business in than the US.
- Bracing and characteristically blunt evisceration of Silicon Valley’s obsession with unicorns from Mark Suster which skewers the short-termism. For Suster, being an entrepreneur is not about the glamour of a billion dollar valuation:
It’s really about love. And sacrifice. And hard work. And putting in the daily things that it takes to achieve great things. And how in the daily routine of being yourself, committing to goals and just living life, you realize that goals were easier to obtain than you had imagined. It’s a love story of sorts.
- Hilarious and sharp alternative Hierarchy of Needs for Silicon Valley:
- Good HBR take on the VW emissions omnishambles outlines three vital lessons the company failed to absorb and now has to act upon really quickly if it is to recover:
- Being clean and green has real, bottom-line value.
- Protecting the environment builds trust, and trust is precious capital.
- Trust comes from transparency, and transparency is the norm today.
- The key takeaway:
“being clean and green is important, but only if it’s authentic. And remember that everyone is watching you.”
- Travis Bradberry in this thoughtful LinkedIn post goes further suggesting that what lies at the heart of the issue is nothing less than the integrity of VW’s leadership, which has this week changed. He outlines some key leadership antipatterns including micromanagement, toleration of poor performance, cult of personality and the tyranny of the urgent:
The tyranny of the urgent is what happens when leaders spend their days putting out small fires. They take care of what’s dancing around in front of their faces and lose focus of what’s truly important—their people. Your integrity as a leader hinges upon your ability to avoid distractions that prevent you from putting your people first.
- Certainly the scale of the scandal and the pace at which it has unravelled is frightening with AdWeek suggesting the company has just destroyed 55 years of brilliant campaigns with one calamitous mistake. Clearly VW senior management was completely out of touch with customer and public sentiment. HBR’s Michael Schrage reckons the fallout will be so damaging it will be the “end of large scale corporate deception“. Then again, many said that after Enron:
This sustained and sophisticated software scam suggests truly pathological levels of managerial desperation and contempt: desperation around a failed promise of clean diesel technology and unsubtle contempt for unsuspecting regulators and customers alike. Precisely how gullible did these wizards of Wolfsburg think people would be?
- The FSF link software disclosure through permissive licences with the VW debacle suggesting that ultimately the company needs to demonstrate trust through transparent actions. Ironically, VW is a contributor to the Linux kernel so ought to be well versed with GPL and the culture it comes from:
If Volkswagen knew that every customer who buys a vehicle would have a right to read the source code of all the software in the vehicle, they would never even consider the cheat, because the certainty of getting caught would terrify them
- NYT shone the spotlight on the growing complexity of car software which is increasingly “the weak spot under the hood” for car OEMs who are probably light years behind handset OEMs never mind the likes of GAFA in their software competency profile, tools and methods.
- It may all serve to accelerate the rise of software-defined automotive most obviously in the form of car industry nemesis Tesla. This week CEO Elon Musk provided a tantalising glimpse of a near future electric car offering 620 mile range on a single charge that can’t come soon enough after Dieselgate:
Software on TV
- Entertaining exploration of the hacker tools used in TV series Mr Robot finds one of the most realistic portrayals yet on celluloid with John the Ripper, Bluesniff and Social Engineer Toolkit all getting a namecheck.
- BBC Click’s Spencer Kelly covered the current state of Artificial Intelligence in a two-parter with the second part this week covering neural networks and IPSoft’s conversational UI (CUI) proposition Amelia:
- Marcus du Sautoy fronted an hour-long BBC special on algorithms called “The Secret Rules of Modern Living”. As befits the current Simonyi Professor for the Public Understanding of Science, it’s a superb and very approachable introduction to an important topic. The program starts with a visual explanation of Euclid’s solution for GCD followed by practical demonstrations of different sorting algorithms and a walk-through the Gale-Shapley approach to the Stable Matching Problem (SMP). Highly recommended.
- AI powered business applications are “about to go mainstream and reshape the workplace” specifically in relation to business process reengineering and workflow optimisation:
We define AI-powered business applications as software in which the user experience is driven by AI. Today, that takes the form of voice (Apple Siri), text-based interfaces (x.ai’s Amy) or intelligent notifications (Google Now). Other examples of AI-driven UX like Google Search’s autofill differ in that they do not automate a step within a workflow.
- The Singularity may arguably be approaching fast but robots are still no better than humans at assembling IKEA furniture. You’ll know ASI has arrived when this life skill is mastered:
Cloud, Infrastructure and DevOps
- Splunk’s annual developer conference .conf took place in Las Vegas this week. It’s a company that’s growing fast – 1000 employees and 10,000 customers many of whom attended the conference. The conference was notable for a specific focus on machine learning value adds to the core Splunk operational intelligence proposition focussed on unstructured data.
- InfoQ special on Patterns of DevOps culture presents a series of insights on organisational structure and behaviours from seasoned DevOps practitioners.
- Hashicorp are one of the leading players in the DevOps commercial scene. Founded just 5 years ago, the company is already behind key tools like Vagrant and Packer. Now they’ve announced a meta-orchestration tool called Otto that seems to be positioned as a competitor to Docker Compose.
- Interesting presentation from Talko on how the backend to their Voice 2.0 service platform is built on Erlang in the cloud.
- Matt Asay on the rise of graph databases like OrientDB and what they have to offer in terms of Enterprise data management:
By giving developers a way to express relationships between data rather than fixating on the data itself, graph databases offer a powerful new way to tame the growing complexity of big data.
- Enterprise should embrace Shadow IT as a source of innovation not fear it or, worse still, try and stamp it out:
The rise of the citizen developer will be a common theme over the next few years, and enterprises that embrace the potential of Shadow IT will be best positioned to innovate and thrive.
- With so much change going on in the digital technology space, no wonder HBR are convinced that all corporate Boards need at least one technology expert, a situation that is far from being the case today with an over-reliance on well-paid advisors with typically no skin in the game.
- Disruptive Communications wonders if PR execs in the future will need to be become data scientists to remain relevant. The rationale is hiding in plain sight and applies more broadly to pretty much any function within the modern enterprise:
Any significantly sized business will generate a huge amount of data from across its various operations, and the task of finding actionable intelligence hidden inside it all is not to be underestimated.
- It’s the definitive endpoint for a certain strain of techno-utopian thinking to assume that humans will eventually require their own IP address (or “HIP”). Here’s a canonical defence of the meme from a correspondent who suggests it might happen as soon as 2025:
The evolution of wearable technology points to devices and chips which will be embedded by 2025, rather than worn explicitly. If a single chip is embedded within the human body, or encoded within cells, then in theory every human would become a hub or a node, addressable and traceable within the larger IOT ecosystem and every owned device assigned to that Human IP address. Your ‘social graph’ will become more valuable to the Government than your Social Security or NIS number and it’ll supercede these eventually. … In other words, humans could well become the ultimate ‘thing‘ within The Internet Of Things.
Wearables and Insideables
- Pebble have announced a new product, the Pebble Time Round, with the claim that it is “the world’s lightest and thinnest smartwatch“. It will retail at €300 and sacrifice battery life for style.
- A Huawei waterproof pedometer with iOS and Android compatible notification UI is available online for sub-£60:
- Fusion.net ran what seemed to be a sensationalist story about DARPA “implanting chips in soldiers brains“. DARPA responded:
DARPA responded to a request from Fusion that “brain-neural interfaces” have not yet been implanted in soldiers, though test devices have been implanted in the brains of volunteers already undergoing brain surgery. We’ve changed the headline to reflect that implantation of chips in soldiers’ brains has not happened yet.
Space and Science
- Millions of planets orbiting dwarf stars are more habitable than previously thought because the presence of even a thin atmosphere would cause them to spin:
The researchers believe that an Earth-sized planet with an atmosphere similar to Earth’s would indeed spin as it rotates around small orange dwarf and some red dwarf stars.
- Evidence this week suggests that water may well flow on Mars which significantly increases the possibility of microbial life on the red planet:
- Meanwhile, a real-life astronaut reviews Hollywood blockbuster The Martian for Quartz and finds much to admire:
The Martian paints what I believe is a much truer picture of our nation’s space program, one that requires the collaborative efforts of all teams in order to successfully explore whatever it is that’s potentially “out there.”
- Cue a teaser for the trailer no less, plus an xkcd take on the film’s attractions – xkcd is 10 years old this week:
Startups, Work and Management
- Fantastic must-read from Twitter’s Engineering Effectiveness (EE) team lead, Peter Siebel, on the importance of having engineering resources that are working on tools and productivity support to make the life of everyone else in engineering easier. What is outlined here is essentially a concept used in innovation theory called “absorptive capacity”. To succeed at RnD, particularly at scale, it is essential to have resources that can support and improve the methods and tools used within engineering and not just put everyone on features or bugs. It’s a lesson that has been unevenly learned in the past and one which would-be unicorns ignore at their peril:
- FirstRound profile piece on GiveForward highlights the central role that a unique and authentic culture plays within the organisation and how it colours their approach to fundraising, hiring and retention:
Weird job descriptions weed out the boring, stuffy people and bring us the weird, creative and fun folks we want to work with.
- HBR opinion piece outlines the three key things managers should be doing every day namely building trust, building a team and building a network:
“They build trust by taking the opportunity to demonstrate their ability as they do their daily work, by asking knowledgeable questions and offering insightful suggestions. .. They build a team by using problems and crises in the daily work to remind members of the team’s purpose and what it values most. … They build a network by taking opportunities afforded by routine activities to build and maintain relationships with colleagues outside their group.”
- The Information profile Thuan Pham, Uber’s CTO, and Uber’s engineering struggles highlighting an all-too familiar antipattern behind a major outage in May:
Internal emails and interviews with people who’ve worked on that system show that as CEO Travis Kalanick and operations executives pressed engineers to build new features for the business, it came at the expense of stability in the back-end systems that power them.
- And also provides this rather interesting insight:
Among the tidbits i picked up in reporting this article: 38% of Uber traffic now comes from China, per CTO Thuan. http://t.co/gVhDNKjHHi
— Amir Efrati (@amir) September 22, 2015
Culture and Society
- This powerful NYT piece explains why we should “stop googling and start talking” if we want to reclaim conversation and escape the tether of being permanently online. The prescription includes distinct echoes of Thoreau:
One start toward reclaiming conversation is to reclaim solitude. Some of the most crucial conversations you will ever have will be with yourself. Slow down sufficiently to make this possible. And make a practice of doing one thing at a time. Think of unitasking as the next big thing. In every domain of life, it will increase performance and decrease stress.
- Time on why “sitting is the new smoking” and how fidgeting rather than being a cause for admonishment is actually good for you.
- A tremendous Quora post on historical counterfactuals (“what if” scenarios) asks “what is the smallest amount of believable change in historic facts that would have had the greatest impact on the world as we know it today“. One respondent suggests the haunting possibility of Czar Nicholas’ son being born without haemophilia and all that entailed:
If you want to know how major world events can hinge on one event? That’s pretty much it… a baby born (Alexei) born with Hemophilia with a pair of loving and worried parents…. whose love and concern would throw the world into a century of cataclysm and catastrophe.
- Staying on the theme of Russia, a recently-published book called The Red Web outlines how the country has essentially adapted the intimidatory methods and tactics used the old Soviet era to the digital age co-opting ISPs to host surveillance hardware and support their deep packet inspection goals. The end goal being to establish “digital sovereignty” over Russian citizens. One wonders what privacy activist Edward Snowden makes of it all:
- It’s not usual for a tech blog to review a pop record but then again, Chvrches are not an ordinary band. Their first album, The Mother We Share, was widely feted not least for its emotional content. Their second (traditionally difficult) album, Every Open Eye, is also a triumph according to the Verge. The Atlantic review the band’s progress and the influence of Depeche Mode on their music – quite literally in the case of Clearest Blue:
“In today’s musical climate of disposable pop with little to say versus deliberately obscure electro where all hints of a topline are buried beneath layers of effects and fear of seeming mainstream, Depeche Mode still stand alone, unafraid to foreground melody and imbue music with emotion in a way few other songwriters can.”