Week 44

Published on Author malmLeave a comment

Self-referential data visualisation VI: Double Docker

[avatar user=”malm” size=”small” align=”left” link=”file” /]

Over the last five weeks I’ve developed a simple Python-based data visualisation pipeline for my blog stats culminating in last week’s basic Flask app.  This week, I will further evolve that app by reorganising the code and adding templates and styling. This improved version will then be hosted on the same Digital Ocean instance coexisting with, but isolated from, the first by means of a second Docker container.

Refactoring, templates and CSS

Our starting point last week was a single app.py using a utility file called blog6.py.  In this week’s refactored web2 app, app.py has been modified to use Flask templates.  In addition the old utility file has been split into two separate ones called mongolab.py and graphs.py which manage the mongolab.com cloud database handling and graph data creation respectively.  The corresponding Flask templates reside in a new ‘templates‘ subdirectory and a simple CSS file, style.css, lives in a ‘static‘ subdirectory.  Here’s an example of how the new app.py invokes a styled template which uses jinja2 markup in this case to create a Vincent graphic:

@app.route('/vincent/')
def vincent():
    title = 'malm.teqy.net 2015 blog statistics using vincent'
    jsondata = graphs.vincentVisualisationData(df,width,height,title)
    return render_template('vincent.html',\
      title=title, width=width, height=height, data=jsondata)



<!doctype html>
<title>{{ title }}</title>

<link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">

<script src="http://d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="http://d3js.org/topojson.v1.min.js"></script>
<script src="http://d3js.org/d3.geo.projection.v0.min.js" charset="utf-8"></script>
<script src="http://trifacta.github.com/vega/vega.js"></script>

<div class=page>
    <h1> {{ title }} </h1>
    {% block body %}
        <div id="vis"></div>
    {% endblock %}
</div>

<script type="text/javascript">
    // parse a spec and create a visualization view
    function parse(spec) {
        vg.parse.spec(spec, function(chart) { chart({el:"#vis"}).update(); });
    }
    parse({{ data|safe }});
</script>

The end result is structurally similar to last week’s example but more modular and consistently styled:

visualisation

Double Docker

A second Docker container has been built and deployed onto same the Digital Ocean VM running last week’s container using the following commands:

$ docker build -t flaskapp2 .
$ docker run -d -p=5001:5001 --name web2 flaskapp2

And now we have two Docker containers running separate Flask apps on the VM, the older one on port 5000 and the new one built this week on port 5001 verified with this Docker command:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f59f538c1e72 flaskapp2 "python /usr/src/flas" 6 seconds ago Up 4 seconds 0.0.0.0:5001->5001/tcp web2
676f987989ce flaskapp "python /usr/src/flas" 6 days ago Up 6 days 0.0.0.0:5000->5000/tcp web

You should also be able to browse the second container at http://labs.malm.co.uk:5001.  To pull a copy of all the code for building both the web and web2 containers, run the following command:

$ hg clone https://bitbucket.org/malminhas/blog

There are several other directions that could be explored from here. One useful one would be to replace the cloud mongolab database with a local redis store. However, I’m going to park the extended data visualisation with Docker exercise here for now.

Mobile Devices and Platforms

smartphone-market-pyramid-research

Apple’s increasing monopoly on the high-end of the market is creating a virtuous cycle that ensures they will own the high-end indefinitely. From an app perspective, new and updated apps launch first on iOS, which means people who care buy iPhones, which means future new and updated apps launch first on iOS. From a component perspective, Apple is increasingly the only manufacturer that can even afford to buy the best components, and they have massive scale which ensures they get first dibs on what is new.

And Apple’s iPhone plans are clearly working. The company announced that 30% of iPhone sales last quarter come from Android switchers. It’s likely that many iPhone buyers were able to take advantage of various iPhone upgrade programs available from retailers – Apple’s program only launched last quarter as well – and score contract-free new iPhones for cheaper prices than they’d be otherwise used to.

  • Meanwhile Android manufacturers remain under the cosh.  HTC “has stopped issuing future guidance going forward on its financials” citing “uncertainty”.   LG’s successes to date in 2015 were reversed in Q3 as “the firm’s mobile division lost $67.8 million in the last three months“.   And this leaked internal slide reveals what passes for innovation at Samsung and all too many other OEMs it seems – tweaking Android until it looks like iOS:

Samsung's internal slide deck comparing to the iPhone

  • Innerexile have a “magical” instant repair screen protector for the iPhone6s that uses a “hydro-oleophobic coating”:

So what’s the secret sauce behind this new coating? Well, it’s dotted with microcapsules that contain an adhesive-like liquid, and when damaged, the liquid will fill the void so quickly that you probably won’t even realize you’ve just scratched your case or screen protector.

innerexile-instant-self-repair-screen

Google and Android

“Combining the two operating systems means setting up Android to run on laptops and desktop computers, which would require big changes, as well as supporting the Google Play Store”

  • Meanwhile Chrome for Android has a new eight tab view.  Here it is on my phone together with a still image from the George Boole Google Doodle.

Chrome

microsoft-arrow-launcher

people don’t think so much about the operating system on their phone. But those who produce phones or sell phones or develop applications, they are very preoccupied with the operating system.

Mobile Apps and Services

  • Staying with Microsoft, they’ve also added polish to their mobile Outlook app suite offering built on the acquisition of Acompli last year and rebranding of its technology.  That allowed Microsoft to bypass all the tribulation of building Outlook from scratch for rival platforms.  By all accounts it’s been a spectacular success and vividly demonstrates why acquisition can often be a strategic game changer.  Microsoft are expanding on their foothold by integrating further acquisitions including the Sunrise smart calendar.  The screenshots below show Outlook on Android on the left and iOS on the right:

[Outlook] has almost 30 million users on smartphones and tablets, who use the app for 1.2 billion sessions a month, each 22 seconds long on average. Stats on mobile email market share are tough to come by, but that represents a huge jump from Acompli, which had fewer than 200,000 users before it morphed into Outlook. The company has continued with the strategy it started with the Acompli acquisition by buying the Sunrise smart calendar app and Wunderlist task manager.

“you can use the app to be notified when people change email address and other information, and update your records for them easily across all your connected address books”.

FullContact

  • TNW opinion piece from a TNW correspondent on why he dropped Dropbox “and you should too”.  Dropbox, rather like Flipboard, is in danger of finding itself dangerously placed at the nexus of the colliding tectonic plates of the giant platform players:

As Apple, Microsoft and Google have invested in integrating their cloud storage neatly into their ecosystems, it’s gradually negated the need to use Dropbox at all, and made its pricing far less appealing.

  • Bulk SMS gateway API vendor Nexmo will power KLM’s integration with WeChat.   Which reminds me, I looked at it exactly a year ago.  Out of curiosity I just re-ran my “hello world” SMS sender Python script and it still works.  Props are due.   To use this script you need to register with Nexmo and get an API_KEY and API_SECRET.  SOURCE and DEST are phone numbers.
from nexmomessage import NexmoMessage

msg = {
'reqtype': 'json',
'api_key': API_KEY,
'api_secret': API_SECRET,
'from': SOURCE,
'to': DEST,
'text': 'Hello, world!'
}

print("Sending message:\n%s" % msg)
sms = NexmoMessage(msg)
sms.set_text_info(msg['text'])
response = sms.send_request()
print(response)

{u'message-count': u'1', u'messages': [{u'status': u'0', u'network': u'23415', u'remaining-balance': u'1.93960000', u'to': u'xxxxxxxxxx', u'message-price': u'0.03140000', u'message-
id': u'020000008665F94A'}]}
  • This exhausting but highly insightful FirstRound post distills the collective wisdom of Kamo Asatryan on how to grow your mobile app user base.  He is billed as “one of a small handful of people who have observed hundreds of mobile apps, thought deeply and scientifically about their mechanics, and determined what they could change to grow faster“:

In this exclusive interview, Asatryan not only provides the formulas he relies on to help mobile apps optimize their entire user funnel, but the counterintuitive lessons and mistakes he’s learned on the job. Anyone who works on a mobile app in any capacity has something to gain from this clear-cut look at what drives true, meaningful growth.

  • Astrayan’s eight key takeaways:
    1. Don’t over-educate your users.
    2. Bust unexpected sources of friction.
    3. Find opportunities for positive reinforcement.
    4. Find and focus on your best referral channels.
    5. Make referrals a valuable win-win.
    6. Consider double-sided incentives.
    7. Don’t rely on standard social media sharing.
    8. Aim for steady growth.

Artificial Intelligence

The ability to continue publishing and otherwise maintain a presence in the scientific community is the most important factor for top students making career decisions,

  • BrainPickings published a review of a book rambling through the varied views of leading thinkers on Artificial Intelligence and what it means to be human.   It’s a bit of a mixed bag with a fair degree of that most human of traits, contradiction.  This is from  a much longer quote from Kevin Kelly in which he elaborates on the reality that “the kind of thinking done by today’s emerging AIs is not like human thinking” and what that means:

“AI could just as well stand for Alien Intelligence. We cannot be certain that we’ll contact extraterrestrial beings from one of the billion Earthlike planets in the sky in the next 200 years, but we can be almost 100 percent certain that we’ll have manufactured an alien intelligence by then. When we face those synthetic aliens, we’ll encounter the same benefits and challenges we expect from contact with ET. They’ll force us to reevaluate our roles, our beliefs, our goals, our identity.”

Yes, humans can retrain over time, to learn new skills, in readiness for new occupations when their former employment has been displaced by automation.  However, the speed of improvement of the capabilities of automation will increasingly exceed that of humans.  Coupled with the general purpose nature of these capabilities, it means that, conceivably, from some time around 2040, very few humans will be able to find paid work.

Tech unemployment curves

The Internet of Things

Wearable developers are more likely than other IoT developers to be young (80% are under 35), female (9.7% vs 6.7%) and based in Asia (44%, almost as much as North America and Europe combined).

W151020_WILSON_WHATPEOPLE

  • The security holes in IoT need to be dealt with if we are to avoid catastrophic consequences given the scale increase in attack surface represented by having the “internet on things“.

Security

Getting confused between “sequential attack” and a SQL injection attack is an easy enough mistake to make for a firm outside the technology sector. But TalkTalk is a telco so customers are entitled to expect it to have a clearer grasp of standard infosec practices.

  • The FT also weighed in suggesting TalkTalk only had themselves to blame following what appeared to be a lethal combination of slack security and an alarmingly loose disclosure without any apparent lockdown:

“Interest in TalkTalk as a possible target for hacking may have been piqued after one of the company’s customer service representatives tweeted information that indicated the company stored customer’s login credentials in an unencrypted format, sparking more than 2,400 responses on the social media site, according to Recorded Future. Discussions about TalkTalk’s unencrypted databases and at least 11 so-called cross-site scripting vulnerabilities took place on online forums used by hackers weeks before the actual attack on the company was announced.”

  • Perhaps TalkTalk should take note of this advice from APMG that responsibility for cyber security “should start and stop at Board level”:

Boards must  become fluent in the language of cyber security to improve  the way their companies deal with threats.  When it comes to cyber security, when does ignorance become negligence?

Software Engineering

First of all, not all 2B lines of code were in one language. Maybe about 50% of the code was in C++, 25% in Java, and 25% in python and other random small languages like sawzall and protocol buffer declarations. (Maybe Go as well, nowadays)

  • An awesome BSOD:

  • This TechCrunch perspective piece on why coding academies are “nonsense” makes for interesting reading.  There are some valid points in amongst the abrasive commentary.   The essential point being that coding day in, day out is not a career for everyone and is in any case exposes you to rapid skills obsolescence:

In 20+ years of professional coding, I’ve never seen someone go from novice to full-fledged programmer in a matter of weeks, yet that seems to be what coding academies are promising, alongside instant employment, a salary big enough to afford a Tesla and the ability to change lives.

Digital Enterprise

W150210_CHAKRAVORTI_COUNTRIESBUILDINGDIGITAL1

Science

Startups

  • You’ve heard of DevOps.  This FirstRound post extols the virtues of Sales Ops.   When done right they can make the difference between success and failure.

unicorn-exits-1024x583

Work

  • This must read post from Tim O’Reilly is part of his series on what he terms the WTF economy.  In it he outlines why high freedom environments “where employees are given a great deal of discretion to “do the right thing”  for customers, and where the company seeks to do the right thing for employees” represent the future of work.  It may be the only element of the psychological contract we can hold onto as meaningful in a work environment increasingly tilting to a transactional vs. loyalty-led model:

“The employer-employee relationship is broken, and managers face a seemingly impossible dilemma: the old model of guaranteed long-term employment no longer works in a business environment defined by continuous change, but neither does a system in which every employee acts like a free agent. The solution? Stop thinking of employees as either family or as free agents. Think of them instead as allies.”

  • The reality is that all too often companies are not very good at treating employees like people either as this HBR post points out. It underscores the importance of establishing a high freedom culture in the workplace:

There is little doubt that discretionary effort by people who are empowered to give their best produces not just better morale but also better products and services.

  • Even Google with its famed working environment has to deploy the Performance Improvement Plan (PIP) playbook on occasion.   In this dismissal detailed in a Quora response, the end came swiftly for a burnt-out engineer.  Least he got a coffee out of it:

In the end they didn’t wait for PIP to run through and gave me a termination notice in another week. I returned my Google hardware and badge, and my manager walked me out of the building – all in 10 minutes or so. My Google employee account was revoked at the same time. We went to a nearby cafe with him, and he even paid for a drink though 🙂

Culture and Society

If James Bond actually worked in MI6 today, he’d spend a large amount of time behind a desk doing paperwork and making sure everything was properly cleared and authorized.

  • Spectre involves a shadowy global criminal network that exploits information networks to build a digital panopticon.  Apparently even busy Bond villains manage full quorum face to face meetings.   The capitalist villains and cronies at the heart of John Kay’s fluent 45-minute Google Talk on “Other People’s Money” are arguably equally shadowy and omnipotent.  In his lecture, Kay covers the history and operation of the financial sector and the negative consequences of “financialisation” of the global economy over the last few decades. The title of his eponymous book comes from a 1914 classic that remains valid today:

The goose that lays golden eggs has been considered a most valuable possession. But even more profitable is the privilege of taking the golden eggs laid by somebody else’s goose. The investment bankers and their associates now enjoy that privilege. They control the people through the people’s own money

  • Quartz outline evidence suggesting that dancing with friends is good for your health.   Turns out Bez was onto something all along:

So the next time you find yourself at an awkward Christmas party or wedding dance floor, wondering whether or not to get up and groove, just do it.

Leave a Reply