Adventures with Docker I: Virtual Machine creation

[avatar user=”malm” size=”small” align=”left” link=”file” /]

Over the last week, this blog has been migrated from dedicated hardware to the Digital Ocean cloud hopefully without readers noticing too much difference.   Special mention needs to be made here of Ian @ Projany Limited who have been kindly hosting the site entirely for free from the outset. Thanks very much for undercutting every cloud provider out there! Moving to Digital Ocean should involve a PAYG cost of around $10/month on current projections.  The initial migration of WordPress was managed using a recipe provided here involving these steps:

• Back Up all files on old website
• Export the old WordPress database
• Create a WordPress database on new host
• Edit the corresponding wp-config.php file
• Import old database to new host
• Upload WordPress files to new host
• Setup up new DNS details and wait for propagation

A key benefit of moving to the Digital Ocean platform is that it opens up the possibility of automated site management using Docker, a tool that has had a meteoric rise in popularity over the last year amongst devops practitioners.  In order to get started you need to first create a personal access token using the Digital Ocean web admin GUI:

Then you need to install a co-ordination tool called Docker Machine to manage the interaction between the Docker command line client and a host platform for running docker on which can be either in the cloud or in a local environment.  Instructions on how to do that are available here. In my case I installed Docker Machine on a standard-issue Windows 7.1 laptop.

Docker Machine works with different infrastructure options through the means of virtual “drivers”.  There are drivers for AWS, Digital Ocean, Azure and VirtualBox.   It’s a very powerful abstraction mechanism that makes it straightforward to try out different cloud provider platforms.   It’s helpful do understand a little more about Docker Machine’s architecture and how it works under the hood with containers and images.  Two useful starting points are docker’s own description and this resource.   Installing Docker Machine gives you all the equipment you need to fully operate docker end to end via a command line tool called docker-machine:

One thing you may want to do post-installation on Windows is configure Docker and Docker Machine git access from the cmd command line thus:

C:\> set PATH=%PATH%;"c:\Program Files (x86)\Git\bin"

At this point creating and launching a new cloud-based Digital Ocean Virtual Machine (VM) called ‘dodocker‘ from within the standard Windows cmd command line takes less than a minute:

C:\> docker-machine create \
--driver digitalocean \
--digitalocean-access-token <access-token> \
 dodocker

C:\> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
dodocker digitalocean Running tcp://104.131.84.90:2376

Once you have this VM running you can interact with it using the standard Docker command line client simply called docker.   This requires configuring some additional environment variables in your Windows cmd session as follows:

C:\> docker-machine env --shell cmd dodocker
set DOCKER_TLS_VERIFY=1
set DOCKER_HOST=tcp://104.131.84.90:2376
set DOCKER_CERT_PATH=\machines\dodocker
set DOCKER_MACHINE_NAME=dodocker
# Run this command to configure your shell:
# copy and paste the above values into your command prompt

With the Docker client thus configured, you can run a command in a new container on the Digital Ocean virtual machine.  Here we try out a vanilla nginx image which brings up an nginx web server on the target VM on external port 8000:

C:\> docker run -d -p 8000:80 nginx
5cb1893cedf093705d2a4af3b07ba4a97f3a79f501bfb2574c45fc3746e0d27b

C:\> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
73c91ccca3a9 nginx "nginx -g 'daemon off" About a minute ago Up About a minute 443/tcp, 0.0.0.0:8000->80/tcp

C:\> docker ip dodocker
104.131.84.90

If you now direct your browser to http://104.131.84.90:8000/ you should see a “Welcome to nginx!” page.

Using the command line to orchestrate all this cloud interaction on Windows feels impressive.   So does the cornucopia of extant images to play with.  A couple of months ago, I outlined a convoluted recipe for pulling together a deep dreaming inceptionism setup. I needn’t have bothered.  Many people have been there, done it and dockerized it already:

C:\> docker search deepdream
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ryankennedyio/deepdream A container to run the deep dream python n... 12
mjibson/deepdream Google Deepdream visualization 7
...

The basic workflow with Docker consists of starting collections of containers on a VM.  In that way, it’s possible to run potentially a dozen or so small web sites within corresponding nginx containers running on a single VM.  Posit: if you charged each owner $1/month you’d still be making money…  Before finishing off, it’s important to be a responsible cloud citizen and switch off (and ultimately remove) an instance you’re no longer using:

C:\> docker kill 73c91ccca3a9
73c91ccca3a9

C:\> docker-machine kill dodocker

C:\> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
dodocker digitalocean Stopped tcp://104.131.84.90:2376

C:\> docker-machine rm dodocker
Successfully removed dodocker

There’s a huge lot more you can do with Docker.   The next step for me is to build on this starting point to automate the backup and restore of the entire WordPress site.  @charltones recently outlined a recipe to broadly do just that using a tool called docker compose.

Manufacturers and Devices

• Ben Evans on why mobile is rapidly becoming the go to channel for accessing the internet particularly for Millennials and teens:

What happens when almost everyone on earth has a pocket supercomputer connected to the internet? It’s not a subset of the internet – it IS the internet.

• Wired offer cold comfort to HTC in suggesting that their “epic tailspin” is not their fault.   They have assiduously followed the premium Android OEM playbook, released great devices and it still hasn’t been good enough:

But HTC was neither stagnant nor incompetent.  It regularly introduced novel features, many of which were smart and useful. It created consistently good smartphones. It seemingly did enough to, if not lead the pack, at least stay in its general vicinity. … In fact, what did HTC in wasn’t HTC. It was a smartphone market that simply doesn’t have room for more than a handful of players, one in which being different no longer counts for much.

• It’s a problem afflicting all premium Android OEMs.  The whole segment seems increasingly like a $0 billion dollar industry as Charles Arthur highlighted in a compelling post outlining the current “dire” state of affairs within premium Android.  In it he highlighted the fundamental issue – a massive gap in the ASP that Apple are able to extract from their customer base relative to Samsung, HTC, LG and Sony:

“More than ever, the smartphone business is turning into one where only two companies make money from handsets – and increasingly, only one gets the top-end business. For the rest, they need something new to come along that they can get into and make some profit. Android Wear smartwatches don’t seem to be setting the world alight; will VR be the next big thing? Or IoT? Or self-driving cars? At this point, anyone that isn’t Apple or Samsung seems to be praying for a miracle.”

• It’s a message underlined by the largely uninspiring products on display at IFA Berlin.  What seems constitutes innovation in the smartphone form factor no longer seems to move the dial in terms of wonder at least.

“these companies need to look toward innovations that can actually make the human experience more enjoyable. The first smartphones did this by opening up the internet in all of its glory; the first portable music players freed us from having to carry around physical media; wearables promise to make us healthier; and the Internet of Things paves the way for a fully automated home. Genuine usefulness is what these companies need to focus on because simply adding another layer of gloss on what already exists is simply not good enough.”

• One regularly wheeled out strategic approach is for OEMs to move into cross-platform services which is easier than it sounds as it involves quite different skillsets. It at least helps explains why Blackberry is buying rival Good Technology “to boost its ability to help corporate clients manage smartphones running on different operating systems.” Most of Good’s customers use Apple devices.

Google and Android

• The Information are reporting that Google is nearing a re-entry to mainland China after a five-year absence after reaching a deal with state authorities. If true it will be a momentous development.  In the intervening years a vibrant collection of alternative so-called Chinadroid platforms built on vanilla AOSP Android have taken control of the smartphone market with their own competing service layers filling in the missing Google properties.  The re-entry is apparently based on a limited region-specific Google Play Store.

• That new Google logo received more coverage and considerable positive reaction online particularly in respect of the claim that the change resulted in the amount of data required for the logo shrinking from 14kbytes for the old serif format to just 305 bytes.  This insightful post on Quora includes the source for a sub-300 byte svg “constructed almost entirely from circles and rectangles” that very closely approximates the new logo.  Here’s what an online renderer comes up with:

Asia

• Interesting HBR piece connecting the Chinese slowdown to US firms shifting to a blended manufacturing approach that involves more nearshoring.

• WSJ report Huawei suggesting the Chinese smartphone sector is due for some major consolidation as the economy cools.

Cloud and DevOps

• Great post courtesy of SpaceApe Games on the devops hierarchy of needs which provides a glimpse at their journey to a devops way of working.  It also posits somewhat cheekily that the top of the triangle is attending and speaking at conferences.  Actually this is more plausible – a view of devops as akin to gardening with no fixed end to aim at only continuous improvement:

More likely is that there is no next level, just a continuous re-hashing of the foundations beneath us as new technologies and challenges enter the fray.

 

• Serving to underline that message, this Reddit devops thread explains why there isn’t really a “one size fits all” product for the devops space.  Each situation and context is different.  What unites devops practitioners is a shared collection of process, tools and above all shared culture.  It’s really more a movement than any specific technology or product:

Labeling something “DevOps” doesn’t really make a lot of sense, and your use of the word will put you at odds with a significant number of people. Continuous Delivery is OK, I guess, but you better be damned ready to tell me how your product meets the criteria and why Jez Humble and Dave Farley would sing your praises.

• CapGemini on the existential threat digital transformation represents for CIOs.

As back-end IT is increasingly commoditised and delivered as a utility, as employees increasingly use mobile devices and ubiquitous network access the requirement for the CIO’s huge and costly apparatus diminishes.  DevOps and continuous delivery are similarly rendering the old structures obsolete.

Big Data and Privacy

• Fascinating and rather disturbing post on what happened next when Australian reporter Will Ockenden exercised his rights to pull a year’s worth of his smartphone network data courtesy of Telstra and then provided it to the general public for data mining purposes.  Among other highly sensitive personal information it revealed his daily location “heat map” and contact graph from an apparently fairly innocuous (at first sight) dataset:

The dataset contained a year’s worth of outgoing call and SMS records, and six months of his ‘data sessions’, which are the records kept every time his phone connected to the internet over the mobile network.

• All the more reason for greater scrutiny and exploration of the implications of “algorithmic determinism” as sounds likely at this forthcoming conference:

“Addressing issues of the politics of algorithms, normative classifications and algorithmic governmentality “Algorithmic Regimes and Generative Strategies” wants to mobilize the critical perspectives of researchers, artists and activists, to open the field for a wider and more diverse debate.”

Product Analytics

• Really interesting post from Monica Rogati, the VP of Data at Jawbone, on how the “digital native” is now being eclipsed by a new arrival, the “data native” and the profound implications that raises for consumer product companies like hers:

A data native is someone who expects their world to not just be digital, but to be smart and to adjust immediately to their taste and habits,” she says. “For example, a magazine should not only be digital and interactive — it should be personalized. It should tell you what you need to know based on your interests, location, preferences. The expectations have shifted.

• Product analytics increasingly help “eliminate the noise from the signal” in new product development. Literally in the case of a tool proposition called Signals.

Wearables, the Internet of Things and Hardware

• The “Wena Wrist” smartwatch received positive coverage from Engadget who went as far as to suggest it is “the smartwatch that luxury brands should be making“.  The watch itself is fairly traditional and low tech with all the smarts residing in the strap. The product has been developed on Sony’s First Flight crowd-funding platform.

• Estimote outline their indoor location platform play built on beacons and “nearables” which seems to be their term for geolocated beacons:

• TechCrunch propose a “hierarchy of IoT needs” that adheres to entirely predictable lines – hardware and power at the bottom, connectivity next layer up and data/analytics at the top.

• Orange have announced their Raspberry Pi rival is now intended to cost just half the price, only $15.  Hackaday previewed the board this week:

• VSP is the leading provider of vision care insurance in the US. They’re also running an interesting glasses technology incubator which Engadget recently visited where they are developing a futuristic eyewear platform called Project Genesis:

• Panasonic are apparently reviving the legendary Technics SL-1200 vinyl turntables much loved by old-school DJs:

Artificial Intelligence and Robotics

• Provocative and thought-provoking Kernel post on the impact ASI could have on organised religion.  A range of opinions are provided from those that think religion will adapt and evolve as it always has to those that believe artificially superintelligence is fundamentally incompatible with religiosity and will result in a crisis of meaning:

There is no specific religious suggestion that we’ll face a technological singularity. In fact, ancient scriptures from various religions say virtually nothing about science and technology, and what they do say about them is usually wrong (the Earth doesn’t move, is at the center of the solar system, is 6,000 years old).

• Bloomberg cover the latest developments with Brett, the Berkeley robot being developed by Pieter Abbeel’s team that acquires knowledge through reinforcement learning akin to a toddler which was introduced in this blog a few months ago:

What makes Brett’s brain tick is a combination of two technologies that have each become fundamental to the AI field: deep learning and reinforcement learning. Deep learning helps the robot perceive the world and its mechanical limbs using a technology called a neural network. Reinforcement learning trains the robot to improve its approach to tasks through repeated attempts.

• The post is cutely entitled “pre-school for robots” and comes with standard issue robot playing with lego shots. Human analogies stop there though, as the author points out:

“it’s important to remember that A.I. is, and will likely remain, deeply alien”

• As a counterpoint to the surfeit of robotic determinism and automation dystopia, John Markoff the author of “Machines of Loving Grace” explains why “humanity is essential to the future of artificial intelligence.”   And this interesting Nautilus post explains why smart machines rather than being in binary opposition to humanity seeking our destruction will instead “take us with them” resulting in a complex spectrum of post-human enhancements including genomic tinkering to produce “super humans”:

The feedback loop between algorithms and genomes will result in a rich and complex world, with myriad types of intelligences at play: the ordinary human (rapidly losing the ability to comprehend what is going on around them); the enhanced human (the driver of change over the next 100 years, but perhaps eventually surpassed); and all around them vast machine intellects, some alien (evolved completely in silico) and some strangely familiar (hybrids)

• Exoskeleton and electricity “help a paralysed man to walk again“:

• In a similar spirit of human-machine collaboration, IEEE outline Toyota plans for a “major push into AI and robotics“.  The focus is not on autonomous vehicles per se but rather on using AI to make more reliable cars:

The initial focus will be on accelerating the development of AI with applications to smarter and safer vehicles, as well as robots that can make our lives better at home, especially as we age.

Management

• Useful post on the benefits and challenges of managing a distributed engineering team including some tips on particular areas requiring focus – notably communication and alignment.

• HBR on the importance of paying attention and the resulting “powerful effect of noticing good things at work“.  It isn’t necessarily a natural inclination for many leaders:

“For most people, this doesn’t come naturally. Evolution and the survival imperative have attuned human beings to pay careful attention to all things negative; in addition, people quickly become so accustomed to positives in their environments that they hardly notice them. But a simple intervention can help overcome both tendencies.”

• Too many leaders are instead prone to concentrating on what went wrong and exercising “negativity bias” in spite of the often immensely corrosive impact it has on longer term morale and employee loyalty:

There is something called the negativity bias, which says basically, we generally spend more time thinking about negative things than we spend on positive.  So, even if you have appreciated folks you work with it is likely you can appreciate them even more.  It can’t hurt to appreciate more even if you are doing it now.

Startups

• Vanity Fair outlines the evidence suggesting we’re on the verge of another tech bubble about to burst driven by stratospheric VC driven funding, FOMO (fear of missing out) and a need to “run with the unicorns” in order to hire the best engineers.  Logic suggests it must end somewhere in what the author calls a “kaboom!”.   However, if and when it does, the least likely victims will be VCs who are well-protected against any imminent downside:

The problem with being a unicorn, indeed, is that there aren’t many exit strategies. Either you can go public, which is inadvisable without a lot of revenue, or you can sell, which is difficult given the paucity of companies that can afford to make such an offer. So, for many, the choice becomes fairly simple. You continue to raise more and more money, or you die.  KABOOM!

• Tech journalist Biz Carson reviewed 146 recent pitch decks and came up with some conclusions that are worthy of enumeration:
  • Security is the new hot ticket.
  • Hardware startups are growing.
  • How we hire and recruit employees is about to change dramatically.
  • Social startups are dead.
  • Money is falling from trees, and even small businesses want a part of it.
  • Everyone is now an “X for Y.”
  • Startups are looking outside Silicon Valley — way outside.
  • Startups are going niche, in both their product and their market.

Science

• The possibility that we live in a multiverse of universes is one which arises as a consequence of quantum theory and string theory.  This article outlines an approach for how it can be scientifically tested:

The universes predicted by string theory and inflation live in the same physical space (unlike the many universes of quantum mechanics which live in a mathematical space), they can overlap or collide. Indeed, they inevitably must collide, leaving possible signatures in the cosmic sky which we can try to search for.

• Astronauts and the transcendental “overview effect” induced by travelling beyond the bounds of earth. Apparently you can experience the same effect through meditation on the ground.

• Caltech researchers have found a galaxy 13.2 billion light years away raising a question about the accuracy of 13.8 billion year estimate for the current age of the universe.

Work Antipatterns

• Sparky LinkedIn post on “insane” hiring practices:

A guy who wrote a share market algorithm with 67% accuracy statistically failed in Maths aptitude test in on-campus placement four times in a row.

• Again from a LinkedIn opinion piece, not giving employees a say in how they do their jobs is a primary cause of unhappiness at work:

“By giving employees more of a say in how they do their jobs.  By making sure we offer them opportunities to learn and grow. And by encouraging them to suggest improvements to the production process and listening to what they say.”

• HBR on why overwork could be killing you and likely adding to the unhappiness quotient.

• No wonder NYT highlight the extent to which work as a source of friendship is on the wane as it becomes more transaction-oriented and in some cases a fully signed up to a gig economy which tends to discourage personal investment.  It certainly resonates with me that many of the strongest work relationships I have are with co-workers I last worked with in many cases over ten years ago:

The economic explanation is that long-term employment has essentially vanished: Instead of spending our careers at one organization, we expect to jump ship every few years. Since we don’t plan to stick around, we don’t invest in the same way. We view co-workers as transitory ties, greeting them with arms-length civility while reserving real camaraderie for outside work. At best, as the movie “Fight Club” termed our conversation partners on airplanes, colleagues become “single-serving friends.”

• Somewhat confusingly, the “only two rules you need to know to be successful in work and life” break down into about four subrules for each.  tl;dr is – look after your body, nourish your spiritual side, keep learning and stay curious.   Something to bear in mind to counter the challenges above.

Culture and Society

• Excoriating WashPo post on Syria asks why some Arab nations which are physically and culturally closer to would-be migrants aren’t doing more to support them leaving many to risk all on a perilous journey to Europe.

• Therapist “secrets” of human nature reveal some predictable tropes:

Many people are afraid to be honest about what they feel and don’t want to do because they believe telling the truth will disappoint and deeply hurt others.

People who are afraid of being treated unfairly will unintentionally treat others unfairly in an attempt to ensure that no one is taking advantage of them.

• Engadget previewed the newly released Alex Gibney biopic of Steve Jobs.  Much of it covers familiar ground.  As ever with Jobs, it’s complicated. It’s worth reflecting at a time like this that Jobs was also the son of a Syrian migrant.  And his subsequent life story would generate much to keep the therapists busy – a million different people from one day to the next:

The Man in the Machine’s most affecting and illuminating moment occurs during an interview with Bob Belleville, a former Apple engineer from the original Mac team. He describes a fairly demanding work environment, where everyone slaved around the clock to get the Mac up and running. That job eventually cost Belleville his marriage and his relationship with his children. But he also had a special connection with Jobs — they traveled to Japan together several times and worked closely for years. Belleville’s dueling mixture of love and underlying contempt for his former boss are plainly visible when, in tears, he reads a eulogy he wrote for Jobs on camera.