uniontown labs

Using dsh to Transfer Files to Multiple Machines

Thu, 13 Apr 2017 05:00:00 -0400

If you need to transfer a config file to multiple remote machines, dsh will make your life easier. Check out my previous post about dsh if you need more info about setting up dsh.

First, let’s look at the example file we plan on transferring:

$ cat example-file
example file line 1
example file line 2

We will pipe the output of that file to a dsh group using the -i and -c options. Let’s look at the docs for those options first.

--concurrent-shell | -c
       Executes shell concurrently.

--duplicate-input | -i
       Duplicates the input to dsh process to individual process
       that are remotely invoked.

Now, let’s use those options to tee the contents to a file on the remote servers:

$ cat example-file | dsh -g ex-group -i -c 'tee ~/example-file'
server-2: example file line 1
server-2: example file line 2
server-1: example file line 1
server-1: example file line 2
server-4: example file line 1
server-4: example file line 2
server-3: example file line 1
server-3: example file line 2

You will notice that the output is in random order due to the -c concurrent option. Let’s check our work using -w so that the output is in the proper order:

$ dsh -g ex-group -w 'cat ~/example-file'
server-1: example file line 1
server-1: example file line 2
server-2: example file line 1
server-2: example file line 2
server-3: example file line 1
server-3: example file line 2
server-4: example file line 1
server-4: example file line 2

SSH Config Includes

Tue, 27 Sep 2016 08:00:00 -0400

OpenSSH version 7.3 introduced a very handy Include feature, which is great for people who have to manage connection info for multiple servers. This makes it easy for me to generate updated SSH configs via AWS CLI for the multiple EC2 instances that serve Adafruit IO.

Here is how you can use Include to pull in separate SSH config files from your main ~/.ssh/config. First, you will need to install OpenSSH version 7.3 or higher. If you are using Linux, you will need to install version 7.3+ via your package manager (yum,apt-get, etc), or build it from source.

On OS X, you can do this via homebrew:

$ brew install homebrew/dupes/openssh

Confirm that you are now running 7.3 or higher by running ssh -V:

$ ssh -V
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016

Now you can create a new child config file in ~/.ssh using a text editor. For example, we can create an example child config at ~/.ssh/pi_config and add configuration info just as we would in the main SSH config file:

Host pi-1
  HostName 10.0.0.10
  User pi
  IdentityFile ~/.ssh/pi_cluster

Host pi-2
  HostName 10.0.0.11
  User pi
  IdentityFile ~/.ssh/pi_cluster

Host pi-3
  HostName 10.0.0.12
  User pi
  IdentityFile ~/.ssh/pi_cluster

From your main ~/.ssh/config, add the following line at the top:

Include ~/.ssh/pi_config

You should now be able to connect to your servers as you normally would:

$ ssh pi-1

That’s it! Check out this OpenSSH feature request if you would like more info about the new feature.

Tricking Instagram's Audio Fingerprint System with Infrasound

Sun, 18 Sep 2016 04:00:00 -0400

I recently attempted to share an old video of one of my dogs on Instagram, and I immediately received this notice via email after upload:

We’ve removed the video you posted at 12:24 PM on September 18, 2016 because it included the following content:

Gun Street Girl by Tom Waits

If you have permission to share everything in the video including the audio, like the soundtrack or music, you can appeal the removal and have your video re-posted. Remember that people should only post videos they have the right to share.

I understand why sites like YouTube and Instagram need to block unauthorized use of copyrighted works, but I think that it’s overkill to block non-commercial use of a short piece of audio in a puppy video. I can’t speak for Tom Waits, but doubt he’s going to mind if he doesn’t get license fees for the small number of people (30-40) that hear a short clip of his song in my post. It’s not like I’m uploading a full album, or even a full song.

I normally would have just moved on and not posted the video, but I was curious if there were some ways around the audio fingerprint system. My initial thought last night was that I could overlay a subaudible tone, and hopefully that would be enough to trick their system. When I woke up this morning, I did a bit of searching and I was unable to find any examples of this approach, but I was able to find some other methods that people have tested with YouTube’s Content ID system. These methods include reversing the song, altering the pitch, altering the playback speed, resampling, applying noise, volume changes, and tweaking the stereo imagery.

The author of the article concluded:

It is quite possible to thwart the YouTube Content ID system, but some methods mangle the song too much to be used in anything useful.

I decided I’d give my infrasound method a try, so I loaded the video in Audacity. Here’s what the original track looks like:

I then added a new stereo track, and generated a 1Hz sine wave at 0.8 amplitude. Here’s what that track looks like:

Here’s what the two tracks look like mixed together:

I played the track back on my laptop and phone, and couldn’t hear a difference between the source track and the new track with the 1Hz sine wave overlaid. It passed the ear test, but the real test was to upload it to Instagram.

A video posted by Todd Treece (@uniontownlabs_test) on Sep 18, 2016 at 1:16pm PDT

It works! You should see the 1Hz test video above unless Instagram manually removes the video. The nice thing about this approach is that there is no discernible difference between the source and modified tracks. I’ll be testing this method with YouTube’s Content ID system in a follow-up post.

Hardware Testing with Jenkins & TAP

Wed, 31 Aug 2016 08:00:00 -0400

At Adafruit, when we push or merge changes to one of our hundreds of Arduino libraries on GitHub, we have to ask ourselves “Does this change work?”. In a lot of cases, the best answer we can give, even after extensive manual testing, is “Maybe?”. For Adafruit IO, it is important that we are able to easily answer “Yes.” to that question. A large portion of Adafruit IO users could be impacted by a bug in a library such as the Adafruit IO Arduino Library, and we want to make sure our client code is reliable.

The hard truth is that there aren’t enough hours in the day to test each change to every library on all of the supported platforms, so most of the time we have to settle with testing one or two platforms manually, and hoping for the best on the rest of the supported platforms.

What do we mean by platform? In this case, we are talking about the multiple microcontrollers that are supported in the Arduino IDE via the Arduino 1.6.x Third Party Hardware Specification. Arduino Uno, ESP8266, Feather M0 (ATSAMD21), and the Feather WICED (STM32F205) are all good examples different platforms that require separate testing.

First Attempt: Build Verification using Travis CI

We had previously attempted automated build checking using Travis CI, and that method works great for checking if sketches build on multiple platforms.

As you may have guessed, this leaves us with a huge blind spot. A passing build does not mean the code will do what you expect it to do.

Second Attempt: Manual Compatibility Testing

Our next attempt involved building a ‘compatibility matrix’ web app that allowed for easy logging of the results of manual tests on supported platforms. The app pushes test results to a GitHub repo as JSON (in case we need to access it programatically). It also automatically adds/updates a compatibility table in the README of the target library.

Here’s an example of the README output:

This method is very effective for answering “Does this change work?”, but it is very impractical to manually test the daily changes to libraries on every supported platform.

Latest Attempt: Running Tests on Hardware

Our latest approach to this problem uses Jenkins CI running on an AWS EC2 instance, and small fleet of local Raspberry Pi test nodes. Each node uploads and runs unit & integration tests on hardware attached to USB ports.

Each Raspberry Pi can support multiple platforms, and each platform can be targeted by the test. The tests themselves output results using the TAP protocol over a serial connection, and the Raspberry Pi sends the results back to Jenkins so we are aware when builds fail.

Here’s a visual overview of how it works:

Test status can be monitored in real time from anywhere, and we are able to view the raw console output from the test nodes when needed.

TAP results are also parsed and attached to each build:

We will be sharing the details of this in a tutorial soon, but if you would like to take a look at our early experiments with Jenkins, head to jenkins.adafruit.com or check out our Jenkins Pipeline Library on GitHub.

Sending Commands to Multiple EC2 Instances Using dsh

Fri, 26 Feb 2016 04:00:00 -0500

We are moving Adafruit IO’s node.js services to a group of t2.nano EC2 instances, and I wanted to document my dsh setup. I used to use it for managing the backend workers when working on data.sparkfun.com. It came in really handy when I needed to quickly check or kick anything on the servers.

First, you will need to install dsh using brew, apt-get, yum, etc:

$ brew install dsh

You will need to make a couple config directories for your user:

$ mkdir -p ~/.dsh/group

Create a new dsh.conf file using your favorite text editor:

$ vim ~/.dsh/dsh.conf

Paste the following config into ~/.dsh/dsh.conf, and save the file:

showmachinenames = 1
remoteshell = ssh

Now, you can create a dsh group for the set of servers you would like to send commands to. Create a new file called example in the ~/.dsh/group folder using your favorite text editor:

$ vim ~/.dsh/group/example

Enter the host names for your servers, and save the file:

example-1.uniontownlabs.org
example-2.uniontownlabs.org
example-3.uniontownlabs.org
example-4.uniontownlabs.org

Now you can send commands (like uptime) to all servers in the group using the following command:

$ dsh -g example -w 'uptime'
example-1.uniontownlabs.org:  14:17:44 up 17:15,  0 users,  load average: 0.08, 0.03, 0.05
example-2.uniontownlabs.org:  14:17:43 up 16:36,  0 users,  load average: 0.00, 0.01, 0.05
example-3.uniontownlabs.org:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
example-4.uniontownlabs.org:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05

Using -w will send the command to each server in the order they appear in the group config, and will wait for a response before continuing. If you want to send the commands concurrently to all servers in the list, use the -c option to send the command:

$ dsh -g example -c 'uptime'
example-1.uniontownlabs.org:  14:22:42 up 17:20,  0 users,  load average: 0.00, 0.01, 0.05
example-3.uniontownlabs.org:  14:22:42 up 16:46,  0 users,  load average: 0.00, 0.01, 0.05
example-2.uniontownlabs.org:  14:22:41 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
example-4.uniontownlabs.org:  14:22:42 up 16:46,  0 users,  load average: 0.10, 0.05, 0.05

You can get an interactive shell for all servers in the group by using the -c and -i commands together. Use CTRL-D to exit the session:

$ dsh -g -c -i

Caching YouTube Videos

Thu, 07 Jan 2016 04:00:00 -0500

Here are some quick instructions if you would like to grab high resolution copies of all of the videos in a YouTube playlist.

Install dependencies (replace brew with apt-get or yum for linux):

$ brew install youtube-dl ffmpeg

You can get a muxed 1080P version of all videos in a playlist by running the following command and replacing the URL:

$ youtube-dl -w --restrict-filenames -f 137+140 -o '~/%(playlist_title)s/%(upload_date)s.%(ext)s' https://www.youtube.com/playlist?list=PLjF7R1fz_OOU08_hRcayfVZSmTpBCGJbL

The -w option will disable overwrites, and --restrict-filenames will make the saved filenames URL safe. The -f 137+140 option tells youtube-dl to grab the 1080P video and AAC audio files, and mux them using ffmpeg.

Raspberry Pi dwc2 Gadget Notes

Thu, 17 Dec 2015 11:48:00 -0500

Here are some notes from experiments with dwc2 otg enabled on a Raspberry Pi Zero.

Dependencies

$ sudo apt-get install -y device-tree-compiler bc ncurses-dev

Files

$  git clone --depth 1 --branch 4.1.y-dwc2 https://github.com/adafruit/adafruit-raspberrypi-linux.git

gadget.dts - device tree overlay to enable dwc2

/dts-v1/;
/plugin/;

/{
    compatible = "brcm,bcm2835";

    fragment@0 {
        target = <&usb>;
        #address-cells = <1>;
        #size-cells = <0>;
        __overlay__ {
            compatible = "brcm,bcm2835-usb";
            reg = <0x7e980000 0x10000>;
            interrupts = <1 9>;
            dr_mode = "otg";
            clocks = <&clocks>;
            clock-names = "otg";
        };
    };
};

.config - kernel config

Building

compile gadget.dts:

sudo dtc -@ -I dts -O dtb -o /boot/overlays/usb-otg.dtb gadget.dts

enable the usb-otg overlay by adding the following to your /boot/config.txt:

dtoverlay=usb-otg

Message Queue Link Dump

Thu, 17 Dec 2015 04:48:00 -0500

We use Redis fairly heavily for Adafruit IO, and I started wondering if there might be some better/faster options for our MQTT Redis message queue. It looks like there are many options that are much faster than Redis.

Here are some of the options:

ZeroMQ - No server needed, but setup might become more complex. Mosca support exists.
nanomsg - Same as zeromq, but a younger project. Performs better in benchmarks. POSIX compliant & written in C.
NATS - Requires server, but still much faster than Redis. Can be scaled horizontally.

I think ZeroMQ might be our best bet. Here’s a post with some benchmarks.

Redis Killed by OS?

Fri, 11 Dec 2015 19:00:00 -0500

Redis Server has been killed a few times this week by the OOM. More info in this post on stackoverflow.

After setting the maxmemory limit to 3GB, and maxmemory-policy to allkeys-lru, things should have gotten better. For some reason node.js workers would die as soon as they were restarted, reporting that they couldn’t connect to Redis.

Starting the process manually would connect successfully, so the problem seemed to be with pm2, which manages the node cluster.

Fix

I tried killing pm2 by running the following command.

$ pm2 kill

Then, I restarted the node cluster, and everything was back to normal.

$ sudo service io-server-cluster restart

Copy Raspbian Image to SD Card with Progress Bar on OS X

Fri, 11 Dec 2015 10:00:00 -0500

Find the SD card using diskutil list:

todd:~ todd$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:          Apple_CoreStorage                         250.1 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.1 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *249.8 GB   disk1
                                 Logical Volume on disk0s2
                                 5931BF6D-C779-4723-94C6-5831AF22BC00
                                 Unlocked Encrypted
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *17.5 MB    disk2
   1:        Apple_partition_map                         32.3 KB    disk2s1
   2:                  Apple_HFS Flash Player            17.4 MB    disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *15.9 GB    disk3
   1:             Windows_FAT_32 boot                    62.9 MB    disk3s1
   2:                      Linux                         15.9 GB    disk3s2

Unmount the SD card:

todd:~ todd$ diskutil unmountDisk /dev/disk3
Unmount of all volumes on disk3 was successful

Copy the image to the SD card using dd and use pv to display progress:

todd:~ todd$ pv ~/Downloads/2015-11-21-raspbian-jessie.img | sudo dd bs=16m of=/dev/disk3
837MiB 0:05:19 [3.21MiB/s] [=======>                              ] 22% ETA 0:18:29

Notes on Raspberry Pi USB Gadget Mode

Fri, 11 Dec 2015 03:00:00 -0500

via raspberrypi/linux#884:

Use the upstream dwc2 driver in gadget mode - this in theory “should work” but hasn’t really been tested in device mode. The hardware is similar to the type underneath the s3c_hsotg driver in upstream and there is an ongoing effort to consolidate both codebases. This would be the ideal solution if you just require device mode.

Other useful threads:

dwc_otg can’t coexist with other USB drivers raspberrypi/linux#881
RPi zero Support for USB Gadget Mode (pi shows up as keyboard, disk, net, camera etc) raspberrypi/linux#1212

Simple Let's Encrypt Instructions for nginx

Sun, 06 Dec 2015 04:00:00 -0500

Here are the steps I used to setup a new Let’s Encrypt certificate for nginx running on a Debian EC2 instance.

Stop nginx:

$ sudo service nginx stop

Clone the Let’s Encrypt client git repo:

$ git clone https://github.com/letsencrypt/letsencrypt && cd letsencrypt

Run the client using the following options:

~/letsencrypt$ ./letsencrypt-auto --standalone-supported-challenges tls-sni-01 certonly

You will then be presented with a GUI that will ask you a couple questions about the domain you wish to register.

After completing the registration, the client will output the path to your new cert. You will need to copy the full path for the nginx config.

Add the fullchain.pem and privkey.pem to your nginx site config using the ssl_certificate and ssl_certificate_key options. Here’s what my config looks like for uniontownlabs.org:

server {
  listen 443;
  server_name uniontownlabs.org;

  root /var/www;
  index index.html index.htm index.txt;
  error_page    404 = /404;


  location ~ /\. { deny all; }
  location ~ ~$ { deny all; }

  ssl on;
  ssl_certificate /etc/letsencrypt/live/uniontownlabs.org/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/uniontownlabs.org/privkey.pem;

  ssl_ciphers 'AES256+EECDH:AES256+EDH:!aNULL';

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_session_cache shared:SSL:10m;

  ssl_stapling on;
  ssl_stapling_verify on;
  resolver 8.8.4.4 8.8.8.8 valid=300s;
  resolver_timeout 10s;

  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/ssl/certs/dhparam.pem;

  add_header Strict-Transport-Security max-age=63072000;
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;

  location ~* \.(?:html)$ {
    expires 0;
  }

  location / {
    expires max;
    log_not_found off;
    add_header Cache-Control public;
    try_files $uri $uri/ =404;
  }
}

Finally, start nginx:

$ sudo service nginx start

Mosca MQTT Persistence Performance

Wed, 02 Dec 2015 05:43:00 -0500

While quietly rolling out the public beta of Adafruit IO, we noticed some CPU spikes in our Node.js processes that handle MQTT. We currently use a slightly modified version of Mosca by @mcollina as our MQTT broker. It is a very solid package, but we seemed to be pushing the limits of Mosca’s Redis persistence module.

Debugging

After examining the output of node --perf in chrome://tracing, I seemed to have found the culprit. It looked as if Mosca was waiting on Redis to return the result of HKEYS for retained topics.

v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  InnerArraySort native array.js:670:24 {1}
v8  sort native array.js:885:19 {2}
v8  /home/deploy/io/releases/20151202164744/node/node_modules/mosca/lib/persistence/redis.js:220:44
v8  RedisClient.return_reply /home/deploy/io/releases/20151202164744/node/node_modules/redis/index.js:608:47 {2}
v8  /home/deploy/io/releases/20151202164744/node/node_modules/redis/index.js:306:34
v8  doNTCallback0 node.js:416:27
v8  _tickCallback node.js:335:27 {1}

We retain all current values for topics, so they are immediately returned when a user connects to Adafruit IO and subscribes to a topic. This enables the user to return to a previous state on startup without making any calls to the REST API.

Improving Performance with Redis using Lua

In order to improve performance, I decided to do some preprocessing in Lua on the Redis server before returning hash keys to Node.js. MQTT has two standard wildcard characters: + is a wildcard for matching one topic level, and # can be used for matching multiple. Given Lua’s limited string pattern matching abilities, I decided to narrow down the results using Lua and do the final matching in Node.js.

Here’s the Lua script:

local cursor = 0
local response = {}
local match
local hscan
repeat
  hscan = redis.call('HSCAN', KEYS[1], cursor, 'MATCH', ARGV[1])
  cursor = tonumber(hscan[1])
  match = hscan[2]
  if match then
    for idx = 1, #match, 2 do
      response[match[idx]] = match[idx + 1]
    end
  end
until cursor == 0
return cjson.encode(response)

You can call the script from Node.js using Redis EVAL like this:

'use strict';

// this example is es6, so it will only work in node >=4.0.0.
// it assumes you have a redis hash created with at least topic/1/1
// and topic/2/1 as keys.
const redis = require('redis'),
      async = require('async');

const client = redis.createClient(6379, '127.0.0.1');

// syntax highlighting is currently broken for es6 template strings :\
const script = "\
local cursor = 0 \
local response = {} \
local match \
local hscan \
repeat \
  hscan = redis.call('HSCAN', KEYS[1], cursor, 'MATCH', ARGV[1]) \
  cursor = tonumber(hscan[1]) \
  match = hscan[2] \
  if match then \
    for idx = 1, #match, 2 do \
      response[match[idx]] = match[idx + 1] \
    end \
  end \
until cursor == 0 \
return cjson.encode(response)";

client.eval(script, 1, 'test', 'topic/*/999997', function(err, topics){
  if (err) throw err;
  console.log('found', topics);
  client.end();
});

The HSCAN wildcard character is *, so I just replaced the MQTT wildcard characters with *, and did the final pattern matching in Node.js.

Benchmarks

I was benchmarking the change with 1 million retained hash keys, and it looks like HKEYS method is about 258% slower than preprocessing with Lua. The results are listed below so you can compare the results of the current HKEYS method vs the lua script:

created 1000000 hash keys

real  0m23.453s
user  0m11.763s
sys  0m11.694s

--------------------------------------  HKEYS  --------------------------------------
retrieved 1000000 hash keys via HKEYS
JS matched topic: 'topic/0.09308264800347388/999997'
got value via HGET: 0.3918832363560796
result {"topic\/0.09308264800347388\/999997":"0.3918832363560796"}

real  0m3.460s
user  0m2.729s
sys  0m0.194s

----------------------------------------  LUA  ---------------------------------------
retrieved 1 hash keys via Lua
JS matched topic: 'topic/0.09308264800347388/999997'
result {"topic\/0.09308264800347388\/999997":"0.3918832363560796"}

real  0m1.337s
user  0m0.116s
sys  0m0.024s

The pull request to mosca can be found here: mcollina/mosca#379