Scott Hanselman

Bridging Dexcom Share CGM Receivers and Nightscout

March 14, '15 Comments [17] Posted in Diabetes | Hardware | Open Source
Sponsored By

Dexcom Share

I've long lamented the sad state of Diabetes technology. For the last 20 years I've been told that it'll be cured in the next few years. (Spoiler: That hasn't happened.)

Fortunately some technological breakthroughs have happened, like the CGM (Continuous Glucose Meter). This device has a transmitted embedded in my belly that transmits to a small receiver. However, my wife couldn't see my blood sugar remotely, so the Nightscout open source project pretends to be computer connected to the receiver, then uploads it to your own website. Then you can see your blood sugar on your watch, or family and friends can log in also. This project has been moving along nicely for a year or so now.

Just last month Dexcom, the CGM maker, released an update to their receiver that includes Bluetooth, called the Dexcom Share. Now my transmitter goes to my Dexcom device which then bounces via Bluetooth LE to my phone, which is then uploaded to the Dexcom site. The Dexcom iPhone app will support the Apple Watch in the future as well, they say.

However, I'd like more control over my data. Dexcom's solution (as of the time of this writing) is iPhone/iPad only. Not everyone can afford an iWatch and iDevices. I'd like to use my Pebble Watch, for example, which is supported in Nightscout today.

I got the Dexcom Share at 3:30pm today in the mail. By 4:40pm it was paired to my iPhone and working nicely. So what I really need is a simple bridge that takes my Dexcom Share data and copies it to Nightscout. From there I can analyze it, send it to my Pebble, or do whatever.

Watching iPhone Traffic from a Windows Machine

First, I need to understand the Dexcom Api. Let's watch the iPhone talk to Dexcom. I'll install Fiddler on my Windows machine and configure Fiddler as a proxy server. I'll need to trust the Fiddler SSL cert (only for dev purposes) on both the iPhone and the Windows machine. My machine is called Hexpower7 and the proxy is on port 8888. I'll visit http://hexpower7:8888 on my iPhone and install the cert there also, which will allow me to watch the traffic and learn about the API.

I learned a few things by watching the traffic.

Watching Traffic

Calling Dexcom with CURL

First, when you login to the Dexcom API you get a Session ID, which is common and to be expected. With that Session ID you can get your sugar values. After the login I retrieved my latest sugar number:

curl -k -X POST "https://share1.dexcom.com/ShareWebServices/Services/Publisher/ReadPublisherLatestGlucoseValues?sessionID=GUID&minutes=1440&maxCount=1" -H "Accept: application/json" -H "Content-Length: 0"

resulting in:

[{"DT":"\/Date(1426290216000-0700)\/","ST":"\/Date(1426293817000)\/","Trend":4,"Value":113,"WT":"\/Date(1426290240000)\/"}]

Here's a screenshot:

Talking to Dexcom Share from CURL

Cool. So I pair-programmed with Benjamin West from the Nightscout project and we spent an hour writing a script to get my Dexcom Share data and bridge/POST it to Nightscout.

I put the script in an Azure WebJob and it's pulling my Share data and putting it into Nightscout every few minutes. I won't post the code here, rather the Nightscout team will take our prototype from here, but the result is lovely.

I don't have to carry an extra Android device anymore, I just use my Dexcom Share and its supported iPhone uploader application. Very cool.

Now it's 7:30pm, just a few hours after I got my Dexcom Share and I've got the best of both worlds. The API was easy to use and we didn't spend more than two hours on it. Most of the time was waiting for the transmitter to complete its warmup cycle.

My Blood Sugar in the Cloud

I'll do a formal Dexcom review soon, but I can already tell you it's a winner. Everyone who can get a CGM should get a Dexcom Share. It's a thrilling device. I would like the iPhone app to support iPhone 6 and 6+ screen-sizes better, and a nicer UI, but all in all, it's a great device.

Don't forget, visit http://marchisformakers.com, tell your friends and tweet us at #MarchIsForMakers!


Sponsor: Big thanks to Aspose for sponsoring the blog feed this week! Are you working with Files?Aspose.Total for .NET has all the APIs you need to create, manipulate and convert Microsoft Office documents and many other formats in your applications. Start a free trial today.

RELATED READING

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Arduino 101 with an Intel Edison - Hooking up JSON to an LCD Screen

March 10, '15 Comments [20] Posted in Hardware | Open Source
Sponsored By
Playing with Arduino and Nightscout

Hanselminutes and CodeNewbie are teaming up to produce two new podcast episodes a week for the month of March, featuring incredible makers in the hardware space. There will be new content every weekday in March, so bookmark http://www.marchisformakers.com and join us!

  • CodeNewbie is the most supportive community of programmers and people learning to code. Tune in every week for stories about people on their coding journey.
  • Hanselminutes: Fresh Air for Developers is a weekly talk show that brings interesting people together to talk about the web, culture, education, technology and more.

Our hashtag for this event is #MarchIsForMakers and YOU, Dear Reader, can help us out by being our Street Team.

Share our website http://marchisformakers.com!

...with friends, kids, teachers, family, anyone you think my be benefit from learning about hardware and today's maker culture.

This is Week 2! At the end of March we'll have as many as 10 great podcasts, amazing guests, Live Google Hangouts, blog posts, Twitter Chats on Wednesdays, and a huge collection of links and projects for you to explore.


Please note that I'm learning. I'm sure some of you are fantastic experts, while others are working on Hello World. If you find errors, naïve or otherwise, in my code or solution, DO let me know in the comments and I'll update this post with notes and asides so we can all learn!

I wanted to learn a little about Arduino this week. It's an huge and enthusiastic community based around an open-source electronics platform. The hardware is small and relatively inexpensive, and it brings hardware hacking to folks (like myself) that may not feel up to doing really low level electronics work. There or stackable "shields" you can plug on top and easily add new features, screens, sensors, and more.

Arduino_Uno_-_R3

There's lots of different choices for Arduino development, including some more interesting versions like the Intel Edison with Arduino Breakout Board. The Intel Edison supports not just Arduino, but also can run a full version of Yocto Linux, and can run Python and node.js. I have an older Arduino Atmega328 which was like $12.99, but I wanted a more flexible option that included on board Wi-Fi. Getting Wi-Fi connectivity is kind of a hassle if it's not built in.

The Arduino Yún is a great choice, but I figured I'd spend more for the Edison and get a lot more options. I also got the "Seeed Studio Grove Starter Kit." This is a cool Arduino Shield that lets me (and the kids) attached sensors, buttons, screens, and lots of other stuff without soldering!

intel-lot-edition-gen2

I download the Arduino software, but also found that Intel now has a complete Integrated IoT Windows 64 Installer that will get everything you need to get started with the Intel Edison. It makes it REALLY easy to start.

I tried a few small "Sketches" out, turning on a light with a button press, and such.

image

But I wanted to make something more interesting to me personally. I'm a Type 1 Diabetic, and I wear an insulin pump and Dexcom Continuous Glucose Meter. They are connected to the cloud via a project called Nightscout.  Nightscout takes my sugar values and pushes them into Azure, where they are available via a JSON web service.

NOTE: I've put my Arduino Sketch on GitHub here. https://github.com/shanselman/NightscoutArduinoPlayground

Nightscout has a /pebble endpoint that is used to feed the Pebble Watch Face and show folks their blood sugar on their wrist. I thought it would be cool to hook up an Arduino to get my blood sugar from the cloud and display it on an LCD. It'll show the current value, change the background to green/yellow/red to display risk, and then a custom character to show trends (flat, up, down, etc).

The JSON the Nightscout service returns looks like this, but I just want the "sgv" value and the "direction."

{
status: [{
now: 1426017007130
}],
bgs: [{
sgv: "102",
bgdelta: 2,
trend: 4,
direction: "Flat",
datetime: 1426016912000,
filtered: 115232,
unfiltered: 118368,
noise: 1,
rssi: 191,
battery: "59"
}]
}

I needed to bring in three libraries to achieve my goal. Remember I want to:

  • Connect to my Wi-Fi network
  • Download some JSON and parse it
  • Display values on an LCD screen

So I needed:

As with all problems when you're learning new things, you'll want to break them down one at a time.

I wanted to just display a string on the LCD, that was easy, there's lots of examples online in the Arduino community.

lcd.clear();
lcd.setCursor(0, 0);
lcd.print("Hello World");

Connecting to Wi-Fi was pretty easy also:

while (status != WL_CONNECTED) {
// Connect to WPA/WPA2 network. Change this line if using open or WEP network:
status = WiFi.begin(ssid, pass);

// wait 3 seconds for connection:
lcd.setCursor(0, 0);
lcd.print("Waiting...");
delay(3000);
}

When doing an HTTP call (given the libraries I was learning about) it's pretty low level. Constrained memory Arduinos (not like the Edison) don't seem to have SSL support, nor do they have high-level HTTP libraries. I had to craft the HTTP headers manually:

lcd.print("Connecting...");
if (client.connect(server, 80)) {
lcd.clear();
lcd.print("Server connected");
// Make a HTTP request:
client.println("GET /pebble HTTP/1.1");
client.println("Host: hanselsugars");
client.println("Connection: close");
client.println();
}

Then I spin through the HTTP Response looking for the start of the JSON, and store it away. At this point I'm wondering if I'm doing it wrong. It isn't very robust.

boolean jsonFound = false;
int bytes = 0;
while (client.available()) {
char c = client.read();
//look for first {, yes it could be in a cookie but I'm thinking positively.
if (c == '{') jsonFound = true;
if (!jsonFound) continue;

stringBuffer[bytes++] = c;
if (bytes >= MAXBUFFER) break; //that's all we have room for or we're done
}

Then once I've got the stringBuffer, I use the aJSON library to get the values I need. This isn't pretty, but it's how you use this library.

I also added a little bit to turn the color of the screen red/green/yellow.

if (root != NULL) {
aJsonObject* bgs = aJson.getObjectItem(root, "bgs");
if (bgs != NULL) {
aJsonObject* def = aJson.getArrayItem(bgs, 0);
if (def != NULL) {
aJsonObject* sgv = aJson.getObjectItem(def, "sgv");
String bg = sgv->valuestring;
int bgInt = bg.toInt();
if (bgInt > 180) lcd.setRGB(255, 0, 0);
if (bgInt > 150 && bgInt <= 180) lcd.setRGB(255, 255, 0);
if (bgInt <= 150) lcd.setRGB(0, 255, 0);
lcd.clear();
lcd.setCursor(0, 0);
lcd.print("Glucose (mg/dl)");
lcd.setCursor(0, 1);
lcd.print(bg);
}

I also created some custom characters to show trends, based on a string (pretending to be an enum) that comes back from the JSON service. For example, an up arrow:

byte SingleUp[8] = {
B00000,
B00100,
B01110,
B10101,
B00100,
B00100,
B00100,
B00000
};

All the questionable code is up at https://github.com/shanselman/NightscoutArduinoPlayground

At this point I'm confused/learning about a few things:

  • It runs for a while and then stops.
    • I don't know what state it's in.
    • I would like it to call out to the web service every 5 min and update the LCD. I suspect I am not managing Wi-Fi connection state correctly (as in, not at all).
    • So the question is, how does one take a working prototype and turn it into a real appliance that can run forever?
  • I'm reusing the stringBuffer which is a bad idea. I need to clear it out.
  • Sometimes the Custom Characters are corrupted on the screen.
    • No idea why. It works 8 out of 10 times. Perhaps it's an Edison bug or a SeeedStudio bug.

When I get it working reliably, I'd like to 3D Print a case and mount it somewhere. :)

Arduino for Visual Studio

One other thing I found out, there's a fantastic add-in for Visual Studio that will give you a great Arduino Development Environment inside of Visual Studio. It includes simple debugging, breakpoints, a nice serial monitor and more. I'm still finding debugging to be challenging as local watches and step over isn't supported, but it's vastly superior to the tiny Arduino IDE.

Visual Micro - Arduino for Visual Studio

Arduino Emulator

Don't have an Arduino or a breadboard? Check out http://123d.circuits.io online. It's amazing. It's an Arduino Circuit Simulator online. Check this out!

image

You can setup boards, write code, and practice in software before you test it out on hardware.

image

Don't forget, visit http://marchisformakers.com, tell your friends and tweet us at #MarchIsForMakers!


Sponsor: Big thanks to Aspose for sponsoring the blog feed this week! Are you working with Files? Aspose.Total for .NET has all the APIs you need to create, manipulate and convert Microsoft Office documents and many other formats in your applications. Start a free trial today.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

How to run ASP.NET 5 Beta 3 or GoLang on a Raspberry Pi 2

March 3, '15 Comments [48] Posted in Hardware | Open Source
Sponsored By
Hanselman's Desk of Hardware

Hanselminutes and CodeNewbie are teaming up to produce two new podcast episodes a week for the month of March, featuring incredible makers in the hardware space. There will be new content every weekday in March, so bookmark http://www.marchisformakers.com and join us!

  • CodeNewbie is the most supportive community of programmers and people learning to code. Tune in every week for stories about people on their coding journey.
  • Hanselminutes: Fresh Air for Developers is a weekly talk show that brings interesting people together to talk about the web, culture, education, technology and more.

Our hashtag for this event is #MarchIsForMakers and YOU, Dear Reader, can help us out by being our Street Team.

Share our website http://marchisformakers.com!

...with friends, kids, teachers, family, anyone you think my be benefit from learning about hardware and today's maker culture.

This is just Day 1! At the end of March we'll have as many as 10 great podcasts, amazing guests, Live Google Hangouts, blog posts, Twitter Chats on Wednesdays, and a huge collection of links and projects for you to explore.


How to run ASP.NET Beta 3 running on a Raspberry Pi 2

I love Raspberry Pi, the tiny $35 computer. I've gone through 6 at least count. Have they died? Not at all! They've been gifted forward. Right now I've got one running my 3D Printer, one running as a media center, and a Raspberry Pi 2 that my kids are using as their primary computer. There's so many Raspberry Pi projects - How can you not love a tiny computer?

This actually a dual tutorial/how-to. I've been so impressed with the Raspberry Pi 2 I've wanted to see how far one can take it. It's still a modest little machine, but it's definitely twice as fast or more in single-tasking and perhaps 6x faster in multitasking in my experience than the previous Raspberry Pi.

Basic Raspberry Pi set up

I use the Raspbian Operating System image for my Raspberry Pi 2. It's a Debian Wheezy image for techies, that's a Unix for non-techies.

I use this hardware (these are Amazon links) that I put together myself, although you can get a kit that includes memory, power, wifi, cables, case, etc.

You can get the disk image and follow the setup instructions here. I also added TightVNC so I could remote into my Raspberry Pi from my desktop. This also allowed me to run it "headless" without a monitor, but it's up to you.

You can see me VNC'ed into my Raspberry Pi 2 here. Of course, you can always connect it to your monitor or TV.

VNC'ed into a Raspberry Pi

I wanted to see how hard it would be to run .NET on this Raspberry Pi. Depending on how deep you want to go, it's not hard at all.

Running ASP.NET on a Raspberry Pi 2

NOTE/DISCLAIMER: This is a point in time. It's a beta/daily build of an early thing. I'm sure this will get down to a few simple lines in the future, so don't panic thinking that ASP.NET on Linux will suck. It's early.

Frist, you can get an old (3 years old) version of the open source Mono runtime with the stable standard repositories

sudo apt-get mono-complete

And this will get you version 3.2.8. You can do basic stuff. Make a HelloWorld.cs and run it with

gmsc HelloWorld.cs
mono HelloWorld.exe

Debian likes to be very stable, I'm told, so if you want to get a very NEW version of Mono like version 3.10 (that's "Three Point Ten") and as such, run things like ASP.NET 5, you'll need to do a little more work.

UPDATE and IMPORTANT NOTE: There's two options here. Build Mono from source, or use a custom repository from the Mono folks.

Option 1: Install Mono from The Mono Project's repositories

The Mono Project has their own repository for Debian distributions like Raspbian Wheezy. If you've put on an early mono, you'll want to sudo apt-get remove mono-complete first to tidy up.

Per their instructions from the Mono site, you'll then

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF 
echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list
sudo apt-get update && apt-get upgrade
sudo apt-get install mono-complete

And you should be all set!

Option 2: Build Mono from Source (slow and advanced and of questionable value)

You could git clone the Mono repository and build it. I found this process slow, so I downloaded the source from an archive instead, then unzipped it into a folder.

mkdir ~/monosource
cd ~/monosource
wget http://download.mono-project.com/sources/mono/mono-3.10.0.tar.bz2
tar -xjvf mono-3.10.0.tar.bz2

Then here's a bunch of prerequisites I needed, plus htop because I like it.

sudo apt-get install build-essential autoconf automake binutils libtool libglib2.0-dev libxrender-dev libfontconfig1-dev libpng12-dev libgif-dev libjpeg8-dev libtiff5-dev libexif-dev gettext libcairo2-dev htop 

Then start the build. This too an hour or so, and is dependent on the speed of your Pi AND the speed of your SD card. It should be a Class 10 if possible.

./configure --prefix=/usr/local; sudo SKIP_AOT=true make; sudo SKIP_AOT=true make install;

Note this is three lines in one. Do a sudo reboot when you're done. If you can run mono -V and see version 3.10.0 then you're in a good place.

Mono 3.10.0 on a Raspberry PI

Adding ASP.NET 5

You can add ASP.NET 5 at this point by downloading the samples directly from Github and running the "kvminstall.sh" to setup the ASP.NET runtime manager.

mkdir ~/sources/aspnet5 
cd ~/sources/aspnet5
git clone git://github.com/aspnet/home.git
sh ~/sources/aspnet5/kvminstall.sh
source ~/.k/kvm/kvm.sh
kvm upgrade

Then per this GitHub issue you need to tell your system about the SSL certs for NuGet to restore correctly.

CERTMGR=/usr/local/bin/certmgr
sudo $CERTMGR -ssl -m https://go.microsoft.com
sudo $CERTMGR -ssl -m https://nugetgallery.blob.core.windows.net
sudo $CERTMGR -ssl -m https://nuget.org

mozroots --import --machine --sync

Then go to one of the samples like ~/sources/aspnet5/home/samples/HelloMvc and run "kpm restore." Note this uses about 400 megs of RAM for a minutes so you'll want a newer Raspberry Pi.

Running kpm restore on Raspberry Pi

NOTE: Make sure the sample version in project.json match your local runtime version. I needed to update version strings to beta3 to match what "kvm list" said. I'm sure this will get fixed soon.

The "Kestrel" web server uses libuv, an HTTP library. Here is how to build libuv. I found this on Punit Ganshani's blog, which I'm actually wishing I'd found earlier in this blog post. ;)

sudo apt-get install gyp
wget http://dist.libuv.org/dist/v1.0.0-rc1/libuv-v1.0.0-rc1.tar.gz
tar -xvf libuv-v1.0.0-rc1.tar.gz
cd libuv-v1.0.0-rc1/
./gyp_uv.py -f make -Duv_library=shared_library
make -C out
sudo cp out/Debug/lib.target/libuv.so /usr/lib/libuv.so.1.0.0-rc1
sudo ln -s libuv.so.1.0.0-rc1 /usr/lib/libuv.so.1 Then run

Then run "k kestrel" and hit the port mentioned in the project.json.

ASP.NET 5 Beta 3 on a Raspberry Pi 2

How to run Go on a Raspberry Pi 2

I wanted to Go running as well. Go has fewer dependencies but no official ARM builds. The Raspberry Pi 2 is an ARMv5. However, a very kind gentleman named Dave Cheney has been building and hosting his own unofficial ARM tarballs for Go. You take the instructions from the GoLang site and his links and you're all set on your Raspberry Pi or Pi 2.

Here's what I did for a Raspberry Pi 2.

wget http://dave.cheney.net/paste/go1.4.2.linux-arm~multiarch-armv7-1.tar.gz
sudo tar -C /usr/local -xzf go1.4.2.linux-arm~multiarch-armv7-1.tar.gz

Then, add it to your path, or .profile, or whatever.

export PATH=$PATH:/usr/local/go/bin

Make a HelloGo.go, compile and run.

Go on a Raspberry Pi

Personally, I'd love to see "dotnet" be as easy to get running on Linux as Go.

package main

import "fmt"

func main() {
fmt.Println("Hello Go on Raspberry Pi 2")
}

The Raspberry Pi is just a little joy. It's a lot of fun and has a lot of potential. Definitely pick up some for the kids (yourself.)

Don't forget, visit http://marchisformakers.com, tell your friends and tweet us at #MarchIsForMakers!

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Running the Ruby Middleman Static Site Generator on Microsoft Azure

February 24, '15 Comments [17] Posted in Azure | Open Source | Ruby
Sponsored By

Middleman is "a static site generator using all the shortcuts and tools in modern web development." With any static site generator you can run it all locally and then push/FTP/whatever the resulting HTML to any host. However, static site generators are even more fun when you can host the source code in Git and have your static site build and deploy in the cloud.

Middleman uses Ruby for its build system and views, and some of the Gems it uses are native gems. That means if you are a Windows user, your system will need not just Ruby, but the Ruby DevKit so you can build those native gems. The DevKit is a lovely set of tools that "makes it easy to build and use native C/C++ extensions such as RDiscount and RedCloth for Ruby on Windows."

Azure Websites supports not just ASP.NET today, but also node.js, PHP, Python, and Java, all built in. But not Ruby, directly, yet. Also, Azure Websites doesn't know formally about the idea of a static site generator. You might be thinking, oh, this'll be hard, I'll need to use a VM and do this myself.

However, even though Azure Websites are totally "platform as a service" there's still a Windows Virtual Machine underneath, and you can use the disk space however you like. You've got a LOT of control and can even get a hold of a console where you can run commands and install stuff. The Azure Portal lets you open a command line from your website.

The New Azure Portal

Check me out, here in the new Azure Portal. This is where I did my practice work to see if I could programmatically download and install Ruby via a script. I tried a number of different commands, all from the browser, and explored a number of ideas. When I got it working, I put together a batch file called GetRuby. I could have also used a shell script or PowerShell, but Batch was easy given what I was doing.

ASIDE: You may recognize that console from this video I did about the "Super Secret Debug Console" in Azure. It's not so secret now, it's a feature.  There is still a cool debug "sidecar" website for every Azure site, it's at http://YOURSITENAME.scm.azurewebsites.net/DebugConsole but now a version of the console is in the portal as well.

Azure Websites uses an open source project called Kudu to deploy from locations with source like Github. Kudu supports custom deployment scripts where you can jump in and do whatever you like (within the limits of the security sandbox)

Basically I needed to do these things before running Middleman on my source:

  • Ensure Ruby is installed and in the path.
  • Ensure the DevKit (which includes native compilers, etc) is installed
  • Initialize and setup DevKit for builds
  • Update RubyGems to 2.2.3 until the Windows version of Ruby has this included
  • Install eventmachine 1.0.7, a problematic gem on Windows
  • Run the Ruby Bundler's update
  • Install Middleman

And then, every deployment run the Middleman static site generator.

  • Middleman build

The first part is a one time thing for a new website. I just need to make sure Ruby is around and in the path. The second part is what runs every time a source file for my static site generator is checked in. It runs middleman build. Then at the very end, Kudu takes the results from the /build folder and moves them to /wwwroot, which makes the changes live.

Here's an annotated part of the first bit, but the actual file is on GitHub. Note that I'm putting stuff in %temp% for speed. Turns out %temp% a local drive, so it's a few times faster than using the main drive, which makes this deployment faster. However, it's cleared out often, so if I wanted things to be persistent but slower to deploy, I'd put them in D:\deployments\tools. As it is, the deploy is fast (less than a minute) when Ruby is there, and just about 3 minutes to get and setup Ruby when it's not. The exists check handles the case when a deploy happens but %temp% has been cleared so it'll get Ruby again.

NOTE: If this seems confusing or complex, it's because I like to give folks LOTS of detail. But just look at my repository. All we have is a standard "Middleman init" site plus the Azure-generator deploy.cmd and my getruby.cmd. That's all you need, plus a Basic Azure Website. The getruby.cmd is my automating what you'd have to any way on a Windows machine without Ruby.

REM Note that D:\local\temp is a LOCAL drive on Azure, and very fast
SET PATH=%PATH%;D:\local\temp\r\ruby-2.1.5-x64-mingw32\bin

pushd %temp%
REM If you need things to be persistent, then put them elsewhere, not in TEMP
if not exist r md r
cd r
if exist ruby-2.1.5-x64-mingw32 goto end

echo No Ruby, need to get it!

REM Get 64-bit Ruby
curl -o ruby215.zip http://dl.bintray.com/oneclick/rubyinstaller/ruby-2.1.5-x64-mingw32.7z?direct
ECHO START Unzipping Ruby. 7Zip is already on Azure Websites
REM Note Azure deployments run faster with 7Zip not spewing so much. Redirect to a file.
d:\7zip\7za x -y ruby215.zip > out

REM Get DevKit to build Ruby native gems
REM If you don't need DevKit for your Gems, rem this out.
curl -o DevKit.zip http://cdn.rubyinstaller.org/archives/devkits/DevKit-mingw64-64-4.7.2-20130224-1432-sfx.exe
ECHO START Unzipping DevKit
d:\7zip\7za x -y -oDevKit DevKit.zip > out
ECHO DONE Unzipping DevKit

ruby DevKit\dk.rb init

REM Tell DevKit where Ruby is
echo --- > config.yml
echo - d:/local/temp/r/ruby-2.1.5-x64-mingw32 >> config.yml

REM Setup DevKit
ruby DevKit\dk.rb install

REM Update Gem223 until someone fixes the Ruby Windows installer https://github.com/oneclick/rubyinstaller/issues/261
curl -L -o update.gem https://github.com/rubygems/rubygems/releases/download/v2.2.3/rubygems-update-2.2.3.gem
call gem install --local update.gem
call update_rubygems --no-ri --no-rdoc > updaterubygemsout
ECHO What's our new Rubygems version?
call gem --version
call gem uninstall rubygems-update -x

REM This is needed on Windows, why is this gem such a problem?
ECHO Install eventmachine 1.0.7
call gem install eventmachine -v '1.0.7' --no-ri --no-rdoc > updateventmachineout

call bundle update

ECHO Install middleman...the whole point!
call gem install middleman --no-ri --no-rdoc

:end
popd

call middleman build

REM KuduSync and actual /build to /wwwroot is after this in deploy.cmd!

And in the Deploy.cmd all I needed to change was this under SETUP. This is where YOU can do whatever you like. Note since I'm using Batch, I need to put CALL in front of other Batch files (and Ruby uses them also!) otherwise my script will just end early.

ECHO CALLING GET RUBY

call getruby.cmd

ECHO WE MADE IT

Then later, still in Deploy.cmd, I just added \build to the source directory name.

call :ExecuteCmd "%KUDU_SYNC_CMD%" -v 50 -f "%DEPLOYMENT_SOURCE%\build" -t "%DEPLOYMENT_TARGET%" -n "%NEXT_MANIFEST_PATH%" -p "%PREVIOUS_MANIFEST_PATH%" -i ".git;.hg;.deployment;deploy.cmd"

And that's it.  Now whenever I updated my views or other things in my Middleman source on GitHub, it automatically deploys to my live site.

Yes, again, to be clear, I realize it's a static site generator that I could run locally and FTP the results, but I'm working in a small team and this is a great way for us to collaborate on our static site. Plus, when it's done, it's all done and I don't have to mess with it again.

Middleman Static Site Generator on Azure

Debugging Custom Azure Website Deployments

I thought debugging my GetRuby batch file was going to be a nightmare. However, it turns out that the Azure cross-platform command line (the Azure x-plat CLI, open source, and written in nodejs, BTW) can connect to Azure's log streaming service. "Azure Site Log Tail" lets me see the LIVE console output as the deploy happens!

Azure Site Log Tail

Now, note that the need for this whole "getruby.bat" file totally goes away if the Azure Websites folks start including Ruby and DevKit in the Azure Websites VM image by default. That would make Ruby, Rails, Gems, DevKit, etc. available to everyone. Do you want Ruby on Azure? Do you care? Sound off in the comments!

HELP: The batch file could use more testing, especially for robustness as well as Ruby-correctness as I'm likely confused about a few things, but it works for me and it's a great start. Sometimes different native gems don't build, though, or Gems complains about conflicting versions and asks me to run Bundler. I have no idea why. Running things twice clears it. It's either my bug or someone else's. :)

I'm just happy that Azure Websites is as flexible as it is that I was able to console into it from the browser, look around, add my own custom deployment hook, and do something I initially didn't think was possible!

Give Azure Websites a try FOR FREE, no signup, no credit card for an hour in a sandbox with PHP, Node, ASP.NET, or Java at http://try.azurewebsites.net. (Full Disclosure, I helped a little with this site, so I'm a fan.)

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Proper benchmarking to diagnose and solve a .NET serialization bottleneck

February 20, '15 Comments [25] Posted in ASP.NET | Learning .NET | Open Source
Sponsored By

From http://adrianotto.com/2010/08/dev-null-unlimited-scale/Here's a few comments and disclaimers to start with. First, benchmarks are challenging. They are challenging to measure, but the real issue is that often we forget WHY we are benchmarking something. We'll take a complex multi-machine financial system and suddenly we're hyper-focused on a bunch of serialization code that we're convinced is THE problem. "If I can fix this serialization by writing a 10,000 iteration for loop and getting it down to x milliseconds, it'll be SMOOOOOOTH sailing."

Second, this isn't a benchmarking blog post. Don't point this blog post and say "see! Library X is better than library Y! And .NET is better than Java!" Instead, consider this a cautionary tale, and a series of general guidelines. I'm just using this anecdote to illustrate these points.

  • Are you 100% sure what you're measuring?
  • Have you run a profiler like the Visual Studio profiler, ANTS, or dotTrace?
  • Are you considering warm-up time? Throwing out outliers? Are your results statistically significant?
  • Are the libraries you're using optimized for your use case? Are you sure what your use case is?

A bad benchmark

A reader sent me a email recently with concern of serialization in .NET. They had read some very old blog posts from 2009 about perf that included charts and graphs and did some tests of their own. They were seeing serialization times (of tens of thousands of items) over 700ms and sizes nearly 2 megs. The tests included serialization of their typical data structures in both C# and Java across a number of different serialization libraries and techniques. Techniques included their company's custom serialization, .NET binary DataContract serialization, as well as JSON.NET. One serialization format was small (1.8Ms for a large structure) and one was fast (94ms) but there was no clear winner. This reader was at their wit's end and had decided, more or less, that .NET must not be up for the task.

To me, this benchmark didn't smell right. It wasn't clear what was being measured. It wasn't clear if it was being accurately measured, but more specifically, the overarching conclusion of ".NET is slow" wasn't reasonable given the data.

Hm. So .NET can't serialize a few tens of thousands of data items quickly? I know it can.

Related Links: Create benchmarks and results that have value and Responsible benchmarking by @Kellabyte

I am no expert, but I poked around at this code.

First: Are we measuring correctly?

The tests were using DateTime.UtcNow which isn't advisable.

startTime = DateTime.UtcNow;
resultData = TestSerialization(foo);
endTime = DateTime.UtcNow;

Do not use DateTime.Now or DateTime.Utc for measuring things where any kind of precision matters. DateTime doesn't have enough precision and is said to be accurate only to 30ms.

DateTime represents a date and a time. It's not a high-precision timer or Stopwatch.

As Eric Lippert says:

In short, "what time is it?" and "how long did that take?" are completely different questions; don't use a tool designed to answer one question to answer the other.

And as Raymond Chen says:

"Precision is not the same as accuracy. Accuracy is how close you are to the correct answer; precision is how much resolution you have for that answer."

So, we will use a Stopwatch when you need a stopwatch. In fact, before I switch the sample to Stopwatch I was getting numbers in milliseconds like 90,106,103,165,94, and after Stopwatch the results were 99,94,95,95,94. There's much less jitter.

Stopwatch sw = new Stopwatch();
sw.Start();

// stuff

sw.Stop();

You might also want to pin your process to a single CPU core if you're trying to get an accurate throughput measurement. While it shouldn't matter and Stopwatch is using the Win32 QueryPerformanceCounter internally (the source for the .NET Stopwatch Class is here) there were some issues on old systems when you'd start on one proc and stop on another.

// One Core
var p = Process.GetCurrentProcess();
p.ProcessorAffinity = (IntPtr)1;

If you don't use Stopwatch, look for a  simple and well-thought-of benchmarking library.

Second: Doing the math

In the code sample I was given, about 10 lines of code were the thing being measured, and 735 lines were the "harness" to collect and display the data from the benchmark. Perhaps you've seen things like this before? It's fair to say that the benchmark can get lost in the harness.

Have a listen to my recent podcast with Matt Warren on "Performance as a Feature" and consider Matt's performance blog and be sure to pick up Ben Watson's recent Book called "Writing High Performance .NET Code".

Also note that Matt is currently exploring creating a mini-benchmark harness on GitHub. Matt's system is rather promising and would have a [Benchmark] attribute within a test.

Considering using an existing harness for small benchmarks. One is SimpleSpeedTester from Yan Cui. It makes nice tables and does a lot of the tedious work for you. Here's a screenshot I stole borrowed from Yan's blog.

image11

Something a bit more advanced to explore is HdrHistogram, a library "designed for recoding histograms of value measurements in latency and performance sensitive applications." It's also on GitHub and includes Java, C, and C# implementations.

PercentileHistogramExample

And seriously. Use a profiler.

Third: Have you run a profiler?

Use the Visual Studio Profiler, or get a trial of the Redgate ANTS Performance Profiler or the JetBrains dotTrace profiler.

Where is our application spending its time? Surprise I think we've all seen people write complex benchmarks and poke at a black box rather than simply running a profiler.

Visual Studio Profiler

Aside: Are there newer/better/understood ways to solve this?

This is my opinion, but I think it's a decent one and there's numbers to back it up. Some of the .NET serialization code is pretty old, written in 2003 or 2005 and may not be taking advantage of new techniques or knowledge. Plus, it's rather flexible "make it work for everyone" code, as opposed to very narrowly purposed code.

People have different serialization needs. You can't serialize something as XML and expect it to be small and tight. You likely can't serialize a structure as JSON and expect it to be as fast as a packed binary serializer.

Measure your code, consider your requirements, and step back and consider all options.

Fourth: Newer .NET Serializers to Consider

Now that I have a sense of what's happening and how to measure the timing, it was clear these serializers didn't meet this reader's goals. Some of are old, as I mentioned, so what other newer more sophisticated options exist?

There's two really nice specialized serializers to watch. They are Jil from Kevin Montrose, and protobuf-net from Marc Gravell. Both are extraordinary libraries, and protobuf-net's breadth of target framework scope and build system are a joy to behold. There are also other impressive serializers in including support for not only JSON, but also JSV and CSV in ServiceStack.NET.

Protobuf-net - protocol buffers for .NET

Protocol buffers are a data structure format from Google, and protobuf-net is a high performance .NET implementation of protocol buffers. Think if it like XML but smaller and faster. It also can serialize cross language. From their site:

Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:

  • are simpler
  • are 3 to 10 times smaller
  • are 20 to 100 times faster
  • are less ambiguous
  • generate data access classes that are easier to use programmatically

It was easy to add. There's lots of options and ways to decorate your data structures but in essence:

var r = ProtoBuf.Serializer.Deserialize<List<DataItem>>(memInStream);

The numbers I got with protobuf-net were exceptional and in this case packed the data tightly and quickly, taking just 49ms.

JIL - Json Serializer for .NET using Sigil

Jil s a Json serializer that is less flexible than Json.net and makes those small sacrifices in the name of raw speed. From their site:

Flexibility and "nice to have" features are explicitly discounted in the pursuit of speed.

It's also worth pointing out that some serializers work over the whole string in memory, while others like Json.NET and DataContractSerializer work over a stream. That means you'll want to consider the size of what you're serializing when choosing a library.

Jil is impressive in a number of ways but particularly in that it dynamically emits a custom serializer (much like the XmlSerializers of old)

Jil is trivial to use. It just worked. I plugged it in to this sample and it took my basic serialization times to 84ms.

result = Jil.JSON.Deserialize<Foo>(jsonData);

Conclusion: There's the thing about benchmarks. It depends.

What are you measuring? Why are you measuring it? Does the technique you're using handle your use case? Are you serializing one large object or thousands of small ones?

James Newton-King made this excellent point to me:

"[There's a] meta-problem around benchmarking. Micro-optimization and caring about performance when it doesn't matter is something devs are guilty of. Documentation, developer productivity, and flexibility are more important than a 100th of a millisecond."

In fact, James pointed out this old (but recently fixed) ASP.NET bug on Twitter. It's a performance bug that is significant, but was totally overshadowed by the time spent on the network.

Thanks to Marc Gravell and James Newton-King for their time helping with this post.

What are your benchmarking tips and tricks? Sound off in the comments!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.