Scott Hanselman

RFC: Web Platform Installer (WebPI) with an offline feed mode for bandwidth savings

September 24, 2011 Comment on this post [36] Posted in Tools
Sponsored By

My wife is in the Kingdom of Swaziland this evening. If she was a developer over there and wanted to use Web Platform Installer to get Visual Studio or some large download she would have to patiently bring the bits down over 3G. If she was very well off, she might have ASDL with a maximum speed of 512kbps down, most likely with a cap. I've talked about limited connectivity in African countries and other nations (*cough* New Zealand *cough*) before on this blog and it's no fun. I've told the wife in the past that I'm happy to live overseas but I needs me some bandwidth. It sucks when tools you like don't conserve your bandwidth.

Making sure products are available and convenient for customers without a "fat pipe" is a passion of mine and I'm always the guy in the meetings at work to remind people within the Redmond Reality Distortion Field that not everyone in the world has 35 megabit fiber to their house.

We've also been trying to push for transparency and openness in all things (no more secrets, I like to say) and it usually comes down to this: They don't like to announce stuff if they aren't sure they are going to ship it. That's cool, and I get it.

But, when I saw that the folks on the WebPI team have heard you and I and told me they are working on an offline experience for WebPI today I had to share it with you, Dear Reader. Here's the deal, I'm looking at daily builds off the developer's machine. This IS NOT the final experience and this IS NOT anything other than a daily. They are looking at shipping this soon, but they told me to put lots of disclaimers. Disclaim! There it is. As this is a daily build, there are likely more questions than answers, so I'll show you how it works now, with my commentary throughout.

Web Platform Installer Offline

The problem is, of course, that the Web Platform Installer assumes that you and the 50 people at your company sharing internet are cool with downloading the same x gigs 50 times. It also assumes that your system IS connected to the internet in some way. It's a wasteful way for us to get you bits and everyone loses.

I think the ideal use case is that someone who has the bandwidth and the time downloads all the apps they want then puts them on a DVD or a network share and then all the consumers point to that offline cache. Companies could have a share or feed that's refreshed monthly, for example.

Products in WebPI have an "AppID" or Product ID. You'll need to know the AppID for the thing or things you want to install. For example, with the online Web Platform Installer if you want to install "all the web stuff" in one click, you'd use this URL http://www.microsoft.com/web/gallery/install.aspx?appid=VWDorVS2010SP1Pack.

See the "VWDorVS2010SP1Pack" part? That refers to an AppID inside of the Feed that Web Platform Installer consumes. Everything in the feed has an AppID and you can string them together. For example, you can say http://www.microsoft.com/web/gallery/install.aspx?appid=MVC to get ASP.NET MVC 3 for VS2010. You can add lots of AppIDs separated by semicolons ";" to get multiple products and all their dependencies.

I am testing a spike of an offline command line tool called wpiutil.exe but the plan is to merge it with the Web Platform Installer Command Line tool. So, even though I'm showing two tools, pretend there are one.

You can find an AppID inside the Atom Feed that the Web Platform Installer consumes. The main feed is at https://go.microsoft.com/?linkid=9767054 which isn't really discoverable, so you can use the command line tool to find what you want.

I can run WebpiCmdLine.exe /list:all and get a big list of products and applications, then collect the IDs I want to install or cache offline for later.

Current language of installers is English

--Previously Installed Products
ID Title
----------------------------------------
AppFabric Windows Server AppFabric
AppFabricSetup Windows Server AppFabric Setup
*SNIP*
MVC2 ASP.NET MVC 2
MVC3 ASP.NET MVC 3 (Visual Studio 2010)
MVC3Loc ASP.NET MVC 3 Tools Update Language Packs
MVC3Runtime ASP.NET MVC 3 Runtime
*SNIP*
--Applications
ID Title
----------------------------------------
AcquiaDrupal Acquia Drupal
Atlas Atlas
*SNIP*
Umbraco Umbraco CMS
WordPress WordPress

Scott Feedback: I told the team that I thought these AppIDs could be also on this page in the GUI. We could add "Product ID:" to screens like this?

An example Product Details page that could potentially include more data

So now I can do this:

C:\>wpiutil.exe /create-shadow /shadow-path:c:\webpicache /product-ids:MVC3;IISExpress

Loading products in webpi feeds ...

Creating shadow for following products:
MVC3
NETFramework4
WindowsImagingComponent
MVC3Installer
IISExpress

Processing feed: 'https://go.microsoft.com/?linkid=9767054'

Processing resources ...

Processing product entries ...
Processing product entry: 'MVC3Installer'.
Processing product entry: 'MVC3'.
Processing product entry: 'NETFramework4'.
Processing product entry: 'IISExpress'.
Processing product entry: 'WindowsImagingComponent'.

Done.

See how the tool got the the dependencies as well? Here's the folder structure that was created locally:

The folder structure created by WebPI that has all the installers and custom feed

Now I can point the command line tool at my new custom feed. My custom feed contains ONLY what I've downloaded. It's a subset, although I suppose a company COULD make a complete (or at least a large) shadow copy of everything and keep it on a server.

C:\>WebpiCmdLine.exe /xml:"C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml" /List:All

Successfully loaded metadata from feed C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml

Current language of installers is English

--Available Products
ID Title
----------------------------------------
MVC3Installer ASP.NET MVC 3 Tools Update Installer
IISExpress IIS 7.5 Express
MVC3 ASP.NET MVC 3 (Visual Studio 2010)
NETFramework4 Microsoft .NET Framework 4

Then I can install the products I want like this:

C:\>WebpiCmdLine.exe /xml:"C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml" /Products:MVC3

Here's some of the my personal feedback to the team in no order:

  • There should be as much visual feedback as possible from the command line tool, given that it'll be run over slow networks (even 3G) so something like a (345kb of 56 megs) updating text area...
  • A way to switch the Web Platform GUI to be completely offline by pointing it to a local feed.
  • The command line tool should be smart about relative vs. absolute paths, as well as UNC, or even running your company's custom internal Web PI server.
  • Maybe consider using BITS (background internet transfer service) like Windows Update so folks could run it in the background over many days.
  • "Shadow" isn't a good name as it sounds like the Shadow Copying feature in Windows. Just call it offline.
  • More documentation on how customers can make custom feeds for their products. Just as NuGet gets libraries for Visual Studio 2010 projects, WebPI consumes feeds of MSIs and their dependencies.
  • It must be smart about resuming after a dropped connection or call and must retry corrupt downloads.
  • Group policy support so that IT Administrators could control how WebPI works inside a large enterprise and where it loads files from.
  • Should have a smart default path like c:\webpicache or .\webpicache if it's run without a full path.

Now, this is your (and my) opportunity to provide comments and suggestions to the team and affect the final direction of the product or future versions.

From what I've shown you, Dear Reader, my questions to YOU are:

  • Will this be a good feature?
  • Will it help you?
  • Is it useful or useless?
  • Is it important to be a GUI or is a Command Line good too?
  • What has been forgotten?
  • What's a "must have" and what's a "don't need?"

Discuss, they are listening!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Hosting By
Hosted in an Azure App Service
September 24, 2011 12:15
How about enhancing NuGet so that WebPI can use NuGet to install all the apps using offline images ... just a thought
September 24, 2011 13:17
I always prefer direct download method using IDM as i am on 512 kbps internet connection which is not stable.

Also what about pause resume functionality ? Does WebPI picks it up from where it left in case i get disconnected ?
September 24, 2011 13:34
Offline feed mode will also be very useful for using behind corporate firewall.

Can we also have the ability to turn off the online feed mode?
September 24, 2011 13:35
OK it's half time in the All Blacks vs. France game so I have time to comment :-)
As a developer who lives in *cough* New Zealand *cough* (I was at Tech Ed 2008 when you encountered the joy of NZ internet access...couldn't resist thinking "Welcome to our world" :-) ) I do think this would be a good feature (and would help avoid wasting bandwidth in corporate environments).
It needs to be GUI at least or it won't appear on some people's radar, but I'd like to see command line as well.

Oops, gotta go, the game's back on...I'll comment again later if I think of anything that's been left out.
September 24, 2011 14:55
I hear they are not very excited about this in Korea :)))) but for the rest of us in 3rd world, it is a good news.
September 24, 2011 14:56
Hi Scott, Thanks for the post ;)

I believe that this is a great feature. I ran to a problem with this when doing a presentation about WebMatrix and I wanted to show the download process, but the internet connection went off. I prayed there would be such a thing as an offline cache ;)

For my suggestions, I believe that a GUI would be a good addition. Not all users prefer the command line. One more thing that you haven't stressed enough is "updating the cache". Will we have to re-download all the stuff again, or would there be an "update cache" kind of command to intelligently select the packages that have updates?
September 24, 2011 15:13
What about when WebPI is installed on a server (or desktop) with IIS it would add a handler similar to Web Deploy so you could point to that machine or URL as the gallery for other machines. It would be like WSUS's spoke and hub architecture but for WebPI.
September 24, 2011 15:57
Thanks Thanks Thanks! Thanks a lot, it would really save time.
September 24, 2011 16:38
Hello Scott,

I for one have had trouble with WebPI in the US of A as well, thanks to ma bell's (supposedly 6Mbps) DSL (which happens to be the 'best' and only option where I stay). God help when I go back to India (no coughing required)!

So the features above are useful, but get watered down unless:
1. The WebPI bootstrapper can resume downloads
2. Verifies downloaded bits before starting installation (so that it doesn't discover halfway something got busted because the hashes don't match).

As you mentioned, visual feedback on download progress in GUI/Command line is very important.

Couple of other points I noted recently:
The last WebPI setup for MVC4 did not check for available installs but when I tried to install an already available component it just showed installation failed. Either it shouldn't allow selection or say already installed hence skipped. On the brighter side MVC4 setup was the first successful WebPI setup I managed ;-).

Overall, ability to manage offline cache is a big win if it comes with resumable downloads specifically for those challenged by bandwidth/unstable connectivity.

Cheers,
Sumit.
September 24, 2011 16:45
Just to clarify my above comment, WebPI for MVC4 said 'Installation Failed' ONLY for the component that was already installed. It installed rest of the selections perfectly.
September 24, 2011 17:56
Thanks Scott for another great article.

I remember well the pains of New Zealand internet speeds. I was back home in November 2009 for a funeral, and is usual I had to fix Dad's laptop. It took me 2 days to complete Windows Update on his XP machine. What they call ADSL is a joke.

I used to work for a large ISP in New Zealand in the early 90's. At the time, the only Internet access to the rest of the world was a T1 from Hawaii to the University of Waikato. Painful does not begin to describe it.

Cheers


Mark
September 24, 2011 19:27
An offline Web Platform Installer option is critical feature. I recently had to deal with this very issue while trying to use the installer on several IIS boxes in a DMZ that didn't have outbound internet access (not fun). If it's possible, it would be great if all of the shadow cache artifacts could go into a single .cab file instead.
September 24, 2011 23:19
I totally agree with James about servers in DMZ. Right now I am trying to figure out how to install 'LightSwitch 2011 Server Runtime' on a DMZ server that is running 2008 R2 Server Core. Most instructions say use Web Platform Installer. Being able to redirect WPI to use a local file system based cache, network share or internal web server would be ideal.
September 24, 2011 23:49
For me you should try and match the experience offered by the cygwin setup program, run it to download/install straight from the web or from a previously downloaded tree.

It shows all available packages and handles the dependencies when you try to install all in a rudimentary gui.
September 24, 2011 23:55
I would like a subscription service to WebPI. Where I can say what tools/sdks I am interested in keeping up to date and the install package trickle down whenever I have a connection.

Then when I am ready to install I can install from the cache, even if I am offline. I can also take that cache and copy it to a network share and let someone else point to it and do installs as well.
September 25, 2011 0:01
Scott,
Thank you for looking out for us. Due to our sucky bandwidth I have never gotten to use vs for wp7 development much to Malisa's
Frustration.
I think the ability to resume downloads and the ability to share the downloads would also be great so that sharing the installers at the dnug becomes simpler.
Things in uganda are so bad that whenever I need the latest devexpress update, I buy 512mb of data and do the download on my phone.
The offline mode will be of help but if it cannot be paused and resumed there is no point. Also, CONSIDER TORRENTS!!!
Thanks Scott
September 25, 2011 0:20
Personally I would love if Microsoft offered simple .ISO files for us to download via Torrents.

A torrent properly seeded by Microsoft would be fantastic and would give everybody amazing speeds instead of relying on a static file server somewhere Stateside.

I've used the web platform installer and it runs fine 20% of the time. Most of the time I don't know if it's still running, or crashed, or downloading very slowly or what.

September 25, 2011 0:27
Being able to run the command line tool via xcopy would be ideal. A lot of server administrators are very particular about which tools are installed, especially the ones that are used to setup up the server, but should be removed after all the components are setup and configured.

Because the user could be creating the offline files on a machine that does not match the target machine, the user should be able to specify the /architecture=x86,x64 /language=en,es,fr,zh,... target OS=(2008/2008 R2/Windows 7 or 6.0.0/6.1.0) and the OSType= (client vs server). These options should be flexible to allow to specify all or a list and have logical defaults.
September 25, 2011 11:38
I'm quite excited about this as I use a multitude of different PCs and being able to "download once, use anywhere" will be a great addition to WebPI.

I have nothing against using the command-line to shadow copy and create an alternative feed, but when it comes to installing the products, using the WebPI GUI is a must. Having multiple feeds would be great so that WebPI can favour local/network cahced copies and fall back to the web if it's out of date or not found.

+1: "Group policy support so that IT Administrators could control how WebPI works inside a large enterprise and where it loads files from."
September 25, 2011 15:22
MSDN Subscription team is working on support for the "bandwidth limited" as well. Thanks to all!

As a U.S. satellite user, my requirement is for big downloads to run between 0300 and 0700, when Hughesnet doesn't count your bits.

Will I be able to schedule the command to run at a certain time, and will the downloads be restartable if I have to stop them at 0700?

Thanks again!
September 25, 2011 18:56
+1 for allowing the GUI to reference the local feed. I've been working on getting a local NuGet repository up and running and it works really well (<a hred=http://www.chrispont.co.uk/2011/09/creating-a-private-nuget-repository-and-creating-packages/">see my blog</a>). Something similar for WebPI, so that software can just be downloaded once and published to the feed would be awesome.
September 26, 2011 4:23
"Group policy support so that IT Administrators could control how WebPI works inside a large enterprise and where it loads files from."

No!!!!!!!!!!!!!! Administrators already make our lives hell inside large enterprises, let's not give them more power! :p
September 26, 2011 10:07
Scott

Yes. This is a required thing to have. In my opinion they should also provide better GUI tool to do this instead of command line [ not everybody likes cmd.exe ;) ].

Krunal.
September 26, 2011 11:33
Hi Scott

As I tweeted to you yesterday, I'm in South Africa and have a 1Mb ADSL line at work (and a 4Mb ADSL line at home) and WPI is dog slow on both connections!

The initial install of WPI can take 15-20 minutes (if it doesn't time out) and then browsing the catalog there after is also painfully slow, and installing anything is also a bit of a gamble.

To be honest, I don't think it has so much to do with bandwidth as it does with the poor infrastructure MS has setup to host WPI (which is surprising considering that it's rumoured that Apple's iCloud is supposed to be running on Windows Azure).

Just my 2c. Hope to see you in SA against some time soon.

Dieter
September 26, 2011 16:40
I wrote a lot more than this and then I realized you are way ahead of me on this thinking. For me as a developer, WebPI as an offline tool isn't that valuable. I like how WebPI lets me build up a test image or server with the pieces I want (or even just want to try) without any more input from me.

I don't really seem to be the target audience for this. And it really looks like you are putting care and thought into that target audience. As a developer, I think the value of WebPI is it's "let-me-do-that-for-you"itiveness and an offline installer wouldn't be quite as valuable to me (not worthless, mind you... as it can still be as helpful, just from a snapshot in time of product/server/framework/app builds).
September 26, 2011 19:10
Scott,

The team should also really look into SQL Server dependencies. It seems every product or component I install thinks the dependency on some SQL Server component is not yet met: CLR types, management objects etc.

Really annoying!
September 26, 2011 20:00
Scott,

This is a very important feature for me.
We have very limited download speeds and need to install Microsoft stuff in a lot of PCs.
The over reliance on the web installer is a total pain.

Please, please, please make this happen.

Sincerely
Alvaro from a south american country far, far away
September 26, 2011 20:24
I would live it if the WebPlatformInstaller (and all the update systems) would use some system to check if someone else on the same LAN as me has done the download and if so would get the download from their machine rather than the web. It is effort to coordinate putting downloads in a shared folder, why can’t it be automatic?
September 26, 2011 20:46
Scott,

as a counter point to "Mr. Dot NET", group policy configuration is a fantastic idea that can (and should) be implemented/ extended. If we can really go wild and add SCCM, MDT, and WSUS integration, admin types the world over will sing your praises (even more). Bonus points if the API is straight forward, documented, and can be plugged into SpiceWorks/ Nagios/ 3rd party tools.
Location/ situation awareness (ex: if no connectivity, use offline cache) would be a nice to have, maybe for the next rev; configurable hours of activity with bandwidth restrictions (I'm cribbing most of this from utorrent settings); "tracker" locations - attempt to download from internal site "1", "2", "3" and network share "x" for package "n".
September 27, 2011 0:53
So you had to bring that up. I guess the bad experiences do tend to linger but 3 years? Three years ago I was getting reliable 10Gb/month 3Mbit ADSL download speeds in a semi-rural location in the shadow of Mt Taranaki for around USD30. Now I'm getting ADSL2 speeds in the same location. Yes I would absolutely like to get more data, faster and cheaper but as an independent software developer I've generally considered the bandwidth available to be fairly adequate and reasonably affordable. And yes I do download iso's from MSDN. The point I'm trying to make is that the brush you're painting with might be too broad for the subject you're attempting to capture. Just saying.
September 27, 2011 3:53
We have another reason to like the offline functionality, in our enterprise every download from the internet needs to go thru an approval process, infact it needs to be approved on a per download per user granularity. Getting approval once and putting the packet into a seperate cache would be a greate idea, adding security so that each download could be only used by members of an approved windows group would be even better.
THis would also be a great idea for Nuget if anyone is listening.
Finnaly I am a Microsoft Developer we need a GUI (command line yuckky) preferably a touch enable win8 UI :)
September 27, 2011 23:43
Cannot stress enough the importance of smart resume--over any IP. For example, I have a router that fails to 3G when my (384/192 baby!) WiMax fails because somebody dug up the over-land fiber optics for copper.

Smart resume seems like a no-brainer, but I've found tha that the partner portal (a gnat's eyeball, may it be kicked in the head until dead) downloader is abysmal here.

The visual feedback is valuable, but I'm realistic: I don't expect per-second updates. Just don't leave me completely in the dark. That said, "even 3G" made me want to both laugh and cry. As often as not, the little blue light on my 3G stick turns green and it's off to GPRS land.

Sean.
September 28, 2011 12:22
Hey Scott,

several points that could lead to a *great* product:

a) Why is it called "WebPI"? DevPI would actually make a bit more sense, as I use it without even ever touching Web dev at the moment. Or give it a cool trendy name like "Silverlight"...

b) Every download should be *ALSO* downloadable with HTTP from the master repository. There are situation where this is more handy, and as WebPI does nothing but DL the MSI and launch it, please make the URLs public. You have nothing to lose but bandwidth usage.

c) Why are not all the free installable dev things from MS in the platform? Are there plans? Think about pnp stuff, sql stuff, blend stuff,...
It's usually "Okay: that's commercial, so it's MSDN, okay: that is direct download, okay: that is WebPI for some random reason, okay: that is nuget for another reason"...

d) Recently I thought I am an auto-update trigger bot:
- Windows Updates (Win + Office, but also SPs)
- Web PI
- Extension Manager in VS 2010
- NuGet for my used packages

Don't know what you think but I think this could consolidated a bit more in terms of cmd usage, UI style etc. pp. I think even these teams could talk to each other every now and then.

How many package distribution solutions do we need for Windows?

Beside that, I like what WebPI is doing even on 3 (!) different system I am not able to get Visual Studio Tools for Silverlight shown as "installed" with SP1. It was much worse with older versions though.

Regards
Rick
October 11, 2011 22:14
well, i have seen the proxy server problems of the past. seems it would almost be better to let a companies webserver take care of the proxy loads instead of adding another level of caching traps.
October 12, 2011 9:30
YES!!! Please make sure this makes it into the API as well. I use the API for a custom webapp installer, and have had issues installing behind proxies!!!!!
October 12, 2011 17:15
This new mechanism would also have significant positive impacts to configuration-management / application-life cycle.

Large shops that create development environments and build environments need an efficient way to create the right build-run time environment. The fact that we could stash the output of this tool locally and reuse it again and again (years later, if we have to patch!) to create / validate our build environment is huge.

Howard Hoffman

Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.