Scott Hanselman

CSI: Visual Studio - Unable to translate Unicode character at index X to specified code page

June 8, '13 Comments [17] Posted in Bugs
Sponsored By
A crazy internal error from Visual Studio

A customer emailed me a weird one. I tend to have a sense for when something is up and when an obscure thing will turn into something interesting.

The person says:

...mysteriously most of my projects refuse to build.  "The build stopped unexpectedly because of an internal failure... something about unicode... blah blah"

There are a few messages out there on the web about it -- even a really old hot fix.  What's the best way to proceed with the VS team / MS?  Is there anyone actively interested in glitches like this?

My spidey-sense is tingling. First, when something says "internal failure" it means some fundamental expectation wasn't met. Garbage in perhaps? He says "most of my projects" which implies it's not a specific project. There's also the sense that this is a "suddenly things stopped working" type thing. Presumably it worked before.

I say:

"Have you checked all the source files to make sure one isn't filled with Unicode nulls or something?"

And says no, but sends a call-stack (which is always nice when it's sent FIRST, but still):

Error    1    The build stopped unexpectedly because of an internal failure.
System.Text.EncoderFallbackException: Unable to translate Unicode character \uD97C at index 1321 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder)
at System.Text.UTF8Encoding.GetByteCount(String chars)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.Build.BackEnd.NodePacketTranslator.NodePacketWriteTranslator.TranslateDictionary(Dictionary`2& dictionary, IEqualityComparer`1 comparer)
at Microsoft.Build.Execution.BuildParameters.Microsoft.Build.BackEnd.INodePacketTranslatable.Translate(INodePacketTranslator translator)
at Microsoft.Build.BackEnd.NodePacketTranslator.NodePacketWriteTranslator.Translate[T](T& value, NodePacketValueFactory`1 factory)
at Microsoft.Build.BackEnd.NodeConfiguration.Translate(INodePacketTranslator translator)
at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.NodeContext.SendData(INodePacket packet)
...

OK, so it doesn't like a character. But a character in WHAT? Well, we'd assume a source file, but it's important to remember that there's other pieces of input to a compiler like path names, environment variables, commands passed to the compiler as switches, etc.

It says Index 1321 which seems pretty far into a string before it gets mad. I asked a few people inside and Sara Joiner says:

It looks like the only place in BuildParameters that we call TranslateDictionary is when transferring the state of the environment [variables] across the wire. 

Ah, so this is splitting up name-value pairs that are the environment variables! David Kean says "ask him what his PATH looks like." I ask and I get almost 2000 bytes of PATH! It's a HUGE path, it looks like it may even have been duplicated and appended to itself a few times.

Here's just a bit of the PATH in question. See anything?

\;C:\PROGRA~1\DISKEE~1\DISKEE~1\;C:\Program Files (x86)\Windows Kits\8.0\Windows
Performance Toolkit\;C:\Program Files\Microsoft SQL
Server\110\Tools\Binn\;C:\Program Files\Microsoft\Web Platform
Installer\;C:\Program Files\TortoiseSVN\binVN\???p??;C:\Program
Files\TortoiseSVN\bin;C:\PHP\;C:\progra~1\NVIDIA
Corporation\PhysX\Common;C:\progra~2\Common Files\Microsoft Shared\Windows
Live;C:\progra~1\Common Files\Microsoft Shared\Windows
Live;C:\q\w32;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;
C:\Windows\System32\WindowsPowerShell\v1.0\;C:\progra~2\WIDCOMM\Bluetooth
Software\;C:\progra~2\WIDCOMM\Bluetooth

See those ??? marks? That doesn't feel like question marks to me. I open the result of "SET > env.txt" as a binary file in Visual Studio and it looks like it's 3Fs, which are ? marks.

I think the text file was converted to ANSI

This makes me think that there's unicode goo in the PATH that was converted to ANSI with it was piped. Phrased differently, this text file isn't reality.

However, elsewhere in the Windows UI his PATH variable looks like different.

C:\Program Files\TortoiseSVN\binVN\�侱ᤣp䥠؉;

Sometimes that corruption in the path looks like this and you might assume it's Chinese. No, it's corruption that's getting interpreted as Unicode. Interestingly the error said the naughty character was 0xD97C which is &#0xD97C; � which implies to me that something got stripped out at some point in processing and turned into the Unicode equivalent of 'uh...' Regardless, it's wrong and it needs to be removed.

I ask him if cleaning his PATH worked and the customer just send me a one line response via email...the best kind of response:

========== Build: 12 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

Yay! I hope this helps the next person who goes aGoogling for the answer and thought they were alone. Thanks to David Kean, Sara Joiner and Srinivas Nadimpalli for looking at the call stack and guessing at solutions with me!

Any insights, Dear Reader?


Sponsor: Big thanks to RedGate for sponsoring the feed this week! Check out Deployment Manager – app deployment without the stress. Deploy .NET code & SQL Server databases in one simple process from a web-based UI. Works with local, remote and cloud servers. Try it free.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to delete Open or Insecure Wi-Fi HotSpots from Windows 8: Wifi.exe Command Line Utility with Source

June 6, '13 Comments [19] Posted in Open Source | Tools | Win7 | Win8
Sponsored By

image

For the most part I'm happy with Windows 8 but one feature was removed that makes no sense to me - the wireless networks dialog.

Sure, you can "Forget this network" by right clicking on a Wi-Fi Connection, but only when that network is in range. The old Wireless Networks dialog where you could add and remove networks is gone. Who knows how many saved Wi-Fi hotspot profile I have littering my system?

So, The Problem: I want to remove saved Wi-Fi Profiles whenever I feel like it. I wrote a command line util that will work in Windows 7 and Windows 8.

TL;DR Version

There's a build zipped up of Wifi.exe available here and the source is on GitHub.

UPDATE: I've put Wifi-Manager up on Chocolately so you can now "cinst wifi-manager." Thanks to Brendan Forster for the heavy lifting! Learn more about the Chocolatey package manager here!

Caveats and "Ya I know."

First, let me be clear that I have written a command line utility to replace another command line utility. I get it. I knew it when I did it. Others smarter than I have done similar things and written utilities that match their way of thinking rather than learning an unintuitive syntax. Don't hate the playa, hate the Regular Expression.

Aside: This is also a problem with my iPhone. I likely have 50+ saved Wi-Fi spots on my phone and no way to delete them without jail-breaking.

You can access Wi-Fi profiles with the netsh.exe that comes with Windows, so you could list out profiles like this:

c:\>netsh wlan show profiles

Profiles on interface Wi-Fi:

User profiles
-------------
All User Profile : Wayport_Access
All User Profile : HANSELMAN
All User Profile : HANSELMAN-N
All User Profile : HanselSpot
All User Profile : EliteWifi
All User Profile : Qdoba Free Wifi

Then, for each one, call

c:\>netsh wlan show profile "Qdoba Free Wifi"

Profile Qdoba Free Wifi on interface Wi-Fi:
=======================================================================

Profile information
-------------------
Version : 1
Type : Wireless LAN
Name : Qdoba Free Wifi
Control options :
Connection mode : Connect manually

Connectivity settings
---------------------
Number of SSIDs : 1
SSID name : "Qdoba Free Wifi"
Network type : Infrastructure

For each of these profiles, check if they are secure or open, and if you are connecting manually or automatically. Then, if you wanted, you could netsh wlan delete profile name="Qdoba Free Wifi" and remove a profile, even when it's not near you.

In my recent podcast with security expert Troy Hunt, he pointed out that it's easy to create a fake honeypot Wi-Fi spot that has the same name as a common open network, like Starbucks, for example.

  • Given: If my PC or phone is set up to automatically connect to any open hotspot named "Starbucks" then it will just connect to one...even an evil hotspot.
  • Therefore: it would be nice to automatically delete profiles for Wi-Fi spots that are both open (no security) and set to automatically connect.

I was tired, so I thought I'd bang out a little utility to do this. I could have used PowerShell or something but I felt like using C#. It's exercise.

UPDATE: Lee Holmes went and wrote it in PowerShell! Amazing.

Wifi.exe and it's Usage

Tired of reading? There's a build zipped up of Wifi.exe available here and the source is on GitHub. You may need to Right Click | Properties | Unblock the zip.

There's no warranty. The code sucks and I'm a horrible person and you're running a util you found on my blog. However, it works awesome on my machine. Issues appreciated, tidy PRs appreciated more, running Resharper and doing a PR, less so. I'll update the build if good bugs require it.

If you run Wifi.exe (I put it in my path) you'll see something like this:

c:\>wifi
AP-guest manual WPA2PSK
HANSELMAN-N auto WPA2PSK
HANSELMAN auto WPA2PSK
HanselSpot auto WPA2PSK
Qdoba Free Wifi manual open
Wayport_Access auto open Warning: AUTO connect to OPEN WiFi

Delete WiFi profiles that are OPEN *and* AUTO connect? [y/n]
n

Notice the columns, and the prompt. There's a warning when a hotspot is both open and set to auto-connect. If you answer Y to the prompt, the utility will delete that profile. You can also type 'wifi /deleteautoopen' to bypass the prompt and auto-delete just profiles that are auto and open.

A pull request a few minutes after I pushed this code also added the ability to

wifi delete "HOTSPOTNAME"

which is nice also. Thanks!

The Code

One of the great things about writing command line apps like this is that there's literally a dozen ways to do everything. They are trivial and silly but also useful and used daily. In this case I've got command line argument processing to think about, parsing output from a spawned process, doing the parsing in a clean way, making sure it works on a non-English machine (which I thought about but didn't test), as well as cleaning up of generated temp files.

It's hardly impressive code, but some of it was fun or interesting. Here's a few bits I liked.

Making Columns with Console.WriteLine and String.Format

Did you know that you can right- and left-align columns within a fixed with using String.Format? Few people know about this and I've seen whole libraries written custom to do the work that's built right in.

Console.WriteLine(String.Format("{0,-20} {1,10} {2,10} {3,30} ", a.Name, a.ConnectionMode, a.Authentication, warning));

Note the {0,-20} (left aligned) and the {1,10} (right aligned). Those are just like {0} and {1} in a String.Format but they include alignment and width.

Gratuitous use of Linq

It wouldn't be a silly utility without in crazy LINQ, eh? Who needs Regular Expressions when you can when you can do a SQL query over your string? ;) Actually, I don't know if this is a good thing or not. It was fun, though, and it works. Your thoughts?

This takes the output from wlan show profiles (seen above) and parses it into a list of just the AP Names. I think it should work in any language, assuming the : colons are there.

string result = ExecuteNetSh("wlan show profiles");
var listOfProfiles = from line in result.Split(new string[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries)
where line.Contains(":")
let l = line
where l.Last() != ':'
select l.Split(':')[1].Trim();

foreach (string profile in listOfProfiles)
ExecuteNetSh(String.Format("wlan export profile \"{0}\" folder=\"{1}\"", profile, Environment.CurrentDirectory));

Cleaning up the temp XML files

I export a bunch of very specific XML files with a VERY non-specific extension. I can't control their file name and I don't want guess what their name is because I would need to recreate their AP Name encoding scheme. Instead, I look for any XML files in the current folder (given the rare chance that YOU, the utility runner, have XML files in the same folder already) and only delete the ones with the namespace that I know to be present in Wi-Fi profiles. I patted myself on the back for this one, but just lightly.

static XNamespace ns = "http://www.microsoft.com/networking/WLAN/profile/v1";

//Delete the exported profiles we made, making sure they are what we think they are!
foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.xml"))
if (XElement.Load(file).Name.Namespace == ns)
File.Delete(file);

Capturing Command Line Output

Finally, here's how you get the output of a command line process you started:

Process p = new Process();
p.StartInfo.FileName = "netsh.exe";
p.StartInfo.Arguments = arguments ?? String.Empty;
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.Start();

string output = p.StandardOutput.ReadToEnd();
return output;

Pretty basic, but useful to bookmark.

Alternatives

After I wrote this I noticed there are some WinForms utilities to do this. That's great. I wouldn't mind making may own, except I'd want it to look exactly like the Windows 7 dialog. It'd be fun just to see if I could get it pixel perfect.

Feel free to go check out the code, play with it and make fun of me. https://github.com/shanselman/Windows-Wifi-Manager


Get Involved! Check out my latest production with TekPub. A meticulously edited TWO HOURS of video content where we cover everything we think a developer should know to "Get Involved" in the developer community.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Announcing "Get Involved" from Tekpub - Enhance your career by engaging with your peers

June 6, '13 Comments [15] Posted in Musings | Open Source
Sponsored By

Get Involved - The Video

My friend Rob Conery and I work on the This Developer's Life podcast together. You should check it out, we work well together. Last year we created a Technical Speaking Tips Video and launched http://speakinghacks.com on Rob's TekPub site. 

Get Involved!Since then I've done a little more moonlighting on the side with Rob and release not just The Art of Speaking but also done other episodes with Rob including a regular show called "The Source" where we explore the source of popular open source frameworks. Again, you get all of this with a yearly or monthly subscription.

Today we're launched our most ambitious project yet. A meticulously edited episode that's almost TWO HOURS of video content where we cover everything we think a developer should know to "Get Involved" in the developer community.

Announcing "Get Involved"

If you're a fan of This Developer's Life you know how tightly we like to produce things - this video is no exception. Filmed on the streets of Portland and at a Portland user group, we talk about Blogging, Twitter, Github, StackOverflow, Open Source, Speaking, User Groups and Conferences - all of this hoping to make you a happier, more productive, more connected developer. We want to inspire you and perhaps to take your career to the next level.

Additionally, we stretched far beyond Portland to seek out the other people who are very active and well known in the social arena like Jon Skeet and Jeff Atwood:

  • Jeff Atwood and I talk about blogging, writing, and "working your voice free" so people who read your posts hear you loud and clear.
  • Jon Skeet joins us to talk about what a Good Question is on StackOverflow - and also how you can gain reputation by providing Good Answers - and edits to Good Questions!
  • We venture out to the Portland Area DotNet Users Group (PADNUG) and meet a few developers who have just started going - as well as people who have been there for years.
  • While we were there, I gave a 10-minute lightning talk on Azure - a great way to get started speaking if you're not a fan of public speaking. Rob filmed the whole thing.

We put an immense amount of work into this production and I really think you'll enjoy it. You can buy it "ala carte" or as with all Tekpub productions you can get a one year subscription and get access to everything on Tekpub including this video, my speaking video, my show The Source as well as my episodes of Full Throttle with Rob PLUS dozens of other videos on new tech like Backbone, Async C# 5.0 with Jon Skeet, ASP.NET MVC with Sam Saffron, RavenDB with Ayende Rahien, Mastering jQuery, and on and on.

I hope you enjoy this show. We poured our hearts into it.

P.S. Here's a 10% discount coupon if you only want the video alone: KQWFCJUE0SPO

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Blocking Image Hotlinking, Leeching and Evil Sploggers with IIS Url Rewrite

June 3, '13 Comments [60] Posted in IIS
Sponsored By

I recently discovered that a blog called (seriously) "Google Chrome Browser" was reblogging my site. (It of course has NO relationship to Google or the lovely folks on the Chrome team.)

This is a splog or "spam blog." It's less of a blog and more of a 'suck your feed in and reblog it.' Basically every post is duplicated or sucked in via RSS from somewhere else.  I get this many times a week and have for years.

However, this particular site started showing up ahead of mine in searches and that's not cool.

You evil bastards.

Worse yet, they have almost 25k followers on Twitter. I've asked them a few times to stop doing this, but this time I got tired of it.

They're even 'hotlinking' my images, which means that all my PNGs are still hosted on my site. When you visit their site, the text is from my RSS but I pay for the images bandwidth. The irony of this is thick. Not to mention my copyright notice is intact on their site. ;)

When an image is linked to from another domain the HTTP_REFERER header is populated with the location that the image is linked from. That means when my web server gets a request for 'foo.png' from the Google Chrome Browser blog I can see the page that asked for that image.

For example:

Request URL:http://www.hanselman.com/blog/content/binary/Windows-Live-Writer/How-to-run-a-Virtual-Conference-for-10_E53C/image_5.png
Request Method:GET
Referer:http://google-chrome-browser.com/penny-pinching-cloud-how-run-two-day-virtual-conference-10

Because this differentiates the GET request that means I can do something about it. This brings up a few important things to remember in general about the web that I feel a lot of programmers forget about:

That said, I want to detect these requests and serve a different image.

If I was using Apache and had an .htaccess file, I might do this:

RewriteCond %{HTTP:Referer} ^.*http://(?:www\.)?computersblogsexample.info.*$
RewriteHeader Referer: .* damn\.spammers

RewriteCond %{HTTP:Referer} ^.*http://(?:www\.)?google-chrome-browser.*$
RewriteHeader Referer: .* damn\.spammers

#make more of these for each evil spammer

RewriteCond %{HTTP:Referer} ^.*damn\.spammers.*$
RewriteRule ^.*\.(?:gif|jpg|png)$ /images/splog.png [NC,L]

Since I'm using IIS, I'll do similar rewrites in my web.config. I could do a whitelist where I only allow hotlinking from a few places, or a blacklist where I only block a few folks. Here's a blacklist.

<system.webServer>
<rewrite>
<rules>
<rule name="Blacklist block" stopProcessing="true">
<match url="(?:jpg|jpeg|png|gif|bmp)$" />
<conditions>
<add input="{HTTP_REFERER}" pattern="^https?://(.+?)/.*$" />
<add input="{DomainsBlackList:{C:1}}" pattern="^block$" />
<add input="{REQUEST_FILENAME}" pattern="splog.png" negate="true" />
</conditions>
<action type="Redirect" url="http://www.hanselman.com/images/splog.png" appendQueryString="false" redirectType="Temporary"/>
</rule>
</rules>
<rewriteMaps>
<rewriteMap name="DomainsBlackList" defaultValue="allow">
<add key="google-chrome-browser.com" value="block" />
<add key="www.verybadguy.com" value="block" />
<add key="www.superbadguy.com" value="block" />
</rewriteMap>
</rewriteMaps>
</rewrite>
</system.webServer>

I could have just made a single rule and put this bad domain in it but it would have only worked for one domain, so instead my buddy Ruslan suggested that I make a rewritemap and refer to it from the rule. This way I can add more domains to block as the evil spreads.

It was important to exclude the splog.png file that I am going to redirect the bad guy to, otherwise I'll get into a redirect loop where I redirect requests for the splog.png back to itself!

The result is effective. If you visit their site, I'll issue an HTTP 307 (Moved Temporarily) and then you'll see my splog.png image everywhere that they've hotlinked my image.

Not cool, splogger, not cool.

If you wanted to change the blacklist to a white list, you'd reverse the values of allow and block in the rewrite map:

<rewriteMaps>
<rewriteMap name="DomainsBlackList" defaultValue="block">
<add key="google-chrome-browser.com" value="allow" />
<add key="www.verybadguy.com" value="allow" />
<add key="www.superbadguy.com" value="allow" />
</rewriteMap>
</rewriteMaps>

Nice, simple and clean. I don't plan on playing "whac a mole" with sploggers as it's a losing game, but I will bring down the ban-hammer on particularly obnoxious examples of content theft, especially when they mess with my Google Juice.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Hanselman's Newsletter of Wonderful Things: May 13th, 2013

June 3, '13 Comments [4] Posted in Newsletter
Sponsored By

I have a "whenever I get around to doing it" Newsletter of Wonderful Things. Why a newsletter? I dunno. It seems more personal somehow. Fight me.

You can view all the previous newsletters here. You can sign up here Newsletter of Wonderful Things or just wait and get them later on the blog, which hopefully you have subscribed to. Email folks get it first!

Here's the newsletter that I sent out May 13th.


Hi Interfriends,

Thanks again for signing up for this experiment. Here's some interesting things I've come upon this week. If you forwarded this (or if it was forwarded to you) a reminder: You can sign up at http://hanselman.com/newsletter and the archive of all previous Newsletters is here.

Remember, you get the newsletter here first. This one will be posted to the blog as an archive in a few weeks.

Scott Hanselman

(BTW, since you *love* email you can subscribe to my blog via email here: http://feeds.hanselman.com/ScottHanselman DO IT!)

P.P.S. You know you can forward this to your friends, right?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.