Scott Hanselman

Unix Fight! - Sed, Grep, Awk, Cut and Pulling Groups out of a PowerShell Regular Expression Capture

August 1, '11 Comments [15] Posted in PowerShell
Sponsored By

There's a wonderful old programmers joke I've told for years:

"You've got a problem, and you've decided to use regular expressions to solve it.

Ok, now you've got two problems..."

A friend of mine was talking on a social network and said something like:

"That decade I spent in the Windows world stunted my growth. one teeny-tiny unix command grabbed certain values from an XML doc for me."

Now, of course, I took this immediately as a personal challenge and rose up in a rit of fealous jage and defended my employer. Nah, not really as I worked at Nike on Unix for a number of years and I get the power of sed and awk and what not. However, he said XML, and well, PowerShell rocks XML.

Because it's a dynamic language, you can refer to XML nodes just like this:

$a = ([xml](new-object net.webclient).downloadstring("http://feeds.feedburner.com/Hanselminutes"))
$a.rss.channel.item

The first line gets the feed and the second line gets all the items.

However, turns out my friend was actually trying to retrieve values within poorly-formed XML fragments within a larger SQL dump file. There's three kinds of XML. Well-formed, valid, and crap. He was sifting through crap for some values. Basically he had this crazy text file with some fragments of XML within it and wanted the values in-between elements: "<FancyPants>He wants this value</FancyPants>."

Something like this:

grep "<FancyPants>.*<.FancyPants>" test.txt | sed -e "s/^.*<FancyPants/<FancyPants/" | cut -f2 -d">"| cut -f1 -d"<" > fancyresults.txt

I'm old, but I'm not an expert in grep and sed so I'm sure there are ways he could have done it more tersely. There always is, right? With regular expressions, sometimes someone just types $@($*@)$(*@)(@*)@*(%@%# and Shakespeare pops out. You never know.

There's also a lot of different ways to do this in PowerShell, but since he used RegExes, who am I to disagree?

First, here's the one line answer.

cat test.txt | foreach-object {$null = $_ -match '<FancyPants>(?<x>.*)<.FancyPants>'; $matches.x}

But I thought I'd also sort them, remove duplicates...

cat test.txt | foreach-object {$null = $_ -match '<FancyPants>(?<x>.*)<.FancyPants>'; $matches.x} | sort | get-unique

But foreach-object can be aliased as % and get-unique can be just "gu" so the final answer is:

cat test.txt | % {$null = $_ -match '<FancyPants>(?<x>.*)<.FancyPants>';$matches.x} | sort | gu

I think we can agree at they are both hard to read. I still love PowerShell.

Related Links:

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Entity Framework Code First Migrations: Alpha - NuGet Package of the Week #10

July 28, '11 Comments [47] Posted in Data | NuGet | NuGetPOW
Sponsored By

Hot on the heels of my RFC blog post on product versioning, the Entity Framework team has released Entity Framework 4.1 Code First Migrations: August 2011 CTP. Cool. And it's July, too!

Or my preferred product name, "Migrating Magic Unicorns 0.5." It's probably best to think of this as 0.5 Alpha Migrations for EF but that's my guess at a name and not nearly as descriptive.

I showed early daily builds of EF Migrations at a few conferences recently, and encouraged folks to comment on the ADO.NET team blog. Now they've released bits for us to play with. This initial CTP is available via NuGet as the EntityFramework.SqlMigrations package.

EntityFramework.SqlMigrations - 0.5.10727.0

Here's the general idea. Be aware that this is NOT specific to the Web. You can do this in a Console App or whatever. I just like Hello World Web applications.

Short Walkthrough

Make a new ASP.NET MVC app. Go to Manage NuGet Packages from References and search for EntityFramework.SqlMigrations. Note the dependency it has. It'll upgrade your project's EntityFramework package as well.

Entity Framework SQL Migrations In NuGet

Now, make a new class with a simple model:

namespace MvcApplication15.Models
{
public class Person
{
public int PersonId { get; set; }
public string Name { get; set; }
}

public class DemoContext : DbContext
{
public DbSet<Person> People { get; set; }
}
}

Scaffold out a quick Person Controller...

Add Controller

Visit it at /Person and make a few People.

 Index - Windows Internet Explorer (58)

The database that was created looks like this. There's a People table and the First column that is an nvarchar.

image

Now, let's add an Email field. I'll update the Person class to include Email:

public class Person
{
public int ID { get; set; }
public string Name { get; set; }
public string Email { get; set; }
}

Then, I'll compile and from the console inside Visual Studio, I'll do this:

PM> update-database
No pending custom scripts found.
Ensuring database matches current model.
- Performing automatic upgrade of database.
- Starting rebuilding table [dbo].[EdmMetadata]...
- Caution: Changing any part of an object name could break scripts and stored procedures.
- Starting rebuilding table [dbo].[People]...
- Caution: Changing any part of an object name could break scripts and stored procedures.
- Update complete.

I typed update-database, that's all. This is an automatic migration. See how the system compare the .NET type and the database and did what needed to be done:

image

Now, let's rename Email to EmailAddress. If I change the Person...

public class Person
{
public int ID { get; set; }
public string Name { get; set; }
public string EmailAddress { get; set; }
}

And type update-database...

PM> update-database
No pending custom scripts found.
Ensuring database matches current model.
- Performing automatic upgrade of database.
Update-Database : - .Net SqlClient Data Provider: Msg 50000, Level 16, State 127, Line 6 Rows were detected. The schema update is terminating because data loss might occur.
At line:1 char:16
+ Update-Database <<<<
+ CategoryInfo : NotSpecified: (:) [Update-Database], Exception
+ FullyQualifiedErrorId : UnhandledException,System.Data.Entity.Migrations.Commands.MigrateCommand

I'm told that data loss may occur. It can't tell that I want to rename that column or not. It doesn't know what it was before and what it wants to be. Maybe I want to drop Email and add EmailAddress? Who knows. Let me be explicit and give Migrations more context.

PM> Update-Database -Renames:"Person.Email=>Person.EmailAddress"
No pending custom scripts found.
Ensuring database matches current model.
- Performing automatic upgrade of database.
- The following operation was generated from a refactoring log file d5598498-a656-4ccd-1e93-bea562ab6e31
- Rename [dbo].[People].[Email] to EmailAddress
- Caution: Changing any part of an object name could break scripts and stored procedures.
- Update complete.

I'm not sure if I like that Renames: syntax. I'm sure the team would be interested in your opinion. But that works, as I can see in the database.

image

What Changes Can Migrations Detect Automatically?

From the ADO.NET blog:

Here is the full list of changes that migrations can take care of automatically:

  • Adding a property or class
    • Nullable columns will be assigned a value of null for any existing rows of data
    • Non-Nullable columns will be assigned the CLR default for the given data type for any existing rows of data
  • Renaming a property or class
    • See ‘Renaming Properties & Classes’ for the additional steps required here
  • Renaming an underlying column/table without renaming the property/class
    (Using data annotations or the fluent API)
    • Migrations can automatically detect these renames without additional input
  • Removing a property
    • See ‘Automatic Migrations with Data Loss’ section for more information

Moving to Staging/Production/etc

Once dev is correct, when you want to move to production, you would generate a script for your other database by doing a diff between what that DB looks like and what the code looks like.

For example, -script generates a script I can run myself with osql.exe or whatever.

update-database -Script -ConnectionString "SERVER=.\SQLEXPRESS;Database=PersonProd;Trusted_Connection=true;"

Which spits out something like (but more complex than) this:

CREATE TABLE [dbo].[People] (
[ID] INT IDENTITY (1, 1) NOT NULL,
[Name] NVARCHAR (MAX) NULL,
[EmailAddress] NVARCHAR (MAX) NULL,
PRIMARY KEY CLUSTERED ([ID] ASC)
);

Call for Feedback

The EF Team has also this specific call for feedback when it comes to Custom Scripts:

Call for Feedback: From what we are seeing in our own internal use we don’t anticipate that custom scripts will be required very often. However, our efforts are somewhat sheltered from the ‘real world’ so we would love feedback on situations where you need to use custom scripts. In particular we are interested if there are significant scenarios where a code based alternative to writing raw SQL would be beneficial.

Leave your thoughts in my comments, or theirs and I'll make sure the right people get it.

Enjoy!

P.S. Note the EntityFramework.SqlMigrations NuGet package's exposed version number. It's 0.5.10727.0. ;)

Limitations

This is Alpha, so read the Limitations section. They are putting out rough things like this because they know we want to see bits earlier, but the trade off is limitations. Here's a few. Read the list for the rest.

  • There is no provider model, this release only targets SQL Server, including SQL Azure. SQL Compact and other providers are not supported. We are currently working through what the provider model should look like for migrations.
    • Question: What do you think? How important is this?
  • Migrations currently needs to run in full trust. This isn’t an issue when working inside of Visual Studio but if consuming the migrations assembly from custom code you may want to run in medium trust. We are looking at ways to support this in a later release.
  • This release is only available via NuGet. As we support more scenarios such as team build and an ‘outside of Visual Studio’ command line experience we will also support more installation options.
    • Scott: Limitation? That's lovely! ;)
  • Downgrade is currently not supported. When generating custom scripts you will notice that the script is named ‘Up.sql’ but there is no corresponding ‘Down.sql’. We are planning to add downgrade functionality prior to RTM but it is not available in this release.

I'm glad we're getting migrations. It's been a missing LEGO piece for a while.

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to connect to a Wireless WIFI Network from the Command line in Windows 7

July 27, '11 Comments [13] Posted in Tools
Sponsored By

For the humorless amongst you who didn't find these Updated for 2011 - McDonald's WiFi Guide with updates for Mac OS X Lion and Windows 7 to be HIGH-LARIOUS, the question was asked, "well, sir, how do you connect to a Wireless WIFI Network from the Command line in Windows 7?"

The answer, is, ahem, thusly:

C:\>netsh wlan connect name=HANSELMAN-N
Connection request was completed successfully.

Cool.

More Details

What happened there? Well, the command line is netsh wlan and the full one is

netsh wlan connect ssid=YOURSSID name=PROFILENAME

What's a profile? It's the only thing required. You can see them with:

C:\>netsh wlan show profile

Profiles on interface Wireless Network Connection:
.
..snip..
User profiles
-------------
    All User Profile     : Clear Spot b0e
    All User Profile     : HANSELMAN-N
    All User Profile     : Quiznos

These are the same ones that you see in the wireless networks dialog...

Manage Wireless Networks

You can set these up and refer to them by name from the command line, or a batch file, etc. Nice to do for the places you are regularly.

If you have multiple wireless cards (What's wrong with you!?) then you have to be more specific:

netsh wlan connect ssid=YOURSSID name=PROFILENAME interface="WIRELESS NETWORK CONNECTION"

And of course, you can

netsh wlan disconnect

And include the interface optionally, for multiple interfaces. Additionally, interesting things can be seen with

netsh wlan dump

This is nice because you can

netsh wlan dump > myconfig.txt

on one machine and then later on another machine

netsh exec myconfig.txt

All this command line love will work in most versions of Windows, actually, not just 7 AFAIK. There's lots of more detail and docs on managing Wireless Profiles on the Web.

Enjoy!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Request for Comments: Issues with .NET and Microsoft Product Versioning

July 27, '11 Comments [248] Posted in Microsoft | Musings
Sponsored By

Oy.NOTE: I have had this blog post sitting in my drafts for a few months now. I've gotten close to posting it, held back, then gotten close to posting again. Today I noticed that Microsoft published a patch/update to Entity Framework 4.1 (Code First), a product I personally love and support, and they've called it EF 4.1 Update 1. Then I decide to post this. This is not intended as a rant or a complaint, although there is frustration on my part. What I want to know from you, is DO YOU CARE. If you do, then I can make sure your voice is heard. If you don't care, that's cool too.

I really care about how products are versioned and I'm sure you to do, Dear Reader. Naming Things is Step 0 when it comes to understanding Things. I want to talk to you about some things I've noticed around .NET versioning, and see what your thoughts are so that I might share them with TPTB (The Powers That Be).

I noticed recently that Microsoft released something called ".NET 4.0 Platform Update 1." This is an update to the .NET Framework 4 to include new features and functionality around Workflow and Azure. This post isn't specific to this update, but ALL the updates lately, from the .NET Framework, the Entity Framework, SQL Server, and a dozen more.

This was concerning to me for a few reasons. First, Platform Update "1" implies a future Platform Update "N+1." Second, when something that is a platform called 4.0 is updated, you'd expect it to be called 4.1 or maybe 4.0.1. 

I've met with that team, and encouraged them to stick with Semantic Versioning and call these updates .NET 4.0.2, etc. I think they hear me, we shall see.The .NET Framework Team agrees and they've said that will be following Major.Minor.Revision now so the next small release will be 4.0.2! However I may be perceived as Chicken Little as I haven't personally collected broad community opinion.

If you get the Premium version of Visual Studio, you've got a higher SKU than the Professional version. However, if you choose between Premium and Profession versions of Windows 7, get Professional. It's higher.

I recently noticed other products coming out with "Cumulative Update to SP1" and recently "EF 4.1 Update 1" and similar things, not to mention SQL Server. I really think this is confusing to customers. It certainly is to me. I'd like to know if you agree.

In my personal opinion, Microsoft has typically done a (poor) job with naming things - I think that's a fair statement with some exceptions like things like Lync, Kinect, Xbox and NuGet. Sometimes there's engineering reasons for versioning, but mostly it's a combination of marketing, lack of coordination between groups in a large company and a lack of community/customer outrage or just apathy. I think when folks work at a large company for many years it's easy to become complacent and stop fighting what is perceived as "small things."

Versioning and naming isn't set in stone. There isn't a technical reason that I know of to call something a Rollup Update Pack. Only willpower and organizational agreement is needed. If it's important to you, and your voice is heard, it'll become important to the people who make these decisions. Personally, I am a big fan of Semantic Versioning both in concept and in practice and I'd like to see its practical common sense take root at Microsoft.

Here's how the .NET Framework has been versioned so far. You are all familiar with it, probably because you've had to explain it to your boss at some point.

Version Runtime (CLR) Framework
1.0 First Release First Release
1.1 New Small Changes
2.0 New Lots of Changes
3.0 Same as 2.0 WinFX libraries
3.5 Same as 2.0, but new C# 3 Compiler Some Changes
3.5 SP1 Same as 2.0, with C# 3 Whole Lot of Changes
4 New Lots of Changes
4.0 PU1 Same as 4 Workflow and Azure Changes

We can't change the past, but we can influence the future. I would have probably made framework changes Point Releases (.1,.5, etc) and new CLRs or compilers should be Major Releases.

I suspect that product naming folks think that words are easier to understand than numbers, but I think they sell us short. Semantic Versioning would be easier to explain, deal with and sell. I think we need to stop with the Update, Refresh, Rollup, Pack stuff and just change numbers.

Going further, with the release of this new "EF 4.1 Update 1," what should they have called it? I would have said 4.1.5 or 4.2. Probably 4.2 if there were minor new features, and 4.1.5 if it's just patches.

Photo via Creative Commons: Original Photo at http://www.flickr.com/photos/nicmcphee/422442291/

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to write or burn a CD (CD-R) that includes CD-Text with ImgBurn

July 27, '11 Comments [14] Posted in Tools
Sponsored By

CD with CD-Text in my carOne of the greatest, coolest, but sadly least known and least often used tech things about CDs is "CD-Text." What is it? From Wikipedia:

CD-Text is an extension of the Red Book Compact Disc specifications standard for audio CDs. It allows for storage of additional information (e.g. album name, song name, and artist) on a standards-compliant audio CD.

Phrased differently, you'll be able to see the track details on your car CD player! This has been out for 14 years and I can count on one hand the number of times I've actually SEEN a CD in my car have text associated with it.

This evening I was putting together a Mix CD for my brother-in-law's upcoming wedding. I put together a lovely compilation from a list of songs they gave me, and even managed to fit in 79 minutes and 48 seconds. This is extra awesome because a CD holds 80 minutes. I bought the MP3s legally, and then burned them. When I went into the car to test the CD, I saw only DISC ---, TRACK ---, ARTIST ---.

I said to my self, "Self, this is weak sauce." I put all this work into the CD but I get hyphens instead of love? So I started doing some research and trying different tools.

I found ImgBurn (download link) to be the burner that gave me the most control while still maintaining an easy interface.

Start up ImgBurn, and from the Tools Menu select, "Create CUE File." What's a CUE file? It's a track that specifies exactly how to lay out the tracks on a CD before you burn them. The CUE includes not only the MP3s you want to burn, but also the gaps between songs, song length, the song text, etc.

You can make them in Notepad if you are really showoff Linux User who just has to do everything in Vim someone who likes to do things manually. Here's an example snippet of my CUE file, just so you know what one looks like.

TITLE "Vusi and Philile's Wedding CD"
PERFORMER "Various Arists"
FILE "Brenda Fassie - Wedding Day.mp3" MP3
REM FILE-DECODED-SIZE 04:04:33
TRACK 01 AUDIO
TITLE "Wedding Day"
PERFORMER "Brenda Fassie"
INDEX 01 00:00:00
FILE "04 Giving Myself.mp3" MP3
REM FILE-DECODED-SIZE 04:14:56
TRACK 02 AUDIO
TITLE "Giving Myself"
PERFORMER "Jennifer Hudson"
INDEX 01 00:00:00
....etc.... But insi

ImgBurn has a nice CUE File Creator/Editor. Tip: Be sure to set the "Default CD-Text" for the disc and the track to pull from the MP3s ID3 Tags. This is just the default. You can change this for each song later. Click each song and, if need be, change the CD-TEXT to Custom and put in whatever you like. Perhaps rather than song titles, put in messages to your sweetie?

Create CUE File for CD-Text in ImgBurn

When you click OK, you can save your .cue file for later use. Later, in Write Mode in ImgBurn, click the Folder button up there by "Source" and select your .cue file. All your MP3s will automatically be loaded up and queued for write.

ImgBurn Main Screen with my CUE file ready to burn 

Now, when I burn my CD, I see this on my car's dashboard, which gives me GREAT satisfaction, as it should you, Dear Reader.

 CD with CD-Text in my car

Doesn't this just make you want to burn some CDs with CD-Text and go see where they work? It did me. Enjoy.

NOTE: iTunes includes some CD-Text support, but it doesn't always work. Apparently there are two ways to burn CD-Text and iTunes doesn't seem to do the more compatible one. I like a little more control and that's why I prefer ImgBurn.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.