Scott Hanselman

Easily adding Security Headers to your ASP.NET Core web app and getting an A grade

March 24, '20 Comments [17] Posted in ASP.NET | Linux
Sponsored By

Well that sucks.

Score of F on SecurityHeaders.com

That's my podcast website with an F rating from SecurityHeaders.com. What's the deal? I took care of this months ago!

Turns out, recently I moved from Windows to Linux on Azure.

If I am using IIS on Windows, I can (and did) make a section in my web.config that looks something like this.

Do note that I've added a few custom things and you'll want to make sure you DON'T just copy paste this. Make yours, yours.

Note that I've whitelisted a bunch of domains to make sure my site works. Also note that I have a number of "unsafe-inlines" that are not idea.

<configuration>
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Strict-Transport-Security" value="max-age=31536000"/>
<add name="X-Content-Type-Options" value="nosniff"/>
<add name="X-Xss-Protection" value="1; mode=block"/>
<add name="X-Frame-Options" value="SAMEORIGIN"/>
<add name="Content-Security-Policy" value="default-src https:; img-src * 'self' data: https:; style-src 'self' 'unsafe-inline' www.google.com platform.twitter.com cdn.syndication.twimg.com fonts.googleapis.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' www.google.com cse.google.com cdn.syndication.twimg.com platform.twitter.com platform.instagram.com www.instagram.com cdn1.developermedia.com cdn2.developermedia.com apis.google.com www.googletagservices.com adservice.google.com securepubads.g.doubleclick.net ajax.aspnetcdn.com ssl.google-analytics.com az416426.vo.msecnd.net/;"/>
<add name="Referrer-Policy" value="no-referrer-when-downgrade"/>
<add name="Feature-Policy" value="geolocation 'none';midi 'none';notifications 'none';push 'none';sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'self';vibrate 'none';fullscreen 'self';payment 'none';"/>
<remove name="X-Powered-By" />
<remove name="X-AspNet-Version" />
<remove name="Server" />
</customHeaders>
</httpProtocol>
...

But, if I'm NOT using IIS - meaning I'm running my ASP.NET app in a container or on Linux - this will be ignored. Since I recently moved to Linux, I assumed (my bad for no tests here) that it would just work.

My site is hosted on Azure App Service for Linux, so I want these headers to be output the same way. There are several great choices in the form of Open Source NuGet libraries to help. If I use the ASP.NET Core middleware pipeline then these headers will be output and work the SAME on both Windows AND Linux.

I'll be using the NWebsec Security Libraries for ASP.NET Core. They offer a simple fluent way to add the headers I want.

TO BE CLEAR: Yes I, or you, can add these headers manually with AddHeader but these simple libraries ensure that our commas and semicolons are correct. They also offer a strongly typed middleware that is fast and easy to use.

Taking the same web.config above and translating it to Startup.cs's Configure Pipeline with NWebSec looks like this:

app.UseHsts(options => options.MaxAge(days: 30));
app.UseXContentTypeOptions();
app.UseXXssProtection(options => options.EnabledWithBlockMode());
app.UseXfo(options => options.SameOrigin());
app.UseReferrerPolicy(opts => opts.NoReferrerWhenDowngrade());

app.UseCsp(options => options
.DefaultSources(s => s.Self()
.CustomSources("data:")
.CustomSources("https:"))
.StyleSources(s => s.Self()
.CustomSources("www.google.com","platform.twitter.com","cdn.syndication.twimg.com","fonts.googleapis.com")
.UnsafeInline()
)
.ScriptSources(s => s.Self()
.CustomSources("www.google.com","cse.google.com","cdn.syndication.twimg.com","platform.twitter.com" ... )
.UnsafeInline()
.UnsafeEval()
)
);

There is one experimental HTTP header that NWebSec doesn't support (yet) called Feature-Policy. It's a way that your website can declare at the server-side "my site doesn't allow use of the webcam." That would prevent a bad guy from injecting local script that uses the webcam, or some other client-side feature.

I'll do it manually both to make the point that I can, but also that you aren't limited by your security library of choice.

NOTE: Another great security library is Andrew Lock's NetEscapades that includes Feature-Policy as well as some other great features.

Here's my single Middleware that just adds the Feature-Policy header to all responses.

//Feature-Policy
app.Use(async (context, next) =>
{
context.Response.Headers.Add("Feature-Policy", "geolocation 'none';midi 'none';notifications 'none';push 'none';sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'self';vibrate 'none';fullscreen 'self';payment 'none';");
await next.Invoke();
});

Now I'll commit, build, and deploy (all automatic for me using Azure DevOps) and scan the site again:

Score of A on SecurityHeaders.com

That was pretty straightforward and took less than an hour. Your mileage may vary but that's the general idea!


Sponsor: Protect your apps from reverse engineering and tampering with PreEmptive, makers of Dotfuscator. Dotfuscator has been in-the-box with Microsoft Visual Studio since 2003. Mention HANSELMAN for savings on a professional license!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

CSI: The case of the missing WAV audio files on the FAT32 SD Card

March 12, '20 Comments [16] Posted in Linux | Open Source | Win10
Sponsored By

Scott Hanselman and Altered Carbon's Chris Conner who plays PoeBuckle up kids, as this is a tale. As you may know, I have a lovely podcast at https://hanselminutes.com. You should listen.

Recently through an number of super cool random events I got the opportunity to interview actor Chris Conner who plays Poe on Altered Carbon. I'm a big fan of the show but especially Chris. You should watch the show because Poe is a joy and Chris owns every scene, and that's with a VERY strong cast.

I usually do my interviews remotely for the podcast but I wanted to meet Chris and hang out in person so I used my local podcasting rig which consists of a Zoom H6 recorder.

I have two Shure XLR mics, a Mic stand, and the Zoom. The Zoom H6 is a very well though of workhorse and I've used it many times before when recording shows. It's not rocket surgery but one should always test their things.

I didn't want to take any chances to I picked up a 5 pack of 32GIG high quality SD Cards. I put a new one in the Zoom, the Zoom immediately recognized the SD Card so I did a local recording right there and played it back. Sounds good. I played it back locally on the Zoom and I could hear the recording from the Zoom's local speaker. It's recording the file in stereo, one side for each mic. Remember this for later.

I went early to the meet and set up the whole recording setup. I hooked up a local monitor and tested again. Records and plays back locally. Cool. Chris shows up, we recorded a fantastic show, he's engaged and we're now besties and we go to Chipotle, talk shop, Sci-fi, acting, AIs, etc. Just a killer afternoon all around.

I head home and pull out the SD Card and put it into the PC and I see this. I almost vomit. I get lightheaded.

Infinite folders recursing. They are empty.

I've been recording the show for over 730 episodes over 14 years and I've never lost a show. I do my homework - as should you. I'm reeling. Ok, breathe. Let's work the problem.

Right click the drive, check properties. Breathe. This is a 32 gig drive, but Windows sees that it's got 329 MB used. 300ish megs is the size of a 30 minute long two channel WAV file. I know this because I've looked at 300 meg files for the last several hundred shows. Just like you might know roughly the size of a JPEG your camera makes. It's a thing you know.

A 32gig (really 29.1GB) drive with 329 MB used

Command line time. List the root directory. Empty. Check it again but "show all files," weird, there's a Mac folder there but maybe the SD Card was preformatted on a Mac.

Interesting Plot Point - I didn't format the SD card. I use it as it came out of the packaging from Amazon. It came preformatted and I accepted it. I tested it and it worked but I didn't "install my own carpet." I moved in to the house as-is.

What about a little "show me all folders from here down" action? Same as I saw in Windows Explorer. The root folder has another subfolder which is itself. It's folder "Inception" with no Kick!

G:\>dir /a
Volume in drive G has no label.
Volume Serial Number is 0403-0201
Directory of G:\
03/12/2020 12:29 PM <DIR>
03/13/2020 12:44 PM <DIR> System Volume Information
0 File(s) 0 bytes
2 Dir(s) 30,954,225,664 bytes free
G:\>dir /s
Volume in drive G has no label.
Volume Serial Number is 0403-0201
Directory of G:\
03/12/2020 12:29 PM <DIR>
0 File(s) 0 bytes
Directory of G:\
03/12/2020 12:29 PM <DIR>
0 File(s) 0 bytes
IT GOES FOREVER

Ok, the drive thinks there's data but I can't see it. I put the SD card back in the Zoom and try to play it back.

The Zoom can see folders and files AND the interview itself. And the Zoom can play it back. The Zoom is an embedded device with an implementation of the FAT32 file system and it can read it, but Windows can't. Can Linux? Can a Mac?

Short answer. No.

Hacky Note: Since the Zoom can see and play the file and it has a headphone/monitor jack, I could always plug in an analog 1/8" headphone cable to a 1/4" input on my Peavy PV6 Mixer and rescue the audio with some analog quality loss. Why don't I use the USB Audio out feature of the Zoom H6 and play the file back over a digital cable, you ask? Because the Zoom audio player doesn't support that. It supports three modes - SD Card Reader (which is a pass through to Windows and shows me the recursive directories and no files), an Audio pass-through which lets the Zoom look like an audio device to Windows but doesn't show the SD card as a drive or allow the SD Card to be played back over the digital interface, or its main mode where it's recording locally.

It's Forensics Time, Kids.

We have an 32 SD Card - a disk drive as it were - that is standard FAT32 formatted, that has 300-400 megs of a two-channel (Chris and I had two mics) WAV file that was recorded locally by the Zoom H6  audio reorder and I don't want too lose it or mess it up.

I need to take a byte for byte image of what's on the SD Card so I can poke and it and "virtually" mess with with it, change it, fix it, try again, without changing the physical.

"dd" is a command-line utility with a rich and storied history going back 45 years. Even though it means "Data Definition" it'll always be "disk drive" I my head.

How to clone a USB Drive or SD Card to an IMG file on Windows

I have a copy of dd for Windows which lets me get a byte for byte stream/file that represents this SD Card. For example I could get an entire USD device:

dd if=\\?\Device\Harddisk1\Partition0 of=c:\temp\usb2.img bs=1M --size --progress

I need to know the Harddisk number and Partition number as you can see above. I usually use diskpart for this.

>diskpart

Microsoft DiskPart version 10.0.19041.1

Copyright (C) Microsoft Corporation.
On computer: IRONHEART

DISKPART> list disk

Disk ### Status Size Free Dyn Gpt
-------- ------------- ------- ------- --- ---
Disk 0 Online 476 GB 0 B *
Disk 1 Online 1863 GB 0 B *
Disk 2 Online 3725 GB 0 B
Disk 3 Online 2794 GB 0 B *
Disk 8 Online 29 GB 3072 KB

DISKPART> select disk 8

Disk 8 is now the selected disk.

DISKPART> list part

Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 Primary 29 GB 4096 KB

Looks like it's Disk 8 Partition 1 on my system. Let's get it all before I panic.

dd if=\\?\Device\Harddisk8\Partition1 of=c:\temp\ZOMG.img bs=1M --size --progress

IF and OF are input file and output file, and I will do it for the whole size of the SD Card. It's likely overkill though as we'll see in a second.

This file ended up being totally massive and hard to work with. Remember I needed just the first 400ish megs? I'll chop of just that part.

dd if=ZOMG.img of=SmallerZOMG.img bs=1M count=400

What is this though? Remember it's an image of a File System. It just bytes in a file. It's not a WAV file or a THIS file or a THAT file. I mean, it is if we decide it is, but in fact, a way to think about it is that it's a mangled envelope that is dark when I peer inside it. We're gonna have to feel around and see if we can rebuild a sense of what the contents really are.

Importing Raw Bytes from an IMG into Audition or Audacity

Both Adobe Audition and Audacity are audio apps that have an "Import RAW Data" feature. However, I DO need to tell Audition how to interpret it. There's lots of WAV files out there. How many simples were there? 1 channel? 2 channel? 16 bit or 32 bit? Lots of questions.

image

Can I just import this 4 gig byte array of a file system and get something?

Looks like something. You can see that the first part there is likely the start of the partition table, file system headers, etc. before audio data shows up. Here's importing as 2 channel.

image

I can hear voices but they sound like chipmunks and aren't understandable. Something is "doubled." Sample rate? No, I double checked it.

Here's 1 channel raw data import even though I think it's two.

Raw audio data 1 channel

Now THIS is interesting. I can hear audio at normal speed of us talking (after the preamble) BUT it's only a syllable at a time, and then a quieter version of the same syllable repeats. I don't want to (read: can't really) reassemble a 30 min interview from syllables, right?

Remember when I said that the Zoom H6 records a two channel file with one channel per mic? Not really. It records ONE FILE PER CHANNEL. A whateverL.wav and a whateverR.wav. I totally forgot!

This "one channel" file above is actually the bytes as they were laid down on disk, right? It's actually two files written simultaneously, a few kilobytes at a time, L,R,L,R,L,R. And here I am telling my sound software to treat this "byte for byte file system dump" as one file. It's two that were made at the same time.

It's like the Brundlefly. How do I tease it apart? Well I can't treat the array as a raw file anymore, it's not. And I want (really don't have the energy yet) to write my own little app to effectively de-interlace this image. I also don't know if the segment size is perfectly reliable or if it varies as the Zoom recorded.

NOTE: Pete Brown has written about RIFF/WAV files from Sound Devices records having an incorrect FAT32 bit set. This isn't that, but it's in the same family and is worth noting if you ever have an issue with a Broadcast Wave File getting corrupted or looking encrypted.

Whole helping me work this issue, Pete Brown tweeted a hexdump of the Directory Table so you can see the Zoom0001, Zoom0002, etc directories there in the image.

Hexdump of FAT32 Directory Table shows that there ARE directories

Let me move into Ubuntu on my Windows machine running WSL. Here I can run fdisk and get some sense of what this Image of the bad SD Card is. Remember also that I hacked off the first 0-400 Megs but this IMG file thinks it's a 32gig drive, because it is. It's just that's been aggressively truncated.

$ fdisk -u -l SmallerZOMG.img
Disk SmallerZOMG.img: 400 MiB, 419430400 bytes, 819200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
SmallerZOMG.img1 8192 61157375 61149184 29.2G c W95 FAT32 (LBA)

Maybe I can "mount" this IMG? I make a folder on Ubuntu/WSL2 called ~/recovery. Yikes, ok there's nothing there. I can take the sector size 512 times the Start block of 8192 and use that as the offset.

sudo mount -o loop,offset=4194304 SmallerShit.img recover/
$ cd recover/
$ ll
total 68
drwxr-xr-x 4 root root 32768 Dec 31 1969 ./

Ali Mosajjal thinks perhaps "they re-wrote the FAT32 structure definition and didn't use a standard library and made a mistake," and Leandro Pereria postulates "what could happen is that the LFN (long file name) checksum is invalid and they didn't bother filling in the 8.3 filename... so that complying implementations of VFAT tries to look at the fallback 8.3 name, it's all spaces and figures out "it's all padding, move along."

Ali suggested running dosfsck on the mounted image and you can see again that the files are there, but there's like 3 root entries? Note I've done a cat of /proc/mounts to see the loop that my img is mounted on so I can refer to it in the dosfsck command.

$ sudo dosfsck -w -r -l -a -v -t /dev/loop3
fsck.fat 4.1 (2017-01-24)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID " "
Media byte 0xf8 (hard disk)
512 bytes per logical sector
32768 bytes per cluster
1458 reserved sectors
First FAT starts at byte 746496 (sector 1458)
2 FATs, 32 bit entries
3821056 bytes per FAT (= 7463 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8388608 (sector 16384)
955200 data clusters (31299993600 bytes)
63 sectors/track, 255 heads
8192 hidden sectors
61149184 sectors total
Checking file /
Checking file /
Checking file /
Checking file /System Volume Information (SYSTEM~1)
Checking file /.
Checking file /..
Checking file /ZOOM0001
Checking file /ZOOM0002
Checking file /ZOOM0003
Checking file /ZOOM0001/.
Checking file /ZOOM0001/..
Checking file /ZOOM0001/ZOOM0001.hprj (ZOOM00~1.HPR)
Checking file /ZOOM0001/ZOOM0001_LR.WAV (ZOOM00~1.WAV)
Checking file /ZOOM0002/.
Checking file /ZOOM0002/..
Checking file /ZOOM0002/ZOOM0002.hprj (ZOOM00~1.HPR)
Checking file /ZOOM0002/ZOOM0002_Tr1.WAV (ZOOM00~1.WAV)
Checking file /ZOOM0002/ZOOM0002_Tr2.WAV (ZOOM00~2.WAV)
Checking file /ZOOM0003/.
Checking file /ZOOM0003/..
Checking file /ZOOM0003/ZOOM0003.hprj (ZOOM00~1.HPR)
Checking file /ZOOM0003/ZOOM0003_Tr1.WAV (ZOOM00~1.WAV)
Checking file /ZOOM0003/ZOOM0003_Tr2.WAV (ZOOM00~2.WAV)
Checking file /System Volume Information/.
Checking file /System Volume Information/..
Checking file /System Volume Information/WPSettings.dat (WPSETT~1.DAT)
Checking file /System Volume Information/ClientRecoveryPasswordRotation (CLIENT~1)
Checking file /System Volume Information/IndexerVolumeGuid (INDEXE~1)
Checking file /System Volume Information/AadRecoveryPasswordDelete (AADREC~1)
Checking file /System Volume Information/ClientRecoveryPasswordRotation/.
Checking file /System Volume Information/ClientRecoveryPasswordRotation/..
Checking file /System Volume Information/AadRecoveryPasswordDelete/.
Checking file /System Volume Information/AadRecoveryPasswordDelete/..
Checking for bad clusters.

We can see  them, but can't get at them with the vfat file system driver on Linux or with Windows.

The DUMP.exe util as part of mtools for Windows is amazing but I'm unable to figure out what is wrong in the FAT32 file table. I can run minfo on the Linux command land telling it to skip 8192 sectors in with the @@offset modifier:

$ minfo -i ZOMG.img@@8192S
device information:
===================
filename="ZOMG.img"
sectors per track: 63
heads: 255
cylinders: 3807

mformat command line: mformat -T 61149184 -i ZOMG.img@@8192S -h 255 -s 63 -H 8192 ::

bootsector information
======================
banner:" "
sector size: 512 bytes
cluster size: 64 sectors
reserved (boot) sectors: 1458
fats: 2
max available root directory slots: 0
small size: 0 sectors
media descriptor byte: 0xf8
sectors per fat: 0
sectors per track: 63
heads: 255
hidden sectors: 8192
big size: 61149184 sectors
physical drive id: 0x80
reserved=0x0
dos4=0x29
serial number: 04030201
disk label=" "
disk type="FAT32 "
Big fatlen=7463
Extended flags=0x0000
FS version=0x0000
rootCluster=2
infoSector location=1
backup boot sector=6

Infosector:
signature=0x41615252
free clusters=944648
last allocated cluster=10551

Ok, now we've found yet ANOTHER way to mount this corrupted file system. With mtools we'll use mdir to list the root directory. Note there is something wrong enough that I have to set mtools_skip_check=1 to ~/.mtoolsrc and continue.

$ mdir -i ZOMG.img@@8192S ::
Total number of sectors (61149184) not a multiple of sectors per track (63)!
Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
$ pico ~/.mtoolsrc
$ mdir -i ZOMG.img@@8192S ::
Volume in drive : is
Volume Serial Number is 0403-0201
Directory for ::/

<DIR> 2020-03-12 12:29
1 file 0 bytes
30 954 225 664 bytes free

Same result. I can run mdu and see just a few folders. Note the ZOOMxxxx ones are missing here

$ mdu -i ZOMG.img@@8192S ::
::/System Volume Information/ClientRecoveryPasswordRotation 1
::/System Volume Information/AadRecoveryPasswordDelete 1
::/System Volume Information 5
::/ 6

Now, ideally I want to achieve two things here.

  • Know WHY it's broken and exactly WHAT is wrong.
    • There's a nameless root directory here and I lack the patience and skill to manually hexdump and patch it.
  • Be able to copy the files out "normally" by mounting the IMG and, well, copying them out.

UPDATE #1 - I'm back after a few minutes of thinking again.

If I use mmls from Sleuthkit, I can see this.

$ mmls HolyShit.img
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

Slot Start End Length Description
000: Meta 0000000000 0000000000 0000000001 Primary Table (#0)
001: ------- 0000000000 0000008191 0000008192 Unallocated
002: 000:000 0000008192 0061157375 0061149184 Win95 FAT32 (0x0c)

If I do the 512*8192 offset again and visualize the FAT32 table in Hexdump/xxd like this:

xxd -seek 4194304 ZOMG.img  | more
00400000: eb00 9020 2020 2020 2020 2000 0240 b205 ... ..@..
00400010: 0200 0000 00f8 0000 3f00 ff00 0020 0000 ........?.... ..
00400020: 0010 a503 271d 0000 0000 0000 0200 0000 ....'...........
00400030: 0100 0600 0000 0000 0000 0000 0000 0000 ................
00400040: 8000 2901 0203 0420 2020 2020 2020 2020 ..)....
00400050: 2020 4641 5433 3220 2020 0000 0000 0000 FAT32 ......
00400060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00400070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00400080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00400090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
004000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
004000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
004000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................

I can see I seek'ed to the right spot, as the string FAT32 is just hanging out. Maybe I can clip out this table and visualize it in a better graphical tool.

I could grab a reasonable (read: arbitrary) chunk from this offset and put it in a very small manageable file:

dd if=ZOMG.img ibs=1 skip=4194304 count=64000 > another.img

And then load it in dump.exe on Windows which is really a heck of a tool. It seems to be thinking thinking there's multiple FAT Root Entries (which might be why I'm seeing this weird ghost root). Note the "should be" parts as well.

FAT Root Entry (non LFN) (0x00000000)
Name: ···
Extension:
Attribute: 0x00
FAT12:reserved: 02 40 B2 05 02 00 00 00 00 F8
FAT32:reserved: 02
FAT32:creation 10th: 0x40
FAT32:creation time: 0x05B2
FAT32:creation date: 0x0002
FAT32:last accessed: 0x0000
FAT32:hi word start cluster: 0xF800
Time: 0x0000 (00:00:00) (hms)
Date: 0x003F (1980/01/31) (ymd)
Starting Cluster: 0x00FF (0xF80000FF)
File Size: 8192

FAT Root Entry (non LFN) (0x00000020)
Name: ····'···
Extension: ···
Attribute: 0x00
FAT12:reserved: 02 00 00 00 01 00 06 00 00 00
FAT32:reserved: 02
FAT32:creation 10th: 0x00
FAT32:creation time: 0x0000
FAT32:creation date: 0x0001
FAT32:last accessed: 0x0006
FAT32:hi word start cluster: 0x0000
Time: 0x0000 (00:00:00) (hms)
Date: 0x0000 (1980/00/00) (ymd)
Starting Cluster: 0x0000 (0x00000000) <--- should be 0x0002 or higher.
File Size: 0

FAT Root Entry (non LFN) (0x00000040)
Name: ··)····
Extension:
Attribute: 0x20 Archive
FAT12:reserved: 20 20 20 20 20 20 46 41 54 33
FAT32:reserved: 20
FAT32:creation 10th: 0x20
FAT32:creation time: 0x2020
FAT32:creation date: 0x2020
FAT32:last accessed: 0x4146
FAT32:hi word start cluster: 0x3354
Time: 0x2032 (04:01:18) (hms)
Date: 0x2020 (1996/01/00) (ymd)
Starting Cluster: 0x0000 (0x33540000)
File Size: 0

FAT Root Entry (non LFN) (0x00000060)
Name: ········
Extension: ···
Attribute: 0x00
FAT12:reserved: 00 00 00 00 00 00 00 00 00 00
FAT32:reserved: 00
FAT32:creation 10th: 0x00
FAT32:creation time: 0x0000
FAT32:creation date: 0x0000
FAT32:last accessed: 0x0000
FAT32:hi word start cluster: 0x0000
Time: 0x0000 (00:00:00) (hms)
Date: 0x0000 (1980/00/00) (ymd)
Starting Cluster: 0x0000 (0x00000000) <--- should be 0x0002 or higher.
File Size: 0

FAT Root Entry (non LFN) (0x00000080)
Name: ········
Extension: ···
Attribute: 0x00
FAT12:reserved: 00 00 00 00 00 00 00 00 00 00
FAT32:reserved: 00
FAT32:creation 10th: 0x00
FAT32:creation time: 0x0000
FAT32:creation date: 0x0000
FAT32:last accessed: 0x0000
FAT32:hi word start cluster: 0x0000
Time: 0x0000 (00:00:00) (hms)
Date: 0x0000 (1980/00/00) (ymd)
Starting Cluster: 0x0000 (0x00000000) <--- should be 0x0002 or higher.
File Size: 0

FAT32 Info Block (0x00000000)
sig: 0x209000EB (' ···') [1] <--- should be 0x41615252.
reserved:
00000004 20 20 20 20 20 20 20 00-02 40 B2 05 02 00 00 00 .........@......
00000014 00 F8 00 00 3F 00 FF 00-00 20 00 00 00 10 A5 03 ....?...........
00000024 27 1D 00 00 00 00 00 00-02 00 00 00 01 00 06 00 '...............
00000034 00 00 00 00 00 00 00 00-00 00 00 00 80 00 29 01 ..............).
00000044 02 03 04 20 20 20 20 20-20 20 20 20 20 20 46 41 ..............FA
00000054 54 33 32 20 20 20 00 00-00 00 00 00 00 00 00 00 T32.............

The most confusing part is that the FAT32 signature - the magic number is always supposed to be 0x41615252. Google that. You'll see. It's a hardcoded signature but maybe I've got the wrong offset and at that point all bets are off.

So do I have that? I can search a binary file for Hex values with a combo of xxd and grep. Note the byte swap:

xxd another.img  | grep "6141"
00000200: 5252 6141 0000 0000 0000 0000 0000 0000 RRaA............
00000e00: 5252 6141 0000 0000 0000 0000 0000 0000 RRaA............

Just before this is 55 AA which is the last two bytes of the 64 byte partition table. mm

Now do I have two FAT32 info blocks and three Root Entries? I'm lost. I want to dump the directory entries.

What does fsstat say about the Root Directory?

File System Layout (in sectors)
Total Range: 0 - 61149183
* Reserved: 0 - 1457
** Boot Sector: 0
** FS Info Sector: 1
** Backup Boot Sector: 6
* FAT 0: 1458 - 8920
* FAT 1: 8921 - 16383
* Data Area: 16384 - 61149183
** Cluster Area: 16384 - 61149183
*** Root Directory: 16384 - 16447

I'll update this part as I learn more. I'm exhausted. Someone will likely read this and be like "you dork, seek HERE" and there's the byte that's wrong in the file system. That LFN (long file name) has no short one, etc" and then I'll know.

UPDATE #2:

I skyped with Ali and we think we know what's up. He suggested I format the SD Card, record the same 3 shows (two test WAVs and one actual one) and then make an image of the GOOD disk to remove variables. Smart guy!

We then took the first 12 megs or so of the GOOD.img and the BAD.img and piped them through xxd into HEX, then used Visual Studio Code to diff them.

We can now visualize on the left what a good directory structure looks like and the right what a bad one looks like. Seems like I do have two recursive root directories with a space for the name.

Bad directory structures

Now if we wanted we could manually rewrite a complete new directory entry and assign our orphaned files to it.

That's what I would do if I was hired to recover data.

7zip all the things

Here's where it gets weird and it got so weird that both Pete Brown and I were like, WELL. THAT'S AMAZING.

On a whim I right-clicked the IMG file and opened it in 7zip and saw this.

Opening an img in 7zip

See that directory there that's a nothing? A space? A something. It has no Short Name. It's an invalid entry but 7zip is cool with it. Let's go in. Watch the path and the \\. That's a path separator, nothing, and another path separator. That's not allowed or OK but again, 7zip is chill.

The recovered files

I dragged the files out and they're fine! The day is saved.

The moral? There are a few I can see.

  • Re-format the random SD cards you get from Amazon specifically on the device you're gonna use them.
  • FAT as a spec has a bunch of stuff that different "drivers" (Windows, VFAT, etc) may ignore or elide over or just not implement.
  • I've got 85% of the knowledge I need to spelunk something like this but that last 15% is a brick wall. I would need more patience and to read more about this.
  • Knowing how to do this is useful for any engineer. It's the equivalent of knowing how to drive a stick shift in an emergency even if you usually use Lyft.
    • I'm clearly not an expert but I do have a mental model that includes (but not limited to) bytes on the physical media, the file system itself, file tables, directory tables, partition tables, how they kinda work on Linux and Windows.
    • I clearly hit a wall as I know what I want to do but I'm not sure the next step.
      • There's a bad Directory Table Entry. I want to rename it and make sure it's complete and to spec.
  • 7zip is amazing. Try it first for basically everything.

Ideally I'd be able to update this post with exactly what byte is wrong and how to fix it. Thanks to Ali, Pete, and Leandro for playing with me!

Your thoughts? (If you made it this far the truncated IMG of the 32 gig SD is here (500 megs) but you might have to pad it out with zeros to make some tools like it.

Oh, and listen to https://hanselminutes.com/ as the interview was great and it's up now!


Sponsor: Have you tried developing in Rider yet? This fast and feature-rich cross-platform IDE improves your code for .NET, ASP.NET, .NET Core, Xamarin, and Unity applications on Windows, Mac, and Linux.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to SSH into a Windows 10 Machine from Linux OR Windows OR anywhere

March 10, '20 Comments [12] Posted in Linux | Open Source | Win10
Sponsored By

I've been shushing all over the place lately. I SSH into Linux from Windows using the built-in OpenSSH Client that Windows 10 has shipped for years that you didn't know about. ;) You don't need Putty to SSH with Windows (unless it makes you happy, then putty on, my friend.)

Adding OpenSSH Server to Windows

From an Administrative PowerShell I'll see what OpenSSH stuff I have enabled. I can also do this with by typing "Windows Features" from the Start Menu.

> Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'

Name : OpenSSH.Client~~~~0.0.1.0
State : Installed

Name : OpenSSH.Server~~~~0.0.1.0
State : NotPresent

Looks like I have the OpenSSH client stuff but not the server. I can SSH from Windows, but not to.

I'll add it with a similar command with the super weirdo but apparently necessary version thing at the end:

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Starting SSHD on Windows as a Service

Once this has finished (and you can of course run this with OpenSSH.Client as well to get both sides if you hadn't) then you can start the SSH server (as a Windows Service) with this, then make sure it's running.

Start-Service sshd
Get-Service sshd

Since it's a Windows Service you can see it as "OpenSSH SSH Server" in services.msc as well as set it to start automatically on Startup if you like. You can do that again, from PowerShell if you prefer

Set-Service -Name sshd -StartupType 'Automatic'

Remember that we SSH over port 22 so you'll have a firewall rule incoming on 22 at this point. It's up to you to be conscious of security. Maybe you only allow SSHing into your Windows machine with public keys (no passwords) or maybe you don't mind. Just be aware, it's on you, not me.

Now, from any Linux (or Windows) machine I can SSH into my Windows machine like a pro! Note I'm using the .local domain suffix to make sure I don't get a machine on my VPN (staying in my local subnet)

$ ssh scott@ironheart.local
Microsoft Windows [Version 10.0.19041.113]
(c) 2020 Microsoft Corporation. All rights reserved.

scott@IRONHEART C:\Users\scott>pwsh
PowerShell 7.0.0
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/powershell
Type 'help' to get help.

Loading personal and system profiles took 1385ms.
⚡ scott@IRONHEART>

Note that when I SSH'ed into Windows I got the default cmd.exe shell. Remember also that there's a difference between a console, a terminal, and a shell! I can ssh with any terminal into any machine and end up at any shell. In this case, the DEFAULT was cmd.exe, which is suboptimal.

Configuring the default shell for OpenSSH in Windows

On my server (the Windows machine I'm SSHing into) I will set a registry key to set the default shell. In this case, I'll use open source cross platform PowerShell Core. You can use whatever makes you happy.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\PowerShell\7\pwsh.exe" -PropertyType String -Force

Now when I ssh into my Windows machine from elsewhere (even my iPad!) I get the shell I want:

$ ssh scott@ironheart.local
PowerShell 7.0.0
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/powershell
Type 'help' to get help.

Loading personal and system profiles took 1854ms.
⚡ scott@IRONHEART>

Even better if I wanted to add a menu item (profile) to my Windows Terminal with an entry for my Windows Machine that would automatically log me into it from elsewhere using public keys, I could do that also!

Additionally, now that this is set up I can use WinSCP (available on the Window Store) as well as scp (Secure Copy) to transfer files.

Of course you can also use WinRM or PowerShell Remoting over SSH but for my little internal network I've found this mechanism to be simple and clean. Now my shushing around is non-denominational!


Sponsor: Have you tried developing in Rider yet? This fast and feature-rich cross-platform IDE improves your code for .NET, ASP.NET, .NET Core, Xamarin, and Unity applications on Windows, Mac, and Linux.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to set up a tab profile in Windows Terminal to automatically SSH into a Linux box

February 25, '20 Comments [7] Posted in Linux | Win10
Sponsored By

A lovely list of Profiles in Windows TerminalBy now I hope you've installed Windows Terminal. If not, go do that, I'll wait. It's time.

You may also have customize your settings. If you tried terminal a few versions ago and haven't gone back in, it's also time to let the Windows Terminal generate you a nice fresh new profiles.json (settings file). It's OK to zero-out/delete yours. Windows Terminal will regenerate it when it next starts.

I have a number of things in my Terminal dropdown. It looks like this.

However, I'd like to be able to have a profile that ssh's into Linux machines that I use regularly. Perhaps those remote machine can have their own cool menu item? Let's see what that would look like and how we'd do it.

Adding a New Profile to Windows Terminal

Click the down arrow in the Windows Terminal top tab bar. Note that there are a ton of great and useful settings so explore the Settings Schema, and when you're editing the settings make sure that Visual Studio Code is set as your default handler for .json files. That's important because the Windows Terminal settings profile.json includes a JSON Schema and you'll want your settings to have autocomplete/intellisense. This will make it easier to create and discover new settings.

I'll add a profile to the "profiles" array. To start, and to learn, let's add the simplest possible profile! I'm just adding the { } as an array item in the larger profiles [] and giving it a name.

"profiles": [
{
"name": "This is a name"
},

This will make a new menu item in Windows Terminal with the same name. It will have no icon and it'll launch cmd.exe as the default shell because I didn't set any other command line! It I add it at the top (as the first) item in the profiles array it'll also appear first in the menu and have the hotkey Ctrl+Shift+1.

This is lame, so let's add more. I'll add a tabTitle and a commandline.

    {
"name": "This is a name",
"tabTitle": "This is a tab title",
"commandline": "powershell"
},

This menu item will appear as "This is a name" in the menu, but the the tab will be called "This is a tab title." It'll launch powershell. Note that I didn't include .exe even though I could have. I wanted to make sure you're clear that Windows Terminal is basically just called Process.Start so you can set a profile tab to call anything in the PATH, or you can be explicit. I could also add "startingDirectory" and a bunch of other options.

Since I can call anything in the PATH, what else can I get away with?

Using OpenSSH on Windows

You may not have heard but OpenSSH has shipped in Windows for a few years now. That means that a lot of the utilities that you might have installed Putty for are already available in Windows. You can open an admin PowerShell and run one command to ensure OpenSSH's client apps are there:

Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0

This installs the the client, but there's an optional server as well if you'd like.

I'm going to focus only on the client. Skip to the next area if you want to do your SSH'ing from Linux, not Windows.

Here's what's installed in c:\windows\System32\OpenSSH

OpenSSH utilties

Here we've got sftp, scp, and most importantly, ssh.exe and ssh-agent. Since ssh is in the PATH when it's installed with Windows I can change my Windows Terminal profile to look like this and log into my Raspberry Pi 4.

{
"name": "ssh hanselPi4",
"tabTitle": "HanselPi4",
"commandline": "ssh pi@hanselpi4"
},

Note in this screenshot I've got the ssh connection listed at the top, and when I click on it it opens ssh.exe and prompts me for a password. I have no ssh keys on my system that would enable auto-login, hence the password is needed.

ssh'ing into a Raspberry Pi

Automatically SSH'ing/logging into a Linux machine from a Windows Terminal profile

Now this is important, so pay attention. If you have WSL or WSL already on your machine you can certainly just use the SSH keys and utils that are included in your preferred Linux distro.

In that case, your command line in your Windows Terminal profile would be something like:

wsl ssh pi@hanselpi4

or if you want a specific distro, you can launch a distro and ssh from there.

wsl -d Ubuntu-18.04 ssh pi@hanselpi4

If you know Linux, then you're familiar with how to set up your public keys to allow this. However, most folks think you need Putty or some 3rd party tool to do this on Windows so I'll focus on how do to that here.

I want to be able to type "ssh pi@hanselpi4" from my Windows machine and automatically be logged in. More specifically I want to click the profile and have it Just Work.

I will

  • Make a key on my Window machine. The FROM machine, in this case, Windows. Then I want to ssh FROM here TO the remote Linux machine.
  • Tell the Linux machine (by transferring it over) about the public piece of my key and add it to a specific user's allowed_keys.

I'll run ssh-keygen to make a key from my command line on Windows. I just hit enter to generate it but you can make your own filename if you want, just use the full path and make sure you keep track of where things are. Defaults are usually best.

>ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\scott/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in hanselpi4.
Your public key has been saved in hanselpi4.pub.

Remember the path is c:\users\yourname because that's the Windows equivalent of the ~ home folder and the keys are in c:\users\yourname\.ssh.

Now I want to transfer what's in id_rsa.pub over to my Raspberry Pi. You can scp (secure copy) if you want, but it's best to append the key to the authorized_keys file on the destination machine.

NOTE: I'm type'ing (cat on Linux is type on Windows) that text file out and piping it into SSH where I login that remote machine with the user pi and I then cat (on the Linux side now) and append >> that text to the .ssh/authorized_keys folder. The ~ folder is implied but could be added if you like.

Run this command once on Windows to output your key and pipe it over to, and append to, the right file on your remote Linux machine. You'll be prompted for your password once.

type c:\users\scott\.ssh\id_rsa.pub | ssh pi@hanselpi4 'cat >> .ssh/authorized_keys'

Make sure you understand what's happening in the line above.

Adding a profile Icon - the raspberry on top

At this point I can click the menu item in Windows Terminal and automatically be ssh'ed/logged into the remote terminal. But, scandalously, the Terminal menu item has no icon. This is clearly unacceptable M$sft sucks, right? I'll go get a nice 32x32 Raspberry Pi Icon and put it somewhere. You might put yours in a Dropbox or OneDrive so they are available everywhere you go.

Now my profile looks like this:

"profiles": [
{
"name": "ssh hanselPi4",
"tabTitle": "HanselPi4",
"commandline": "ssh pi@hanselpi4",
"icon": "c:/users/scott/downloads/icons8-raspberry-pi-32.png"
},

How lovely is this?

A nice Raspbery Pi icon in my profile

Looks good, has a nice title and icon, and I can use a hotkey to automatically SSH into my remote machine.

One final note, you've already got the Azure Cloud Shell in the Windows Terminal (you can get there for free at http://shell.azure.com in your browser and access a free Linux container anywhere anytime with a persistent cloud drive) but now you can follow the instructions in this post and set up one-click SSH to anywhere.

Hope this is useful!


Sponsor: This week's sponsor is...me! This blog and my podcast has been a labor of love for over 18 years. Your sponsorship pays my hosting bills for both AND allows me to buy gadgets to review AND the occasional taco. Join me!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

How to set up Docker within Windows System for Linux (WSL2) on Windows 10

February 21, '20 Comments [13] Posted in Docker | Linux | Win10
Sponsored By

MagicI've written about WSL2 and it's glorious wonders many times. As its release (presumably) grows closer - as of this writing it's on Windows Insiders Slow and Fast - I wanted to update a few posts. I've blogged about a few cool thing around WSL and Docker

Here's a little HanselFAQ and some resources.

I want to run Linux on Windows

You can certainly use HyperV or VirtualBox and run a standard Virtual Machine. Download an ISO and mount it and run "a square within a square." It won't be seamlessly integrated within Windows - it'll be like the movie Inception - but it's time-tested.

Better yet, install WSL or WSL2. It'll take 5-10 minutes tops if your Windows 10 is somewhat up to date.

  • How to install WSL on Windows 10
    • WSL doesn't include a Linux kernel. Its Linux file system access is kinda slow, but it accesses Windows files super fast. If you use Cygwin, you'll love this, because it's really Linux, just the kernel is emulated.
  • How to install WSL2 on Windows 10
    • WSL2 ships an actual Linux kernel and its Linux file system is 5x-10x faster than WSL. WSL2 uses a tiny utility VM that expands contracts its memory and you can manage distros with the wsl command line.
    • Do all your development work inside here, while still using VS Code on Windows. It's amazing. Watch me set up a friend with WSL2, LIVE on YouTube.

I want to SSH into Linux stuff from Windows

There's 15 years of websites telling you to install Putty but you might not need it. OpenSSH has been shipping in Windows 10 for over two years. You can add them with Windows Features, or if you like, grab a release and put it on your PATH.

You can also do things like set up keys to use Windows 10's built-in OpenSSH to automatically SSH into a remote Linux machine. I also like to setup Signed Git Commits with a YubiKey NEO and GPG and Keybase on Windows.

I need a better Terminal in Windows

The new Windows Terminal is for you. Download Windows Terminal now for free. It's open source. You can then run the Win64/Win32 ssh from above, or run any Linux distros SSH. Have fun. It's time.

NOTE: Have you already downloaded the Terminal, maybe a while back? Enough has changed that you should delete your profiles.json and start over.

You can download the Windows Terminal from the Microsoft Store or from the GitHub releases page. There's also an unofficial Chocolatey release. I recommend the Store version if possible.

My prompt and fonts are ugly

Make them pretty. You deserve the best. Go get Cascadia Code's CascadiaPL.ttf and PowerLine and buckle up buttercup. Get a nice theme and maybe a GIF background.

image_e2447ddd-416e-4036-9584-e728455e6d9d

I want to use Docker on Windows and I want it to not suck

Surprise, it's actually awesome. You may have had some challenges with Docker a few years ago on Windows and gave up, but come back. There's been a huge (and fascinating) architecture of Docker on Windows. It's very nicely integrated if you have WSL2.

If you have WSL2 set up nicely, then get Docker Desktop WSL2. This version of Docker for Windows uses WSL2 as its engine allowing you to share your docker context across Windows and Linux on the same machine! As the maker intended!

WSL 2 introduces a significant architectural change as it is a full Linux kernel built by Microsoft, allowing Linux containers to run natively without emulation. With Docker Desktop running on WSL 2, users can leverage Linux workspaces and avoid having to maintain both Linux and Windows build scripts.

So that means

  1. Install Windows 10 Insider Preview build 19018 or higher
  2. Enable WSL 2 feature on Windows. For detailed instructions, refer to the Microsoft documentation.
  3. Download Docker Desktop Edge 2.1.6.0 or a later release.

Ensure your default WSL instances is WSL2. You can do that with wsl -l -v, and then wsl --set-version  <distro> 2

Then within Docker Desktop for Windows you've got two things to check. First, are you using WSL2 as your backend?

Docker for Windows | Enable WSL2

And then, the often missed setup, check under Resources | WSL Integration and tell Docker which WSL2 distros you want to use to access Docker. If you're paying attention you may notice that Docker Desktop tries to prompt you with a notification in Action Center but you might miss it.

Docker | Resources | WSL Integration

NOTE: If you used an early Tech Preview, you might have an extra now-vestigial Docker context named "wsl." You want to use the Default one, not the WSL one.

This isn't intuitive or obvious and you might get weird errors like these

docker wsl open //./pipe/docker_wsl: The system cannot find the file specified.

or

error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_wsl/v1.40/images/json?all=1: open //./pipe/docker_wsl: The system cannot find the file specified.

You can see if you have an extra context from before like below. That "wsl" one is older (if you have it) and you want to use default in both Windows and WSL2.

docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT
default * Current DOCKER_HOST based configuration npipe:////./pipe/docker_engine https://kubernetes.docker
wsl Docker daemon hosted in WSL 2 npipe:////./pipe/docker_wsl

I actually removed that one to avoid confusion with docker context rm wsl.

Here's Ubuntu on my Windows machine

Docker in Ubuntu

And here's my Windows machine. Note that docker images in both instances returns the same list. They are the same Docker backend!

Docker on Windows

I want to code in VS Code on Windows but compile on Linux

At this point once I've set things up I can go bananas. I can do Container-based development, where I use VS Code to run all my developer tools and builds insider a container...maybe I never event install Go or PHP or .NET Core. It's all just inside a container.

Oh, by the way, please Subscribe to my YouTube! I talk a lot about this stuff over there.


Sponsor: Couchbase gives developers the power of SQL with the flexibility of JSON. Start using it today for free with technologies including Kubernetes, Java, .NET, JavaScript, Go, and Python.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Page 1 of 7 in the Linux category Next Page

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.