Scott Hanselman

Dynamically generating robots.txt for ASP.NET Core sites based on environment

June 18, '19 Comments [3] Posted in ASP.NET | DotNetCore | Open Source
Sponsored By

I'm putting part of older WebForms portions of my site that still run on bare metal to ASP.NET Core and Azure App Services, and while I'm doing that I realized that I want to make sure my staging sites don't get indexed by Google/Bing.

I already have a robots.txt, but I want one that's specific to production and others that are specific to development or staging. I thought about a number of ways to solve this. I could have a static robots.txt and another robots-staging.txt and conditionally copy one over the other during my Azure DevOps CI/CD pipeline.

Then I realized the simplest possible thing would be to just make robots.txt be dynamic. I thought about writing custom middleware but that sounded like a hassle and more code that needed. I wanted to see just how simple this could be.

  • You could do this as a single inline middleware, and just lambda and func and linq the heck out out it all on one line
  • You could write your own middleware and do lots of options, then activate it bested on env.IsStaging(), etc.
  • You could make a single Razor Page with environment taghelpers.

The last one seemed easiest and would also mean I could change the cshtml without a full recompile, so I made a RobotsTxt.cshtml single razor page. No page model, no code behind. Then I used the built-in environment tag helper to conditionally generate parts of the file. Note also that I forced the mime type to text/plain and I don't use a Layout page, as this needs to stand alone.

@page
@{
Layout = null;
this.Response.ContentType = "text/plain";
}
# /robots.txt file for http://www.hanselman.com/
User-agent: *
<environment include="Development,Staging">Disallow: /</environment>
<environment include="Production">Disallow: /blog/private
Disallow: /blog/secret
Disallow: /blog/somethingelse</environment>

I then make sure that my Staging and/or Production systems have ASPNETCORE_ENVIRONMENT variables set appropriately.

ASPNETCORE_ENVIRONMENT=Staging

I also want to point out what may look like odd spacing and how some text is butted up against the TagHelpers. Remember that a TagHelper's tag sometimes "disappears" (is elided) when it's done its thing, but the whitespace around it remains. So I want User-agent: * to have a line, and then Disallow to show up immediately on the next line. While it might be prettier source code to have that start on another line, it's not a correct file then. I want the result to be tight and above all, correct. This is for staging:

User-agent: *
Disallow: /

This now gives me a robots.txt at /robotstxt but not at /robots.txt. See the issue? Robots.txt is a file (or a fake one) so I need to map a route from the request for /robots.txt to the Razor page called RobotsTxt.cshtml.

Here I add a RazorPagesOptions in my Startup.cs with a custom PageRoute that maps /robots.txt to /robotstxt. (I've always found this API annoying as the parameters should, IMHO, be reversed like ("from","to") so watch out for that, lest you waste ten minutes like I just did.

public void ConfigureServices(IServiceCollection services)
{
services.AddMvc()
.AddRazorPagesOptions(options =>
{
options.Conventions.AddPageRoute("/robotstxt", "/Robots.Txt");
});
}

And that's it! Simple and clean.

You could also add caching if you wanted, either as a larger middleware, or even in the cshtml Page, like

context.Response.Headers.Add("Cache-Control", $"max-age=SOMELARGENUMBEROFSECONDS");

but I'll leave that small optimization as an exercise to the reader.

UPDATE: After I was done I found this robots.txt middleware and NuGet up on GitHub. I'm still happy with my code and I don't mind not having an external dependency, but it's nice to file this one away for future more sophisticated needs and projects.

How do you handle your robots.txt needs? Do you even have one?


Sponsor: Get the latest JetBrains Rider with WinForms designer, Edit & Continue, and an IL (Intermediate Language) viewer. Preliminary C# 8.0 support, rename refactoring for F#-defined symbols across your entire solution, and Custom Themes are all included.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Making a tiny .NET Core 3.0 entirely self-contained single executable

June 13, '19 Comments [18] Posted in DotNetCore
Sponsored By

I've always been fascinated by making apps as small as possible, especially in the .NET space. No need to ship any files - or methods - that you don't need, right? I've blogged about optimizations you can make in your Dockerfiles to make your .NET containerized apps small, as well as using the ILLInk.Tasks linker from Mono to "tree trim" your apps to be as small as they can be.

Work is on going, but with .NET Core 3.0 preview 6, ILLink.Tasks is no longer supported and instead the Tree Trimming feature is built into .NET Core directly.

Here is a .NET Core 3.0 Hello World app.

225 files, 69 megs

Now I'll open the csproj and add PublishTrimmed = true.

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
</PropertyGroup>
</Project>

And I will compile and publish it for Win-x64, my chosen target.

dotnet publish -r win-x64 -c release

Now it's just 64 files and 28 megs!

64 files, 28 megs

If your app uses reflection you can let the Tree Trimmer know by telling the project system about your Assembly, or even specific Types or Methods you don't want trimmed away.

<ItemGroup>
<TrimmerRootAssembly Include="System.IO.FileSystem" />
</ItemGroup>

The intent in the future is to have .NET be able to create a single small executable that includes everything you need. In my case I'd get "supersmallapp.exe" with no dependencies. That's done using PublishSingleFile along with the RuntimeIdentifier in the csproj like this:

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
<PublishReadyToRun>true</PublishReadyToRun>
<PublishSingleFile>true</PublishSingleFile>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
</PropertyGroup>
</Project>

At this point you've got everything expressed in the project file and a simple "dotnet publish -c Release" makes you a single exe!

There's also a cool global utility called Warp that makes things even smaller. This utility, combined with the .NET Core 3.0 SDK's now-built-in Tree Trimmer creates a 13 meg single executable that includes everything it needs to run.

C:\Users\scott\Desktop\SuperSmallApp>dotnet warp
Running Publish...
Running Pack...
Saved binary to "SuperSmallApp.exe"

And the result is just a 13 meg single EXE ready to go on Windows.

A tiny 13 meg .NET Core 3 application

If you want, you can combine this "PublishedTrimmed" object with "PublishReadyToRun" as well and get a small AND fast app.

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.0</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
<PublishReadyToRun>true</PublishReadyToRun>
</PropertyGroup>
</Project>

These are not just IL (Intermediate Language) assemblies that are JITted (Just in time compiled) on the target machine. These are more "pre-chewed" AOT (Ahead of Time) compiled assemblies with as much native code as possible to speed up your app's startup time. From the blog post:

In terms of compatibility, ReadyToRun images are similar to IL assemblies, with some key differences.

  • IL assemblies contain just IL code. They can run on any runtime that supports the given target framework for that assembly. For example a netstandard2.0 assembly can run on .NET Framework 4.6+ and .NET Core 2.0+, on any supported operating system (Windows, macOS, Linux) and architecture (Intel, ARM, 32-bit, 64-bit).
  • R2R assemblies contain IL and native code. They are compiled for a specific minimum .NET Core runtime version and runtime environment (RID). For example, a netstandard2.0 assembly might be R2R compiled for .NET Core 3.0 and Linux x64. It will only be usable in that or a compatible configuration (like .NET Core 3.1 or .NET Core 5.0, on Linux x64), because it contains native code that is only usable in that runtime environment.

I'll keep exploring .NET Core 3.0, and you can install the SDK here in minutes. It won't mess up any of your existing stuff.


Sponsor: Suffering from a lack of clarity around software bugs? Give your customers the experience they deserve and expect with error monitoring from Raygun.com. Installs in minutes, try it today!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Clever little C# and ASP.NET Core features that make me happy

June 4, '19 Comments [20] Posted in ASP.NET | DotNetCore
Sponsored By

Visual StudioI recently needed to refactor my podcast site which is written in ASP.NET Core 2.2 and running in Azure. The Simplecast backed API changed in a few major ways from their v1 to a new redesigned v2, so there was a big backend change and that was a chance to tighten up the whole site.

As I was refactoring I made a few small notes of things that I liked about the site. A few were C# features that I'd forgotten about! C# is on version 8 but there were little happinesses in 6.0 and 7.0 that I hadn't incorporated into my own idiomatic view of the language.

This post is collecting a few things for myself, and you, if you like.

I've got a mapping between two collections of objects. There's a list of all Sponsors, ever. Then there's a mapping of shows where a show might have n sponsors.

Out Var

I have to "TryGetValue" because I can't be sure if there's a value for a show's ID. I wish there was a more compact way to do this (a language shortcut for TryGetValue, but that's another post).

Shows2Sponsor map = null;
shows2Sponsors.TryGetValue(showId, out map); if (map != null) { var retVal = sponsors.Where(o => map.Sponsors.Contains(o.Id)).ToList(); return retVal; } return null;

I forgot that in C# 7.0 they added "out var" parameters, so I don't need to declare the map or its type. Tighten it up a little and I've got this. The LINQ query there returns a List of sponsor details from the main list, using the IDs returned from the TryGetValue.

if (shows2Sponsors.TryGetValue(showId, out var map))
    return sponsors.Where(o => map.Sponsors.Contains(o.Id)).ToList();
return null;

Type aliases

I found myself building JSON types in C# that were using the "Newtonsoft.Json.JsonPropertyAttribute" but the name is too long. So I can do this:

using J = Newtonsoft.Json.JsonPropertyAttribute;

Which means I can do this:

[J("description")] 
public string Description { get; set; }

[J("long_description")] public string LongDescription { get; set; }

LazyCache

I blogged about LazyCache before, and its challenges but I'm loving it. Here I have a GetShows() method that returns a List of Shows. It checks a cache first, and if it's empty, then it will call the Func that returns a List of Shows, and that Func is the thing that does the work of populating the cache. The cache lasts for about 8 hours. Works great.

public async Task<List<Show>> GetShows()
{
Func<Task<List<Show>>> showObjectFactory = () => PopulateShowsCache();
return await _cache.GetOrAddAsync("shows", showObjectFactory, DateTimeOffset.Now.AddHours(8));
}
private async Task<List<Show>> PopulateShowsCache()
{
List<Show> shows = shows = await _simpleCastClient.GetShows();
_logger.LogInformation($"Loaded {shows.Count} shows");
return shows.Where(c => c.Published == true && c.PublishedAt < DateTime.UtcNow).ToList();
}

What are some little things you're enjoying?


Sponsor: Manage GitHub Pull Requests right from the IDE with the latest JetBrains Rider. An integrated performance profiler on Windows comes to the rescue as well.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Introducing the Try .NET Global Tool - interactive in-browser documentation and workshop creator

May 15, '19 Comments [10] Posted in DotNetCore | Open Source
Sponsored By

Learn .NET or easily author your own workshopIf you find yourself learning C# and .NET and come upon the "Run your first C# Program" documentation you may have noticed a "Try the code in your browser" button that lets you work through your first app entirely online, with no local installation! You're running C# and .NET in the browser! It's a great way to learn that is familiar to folks who learn JavaScript.

The language team at Microsoft wants to bring that easy on-ramp to everyone who wants to learn .NET.

The .NET Foundation has published a lot of free .NET presentations and workshops that you can use today to teach open source .NET to your friends, colleagues, or students. However these do encourage you to install a number of prerequisites and we believe that there might be an easier on-ramp to learning .NET.

Today we're announcing that on ramp - the Try .NET global tool!

Here's the experience. Once you have the .NET SDK - Pick the one that says you want to "Build Apps." Just get the "try" tool! Try it!

  • Open a terminal/command prompt and type dotnet tool install --global dotnet-try

Now you can either navigate to an empty folder and type

dotnet try demo

or, even better, do this!

ACTION: Clone the samples repo with
git clone https://github.com/dotnet/try -b samples
then run
"dotnet try"
and that's it!

NOTE: Make sure you get the samples branch until we have more samples!

C:\Users\scott\Desktop> git clone https://github.com/dotnet/try -b samples
Cloning into 'try'...
C:\Users\scott\Desktop> cd .\try\Samples\
C:\Users\scott\Desktop\try\Samples [samples ≡]> dotnet try
Hosting environment: Production
Content root path: C:\Users\scott\Desktop\try\Samples
Now listening on: http://localhost:5000
Now listening on: https://localhost:5001

Your browser will pop up and you're inside a local interactive workshop! Notice the URL? You're browsing your *.md files and the code inside is runnable. It's all local to you! You can put this on a USB key and learn offline or in disconnected scenarios which is great for folks in developing countries. Take workshops home and remix! Run an entire workshop in the browser and the setup instructions for the room is basically "get this repository" and type "dotnet try!"

Try .NET interactive local documentation

This is not just a gentle on-ramp that teaches .NET without yet installing Visual Studio, but it also is a toolkit for you to light up your own Markdown.

Just add a code fence - you may already be doing this! Note the named --region there? It's not actually running the visible code in the Markdown...it's not enough! It's compiling your app and capturing the result of the named region in your source! You could even make an entire .NET interactive online book.

### Methods
A **method** is a block of code that implements some action. `ToUpper()` is a method you can invoke on a string, like the *name* variable. It will return the same string, converted to uppercase.
``` cs --region methods --source-file .\myapp\Program.cs --project .\myapp\myapp.csproj
var name = "Friends";
Console.WriteLine($"Hello {name.ToUpper()}!");
```

And my app's code might look like:

using System;

namespace HelloWorld
{
class Program
{
static void Main(string[] args)
{
#region methods
var name = "Friends"
Console.WriteLine($"Hello {name.ToUpper()}!");
#endregion
}
}
}

Make sense?

NOTE: Closing code fences ``` must be on a newline.

Hey you! YOU have some markdown or just a readme.md in your project! Can you light it up and make a workshop for folks to TRY your project?

Code Fences within Markdown

Here I've typed "dotnet try verify" to validate my markdown and ensure my samples compile. Dotnet Try is both a runner and a creator's toolkit.

Compiling your workshop

Today "dotnet try" uses .NET Core 2.1 but if you have .NET Core 3 installed you can explore the more complex C# samples here with even more interesting and sophisticated presentations. You'll note in the markdown the --session argument for the code fence allows for interesting scenarios where more than one editor runs in the context of one operation!

image

I'd love to see YOU create workshops with Try .NET. It's early days and this is an Alpha release but we think it's got a lot of promise. Try installing it and running it now and later head over to https://github.com/dotnet/try to file issues if you find something or have an idea.

Go install "dotnet try" locally now, and remember this is actively being developed so you can update it easily and often like this!

dotnet tool update -g dotnet-try

There's lots of ideas planned, as well as the ability to publish your local workshop as an online one with Blazor and WASM. Here's a live example.

Watch for an much more in-depth post from Maria from my team on Thursday on the .NET blog!


Sponsor: Suffering from a lack of clarity around software bugs? Give your customers the experience they deserve and expect with error monitoring from Raygun.com. Installs in minutes, try it today!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Exploring DNS with the .NET Core based Technitium DNS Server

April 18, '19 Comments [12] Posted in DotNetCore
Sponsored By

Earlier this week I talked about how Your Computer is not a Black Box and I spent some time in TCPView and at the command line exploring open ports on my computer. I was doing this in order to debug an issue with a local DNS server I was playing with, so I thought I'd take a moment and look at that server itself.

The Technitium DNS Server is a personal local DNS server (FOSS on GitHub) written in C# and it runs on Windows, macOS, Linux, Raspberry Pi, etc. I downloaded the Portable app.

For Windows folks who aren't used to .tar.gz files, remember to "eXtract Zie Files!" with "tar -xzvf DnsServerPortable.tar.gz -C ./TechnitiumDNS/" and it's also worth reminding you all that tar.exe, curl.exe, wget.exe and more are all included in Windows 10 and have been since 2017. If that's too hard, use 7zip.

Technitium DNS is pretty cool, you just unzip/tar it and run start.sh or start.bat and it "just works." Of course, I did have a process already on port 53 - DNS - so I did a little debugging, but that was my fault.

Here's the local web UI that you can use to administer the server locally. You can forward to whatever upstream DNS server you'd like, with the added bonus that the forwarder can be DNS over HTTPS so you can use things like CloudFlare, Google, or Cloud9. Using DNS over HTTPS means your DNS lookups can be secured with DNSSEC and are far more secure and private than regular DNS over UDP/TCP.

Technitium also includes support for DNS Sinkholes (similar to how I use my Pi-Hole) and Block List URLs. It'll automatically download block lists daily and block ads.

Technitium is a lovely .NET Core based DNS Server

It's also educational to try running your own DNS server and it's fun to read the code! The code for Technitium's DNS Server is up at https://github.com/TechnitiumSoftware/DnsServer and is super interesting from a networking perspective, but also from an C# perspective. It's a very interesting example of some .NET Core code at a very low level and I'm thrilled that it works on every operating system.

There's even bash scripts for setting Technitium up on your RaspberryPi or Ubuntu to make it easy. If you are using Windows and don't care about .NET Core you can use the .NET that's included with Windows and Technitum has a Tray app and Installer as well.

Some of the code isn't "idiomatic" C#/.NET Core but it's interesting to read about. The main DnsWebService.cs is pretty intense as it doesn't use any ASP.NET Core routing or primitives. It's a complete webserver written using only System.Net and its own support libraries, along with some of the lower-level Newtonsoft.Json libraries.

The main DnsServer is also quite low level and very performant. It lives in DnsServer.cs. It opens up n sockets (depending on how many ports you bind to) and starts accepting connections here. DNS Datagrams start getting parsed here, right off the stream. The supporting libraries and networking helper code lives over at https://github.com/TechnitiumSoftware/TechnitiumLibrary which is a wealth of interesting and useful code covering BitTorrent, Mail, and Firewall management. There's a ton of OO representations of networking concepts, and all the DNS records are parsed manually.

Technitium has a DNS Server, client, Mac Address Changer, and open source instant messenger. The developer is extremely prolific. They even host a version of "Get HTTPS for free" that works with Windows and makes getting Let's Encrypt certificates super easy.

Anyway, I've been enjoying exploring DNS again and reminding myself not only that it still works great (since I learned about DNS from sniffing packets in networking class) and it's been updated and improved with caches, DNSSEC, DNS over HTTP and more in the years following.

Here I've set my IPv4 DNS to 127.0.0.1 and my IPv6 DNS to ::1, then I run NSLookup and try some domain lookups.

Looking up domains at the command line with nslookup

Again, to be clear, the local DNS server took these lookups and then forwarded them upstream to another server. However, you have the choice for your upstream lookups to be done over whatever protocols you want, you can use Google, OpenDNS, Quad9 (with DNSSEC or without), and on and on.

Are you running your own DNS Server?


Sponsor: Manage GitHub Pull Requests right from the IDE with the latest JetBrains Rider. An integrated performance profiler on Windows comes to the rescue as well.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Page 1 of 22 in the DotNetCore category Next Page

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.