Scott Hanselman

My one-hundred-and-ninety-ninth podcast is up. I chat with Jeremy Zawodny, a developer at Craigslist on how the system is put together. How many servers do they have? How does it all fit together and what are the major technology problems they have to solve?

Subscribe:

Download: MP3 Full Show

Links from the Show

• Jeremy's Blog and Jeremy on Twitter
• Jeremy Zawodny on Wikipedia

Telerik is our sponsor for this show.

Building quality software is never easy. It requires skills and imagination. We cannot promise to improve your skills, but when it comes to User Interface and developer tools, we can provide the building blocks to take your application a step closer to your imagination. Explore the leading UI suites for ASP.NET AJAX,MVC, Silverlight, Windows Forms and WPF. Enjoy developer tools like .NET reporting, ORM, Automated Testing Tools, TFS, and Content Management Solution. And now you can increase your productivity with JustCode, Telerik’s new productivity tool for code analysis and refactoring. Visit www.telerik.com.

As I've said before this show comes to you with the audio expertise and stewardship of Carl Franklin. The name comes from Travis Illig, but the goal of the show is simple. Avoid wasting the listener's time. (and make the commute less boring)

Enjoy. Who knows what'll happen in the next show?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

I heard someone saying they were having trouble setting up Smooth Streaming for IIS, so I wanted to try it myself.

If you just want to see Smooth Streaming work, visit http://www.smoothhd.com/ for some sample demos. They'll adapt to your bandwidth and look as nice as they can. If you want to fake a low-bandwidth situation, you can use the player at http://www.iis.net/media/experiencesmoothstreaming and play with limiting the bit rate.

Step 0: Got IIS?

I've got IIS 7 because I've got Win 7. If your Win 7 installation doesn't have IIS yet, go to Programs and Features and click "Turn Windows Features On or Off" and select Internet Information Services. Don't forgot to go through the tree and turn on the things you want, like "Static Content" in this case. (Remember, if you've just installed IIS, shut down Skype as it grabs port 80 if it gets there first. Lame.)

Step 1: Add IIS Media Services

I'm running Windows 7 x64 so I installed IIS Media Services 3.0 64-bit, but you can certainly install IIS Media Services for 32-bit also. Even easier, just install it from the Web Platform Installer directly.

Installing the IIS Media Services pack gives you a new icons in the IIS manager:

Step 2: Get (or Make) Some Content (and Encode It)

If you just want to see this work, you can get some pre-encoded content, below in 2a. Later in 2b, I'll encode some new content.

Step 2a: Get Some Pre-Encoded Content to Test

Use At this point, if you're just testing, you can go download the pre-encoded "Buck The Bunny" HD Video Sample Content (about 500 megs of HD video). Run it, extract the files, and put them somewhere like c:\inetpub\wwwroot\bunny.

HOW SMOOTH STREAMING WORKS - SIDE NOTE: You kind have to see it to get it, but it's actually clever in its simplicity. Some folks freak out when they take a, say, 200 meg video file and encode it for Smooth Streaming and the resulting directory is, say, 500 megs. They might say, it's larger! I don't have that kind of bandwidth! In fact, less is sent over the wire than exists on disk. The idea is that Smooth Streaming creates "steps" of bitrates. It encodes and stores your file multiple bit-rates on the disk.
To illustrate this, I will use the DJ Hero video game as an example:
Just as the DJ moves between tracks depending on how the crowd feels, Silverlight between encoded streams depending on the bandwidth is doing. Remember, it's called "Adaptive" Smooth Streaming.

Now I visit http://localhost/bunny/default.html and rather than the standard Silverlight Player, I'm going to select one called "User Experience Simulator." Basically, I want to fake a low bandwidth situation and see the Smooth Streaming react.

NOTE: If you like, or you're demoing this to a boss or someone, there's a prettier/shinier player on the Smooth Streaming Sample Content page that you can swap out. Details on the Getting Started Page for Smooth Streaming.

I start the player, and you'll notice on teh right side there's "Now Downloading" and "Max Bit Rate." I'll force a lower bit-rate (again, I'm simulating congestion here) and seek within the video. Then raise it, and wait, or seak. Notice the chart showing that the Silverlight Player "switched tracks" without dropping frames.

Another metaphor would be changing gears on a bicycle to adapt to a changing road.

Step 2b: Encode Some Content of Your Own.

Ok, so that was just existing content. You can download Expression Encoder 3.0 and encode all the video you want, but if you want to encode Smooth Streaming or H.264, you have to upgrade for $49 to Expression Encoder 3.0 with IIS Smooth Streaming. (By the way, Expression Encoder will also do Screen Capture, which is nice to know. The free version will do up to 10 min of Screen Capture, which is also useful to know.)

I drag a video of ScottGu in HD into the Encoder and select IIS Smooth Streaming. Since my source is HD 720p, I'll select that. By default, the profile will basically encode a high version and low version, but for best results you'll want to decide how many levels of stepping you want. Expression Encoder can be run in a batch from the command-line also, BTW.

Make sure you click the encoding option you want, right-click and say "Apply to Selected Items." You'll know you got it right if the Video section further down (that you can open up) includes multiple streams of various sizes and bitrates. Notice the screenshot. More streams will mean longer encodes and more space on disk taken up, BUT it'll make for more bitrate options for the player to choose and a MUCH better, smoother experience. Again, you might have 1, 5 or 10 streams for a 1 gig source and end up using 5+gigs of space on disk (I made those numbers up) but the end user's experience will be smooth and the bandwidth used will be as small as possible.

It is worth noticing that screencasts should mostly be encoded using an H.264 Screen Encoding profile, while action like sports games work better with other codecs. What looks good on the football field often looks like crap if it's some dude coding on his screen. You'll also want to have a reasonable minimum resolution and a minimum bitrate. Experiment and you'll find what works and save it as a custom profile that's right for your content.

After I encoded all this, I thought to myself, crap, the REALLY LOW bandwidth version is not acceptable. Do I have to re-encode to not include that one? I can just hack the .ism file manifest as it's just XML and remove all knowledge of that lower bit-rate file. UPDATE: There's a nice UI to managed rethe different bit rates, so no need to hack any XML.

It can be tricky to do the math and figure out what resolutions and bitrates to use for your smooth streaming content so you can use this Silverlight-based Smooth Streaming Multi-Bitrate Calculator. There's a WPF one also. To get really good quality you'll vary not only the bitrate, but also the resolution (but you'll stretch it, so the user won't know the resolution changed also.) Ultimately you have complete control as everyone's content has different needs.

The "More Info Than You Asked For" Section: What's Really Happening?

If you want a deep deep dive into what's going on, check out the Smooth Streaming section of Alex Zambelli's blog. It's wonderfully deep. A lot of people don't realize that Smooth Streaming is still just plain HTTP.

Traditionally streaming is a long running stream over a stateful protocol, where a packet is a some milliseconds of video.

Adaptive Streaming is chunked over HTTP where the client can choose different chunk sizes in various bit rates.

From Alex's blog:

A typical Smooth Streaming media asset therefore consists of the following files:

• MP4 files containing video/audio
  • *.ismv – contains video and audio, or only video
    • 1 ISMV file per encoded video bitrate
  • *.isma - contains only audio
    • In videos with audio, the audio track can be muxed into an ISMV file instead of a separate ISMA file
• Server manifest file
  • *.ism
  • Describes the relationships between media tracks, bitrates and files on disk
  • Only used by the IIS Smooth Streaming server – not by client
• Client manifest file
  • *.ismc
  • Describes to the client the available streams, codecs used, bitrates encoded, video resolutions, markers, captions, etc.
  • It’s the first file delivered to the client

Both manifest file formats are based on XML. The server manifest file format is based specifically on the SMIL 2.0 XML format specification.

With IIS7 Smooth Streaming, a client is expected to request fragments in the form of RESTful URLs:

http://video.foo.com/NBA.ism/QualityLevels(400000)/Fragments(video=610275114)

http://video.foo.com/NBA.ism/QualityLevels(64000)/Fragments(audio=631931065)

The values passed in the URL represent encoded bitrate (i.e. 400000) and the fragment start offset (i.e. 610275114) expressed in an agreed-upon time unit (usually 100 ns). These values are known from the client manifest.

But most importantly, therefore I make it bold and in red so you'll read it twice:

...what about dynamic bitrate switching that makes adaptive streaming so effective? This part of the Smooth Streaming experience is implemented entirely in client-side Silverlight application code – the server plays no part in the bitrate switching process.

An important thing to think about when doing network capacity planning is that in a traditional scenario, clients will drop to the bit rate they can sustain on average. In Smooth Streaming, it will continually hop up to the highest they can handle. Smooth Streaming gives each client the highest quality video they can handle. That may result in some network stress for the video provider and network infrastructure if you were expecting video that was less than awesome. ;)

Shiny.

A Q&A from Alex's Smooth Streaming FAQ

Q: Is Smooth Streaming based on Microsoft’s ASF file format? Or does it use some new proprietary file format?
A: No, the Smooth Streaming File Format is based on the ISO-standardized Base Media File Format, better known as the MP4 file format. It follows the ISO/IEC 14496-12 specification

Q: Will Smooth Streaming ensure I can deliver HD video to all my customers?
A: No. Smooth Streaming can’t perform any miracles; it’s simply an enhanced HTTP delivery method. Delivery of HD video is still bound by the efficiency of the codecs used: VC-1, H.264, etc. For example, most video experts would agree that 720p/24 video can’t be delivered at a satisfactory quality level with less than 2 Mbps encoded bitrate. Well, 2 Mbps of encoded video still requires 2 Mbps of network bandwidth in order to be delivered in real-time – and there’s nothing IIS7 or Silverlight can do to change that.

Q: Where can I see Smooth Streaming in action?
A: Visit SmoothHD.com for a demo of on-demand Smooth Streaming, with video content ranging up to 720p HD for those on fast connections (3 Mbps or more). Also check out the IIS.net demo which lets you simulate variable network bandwidth during playback. The only requirement to view Smooth Streaming video is a Silverlight 2 (or better) browser plugin. For a demo of live Smooth Streaming, check out Home Shopping Network’s 24-hr live shopping channel, also offering up to 720p in full-screen mode.

Conclusion

It's really easy to setup and easy to encode for Smooth Streaming. In my case, I just encoded the files off my HD video camera. You'll see more Smooth Streaming around various Microsoft sites that serve video in the coming months.

Have fun!

Related Links

• IIS Smooth Streaming Site

  • IIS *LIVE* Smooth Streaming - you can do this with LIVE content also, BTW.

  • Smooth Streaming Multi-Bitrate Calculator - It can be tricky to do the math and figure out what resolutions and bitrates to use for your smooth streaming content so you can use this Silverlight-based online calculator. There's a WPF one also.

• Silverlight Media Framework with Progressive Download Support - If you're going to do a player, this is where you start.

  • "SMF is built on a proven code base with dozens of leading Smooth Streaming deployments, including Wimbledon, Sunday Night Football on NBCSports, the UEFA Super Cup on Canal+, Roland Garros and the Tour de France on France Télévisions and many others and by providing developers with source code they can more easily build these experiences for their deployments. Key features in the framework include DVR, rewind, alternate language tracks, in-stream data feeds and analytics tracking. The SMF is designed for future third-party extensibility and component modularity, as well as support for other media delivery scenarios beyond Smooth Streaming."

• Adaptive Streaming Comparison - "In addition, IIS Smooth Streaming 4.0 will also provide an option for streaming to iPhone and QuickTime X players."

• Smooth Streaming

  • Smooth Streaming Technical Overview
  • Adaptive Streaming Comparison
  • Smooth Streaming for IIS 7.0 - Getting Started
  • Smooth Streaming for IIS 7.0 - Managing Your Presentations
  • Live Smooth Streaming
    • Live Smooth Streaming for IIS 7.0 - Getting Started
    • Live Smooth Streaming for IIS 7.0 - Creating and Managing Publishing Points
    • Live Smooth Streaming for IIS 7.0 - Syndicating Content between Servers
  • Web Playlists[Show All]
    • Web Playlists for IIS 7.0 - Configuration
    • Web Playlists for IIS 7.0 - Creating a Simple Playlist
    • Web Playlists for IIS 7.0 – Using Playlist With Different Media Players
    • Web Playlists for IIS 7.0 - Serving Media Content From User Folders
  • Bit Rate Throttling
    • Bit Rate Throttling Configuration Walkthrough
    • Bit Rate Throttling Extensibility Walkthrough
  • Smooth Streaming Deployment Guide
  • Windows Media Server or Web Server?
  • Using HTTP for Streaming and Downloading from the Same Computer
  • Edge Caching for Media Delivery
  • Adding Windows Media Services 2008 on an IIS 7.0 Server

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

I was working with a friend recently on a side thing they were doing. They wanted to create an "Export" function for some small bit of data and start it from their website.

• You'd hit a URL after logging in
• Some data would come out of a database
• You'd get a .CSV file downloaded
• You could open it in Excel or whatever.

I spoke to my friend and they said it was cool to share their code for this post. This post isn't meant to be a WTF or OMG look at that code, as is it meant to talk about some of the underlying issues. There's few things going on here and it's not all their fault, but it smells.

• They are using a Page when a IHttpHandler will do.
  • Not a huge deal, but there's overhead in making a Page, and they're not using any of the things that make a Page a Page. The call to ClearContents is an attempt at telling the Page to back off. It's easier to just not have a page.
• They're "thrashing" a bit in the Page_Load, basically "programming by coincidence."
  • They're not 100% sure of what needs to be done, so they're doing everything until it works. Even setting defaults many times or calling methods that set properties and they setting those properties again.
  • This means a few things. First, HTTP is subtle. Second, the Response APIs are confusing (less so in .NET 4) and it's easy to do the same thing in two ways.
• They're not using using() or IDisposable
  • They're cleaning up MemoryStreams and StreamWriters, but if an exception happens, things'll get cleaned up whenever. It's not tight in a "cleaning up after yourself" deterministic (when it needs to be) kind of way.
  • They're calling Flush() when it's not really needed, again programming by coincidence. "I think this needs to be done and it doesn't break..."
• Old school data access
  • Not bad, pre se, but it could be easier to write and easier to read. DataAdapters, DataSets, are a hard way to do data access once you've used Linq to SQL, EF or NHibernate.
• Re-Throwing an Exception via "throw ex"
  • When you want to re-throw an exception, ALWAYS just throw; or you'll lose your current call stack and it'll be hard to debug your code.
• Not reusable at all
  • Now, reuse isn't the end-all, but it's nice. If this programmer wants different kinds of exports, they'll need to extract a lot from the spaghetti.

Here's what they came up with first.

Note that this code is not ideal. This is the before. Don't use it. 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.IO;
using System.Configuration;
using System.Data.SqlClient;
using System.Data;

namespace FooFoo
{
    public partial class Download : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            System.IO.MemoryStream ms = CreateMemoryFile();

            byte[] byteArray = ms.ToArray();
            ms.Flush();
            ms.Close();

            Response.Clear();
            Response.ClearContent();
            Response.ClearHeaders();
            Response.Cookies.Clear();
            Response.Cache.SetCacheability(HttpCacheability.Private);
            Response.CacheControl = "private";
            Response.Charset = System.Text.UTF8Encoding.UTF8.WebName;
            Response.ContentEncoding = System.Text.UTF8Encoding.UTF8;
            Response.AppendHeader("Pragma", "cache");
            Response.AppendHeader("Expires", "60");
            Response.ContentType = "text/comma-separated-values";
            Response.AddHeader("Content-Disposition", "attachment; filename=FooFoo.csv");
            Response.AddHeader("Content-Length", byteArray.Length.ToString());
            Response.BinaryWrite(byteArray);
        }

        public System.IO.MemoryStream CreateMemoryFile()
        {
            MemoryStream ReturnStream = new MemoryStream();

            try
            {
                string strConn = ConfigurationManager.ConnectionStrings["FooFooConnectionString"].ToString();
                SqlConnection conn = new SqlConnection(strConn);
                SqlDataAdapter da = new SqlDataAdapter("SELECT * FROM [FooFoo] ORDER BY id ASC", conn);
                DataSet ds = new DataSet();
                da.Fill(ds, "FooFoo");
                DataTable dt = ds.Tables["FooFoo"];

                //Create a streamwriter to write to the memory stream
                StreamWriter sw = new StreamWriter(ReturnStream);

                int iColCount = dt.Columns.Count;

                for (int i = 0; i < iColCount; i++)
                {
                    sw.Write(dt.Columns[i]);
                    if (i < iColCount - 1)
                    {
                        sw.Write(",");
                    }
                }

                sw.WriteLine();
                int intRows = dt.Rows.Count;

                // Now write all the rows.
                foreach (DataRow dr in dt.Rows)
                {
                    for (int i = 0; i < iColCount; i++)
                    {

                        if (!Convert.IsDBNull(dr[i]))
                        {
                            string str = String.Format("\"{0:c}\"", dr[i].ToString()).Replace("\r\n", " ");
                            sw.Write(str);
                        }
                        else
                        {
                            sw.Write("");
                        }

                        if (i < iColCount - 1)
                        {
                            sw.Write(",");
                        }
                    }
                    sw.WriteLine();
                }

                sw.Flush();
                sw.Close();
            }
            catch (Exception Ex)
            {
                throw Ex;
            }
            return ReturnStream;
        }

    }
}

I don't claim to be a good programmer, but I do OK. I went over my concerns with my friend, and suggested first an HttpHandler. I started with Phil's basic abstract HttpHandler (based on my super basic HttpHandler boilerplate). I could have certainly done by just implementing IHttpHandler, but I like this way. They're about the same # of lines of code. The important part is in HandleRequest (or ProcessRequest).

namespace FooFoo
{
    public class ExportHandler : BaseHttpHandler
    {
        public override void HandleRequest(HttpContext context)
        {
            context.Response.AddHeader("Content-Disposition", "attachment; filename=FooFoo.csv");
            context.Response.Cache.SetCacheability(HttpCacheability.Private);

            var dc = new FooFooDataContext();
            string result = dc.FooFooTable.ToCsv();
            context.Response.Write(result);
        }

        public override bool RequiresAuthentication
        {
            get { return true; }
        }

        public override string ContentMimeType
        {
            get { return "text/comma-separated-values"; }
        }

        public override bool ValidateParameters(HttpContext context)
        {
            return true;
        }
    }
}

At the time I wrote this, I was writing how I wished the code would look. I didn't have a "ToCsv()" method, but I was showing my friend how I though the could should be separated. Even better if there was a clean DAL (Data Access Layer) and Business Layer along with a service for turning things into CSV, but this isn't too bad. ToCsv() in this example is a theoretical extension method to take an IEnumerable of something and output it as CSV. I started writing it, but then decided to Google with Bing, and found a decent direction to start with at Mike Hadlow's blog. He didn't include all the code in the post, but it saved me some typing, so thanks Mike!

namespace FooFoo
{
    public static class LinqToCSV
    {
        public static string ToCsv<T>(this IEnumerable<T> items)
            where T : class
        {
            var csvBuilder = new StringBuilder();
            var properties = typeof(T).GetProperties();
            foreach (T item in items)
            {
                string line = string.Join(",",properties.Select(p => p.GetValue(item, null).ToCsvValue()).ToArray());
                csvBuilder.AppendLine(line);
            }
            return csvBuilder.ToString();
        }

        private static string ToCsvValue<T>(this T item)
        {
            if(item == null) return "\"\"";

            if (item is string)
            {
                return string.Format("\"{0}\"", item.ToString().Replace("\"", "\\\""));
            }
            double dummy;
            if (double.TryParse(item.ToString(), out dummy))
            {
                return string.Format("{0}", item);
            }
            return string.Format("\"{0}\"", item);
        }
    }
}

This creates an extension method that lets me call something.toCsv() on anything IEnumerable. It'll spin through the properties (yes, I know that could have been a LINQ statement also, but I like a nice ForEach sometimes. Feel free to fix this up in the comments! ;) ) and build up the Comma Separated Values.

At some point, it really should format Dates as {0:u} but as of now, it works identically as the before code and attempts to rectify most of the issues brought up. Of course, one could take something like this as far and make it as robust as they like.

As Mike points out, you can also do little projections to control the output:

string csv = things.Select(t => new { Id = t.Id, Name = t.Name, Date = t.Date, Child = t.Child.Name }).AsCsv();

Your thoughts, Dear Reader?

Related Links

• Good Exception Management Rules of Thumb
• Cargo-cult programming
• LINQ to CSV using DynamicObject
• Mike Hadlow's LINQ ToCsv()
• Recommended: Open Source "FileHelpers" for importing and exporting Fixed and Delimited files with .NET from Marco Meli
• Eric White on Linq to Text and Linq to CSV
• Reading data FROM CSV files using LINQ
• Think Linq 

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

My one-hundred-and-ninety-eighth podcast is up. I sit down with Erik Meijer from the Cloud Programmability Team to hear about the Reactive Extensions for .NET (Rx). Rx is a library for composing asynchronous and event-based programs using observable collections. Sound boring? Not even a little. Rx is a prescription for awesome.

Subscribe:

Download: MP3 Full Show

Links from the Show

• Rx Team Blog
• Reactive Extensions for .NET (Rx)
• B# .NET Blog
• Wes Dyer's Blog
• Jeffrey Van Gogh's Blog
• Danny van Velzen's Blog
• Lambda the Ultimate Blog
• Rx on Channel 9

Telerik is our sponsor for this show.

Check out their UI Suite of controls for ASP.NET. It's very hardcore stuff. One of the things I appreciate aboutTelerik is their commitment tocompleteness. For example, they have a page about their Right-to-Left support while some vendors have zero support, or don't bother testing. They also are committed to XHTML compliance and publish their roadmap. It's nice when your controls vendor is very transparent.

As I've said before this show comes to you with the audio expertise and stewardship of Carl Franklin. The name comes from Travis Illig, but the goal of the show is simple. Avoid wasting the listener's time. (and make the commute less boring)

Enjoy. Who knows what'll happen in the next show?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

My one-hundred-and-ninety-seventh podcast is up. Scott sits down with Jimmy Schementi to find out what's the scoop with the DLR. Is it baked? What do I need to do to get started? What's the status of IronRuby - is it done? Will IronPython be a first class language or is it already? All these questions and more will be answered.

Subscribe:

Download: MP3 Full Show

Links from the Show

• IronRuby
• IronPython
• Dynamic Language Runtime
• Jimmy's Blog

Do also remember the complete archives are always up and they have PDF Transcripts, a little known feature that show up a few weeks after each show.

Telerik is our sponsor for this show.

Check out their UI Suite of controls for ASP.NET. It's very hardcore stuff. One of the things I appreciate aboutTelerik is their commitment tocompleteness. For example, they have a page about their Right-to-Left support while some vendors have zero support, or don't bother testing. They also are committed to XHTML compliance and publish their roadmap. It's nice when your controls vendor is very transparent.

As I've said before this show comes to you with the audio expertise and stewardship of Carl Franklin. The name comes from Travis Illig, but the goal of the show is simple. Avoid wasting the listener's time. (and make the commute less boring)

Enjoy. Who knows what'll happen in the next show?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By