Scott Hanselman

Microbenchmarks are evil. Ya, I said it. Folks spend hours in tight loops measuring things trying to find out the "best way" to do something and forget that while they are changing 5ms between two techniques they've missed the 300ms Database Call or the looming N+1 selects issue that has their ORM quietly making even more database calls.

My friend Sam Saffron says we should "take global approach to optimizations." Sam cautions us to avoid trying to be too clever.

// You think you're slick. You're not.
// faster than .Count()? Stop being clever.
var count = (stuff as ICollection<int>).Count;

All that said, let's argue microbenchmark, shall we? ;)

I did a blog post a few months back called "Back to Basics: Moving beyond for, if and switch" and as with all blog posts where one makes a few declarative statement (or shows ANY code at all, for that matter) it inspired some spirited comments. The best of them was from Chris Rigter in defense of LINQ.

I started the post by showing this little bit of counting code:

var biggerThan10 = new List;
for (int i = 0; i < array.Length; i++){
    if (array [i] > 10)
       biggerThan10.Add (array[i]);
}

and then changed it into LINQ which can be either of these one liners

var a = from x in array where x > 10 select x; 
var b = array.Where(x => x > 10);

and a few questions came up like this one from Teusje:

"does rewriting code to one line make your code faster or slower or is it not worth talking about these nanoseconds?"

The short answer is, measure it. The longer answer is measure it yourself. You have the power to profile your code. If you don't know what's happening, profile it. There's some interesting discussion on benchmarking small code samples over on this StackOverflow question.

Now, with all kinds of code like this folks go and do microbenchmarks. This usually means doing something trivial a million times in a tight loop. That's lots of fun and I'm doing to do JUST that very thing right now with Chris's good work, but it's important to remember that your code is usually NOT doing something trivial a million times in a tight loop. Unless it is.

Knaģis says:

"Unfortunately LINQ has now created a whole generation of coders who completely ignores any perception of writing performant code. for/if are compiled into nice machine code, whereas .Where() creates instances of enumerator class and then iterates through that instance using MoveNext method...Please, please do not advocate for using LINQ to produce shorter, nicer to read etc. code unless it is accompanied by warning that it affects performance"

I think that LINQ above could probably be replaced with "datagrids" or "pants" or "google" or any number of conveniences but I get the point. Some code is shown in the comments where LINQ appears to be 10x slower. I can't reproduce his result.

Let's take Chris's comment and deconstruct it. First, taking an enumerable Range as an array and spinning through it.

var enumerable = Enumerable.Range(0, 9999999);
var sw = new Stopwatch();
int c = 0;

// approach 1

sw.Start();
var array = enumerable.ToArray();
for (int i = 0; i < array.Length; i++)
{
    if (array[i] > 10)
        c++;
}
sw.Stop();
Console.WriteLine("Enumerable.ToArray()");
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

The "ToArray()" part takes 123ms and the for loop takes 9ms on my system. Arrays are super fast.

Starting from the enumerable itself (not the array!) we can try the Count() one liner:

// approach 2
Console.WriteLine("Enumerable.Count()");
sw.Restart();
c = enumerable.Count(x => x > 10);
sw.Stop();
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

It takes 86ms.

I can try it easily in Parallel over 12 processors but it's not a large enough sample nor is it doing enough work to justify the overhead.

// approach 3
Console.WriteLine("Enumerable.AsParallel() (12 procs)");
sw.Restart();
c = enumerable.AsParallel().Where(x => x > 10).Count();
sw.Stop();
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

It adds overhead and takes 129ms. However, you see how easy it was to try a naïve parallel loop in this case. Now you know how to try it (and measure it!) in your own tests.

Next, let's do something stupid and tell LINQ that everything is an object so we are forced to do a bunch of extra work. You'd be surprised (or maybe you wouldn't) how often you find code like this in production. This is an example of coercing types back and forth and as you can see, you'll pay the price if you're not paying attention. It always seems like a good idea at the time, doesn't it?

//Approach 4 - Type Checking?
Console.WriteLine("Enumerable.OfType(object) ");
var objectEnum = enumerable.OfType<object>().Concat(new[] { "Hello" });
sw.Start();
var objectArray = objectEnum.ToArray();
for (int i = 0; i < objectArray.Length; i++)
{
    int outVal;
    var isInt = int.TryParse(objectArray[i].ToString(), out outVal);
    if (isInt && Convert.ToInt32(objectArray[i]) > 10)
        c++;
}
sw.Stop();
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

That whole thing cost over 4 seconds. 4146ms in fact. Avoid conversions. Tell the compiler as much as you can up front so it can be more efficient, right?

What if we enumerate over the types with a little hint of extra information?

// approach 5
Console.WriteLine("Enumerable.OfType(int) ");
sw.Restart();
c = enumerable.OfType<int>().Count(x => x > 10);
sw.Stop();
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

Nope, the type check wasn't necessarily in this case. It took 230ms and added overhead. What if this was parallel?

// approach 6
Console.WriteLine("Enumerable.AsParallel().OfType(int) ");
sw.Restart();
c = enumerable.AsParallel().OfType<int>().Where(x => x > 10).Count();
sw.Stop();
Console.WriteLine(c.Dump());
Console.WriteLine(sw.ElapsedMilliseconds.Dump());

That's 208ms, consistently. Slightly faster, but ultimately I shouldn't be doing unnecessary work.

In this simple example of looping over something simple, my best bet turned out to be either the Array (super fast if it was an Array to start) or a simple Count() with LINQ. I measured, so I would know what was happening, but in this case the simplest thing also performed the best.

What's the moral of this story? Measure and profile and make a good judgment. Microbenchmarks are fun and ALWAYS good for an argument but ultimately they exists only so you can know your options, try a few, and pick the one that does the least work. More often than not (not always, but usually) the compiler creators aren't idiots and more often than not the simplest syntax will the best one for you.

Network access, database access, unnecessary serializations, unneeded marshaling, boxing and unboxing, type coercion - these things all take up time. Avoid doing them and when do you do them, don't just know why you're doing them, but also that you are doing them.

Is it fair to say "LINQ is evil and makes things slow?" No, it's fair to say that code in general can be unintuitive if you don't know what's going on. There can be subtle side-effects whose time can get multiplied inside of a loop. This includes type checking, type conversion, boxing, threads and more.

The Rule of Scale: The less you do, the more you can do of it.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By