RFC: Server-side Image and Graphics Processing with .NET Core and ASP.NET 5
The .NET Core and the Core Libraries are open source and run on Windows, Mac, and Linux. In fact, you can run them with support, today, in production on Windows and Linux (Mac is for development, not production).
Head over to http://get.asp.net to get ASP.NET 5 with .NET Core on any platform. Also get the free cross platform editor Visual Studio Code as well.
Because .NET Core runs anywhere, there are things that it doesn't have. It doesn't have a UI layer (no WinForms, for example, because that makes no sense on Linux) or Registry Access (that's only on Windows, so it's simply omitted). While these omissions make sense, others don't exist because of lack of times, changing priorities, or peoplepower.
Currently, as of the time of this blog post's writing, .NET Core has no good built-in option for image resizing or image generation/creation. There is no System.Drawing because there is no Win32 GDI. There are some options, that I'll point out later, but this clearly hasn't been a priority so it's not done yet. There has been some work on System.Drawing.Graphics, but it seems stalled.
Nathanael Jones is the very accomplished author of ImageResizer and the primary at Imazen, an imaging software company. I've talked about ImageResizer before, it's fantastic. Nathanael has been pushing hard to get folks at Microsoft to commit to a story around Server-side Graphics in ASP.NET 5 and recently updated his own roadmap with respect to ImageResizer and the new ASP.NET 5 (that runs on .NET Core as well as the full .NET Framework). Take a moment and read it, there's a lot there. Concerns about how to call native code, how to distribute managed code that has to call native code, and lots more. I've worked with Nathanael for the last year trying to work out some way to solve this problem but it hasn't worked out well. Big company, sigh.
Another library in the .NET imaging space is called ImageProcessor by James M. South. Jim also wants to solve the program of server side image manipulation. He's currently working on a re-write of his ImageProcessor as a cross-platform library and I'm sure he would appreciate your help.
Jim himself says:
Is this wise? Honestly... I don't know. I could be writing code that may be suddenly obsolete. There has been little feedback on questions I've asked but it's a nice learning process if anything and I will definitely be releasing the code for consumption.
Over in the Microsoft corefxlab repo Jim has asked some questions and tried to push as well, but the issue is currently closed. Fortunately it is just closed for lack of interest. Krzysztof Cwalina from Microsoft says:
We have nobody working actively on this. It's a interesting problem space, but it does not align with our current priorities. I will close this for now. Please let me know if anybody wants to work on it and I will reopen.
Enter, you, us, the community. Jim has asked for help, and I'm also asking for help. Is Server-Side graphics manipulation in ASP.NET 5 important? Jim asks:
Please... Spread the word, contribute algorithms, submit performance improvements, unit tests. Help me set up CI for nightly releases.
Performance is a biggie, if you know anything about the new vector types and can apply some fancy new stuff with that it would be awesome.
There's a lot of developers out there who could write this stuff a lot better and faster than I and I would love to see what we collectively can come up with so please, if you can help in any way it would be most welcome and beneficial for all.
So, two things.
First, head over to https://github.com/JimBobSquarePants/ImageProcessor/tree/V3 and talk to James M. South on Twitter. If you can, offer support. Perhaps help with a AppVeyor setup, or offer your own CI server for builds. Do you have a need for server-side image work now or perhaps you have a cache of unit tests? See what you can offer. Even if you're a First Timer to Open Source.
Second, if you want to work on it, either with James, or with Microsoft, check out the closed issues on GitHub at https://github.com/dotnet/corefxlab/issues/86#issuecomment-158459437 and engage there. It'd be awesome to see this problem solved.
Sponsor: Thanks to RedGate for sponsoring the blog this week. They're offering a free .NET eBook! Discover 52 tips to improve your .NET performance! Our new eBook features dozens of tips and tricks to boost .NET performance. With contributions from .NET experts around the world, you’ll have a faster app in no time. Download your free copy.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
Side note: If Vulkan would be used as a basis for a cross platform (Windows and Linux) client side UI library, perhaps we finally could get something "thread safe" in this area.
We all should admire people like James South (aka Jim), for picking up this issue. Now join his gitter channel and help him out!
Thanks for writing this Scott, I really appreciate it! I almost feel famous having featured on your blog twice now!
I thought I'd pitch in and explain a little about my motivation and also the talk about the status of my library to hopefully encourage people to join in.
ImageProcessor has been remarkably successful over the years; I'm incredibly grateful for the support and encouragement people have shown. What started off as a hobby in a field in which I had zero experience had turned into a highly dependable and powerful set of libraries that power a huge number of applications on both the server and desktop. That's amazing! I love the challenge of image processing despite struggling with the mathematics that drives it and I gain a wonderful sense of achievement when I see the result of my effort and how I can help others.
ImageProcessor started off as a simple wrapper around some System.Drawing objects, providing an interface that allowed developers to perform common tasks without worrying about memory management and the various quirks of System.Drawing. What I am attempting to do now though is much more grandiose.
Over the last year, with the recent work from Microsoft organising the development of the cross-platform CoreFx (A monumental and most impressive effort) it appeared to me that someone needed to step up and supply something that would allow cross-platform graphics manipulation. I saw a lot of talk in various threads but very little activity and I fully understand why... Image processing is not a trivial subject.
I saw the work Nathanael Jones had been doing (He's a personal hero of mine) but it appeared to me that there were a lot of issues in the new framework regarding the organisation of native libraries that currently prevent his success. I don't know C either so I could do little to help.
I'd looked at many libraries including GraphicsMagick, libgd, FreeImage, ImageMagick and I couldn't see anything that really satisfied my needs both for simple cross-platform application and ease of use.
So managed C# seemed the way forward and I set about writing something useful with pretty good success so far...
With ImageProcessor V3 I can encode/decode jpeg, bmp, png, and gif (animated also) files. I can resize images with a higher quality than System.Drawing ever could, crop, alter brightness, contrast etc, apply the equivalent of ColorMatrix, blur, sharpen and much more all with a really simple, chainable API that supports multi-threading. I really think I'm onto something.
What I really need with just now is help improving performance and adding algorithms to the library as I believe the API to be pretty solid. I know next to nothing about the new Vector types in System.Numerics as I have no gaming development experience but I have noted great performance boosts with my experiments so far and I'm sure there are now some excellent methods available in the framework that can make it possible to perform manipulation at high speed. I need help designing some of the core primitives (should I use rectangle, point etc?) and I need help fixing my attempts at more complicated things like rotation, text, etc.
As Scott quoted before I really believe there's a lot of developers out there who could write this stuff a lot better and faster than I and I believe that together we could produce something brilliant and also wonderfully open source for everyone to freely use. Microsoft have really knocked on out of the park with the developments over the last year and it would be amazing if we, as a community could fill some of the gaps.
One quibble, you said, "It doesn't have a UI layer (no WinForms, for example, because that makes no sense on Linux) or Registry Access (that's only on Windows, so it's simply omitted)."
IMO, it does make sense if you don't let the name throw you off! Linux is capable of a GUI, and there are similar things like the registry. So supporting the building of GUI apps would be nice, even though I would suspect that it's very low on the priority list, if it even made the list :)
The new SIMD stuff (System.Numerics.Vectors) lacks some key functionality necessary for image processing applications - namely, vectorized type convertors (float -> Int -> Byte) as well as bit-shifting operations (unpack hi / unpack low)...both are necessary for efficient vectorized image processing. I have reached out to Microsoft and they are aware of the problem. I would wait until these holes are patched before bothering with SIMD.
Microsoft opened a proposal/RFC for x-plat image processing in June, but has since been silent as to its plans. https://github.com/dotnet/corefx/issues/2020
A lot of work has gone into that, obviously, and it seems like it would be a waste of resources to re-start from scratch. At this point of .NET Core, it also seems like a better use of Microsoft resources to focus on enabling library developers rather than on reproducing their work. I'd much rather see contributions to Jim's library.
Jim can contact any one of us (my alias is beleroy) if he needs assistance, for example with SIMD, anytime.
For developers already all-in on ASP.NET 5, lack of .NET 4.5 classic or VS2013 support may not be a deal-breaker. And while that segment is not large enough to sustain any kind of commercial support, that doesn't mean than it's not worth pursuing native backends for.
ImageResizer 4 introduced a cross-plat rendering engine in C (C++ tests) that brought a 45-fold speed up compared to GDI+ (System.Drawing - also native, highly optimized C and C++). Many ImageResizer optimizations can actually port to C#; particularly how we increase cache locality by transposing matrices during write operations instead of requiring scattered reads in the Y dimension.
I'd love to see ImageProcessor development reach a point where the most CPU-intensive parts can be swapped out for a native backend when native binaries are present. Interpolation and jpeg (de)coding would be the primary hot-spots. It may be early in the process to establish this API, but I want to go on record that I am very interested in that kind of collaboration, and I urge everyone to contribute to the ImageProcessor project as THE solution for .NET Core.
Benchmarks, harnesses, unit tests, integration tests, sample image files (store externally!), and help with color profile parsing/application are just a few of the things that anyone can jump in and help with. Even if you slept through math classes, you can certainly help set up integration tests. Imaging libraries usually need gigabytes of input and expectation images, and usually those are a separate repository with scheduled CI runs. No push access needed to get started on that :) The core repo usually keeps a set of minimal low-res images for TDD purposes.
I am the author of both Magick.NET and GraphicsMagick.NET. My core focus is on Magick.NET since I am a team member of ImageMagick. Both ImageMagick and GraphicsMagick are crossplatform but the .NET libraries aren't. They are using CLI/C++ which is not supported on Linux. I am planning to rewrite both .NET libraries to use P/Invoke instead. That would make it possible to use them in .NET Core. My plan is to rewrite Magick.NET to use P/Invoke in 2016-Q1 and focus on getting it working on Linux in 2016-Q2. And when that is done GraphicsMagick.NET will also be rewritten.
Nathanael, I'm sure there is a lot we can work out together, I'm especially interested in any performance boosting ideas you have. Bertrand if you could cast your eye over the solution and spot anything that you think would benefit from vectorization that would be great or even if you know a way to incorporate some missing functionality. I'm sure there's things like shear based-rotation that are probably trivial with the right knowhow.
@LKeene Even without those portions of functionality I think there is still a lot to be gained from those libraries.
http://www.graciebarra.com/br/2013/11/significado-oss/
And there is some very radical statements about it, they are all opinions, and formulated as such, so they are not wrong.
I will however say this, I have been engaged in 4 larger Single Page Web Applications by now, and none of them required image processing... And only one of them actually had any raster images at all, and in that particular case, images could just as well be hosted else where... And often was...
Maybe that is because I am a sucker for vector graphics and that these applications all are focused on data management/exchange rather than anything else.
I do agree that Image processing is highly relevant when dealing with systems that publish articles or has to sell products etc, but there are allot of web applications out there that does not fall into either...
And what holds people back from pulling in 3rd party libraries to handle this i wonder?... Besides, we all have a limited amount of time, so does M$... And they have to prioritize within the time they have... I would rather have frequent releases that wait...
There are many many open source image processing sytems with a MS friendly license like MIT/BSD.
Consider ImageJ with its MIT/BSD style license. It's basic Java but will port to C#.
Direct:
- Pick the top 5 image formats for initial support - single frame formats bmp, png, jpeg, jpeg2000, gif. Add multiframe formats in v2 (tiff)
- Pick the top x image operations - read/write, resize, auto/manual levels, reduce bits per pixel, contrast, brightness, historgram equalization
- Pick the top x image color operations - saturation, colorize, apply LUT, cartoon, sepia, ...
Release that as V1
For our image heavy processing systems, we'd put the image + a job info text file in a directory which would be batch processed by a dedicated background process (read job info and call the appropriate command line program like ImageMagick to do the processing and put the output in a completed directory). The calling/requesting system would have to poll for a completion or would get a call back. This asynchronicity is necessary as some image operations would take multiple minutes to run on very large multi-frame tiff/dicom images (e.g., convert a set of images slicing up a 3d space into a 3d volume rendering)
A superset of some of the image/photo manipulation apps for mobile would be a good start.
Microsoft has had nearly 15 years of managed C#/.net to produce a basic imaging library free of com/win32/gdi.
Please share with us what is used by Office, Internet Explorer and Edge for jpeg/png/gif decoding? Is it the basic win32 image codecs or a dedicated coded shipped with the application?
Comments are closed.
And frankly, I don't get why the ASP.NET 5 team is completely ignoring this fact. Take a CMS like Umbraco, which uses ImageProcessor to great effect. I'd certainly not use any CMS that does not have to ability to resize and crop images. Especially with the increasing focus on responsive images, not having the ability to dynamically scale images on the server is just silly.
The lack of a proper server-side image processing library has always plauged ASP.NET, but you could always just fall back on System.Drawing (even if you're not technically supposed to do so). With ASP.NET 5, that is no longer possible. And while third parties (like James M. South) are doing their best to fill that void, the basics should be built in to ASP.NET 5 / .NET Core.