T4 (Text Template Transformation Toolkit) Code Generation - Best Kept Visual Studio Secret
Rob beat me to it. Blogging about T4 (the Text Template Transformation Toolkit) had been on my list literally for a year. He and I were singing its praises last night. Hopefully I can add something small to the conversation.
What's the story? Well, T4 is a code generator built right into Visual Studio. To be clear, you HAVE THIS NOW on your system…go play. Now's the time to introduce code generation to your company. If you're doing something twice or more, manually, in your company, generate it.
However, it's not deep-deep built in, because there's no item templates in File | New Item and there's no intellisense or syntax highlighting.
You don't need this, but if you want really get the most out of T4, first, head over to Clarius Consulting and get their "T4 Editor Community Edition." That'll get you some basic coloring. They have a pay version that gets you more if you want.
Now, go into Visual Studio and make a Console App (or any app) and add a Text File, but name it something with a .tt extension. You'll get a warning since it's a generator, that someone could generate evil. Click OK if you are cool with potential evil. ;)
Now, look in Solution Explorer at the .tt file. If you're using C#, you'll have a sub .cs file, or if you're using VB, a sub .vb file. That's the file that will hold the result of the generation. This is the same visual metaphor used to the express the template/generated file relationship with .designer files you've seen elsewhere in Visual Studio.
If you look in the Properties for the .tt file, you'll see it's using a CustomTool listed as the "TextTemplatingFileGenerator." This custom tool extensibility point is how things like XSD and WSDL code generators hook in to generate their artifacts.
T4 can generate any kind of text file, not just code. Whatever you like. You can use it in your projects, as above, or you can call T4 from the command-line.
Aside from Kzu and the folks at Clarius, Rob notes that Oleg Sych has a great collection of T4 resources. He's got some great Tutorials:
- Creating your first code generator
- Troubleshooting code generation errors
- Debugging code generation files
- Creating reusable code generation templates
- Creating complex code generators
- Reusing code generators on multiple projects
Here's some of the links from Oleg's blog.
- How to create a simple T4 template
- How to use T4 to generate .config files
- How to use T4 to generate Decorator classes
- How to use T4 to generate CRUD stored procedures
- How to use T4 to generate strongly-typed navigation class in ASP.NET (by Kirill Chilingarashvili)
- How to use T4 to generate strongly-typed AzMan wrapper
- How to generate multiple outputs from single T4 template
- T4 template for generating ADO.NET Entity Framework Stored Procedures (by David DeWinter)
- T4 script for generating ADO.NET Entity Framework Views (by ADO.NET team)
- T4 template for generating LINQ to SQL Data Context (by Damien Guard)
- T4 template for generating WiX source files (by New Age Solutions)
- T4 template for generating SQL view from C# enumeration
- MSBuild task for transforming T4 templates (by Elton Stoneman)
- T4 template for generating state machines in C# (by Andrew Matthews)
Oleg also has a CodePlex project called T4 Toolbox that is a library of T4 templates that get added to File | New Item.
Also, check out Damien Guard's T4 templates that are a wholesale replacement of code that LINQ to SQL generates. Here's an example, where I use Damien's T4 templates against the sample Chinook Database.
Notice that I've named the .tt file the same as the .dbml, so Damian's code can find it. I also continue to let original LINQ to SQL generate it's .designer.cs file, but make that file's Build Action "None" so it's not ever compiled. That effectively puts Damian's code in charge.
Here's a screenshot showing a bit of Damian's T4 template using the syntax highlighting from the Clairus T4 Visual Studio free download. If I'd pay them, I'd get intellisense and syntax highlighting inside the code blocks also. It looks like a lot like ASP.NET Web Forms, or Velocity, or any templating language really. The code blocks are where your logic is and outside the codeblocks is the template for whatever you want to generate. Notice how Damien gets input and sets output. You have full control, you can read files off the file system, from with your project, etc. He sets the output extension also. I like to use .g.cs or .g.vb, myself. In this example his generated file is Chinook.generated.cs.
I particularly like Damien's example because he's swapping out parts of LINQ to SQL that he didn't like (the generated code) while keeping the part he did (the general mode, the designer, the dbml file.) If you don't like something, fix it.
Plus, it all works in Visual Studio without installing anything.
If you're doing Code Gen, or thinking about it, check out T4 as it's a great place to start. Also, search my blog for "Code Generation" as I was livign and breathing it with CodeSmith for the many years I worked at Corillian. Have fun!
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
Since there is no way to unload an Assembly from an AppDomain once you load it, the end result is that you have to restart VS every time you compile because VS will report the file is locked :( Once I realized this it was pretty obvious what was happening, but it's definitely a hidden gotcha!
The solution is to create your own AppDomain, dynamically load your source library into it, reflect over the contents, then unload the entire AppDomain - a kind of a pain and it might be good in the future if VS would split T4 into its own executable and just run it instead of trying to do it in-process.
<#@ template language="VBv3.5" hostspecific="True"#>
<#@ output extension=".generated.vb"
Paul - T4 already does run in a separate AppDomain, but it only recycles it occassionally in order to get some better performance. As Victor says, you can control this by writing a host, but Scott's quick and dirty solution is maybe cheaper.
For the default VS Host, the AppDomain is recycled either
a) if 15 of the templates classes are more than 15 minutes older than the last used template
b) if caching is switched off (http://blogs.msdn.com/garethj/archive/2006/06/05/t4caching.aspx) every 25 uses of T4
Oleg discusses the locking issue in his post on generating decorator classes. He suggests an alternative is to use the introspection engine developed by the FXCop team rather than reflection.
In an article on FXCop he links to it says,
"The use of the Introspection engine allows for much faster analysis and supports multithreaded analysis. Unlike the Reflection engine from previous versions, in the Introspection engine the assemblies you're analyzing are not locked so you won't need to shut down FxCop to do a fix and recompile of those assemblies."
1. MSBuild integration so that your code generation can happen as part of the build process and while building from the command line or on your build server.
2. Ability to generate multiple files or single file. You can also control codebehind settings for each file generated.
3. Ability to easily specify metadata or properties to drive the templates and ability to setup multiple templates to run as a batch.
4. Ability to set the build action of generated files. ie. Maybe set a file to be an embedded resource.
5. Templates run in a separate process so that the referenced assemblies are not locked.
6. Ability to execute complex templates as snippet generators from the editor.
7. Source control integration.
8. Ability to automatically add necessary project references to the project.
Those are just a couple of the differentiators from T4 to CodeSmith's Visual Studio integration and doesn't even cover any of the benefits of the template syntax and flexibility that CodeSmith provides. Here is a link to a quick video that goes over the Visual Studio integration features of CodeSmith:
http://www.codesmithtools.com/video/visual-studio-integration.html
Eric J. Smith
Your list may sound like this could be a limitation of T4 technology while it isn't, all the points you list can be achieved using T4 and writing proper code. Moreover, most of them are already out there in form of open source projects like the Web Service Software Factory from p&p which generates multiple outputs from a single template.
On the tooling side AFAIK there is no T4-based product alternative to compare to right now. If you're into T4 there are a few standalone lifesavers as of today like our Clarius T4 Editor which -by the way- IMHO provides a much better IntelliSense experience than CodeSmith as it reuses VS existing IntelliSense support for this instead of re-inventing its own.
And on the engine side, while I'm not familiar with CodeSmith as to tell which one of the two engines is better, I've been using T4 engine extensively and I'm very happy with it. A major point to note is that you can redistribute for free the T4 Engine to VS2005 users and you don't need to distribute at all for VS2008 as it is already built-in while this isn't true for CodeSmith right? Last time I checked one had to pay a redist for this thus I'm may be wrong...
- To achieve even simple things you need to write your own host and directory processor. Paul's problem is one of them that this approach solves, but there are tons of things like that and it requires a lot of codelifting (needless to say that nearly impossible to unit test).
- Visual studio experience isn't the best. With 2008, you have some support(if you think running generation engine in each ctrl+s is indeed a support), with 2005 you are totally out of the game. Other than paying to clarius consulting, who provides a very basic intellisense and higlighting and sends the bill to your house, you have no other option.
- The errors are *really* cryptic. There is no way that you can find which line causes the error without looking at the temporary file, which is most of the time is not even there(cleaned up because of the exception). The messages are obscure (Got an error in Line : 0 ? go figure yourself now).
- Very wierd design decisions, such as relative paths are not relative to template itsself's path but to app's path.
- Documentation quality in MSDN is nowhere near other VS technologies documentations. Very trivial examples.
- In custom hosts you can only communicate with the engine by magic strings and on the fly code generation.
- Templates are hard to test, debugging is a nightmare.
And tons of other things.
The product has a long way to go to be even close to its competitors.
Also, just because that we talk about T4 shouldn't mean that we have to promote code generation. So I'd amend following sentence :
>> If you're doing something twice or more, manually, in your company, generate it.
to my own view
>> If you're doing something twice or more, manually, in your company, refactor it.
Again, you seem to be comparing a complete a full-blown CodeGen product like CodeSmith against the T4 engine which, just in case it's not clear enough, is not a full-blown CodeGen product and as such offers minimal tooling support. This is not really big news isn't it?
My point is that comparing oranges to apples doesn't make much sense -- I also think a company could build upon the current T4 engine and offer a competitive product to any of the CodeGen tools out there, and it is against such a tool what you should compare.
As I worked on the T4 Editor I'm particularly interested in why do you believe the IntelliSense and "highlighting" (I'm guessing this is syntax coloring) of T4 parts + C# code is "very basic" as I believe it's pretty complete offering an experience almost in a par with the ASP.Net editor. Which other tool offers a better experience in this regard?
And yes, we do charge for our Professional Edition (note we also offer a free Community one) as we're part of the weird species that like to get paid for its work :-)
>> Again, you seem to be comparing a complete a full-blown CodeGen product like CodeSmith against the T4 engine which, just in case it's not clear enough, is not a full-blown CodeGen product and as such offers minimal tooling support. This is not really big news isn't it?
I didn't even have CodeSmith in mind while writing this, and didn't mention it once. I won't compare an apple to an apple tree. Most of the problems I listed are not even tool related problems, they are architectural or design issues and related to T4 Engine itsself. These needs are basic ones, I didn't pick them up from any product - they are pains from real life.
However, to play the game fairly people need to know what expects them in the wild forest. They need to know what they are gonna pay for, and what they are not going to get. T4 is definitely an orphaned child of VS product unit line, and comes with the least-if no is rude-support OOTB.
On Clarius T4 Editor:
First I have to thank and say that as it is the first and only work in the arena, this editor eased my life a lot. Though, it is still far from carrying user experience to a next level.
- Intellisense is poor : Do you have intellisense in c# code written in templates ? Do you have autocomplete in the imported assemblies ? Do you have intellisense for included templates ? Or Custom Directive Processors ? Do you have intellisense comments that shows what is used for what ?
- Syntax Coloring is poor: Only directives have appropriate colors, C# codes inside (<#+ +> that is) is not colored. Actually, other than the directives at the top, the template is barely correctly colored.
>> Which other tool offers a better experience in this regard?
There aren't, and that's indeed the main problem. Even with your extra money, you still can't buy a normal visual studio experiment.
>> And yes, we do charge for our Professional Edition (note we also offer a free Community one) as we're part of the weird species that like to get paid for its work :-)
Fair enough, so there is no problem in getting people know this before jumping into T4 generation world, is there ?
Sorry if I wrongly interpreted the first part of your last response, I thought you were -like Eric- comparing the T4 engine against a full-blown codegen product.
Note that I do agree with you on the fact that there are some existing paint points today with T4 engine that you need to workaround and that -as tooling is mostly non-existent- pretty much everything requires coding. I do agree too that parts of it could benefit from some re-design, no doubt, and I hope future T4 versions will improve all these. But at the same time I don't consider these as an adoption showstopper. There are several projects out there that have succesfully used T4 and this technology is fully backed by Microsoft and officially supported through PSS. All this is enough for me to go with it for most cases. I believe this is the point were we disagree and it's ok.
Now regarding the Clarius T4 Editor, it looks like you've not tried our Professional Edition, because:
>>>Do you have intellisense in c# code written in templates?
>>>Do you have autocomplete in the imported assemblies?
>>>Do you have intellisense for included templates?
>>>Or Custom Directive Processors?
>>>Do you have intellisense comments that shows what is used for what?
ALL of the above is supported by our Pro Edition :-) That's why I was surprised to hear you found the IntelliSense support was "poor" as I sincerely think -bias apart now!- is pretty good.
Maybe you were using our old beta or the Community Edition? These two don't support much of the above.
As for error line numbers and relative paths for includes, not really sure where you're coming from on this one. Neither of those are a problem, also with Clarius' editor you can see the generated cs file at any time. (I don't work for Clarius)
There are some things which are pretty annoying in T4 like debugging support but, to be honest, if you get to the point that this is a problem, you need to refactor, massive templates really are hard to work with, but do you write code like that?
Building custom hosts is sometimes necessary, especially if you want build script support but really this is a piece of cake and fairly well documented.
Pete
What is a common usage for such a technology? I would generate ORM stuff with it because we already have so many ORM software including LINQ to SQL with that. Is there any cool templates you guys developed and that you could reuse?
I just need some concrete exemple of using such technology.
Thanks!
Another is the Service Factory (http://www.codeplex.com/servicefactory/) which uses T4 templates to generate code from three DSLs into several projects.
But my question its this:
Does the T4 have an api so i can do code generation at run time? I think any tool should always have the following features:
- API
- Command line
With dynamic application and build tools, continuous integration these two features are essentials. Of course that having an API we can create the command line utility. But withou API i think its a waste of time. Working only in Visual Studio just make it more "license oriented", this means, more money to MS.
Thanks :D
T4 Templates for the .NET Compact Framework
--Jeff
Comments are closed.