Linq

Thinking Different with Linq and Lambdas

H. Steele Price, IV — Tue, 03 Jun 2008 17:06:00 GMT

Today I had to look through a set of controls to extract the Text and build a dynamic search.

Normally, this means looping and/or a bunch of Ifs to do some validation. With Linq to Objects at our disposal I distilled exactly what I needed down to this:

Dim q = From c In SearchCriteria.Controls _
        Where TypeOf c Is RadTextBox _
        AndAlso c.Text.Length > 0

Dim params = q.ToDictionary(Of String, String)( _
                Function(r As RadTextBox) r.ID.Remove(0, 5), _
                Function(r As RadTextBox) r.Text.Trim)

At first glance this is a so what... until you look at what it is really doing for me.

I am looking at a Web Page that I need to extract a bunch of criteria from and many of them will most likely be blank, so why even loop through them? The AndAlso takes care of any problems arising from controls that are not RadTextBox. Additionally, I need to take the results and put them into a Dictionary so that I can pass this onto my Dynamic Query routine.

The code this is replacing is about 30 lines of code (that I didn't write...) and I can see exactly what is going on in this concise format. All I had to do is think a little different from the old habits of coding and apply some of the great new tools we have at our disposal. What the Old code did was to loop through all the controls in the SearchCriteria DIV and check to see if .Text was filled, but to do that, it needed to cast the Control. Then on top of that it needed to add the Dictionary Item.

This new code could actually be written on one line, but for programming clarity and readability, I broke it up.

Now this really reads easy and works perfectly, thanks to Linq. A little refactoring and this could become a permanent addition to a Utilities class.

nb: the Remove leverages the fact that I use "Find_<DBFieldName>" in markup and not have to resort to bindings or strings, it also makes this a candidate for a code generation template.

EntityDB: The next great Microsoft tool (if they would just make it)

H. Steele Price, IV — Tue, 20 May 2008 15:38:00 GMT

Reading in the Industry blogs, official samples and documentation provided with MS SQL Server 2005+, Visual Studio 2008 and Linq to Sql and the Entity Framework, you will get the impression that we have a tremendous ability to efficiently decipher how to interact with a Database. Additionally, Industry leaders have led us down a path of Code Generators, Designers and the separation of UI Design from functional Object code; read "Astoria" and/or "MVC" here...

Didn't CASE do this sort of thing for us years ago? Didn't it also fail? There is huge amount of traffic about the topic of "Falling down when you hit a threshold" and insistence that all things should be built ultra-scalable. OK, that's great if you love to write plumbing. I for one, don't. What if that could be done for you, with the most efficient options always at your disposal; of course with the opportunity to tweak and extend to meet a specific need.

While there are many great arguments to be had over what is exactly ultra-efficient and those who think they have a better method must be able to override and extend the selected generation of code that is offered by not only being able to write that code, but also to inject it into the Generated Code (or at the least allowed to edit the templates) produced by Visual Studio.

If you don't already know why this is so important, we need to first look at the massive shift in Business Software Development and where we are going to be running most business applications. We used to run all our apps on our own hardware, inside our own network and everything was under our tight control. While this model still applies to much corporate development, I believe it will become less and less used as a cost effective alternative to a HOSTED solution. Pricing of hosted solutions have plummeted in recent years, in fact I am almost willing to bet that the cost of electricity alone using hardware just a few years old would cover the cost of a hosted solution. And what happens when I DO need to scale, How do I add 20 servers to the Web Farm overnight? Staffing requirements and maintenance of hardware start to become totally offset when we look at this model. I CANNOT BUILD A SERVER CHEAPER THAN I CAN HOST ONE! That is the real bottom line.

I would wager that MOST applications would run just fine on a hosted solution with one machine doing all the work, at least in the beginning, or at least with a hoster that gives you Web on one machine and your DB on another. Are we talking about every solution here? No, but I think we are talking about a gigantic sector of the market that is currently all but ignored by the provided tools we have at our disposal.

Don't get me wrong here, we have GREAT tools available and more coming, do they target what is really needed?

First lets look at the three new primary tools given to us by Microsoft. WCF, Linq to Sql and the Entity Framework. WCF provides a great pipeline to pass data between our UI, Business Objects and Database, but it just plumbing. Linq to Sql and EF provide a great way to interact with a Database IF and ONLY If you have a proper database in place first.

What does this mean to most data-driven development projects? It means they either need a DBA, or they need to hire one for the portion of the application that defines the database. This is one reason Migrations for Ruby on Rails has gained so much popularity, it hides this chore. Most of us don't care about the DB, it's just a place that holds data. Some of you can go on and on all day about the subtleties of databases but that is completely outside the scope of what I am talking about here. Not only will Migrations create a DB and Schema around what your App needs, it will also keep it up to date with changes made by any developer that can be easily stored in Version Control software like SVN or TFS.

This is what the Designers for Linq to Sql and/or Entity Framework need to do for us. I really don't mind changing my Linq to SQL Apps over to Entity Framework if it will allow us to focus on one great technology. It's great that they can operate against a currently created database, but they should also create one (IN AN OPTIMIZED WAY) when one does not exist. It should create Stored Procedures if I tell it to... Even if we are migrating a legacy application, I would still like to see what a tool spits out for me. I just might use the new techniques. There are tons of ways to move data from one schema to another, again, outside the scope of this article. So back to the scope... EF Does a great job of creating an interaction between Database and Business Objects, but it doesn't help us rapidly create a NEW model without an existing database and I think this is important in two ways, first and foremost, I don't want to write any plumbing code to do this. The tools are smart enough already to do almost all the work for us, it can go just a little bit further and create the expected interaction that solves most of the interaction of the database work for us. I can create a Model with the EF Designer by adding Properties and Associations, but it won't go and create a database around that model.

I should not have to know about how the interaction happens, just that it does and that it does so in a fairly efficient manner. If I lose a certain percentage of performance due to layers, then so be it if I can create my application 10 times faster. If the tool developers choose to implement change tracking in a DataContext or Object Context that is great, do so in a way that makes sense for the CODE developers to use, not require me to go hire Database professionals to argue about the proper plumbing or storage techniques. Linq ALREADY knows how to create efficient SQL, if it did so in code generation vs. dynamically at runtime who cares?

We can host a solution out in the cloud so should we need to be worried about the optimization of how our application persists its data? Certainly there is is need to cover legacy applications and all the DBAs running around out there, that is well covered right now. What we seem to be missing is the ability for competent application developers to create Rich Internet Applications (read Silverlight and Ajax) and have the data persisted without a bunch of plumbing and tweaking.

What is NOT covered is the ability to rapidly envision, design, develop and deploy solutions destined to live on a Hosted Solution -- even if it is "hosted" internally. This solution will most likely have a distinct separation between UI, Business Objects and Database, but do I really need to see those distinctions in my code? Not really. Do I really need to spend the time to wire up all the Databinding when the MetaData already knows what I need to do? No Way, a tool can create that for us. It can already look into our metadata and see what we need to do to hook up a DataGridView to a table.

I would venture to say that the tools are already here, they just don't work the way we need them to so we can build these applications quickly. I spend MOST of my time Creating a Database, then working on plumbing and interactions between the Database and the UI, I should not have to do this any more. Microsoft did a fabulous job of giving us a database that can scale with us as our application grows. We have SQL CE, SQL Server Express and SQL Server Enterprise to grow with us, we can use SQL CE for Mobile Apps and offline stuff, wouldn't it be wonderful if I did not have to architect that portion of the solution any more? I am talking about the people who do not have the luxury of a full time DBA.

Of course, the solution is there, staring us right in the face, the answer is the Designer for Entity Framework. If this Designer could be used to build out the Database, and track the Schema changes, then it would take down part of the battle, just a little more work and it could completely remove the NEED (we should always have the option to extend and modify though) for us to write anything between the Business Objects and the Database.

We should not need to write Serializers for Silverlight, WCF or anything else, the tools already KNOW how to do that, they just don't. The MetaData is there, let's use it. Rob Conery's Subsonic and ActiveRecord shows there is interest in such a tool, but I say this tool needs to come from Microsoft and it needs to be built into the current generation of tools we are using or are being pushed to use (aka Linq). With Rob and Phil working AT Microsoft why can't they contribute some of that knowledge over to the ADO team and get this into the EF Tools we need. Pablo, Mike, Dinesh and the rest of the ADO Team are doing a great job so far, but they are only touching on what we really need. The newest version of EF finally gets a way to do disconnected graphs which is a step towards the goal thanks to Danny Simmons.

These tools need to work seamlessly with Linq and allow me as a developer to write applications with little or no concern for how my data is persisting. I want to create Controls that I can offer to my UI which work seamlessly with Ajax or Silverlight and not require me to write a ton of plumbing just to move things back and forth from one layer to another.

This is the challenge for the new world of development. We have already moved to a hosted world and our tools need to move there with us. But what do I know, I am a VB developer first and foremost... This is what I know, my customers are asking me to create these applications and to create them really fast. I can choose to use the tools I know best or I can branch out and find something new. The problem here is that I have already invested years of time to learn the tools I am using and they are simply not doing what I need them to do to make my job truly easier.

My Invoice Object doesn't really need to know what DataContext my Customer Object lives in and I as a developer should not have to be concerned about that in the slightest. I should be able to freely take a Customer I loaded from Invoice "A" and assign it to be used for Invoice "B" without worrying what context it came from, Invoice "B" just needs to know its a valid Customer or that it is a new one I just created without me forcing something into the plumbing to test it. It should know how to roll itself back when I cancel changes, etc.

With Generics and XML Literals, VB is a Top Notch Code Generator. Kathleen has proven that here and Karl talks about Metadata and what we can do with it over here... We don't need Yet Another Code Generator. We need something that works with the existing modelers and designers and just makes our life as developers easier. It needs to be INSIDE Visual Studio and we need to be able to work with the templates. I should not need LLBLGen or MyGeneration or CodeSmith or SubSonic to do something Visual Studio pretty much already knows HOW to do, it just doesn't. I don't need another designer that works with an existing database, I need the CURRENT Designers to Make the Database AND the plumbing.

I personally think this should be the direction for the next revision of Visual Basic. VB IS the glue that puts the framework together, sure you could also do it in C# (if they added XML Literals...) but VB is already geared for it so why not enhance it's productivity even further by giving us a way to be far more productive than we are today. This does not require any great shift in the tools we already have, it just requires Microsoft to use them smarter and provide us the keys we need to open the door.

Introducing LINQ To Regex

H. Steele Price, IV — Thu, 08 May 2008 10:59:00 GMT

Roy gives us 2 great things here: An easier to understand Regex for people who don't care to learn the ins and outs of regex AND more importantly an example of how to build a Linq Provider. I may tap into this idea for producing a couple Linq Providers I am thinking about, such as Linq to RSS and Linq to NNTP, alas, I really don't want to be a plumber this month so I am hoping someone beats me to the idea for Linq to NNTP which I could really use :-)

Introducing LINQ To Regex

When I first saw Josh Flanagan's Readable Fluent Regex Api, I loved it at first sight. Now I'm working on a demo for the upcoming DevTeach in Toronto, where I'll present how to build custom LINQ providers to various things.
I thought it would be interesting to see if using the fluent Regex API Josh created along with a LINQ query syntax would "work" for me, so I set out to create LINQ to Regex.

Introducing LINQ To Regex on Roy Osherove's Blog

Utility for building Linq Queries

H. Steele Price, IV — Thu, 08 May 2008 10:35:00 GMT

Visual LINQ Query Builder is an add-in to Visual Studio 2008 Designer that helps you visually build LINQ to SQL queries. Functionally it provides the same experience as, for instance the Microsoft Access Query Builder, but in the LINQ domain. The entire UI of this add-in uses Windows Presentation Foundation. The goal of this tool is to help users become more familiar with the LINQ syntax. The tool may also demonstrate to users how to create their own Visual Studio 2008 add-in using Windows Presentation Foundation.

This is very helpful in letting you see the big picture when you have particularly complex queries. I'd really like to see this integrated with something like LinqPad so we could get a better look at how we can structure Linq Queries. I'd also like to see this not need to add something to your Project just to use it, that is a real turn off for me, but you can always delete it when you are done. This is a good effort though on helping to simplify understanding new technologies. I'd also like to see it brach out to support more than Linq to Sql.

On the positive side, this does have a pretty good visual appearance in VS08 and I like how it is put together, building queries is pretty easy to do with the leveraging of intellisense. Some big missing items re Grouping and Aggregates, those tend to be the more difficult queries to visualize and where this tool would be most useful.

I would place this in the beginner category though because it will tend to get in the way of a seasoned Linq developer.

Change Tracking with Linq to SQL

H. Steele Price, IV — Mon, 05 May 2008 10:11:00 GMT

While I am doing this in a VERY Different way... it is worth mentioning the Unplugged LINQ to SQL Generator which is available on CodePlex:

http://pampanotes.tercerplaneta.com/2008/03/implementing-n-tier-change-tracking.html

When designing our application data model, we think some of our entities as an in-memory cache of small pieces of data living in a database (MSSQL, Oracle, Xml files, etc.). This data will be jumping between both worlds.

To keep this "cache" synchronized, we need mechanisms to read data from storage, and update changes back, to perform that we create CRUD operations.

How to use the Unplugged LINQ to SQL Generator

As Chris Rock pointed out, the first release of our Code Project custom tool for LINQ to SQL code generation lacks any usage documentation.

I am not sure this is the approach I want to take, but it does have some interesting possibilities.

I am in the process of writing up the way I am approaching this EXACT SAME DILEMNA in a different way; I am using inheritance of Generic Base Class to accomplish pretty much the same thing. The difference in my approach is that there is no need to do any CodeGen outside the regular way that Linq To SQL already does it, it's just there. This also has pluses and minuses so maybe a combination of the two is in order.

Of course, Mine is also in VB not just C#...

Mocks Nix - An Extensible LINQ to SQL DataContext

H. Steele Price, IV — Mon, 05 May 2008 10:00:00 GMT

mattwar tell us the proper way to0 "extend" DataContext in Linq to SQL.
"I often get asked how LINQ to SQL is supposed to be used with Test Driven Design (TDD).
LINQ to SQL was actually designed to be host to more types of back-ends than just SQL server. It had a provider model targeted for RTM, but was disabled before the release."

I am still reading this, some of it looks good, some of it looks not so good...

DISCLAIMER: Overriding internal implementation details is not a practice recommend or supported by Microsoft. Implementation details are subject to change without warning.

I am not sure that extending the DataContext is really what I need to do for my "Entities with State" model, but I am willing to look at it in that respect.

I think I still prefer what I am doing, ignoring most of the stuff the DataContext tries to help with (in a totally connected type of way...) and move those things to the Entity so I can be completely disconnected much easier.

DebuggerWriter for Linq to SQL

H. Steele Price, IV — Thu, 01 May 2008 13:12:00 GMT

I needed to find out exactly what was being submitted by DataContext.SubmitChanges and didn't want to use SQL Profiler.

I ran across this post by Kris Vandermotten and have converted the code to VB for anyone else who may need it.

This is now always in my Utility.vb file that gets attached to most of my projects.

    
Imports System.Diagnostics
Imports System.Globalization
Imports System.IO
Imports System.Text


''' <summary>
    ''' Implements a <see cref="TextWriter"/> for writing information to the debugger log.
    ''' </summary>
    ''' <seealso cref="Debugger.Log"/>
    Public Class DebuggerWriter
        Inherits TextWriter
        Private _isOpen As Boolean
        Private Shared _encoding As UnicodeEncoding
        Private ReadOnly _level As Integer
        Private ReadOnly _category As String

        ''' <summary>
        ''' Initializes a new instance of the <see cref="DebuggerWriter"/> class.
        ''' </summary>
        Public Sub New()
            Me.New(0, Debugger.DefaultCategory)
        End Sub

        ''' <summary>
        ''' Initializes a new instance of the <see cref="DebuggerWriter"/> class with the specified level and category.
        ''' </summary>
        ''' <param name="level">A description of the importance of the messages.</param>
        ''' <param name="category">The category of the messages.</param>
        Public Sub New(ByVal level As Integer, ByVal category As String)
            Me.New(level, category, CultureInfo.CurrentCulture)
        End Sub

        ''' <summary>
        ''' Initializes a new instance of the <see cref="DebuggerWriter"/> class with the specified level, category and format provider.
        ''' </summary>
        ''' <param name="level">A description of the importance of the messages.</param>
        ''' <param name="category">The category of the messages.</param>
        ''' <param name="formatProvider">An <see cref="IFormatProvider"/> object that controls formatting.</param>
        Public Sub New(ByVal level As Integer, ByVal category As String, ByVal formatProvider As IFormatProvider)
            MyBase.New(formatProvider)
            Me._level = level
            Me._category = category
            Me._isOpen = True
        End Sub

        Protected Overloads Overrides Sub Dispose(ByVal disposing As Boolean)
            _isOpen = False
            MyBase.Dispose(disposing)
        End Sub

        Public Overloads Overrides Sub Write(ByVal value As Char)
            If Not _isOpen Then
                Throw New ObjectDisposedException(Nothing)
            End If
            Debugger.Log(level, category, value.ToString())
        End Sub

        Public Overloads Overrides Sub Write(ByVal value As String)
            If Not _isOpen Then
                Throw New ObjectDisposedException(Nothing)
            End If
            If value <> Nothing Then
                Debugger.Log(level, category, value)
            End If
        End Sub

        Public Overloads Overrides Sub Write(ByVal buffer As Char(), ByVal index As Integer, ByVal count As Integer)
            If Not _isOpen Then
                Throw New ObjectDisposedException(Nothing)
            End If
            If buffer = Nothing OrElse index < 0 OrElse count < 0 OrElse buffer.Length - index < count Then
                ' delegate throw exception to base class
                MyBase.Write(buffer, index, count)
            End If
            Debugger.Log(level, category, New String(buffer, index, count))
        End Sub

        Public Overloads Overrides ReadOnly Property Encoding() As Encoding
            Get
                If _encoding Is Nothing Then
                    _encoding = New UnicodeEncoding(False, False)
                End If
                Return _encoding
            End Get
        End Property

        Public ReadOnly Property Level() As Integer
            Get
                Return Level
            End Get
        End Property

        Public ReadOnly Property Category() As String
            Get
                Return _category
            End Get
        End Property
    End Class

To use this for troubleshooting DataContext issues simply set the DataContext.Log = New DebuggerWriter such as this:

Dim tran As New Transactions.TransactionScope
Using tran
    db.Log = New DebuggerWriter
    db.SubmitChanges(ConflictMode.ContinueOnConflict)
    'rollback by disposing without complete
    tran.Dispose()
End Using

This will submit the changes, spit out all the SQL code to the Debugger Window and roll everything back.