SQL

We don't need no stinkin' DBAs...

H. Steele Price, IV — Mon, 02 Jun 2008 18:02:00 GMT

This is precisely why I think that we need EntityDB. DBA's are great when you have the luxury of hiring one (or having one on staff). MicroISVs and Small companies with a need for custom software do not always have this luxury, nor the insight to hire one. In today's world of Code Generation, Metadata and an awesome pool of brain's at Microsoft - this can all be automated by some the best in the business. Linq Plumbing should not be rocket science, it should be straight-forward and simple to understand. I should not have to have a DBA to write a decent scalable model when the Object Model can generate a proper Database and communication layer all by itself. I say it CAN, it just doesn't because there is not yet enough screaming for it. So in the meantime, I will just keep screaming a little louder each day.

For example, one commercially available survey software that I won't name here, had code in it that retrieved of tens of thousands of rows from the database only to filter them out in a for/each loop in C#.
Needless to say, that ended up being a big bottleneck in terms of CPU use and network bandwidth.
The sales rep swore up and down about the scalability of the solution, when, in reality, it didn't take much traffic to bring the data center to its knees.
The moral of the story: trust your developers to write code, not manage huge amounts of data.
Depending on your developers, you may want to be careful about how much to trust them with the data.
Experienced DBAs can be a real pain to work with and that's what makes them great.
They demand referential integrity and schemas and won't let you make one little change to the schema to simplify what code you have to write.

Andy Leonard on Twitter's Woes

EntityDB: The next great Microsoft tool (if they would just make it)

H. Steele Price, IV — Tue, 20 May 2008 15:38:00 GMT

Reading in the Industry blogs, official samples and documentation provided with MS SQL Server 2005+, Visual Studio 2008 and Linq to Sql and the Entity Framework, you will get the impression that we have a tremendous ability to efficiently decipher how to interact with a Database. Additionally, Industry leaders have led us down a path of Code Generators, Designers and the separation of UI Design from functional Object code; read "Astoria" and/or "MVC" here...

Didn't CASE do this sort of thing for us years ago? Didn't it also fail? There is huge amount of traffic about the topic of "Falling down when you hit a threshold" and insistence that all things should be built ultra-scalable. OK, that's great if you love to write plumbing. I for one, don't. What if that could be done for you, with the most efficient options always at your disposal; of course with the opportunity to tweak and extend to meet a specific need.

While there are many great arguments to be had over what is exactly ultra-efficient and those who think they have a better method must be able to override and extend the selected generation of code that is offered by not only being able to write that code, but also to inject it into the Generated Code (or at the least allowed to edit the templates) produced by Visual Studio.

If you don't already know why this is so important, we need to first look at the massive shift in Business Software Development and where we are going to be running most business applications. We used to run all our apps on our own hardware, inside our own network and everything was under our tight control. While this model still applies to much corporate development, I believe it will become less and less used as a cost effective alternative to a HOSTED solution. Pricing of hosted solutions have plummeted in recent years, in fact I am almost willing to bet that the cost of electricity alone using hardware just a few years old would cover the cost of a hosted solution. And what happens when I DO need to scale, How do I add 20 servers to the Web Farm overnight? Staffing requirements and maintenance of hardware start to become totally offset when we look at this model. I CANNOT BUILD A SERVER CHEAPER THAN I CAN HOST ONE! That is the real bottom line.

I would wager that MOST applications would run just fine on a hosted solution with one machine doing all the work, at least in the beginning, or at least with a hoster that gives you Web on one machine and your DB on another. Are we talking about every solution here? No, but I think we are talking about a gigantic sector of the market that is currently all but ignored by the provided tools we have at our disposal.

Don't get me wrong here, we have GREAT tools available and more coming, do they target what is really needed?

First lets look at the three new primary tools given to us by Microsoft. WCF, Linq to Sql and the Entity Framework. WCF provides a great pipeline to pass data between our UI, Business Objects and Database, but it just plumbing. Linq to Sql and EF provide a great way to interact with a Database IF and ONLY If you have a proper database in place first.

What does this mean to most data-driven development projects? It means they either need a DBA, or they need to hire one for the portion of the application that defines the database. This is one reason Migrations for Ruby on Rails has gained so much popularity, it hides this chore. Most of us don't care about the DB, it's just a place that holds data. Some of you can go on and on all day about the subtleties of databases but that is completely outside the scope of what I am talking about here. Not only will Migrations create a DB and Schema around what your App needs, it will also keep it up to date with changes made by any developer that can be easily stored in Version Control software like SVN or TFS.

This is what the Designers for Linq to Sql and/or Entity Framework need to do for us. I really don't mind changing my Linq to SQL Apps over to Entity Framework if it will allow us to focus on one great technology. It's great that they can operate against a currently created database, but they should also create one (IN AN OPTIMIZED WAY) when one does not exist. It should create Stored Procedures if I tell it to... Even if we are migrating a legacy application, I would still like to see what a tool spits out for me. I just might use the new techniques. There are tons of ways to move data from one schema to another, again, outside the scope of this article. So back to the scope... EF Does a great job of creating an interaction between Database and Business Objects, but it doesn't help us rapidly create a NEW model without an existing database and I think this is important in two ways, first and foremost, I don't want to write any plumbing code to do this. The tools are smart enough already to do almost all the work for us, it can go just a little bit further and create the expected interaction that solves most of the interaction of the database work for us. I can create a Model with the EF Designer by adding Properties and Associations, but it won't go and create a database around that model.

I should not have to know about how the interaction happens, just that it does and that it does so in a fairly efficient manner. If I lose a certain percentage of performance due to layers, then so be it if I can create my application 10 times faster. If the tool developers choose to implement change tracking in a DataContext or Object Context that is great, do so in a way that makes sense for the CODE developers to use, not require me to go hire Database professionals to argue about the proper plumbing or storage techniques. Linq ALREADY knows how to create efficient SQL, if it did so in code generation vs. dynamically at runtime who cares?

We can host a solution out in the cloud so should we need to be worried about the optimization of how our application persists its data? Certainly there is is need to cover legacy applications and all the DBAs running around out there, that is well covered right now. What we seem to be missing is the ability for competent application developers to create Rich Internet Applications (read Silverlight and Ajax) and have the data persisted without a bunch of plumbing and tweaking.

What is NOT covered is the ability to rapidly envision, design, develop and deploy solutions destined to live on a Hosted Solution -- even if it is "hosted" internally. This solution will most likely have a distinct separation between UI, Business Objects and Database, but do I really need to see those distinctions in my code? Not really. Do I really need to spend the time to wire up all the Databinding when the MetaData already knows what I need to do? No Way, a tool can create that for us. It can already look into our metadata and see what we need to do to hook up a DataGridView to a table.

I would venture to say that the tools are already here, they just don't work the way we need them to so we can build these applications quickly. I spend MOST of my time Creating a Database, then working on plumbing and interactions between the Database and the UI, I should not have to do this any more. Microsoft did a fabulous job of giving us a database that can scale with us as our application grows. We have SQL CE, SQL Server Express and SQL Server Enterprise to grow with us, we can use SQL CE for Mobile Apps and offline stuff, wouldn't it be wonderful if I did not have to architect that portion of the solution any more? I am talking about the people who do not have the luxury of a full time DBA.

Of course, the solution is there, staring us right in the face, the answer is the Designer for Entity Framework. If this Designer could be used to build out the Database, and track the Schema changes, then it would take down part of the battle, just a little more work and it could completely remove the NEED (we should always have the option to extend and modify though) for us to write anything between the Business Objects and the Database.

We should not need to write Serializers for Silverlight, WCF or anything else, the tools already KNOW how to do that, they just don't. The MetaData is there, let's use it. Rob Conery's Subsonic and ActiveRecord shows there is interest in such a tool, but I say this tool needs to come from Microsoft and it needs to be built into the current generation of tools we are using or are being pushed to use (aka Linq). With Rob and Phil working AT Microsoft why can't they contribute some of that knowledge over to the ADO team and get this into the EF Tools we need. Pablo, Mike, Dinesh and the rest of the ADO Team are doing a great job so far, but they are only touching on what we really need. The newest version of EF finally gets a way to do disconnected graphs which is a step towards the goal thanks to Danny Simmons.

These tools need to work seamlessly with Linq and allow me as a developer to write applications with little or no concern for how my data is persisting. I want to create Controls that I can offer to my UI which work seamlessly with Ajax or Silverlight and not require me to write a ton of plumbing just to move things back and forth from one layer to another.

This is the challenge for the new world of development. We have already moved to a hosted world and our tools need to move there with us. But what do I know, I am a VB developer first and foremost... This is what I know, my customers are asking me to create these applications and to create them really fast. I can choose to use the tools I know best or I can branch out and find something new. The problem here is that I have already invested years of time to learn the tools I am using and they are simply not doing what I need them to do to make my job truly easier.

My Invoice Object doesn't really need to know what DataContext my Customer Object lives in and I as a developer should not have to be concerned about that in the slightest. I should be able to freely take a Customer I loaded from Invoice "A" and assign it to be used for Invoice "B" without worrying what context it came from, Invoice "B" just needs to know its a valid Customer or that it is a new one I just created without me forcing something into the plumbing to test it. It should know how to roll itself back when I cancel changes, etc.

With Generics and XML Literals, VB is a Top Notch Code Generator. Kathleen has proven that here and Karl talks about Metadata and what we can do with it over here... We don't need Yet Another Code Generator. We need something that works with the existing modelers and designers and just makes our life as developers easier. It needs to be INSIDE Visual Studio and we need to be able to work with the templates. I should not need LLBLGen or MyGeneration or CodeSmith or SubSonic to do something Visual Studio pretty much already knows HOW to do, it just doesn't. I don't need another designer that works with an existing database, I need the CURRENT Designers to Make the Database AND the plumbing.

I personally think this should be the direction for the next revision of Visual Basic. VB IS the glue that puts the framework together, sure you could also do it in C# (if they added XML Literals...) but VB is already geared for it so why not enhance it's productivity even further by giving us a way to be far more productive than we are today. This does not require any great shift in the tools we already have, it just requires Microsoft to use them smarter and provide us the keys we need to open the door.

Inside T-SQL

H. Steele Price, IV — Tue, 05 Jun 2007 05:45:00 GMT

Itzik Ben-Gan once again displays his mastery of T-SQL. He shows us ways to do somedificult or even unconventional things inside Stored Procedures and Functions. He also comments that CLR Functions can be up to 3 times faster at doing String Manioulation than using raw T-SQL. This of course make sense due to the effieciency that the CLR provides for this over the oftem cryptic or limited functionality provided by T-SQL.

His biggest offering was a phenomenal Pivot Stored Procedure that overcomes the limitations of the PIVOT() functionality (such as dynamic columns names) most of which can be found here He also has some pretty interesting uses of the STUFF() Function.

Aggregate Data with XML and SQL2005

H. Steele Price, IV — Thu, 22 Feb 2007 23:50:00 GMT

I was completely wrong about what I understood about GROUP BY with FOR XML PATH() commands.

I specifically asked about this at TechED last year and just never actually tried it, I just took it for granted that it didn't work for some reason. For the heck of it (and out of complete frustration), I did try it a week ago and lo and behold it just worked...

I quickly tore apart some old procedures that FINALLY got upgraded to SQL2005 and the difference in the code and speed where dramatic! I am totally happy with this solution.

The following works perfectly fine (and fast...)

    SELECT
        RequestorID As '@RequestorID', 
        dbo.fn_GetLBRequestorName(RequestorID,@LocationUID) As '@RequestorName',
        Count(LogEntryID) as '@Entries', 
        Sum(Pages) as '@Pages', 
        Sum(Billable) as '@Billable', 
        Sum(CASE Billable WHEN 0 THEN 1 ELSE 0 END) as '@NonBillable', 
        Sum(CASE WHEN ((Date_Completed > '01/01/1900')) THEN 1 ELSE 0 END) as '@Completed',
        Sum(CASE WHEN ((Date_Completed < '01/01/2000')) THEN 1 ELSE 0 END) as '@Incomplete'
    FROM
        LogEntry
    WHERE
        (LocationUID = @LocationUID) AND
        (
            ((@StartDate is Null) OR (@EndDate is Null)) OR 
            (Date_Received between @StartDate and @EndDate)
        )
    GROUP BY RequestorID
    ORDER BY RequestorID
    FOR XML PATH('Requestor'), TYPE, ROOT('Requestors')

I will be putting an updated article up revealing my findings and the sheer delight it is to create Structured XML with SQL 2005. The SQL 2000 Solution is over here and is pretty ugly.

SQL Server 2005 - XML and Recursion

H. Steele Price, IV — Thu, 15 Jun 2006 15:26:00 GMT

I have been thinking about some ways to combine two new features of SQL 2005 to do some very complex Hierarchies without having to resort to using SQL Explicit and really confusing, verbose scripts. Those two Features are For XML and Recursion.

I attended the two deep dive sessions for each of these today and have a few take-aways that are going to help me to formulate my ideas around this.

First, in the XML Deep Dive, Michael Rys gave a little more detail than he was able to at last year's PDC session that I attended. Here are some features that I am going to definitely be using in my development:

Type directive - Enables you to return an XMLDocument datatype to ADO instead of raw text strings, the primary advantage here is that you don't need to use a SQL Datareader to concatenate all the XML results any more. Just tack it on like this… "select * from orders for xml auto, type"
Path Mode - This lets you rename the elements returned easily, path('myelement') returns <myelement> instead of the table name.
With XmlNamespaces - Allows you to set specific namespaces for the elements
Wrapper Elements through Column Aliasing and Nested Subqueries to support additional hierarchies
XslNil to support empty elements
Root directive to support adding a Root element - i.e. root('MyRoot')

One thing I was disappointed to hear is that "Group By" is still not allowed explicitly you have to hide it some way, like in this previous article I wrote for aggregates.

UPDATE: I was totally wrong about this, I guess I misunderstood, I feel like an idiot now for not just trying it... Group By works fine when structured like this

Itzik Ben-Gan gave a great presentation on doing recursion effectively. I really got a lot out of his advanced techniques for using CTEs (Common Table Expressions).

Consider using CTEs when you need to retrieve self joins that exhibit nested hierarchies in your data such as an OrgChart (so and so reports to so and so…) or Part Kits, Kits are a list of parts that may contain other Kits.

Sometimes people resorted to using cursors or temp tables to achieve these results in the past, now there is a very efficient way to do this type of recursion in SQL Server.

Recursion can be inefficient when it needs to replicate a lot of data to call itself, but the SQL Team decided to internally do all recursion as loops which are much more efficient. There is still a safety net to prevent non-terminating loops (usually due to data errors) which you may turn off programmatically if you are really sure you want to do infinite recursion. The MaxRecursion property is not something you should use to limit data either, you should be doing this with a proper recursion level counter as one of your rows and jump out at whatever threshold you set, also, you should be doing boolean tests to find repeating cycles so that infinite loops are avoided.

There is now a RowNumber() function that is extremely fast and with a little though about recursion can be used for on the fly sorting in hierarchical recordsets. This can really improve your performance over sorting by non-indexed text fields and doing multiple sorts in hierarchies. I'll post more about that in a future article.

Now, putting all of this together, I am sure that I can create very efficient and very fast ways of creating Hierarchies that involve nested joins, recursion, subqueries and aggregates (all at the same time) that where either impossible or at best extremely difficult in the past. When I have some time to organize my thoughts about this when I get back from the conference I will write up an article that talks about it in detail.

SQL Server 2005 - Everywhere Edition

H. Steele Price, IV — Thu, 15 Jun 2006 09:01:00 GMT

Microsoft Officially Announces SQL Server 2005 - Everywhere Edition.

Have you every wanted to just access some LOCAL data on your laptop without having to install SQL Express or some other server system just to store some info? Of course you have, most of us have used things like access .MDB files, XML, or even raw text to do this in the past, now we have the option of storing SQL Server data locally and further have the ability to sync and merge with a server on a remote network for occasionally connected systems.

If you have done any Mobile Development for PDAs or Tablets then you may know there is the option of SQL Mobile… That is essentially what this is (renamed), but it is being greatly enhanced. Some of the internal enhancement I can't talk about yet, but here is what I can:

Licensing - Free (essentially the same as SQL Express) now available on XP/2003/Vista as well
Database Size Limit - 4Gb (not set in stone, but most likely)
Runtime Size - 1.4Mb - running In Process with your App (including Office Apps for VSTO developers)
This means no server to install, just some DLLs that are independent from anything else and can just be added/removed without harming any other app on your system.
Single Data File(.sdf) - no transaction log, etc.
Sync with a backend SQL Server through Merge & Replication - (More than just RDA soon)
Differencing Tools in the ADO.NET extensions (MergeUpdated(), MergeInserts(), MergeDeletes())
Complete control over collision handling so you can choose client wins, Server wins or create a custom solution because you have access to both sides of the colliding row.
Multi-connection support so you can run multiple background processes on the same file.
Windows can handle the ACL on the file so you can lock the file to a group of domain users, etc.

For just replicating server data locally for partially disconnected access this will really be useful. I look forward to installing and testing the CTP that is now available here.

Should I do that in T-SQL?

H. Steele Price, IV — Tue, 13 Jun 2006 08:02:00 GMT

Richard Campbell delivered an "as always" great session about SQL Query optimization, this year his special flavor of tools extend to differences in SQL 2005 and where very enlightening. update: Here's the code.

One thing he covers has always been the fabulous Rozenshtein method of doing Crosstabs (Pivot tables) While this is not really necessary since we have the Case Statement; it is quite useful when you need to create something that will run in virtually any database that supports SQL statements or when you need to create dynamic SQL and for some reason can't use a Stored Procedure.

He noted the still persistent problem of crosstabs in SQL 2005 with the new Pivot Command. While this does help, it still forces you to hard code the column names or build the SQL Dynamically (which will bypass the compiler and optimizer). If you want to know the real reason for this, it's because the low-level plumbing is extremely difficult to achieve in a performant way and will most likely be included in a future release when they can pound the algorithms down to improve performance.

Another noted example is to get rid of cursors by using temp tables (and Table Variables I might add).

Also covered was the new Ranking commands in SQL 2005, and how to simulate them (with small differences) in SQL 2000.

Overall a most enjoyable session, Thanks Richard.

Language Integrated Queries

H. Steele Price, IV — Sat, 17 Sep 2005 10:22:00 GMT

The LINQ Project is probably the most exciting announcement of PDC for me.

I have been using an O/R Mapper for quite some time and I couldn't imagine designing a new application without using one.

We all heard about ObjectSpaces way back on 2002 and how it would bring object orientation to relational database UIs. Then we all wondered what happened to it when they decided to pull it from the 2.0 Framework. Well, the ideas of ObjectSpaces were rolled into an even more ambitious project, that is LINQ.

LINQ is broken into several Namespaces: DLinq, XLinq; which target SQL and XML specifically.

LINQ will not render tools such as LLBLGen useless, in fact, it will further enhance what they do best and allow them to focus on how to do the mapping, LINQ will provide a unified approach to provide the plumbing while third party tools can focus on doing the mapping for us.

Then using LINQ we can interact with our data in a very object oriented way that is standardized across implementations and does not require us to remain dedicated to a 3rd party solution to provide proprietary ways to actually do the objectness.

One of the biggest gripes I hear from people about XML is how hard it is to understand and implement things like XPath and XQuery. XLinq and DLinq are going to unify the way we interact with persited data in either XML or SQL, we will essentially access them both in the same way, this is HUGE, it removes the ramp up time developers will need to be able to intuitively write programs that need to interact with data.

As the project progresses, other persistant data will be added so that we can interact with things like the Registry, WMI, LDAP and other structures in the exact same way without having to learn the internal plumbing required to do that sort of interaction.

Community Launch Team Member

H. Steele Price, IV — Wed, 14 Sep 2005 20:40:00 GMT

I am the INETA Community Launch Team Member for Phoenix.

The Community Launch Team Members will be the local experts on the new products and will speak to their user group at their local Community Launch event of Visual Studio and SQL Server after the official product launch.

This means that I will be the guy who has the pleasure of introducing and speaking about Visual Studio 2005 and SQL Server 2005 to the Phoenix area.

Feel free to email me with any questions you may have about these two outstanding tools and how they will impact your development process.

On Dynamic SQL

H. Steele Price, IV — Wed, 22 Jun 2005 11:58:00 GMT

About a Year and a Half ago, I posted this comment during the great StoredProc vs. Dynamic SQL debate between Rob Howard and Frans Bouma. I was at the time in favor of Stored Procs.

After much work, frustration, observation and rational thinking, I have completely reversed my opinion.

The biggest reasons I have reversed that opinion is because I am moving toward embracing a Service Oriented Architecture and Connected Systems. This doesn't have a lot to do with the whole debate until I started generating my Business Logic Layer and Data Access Layer with LLBLGen Pro.

Granted I was sceptical, I strongly debated whether to use CSLA and Stored Procs or a DQE (dynamic query engine). In the final analysis, I chose the DQE.

Why, you ask? The reason is simple, doing this:

WHERE
     (@MyParam1 is NULL or MyParam1_Field Like @MyParam1) AND
     (@MyParam2 is NULL or MyParam2_Field Like @MyParam2)

or this:

WHERE
     MyParam1_Field = COALESCE(@MyParam1,MyParam1_Field) AND
     MyParam2_Field = COALESCE(@MyParam2,MyParam2_Field)

is just stupid, especially when you have a dozen parameters against a table with a million rows.

It's ugly programming any way you look at it and it is just a hack.

Not only does the performance suck, it's hard to maintain (even when generated). Furthermore, in CRUD StoredProcs you are still using a procedure that is saving a whole record instead of just the fields that have changed.

When I need to save a complex data object with elements from different tables, making a Stored Proc do this efficiently and passing back validation/locking errors, etc. is something my DAL should be doing and not something actually inside the database.

When I started looking at my database as just a persistant storage mechanism and not part of my program, then ripping out all the business logic from inside the database just makes sense.

If you want to connect to multiple Database backends, then this starts becoming an even bigger priority. Sure a few of the other players support StoredProcs, you are still going to have to generate them all for each database vendor.

Why would I want to go through that pain when I can just let a DQE do it for me and not worry about maintaining change scripts.

Security is another big issue people throw up when talking about SPs vs. DSQL, well, for the last couple of years, I have ONLY used Integrated Security in my connection string so that is no longer a selling point for SPs any more either.

So I admit it, I have flip-flopped. As we move towards an even more disconnected and distributed world the logical layers we use to architect our applications are going to end up in the hands of automation tools and hopefully hand-tooling things like Stored Procedures will become a piece of architectural history.