13 November 2006

COLLATE

Well, not strictly SQL 2005 related but, as some of our developers have recently noticed, our database servers are set to be case insensitive. SQL Server, by default, is case insensitive. For example, SELECT * FROM myTable WHERE column = 'six' gives the same results as SELECT * FROM myTable WHERE column = 'SiX'.

So, how do we do a case sensitive SELECT statement for something like a password comparison, when you don't want 'six' = 'SiX'?

SQL Server provides a COLLATE clause to do this. You may have seen this clause when generating creation scripts for tables. Here is a quick example:

CREATE TABLE [dbo].[cd_LandCategory] (
[LandCategory_SV] [int] NOT NULL ,
[Desc_TX] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
etc...

The second to last character group in the collation specifies the case sensitivity (CI = Case Insensitive).

It turns out you can also include the COLLATE clause in a query to specify the collation you want your query to use at runtime, completely independent of the DDL used to create the table and store the data (this is similar in function to the CAST function). Using the above table snippet, you may want to write a case sensitive query on the [Desc_TX] column:

SELECT [Desc_TX]
FROM [dbo].[cd_LandCategory]
WHERE [Desc_TX] = 'Old' COLLATE SQL_Latin1_General_CP1_CS_AS

I have changed the CI character group in the collation to CS, providing a case sensitive comparison. One thing to note is that, if I am comparing two columns, I will need to use COLLATE on both sides of the comparison or, I could get a collation error or other unexpected results:

SELECT [Desc_TX]
FROM [dbo].[cd_LandCategory]
WHERE [Desc_TX] COLLATE SQL_Latin1_General_CP1_CS_AS = [SomeOtherColumn_TX] COLLATE SQL_Latin1_General_CP1_CS_AS


The COLLATE clause seems to work the same on both SQL 2000 and SQL 2005.

A readily apparent use for this is comparing passwords in a case-sensitive manner (assuming you didn't create your password column with a case sensitive collation to start with; a common oversight).

Below is a quick code sample you can run on your favorite development server to see how collation, and overriding it with the COLLATE clause, works. It covers case sensitive and insensitive column collations with no COLLATE clause in the query. It also demonstrates overriding the column's collation using the COLLATE clause.

/******** COLLATION EXAMPLE 1.
Case-insensitive column and no COLLATE in where clause ************/

CREATE TABLE #tmp (pwd varchar(100) COLLATE SQL_Latin1_General_CP1_CI_AS not null)
INSERT
#tmp values ('@ssWord')
INSERT
#tmp values ('@ssword')
INSERT
#tmp values ('bit3mE')
INSERT
#tmp values ('bit3me')

SELECT * FROM #tmp
WHERE pwd = 'bit3me'

DROP TABLE #tmp
GO

/******** COLLATION EXAMPLE 2.
Case-sensitive column and no COLLATE in where clause ************/

CREATE TABLE #tmp (pwd varchar(100) COLLATE SQL_Latin1_General_CP1_CS_AS not null)
INSERT
#tmp values ('@ssWord')
INSERT
#tmp values ('@ssword')
INSERT
#tmp values ('bit3mE')
INSERT
#tmp values ('bit3me')

SELECT * FROM #tmp
WHERE pwd = 'bit3me'

DROP TABLE #tmp
GO

/******** COLLATION EXAMPLE 3.
Case-insensitive column and case-sensitive COLLATE in where clause ************/

CREATE TABLE #tmp (pwd varchar(100) COLLATE SQL_Latin1_General_CP1_CI_AS not null)
INSERT
#tmp values ('@ssWord')
INSERT
#tmp values ('@ssword')
INSERT
#tmp values ('bit3mE')
INSERT
#tmp values ('bit3me')

SELECT * FROM #tmp
WHERE pwd = 'bit3me' COLLATE SQL_Latin1_General_CP1_CS_AS

DROP TABLE #tmp
GO

/******** COLLATION EXAMPLE 4.
Case-sensitive column and case-INsensitive COLLATE in where clause ************/

CREATE TABLE #tmp (pwd varchar(100) COLLATE SQL_Latin1_General_CP1_CS_AS not null)
INSERT
#tmp values ('@ssWord')
INSERT
#tmp values ('@ssword')
INSERT
#tmp values ('bit3mE')
INSERT
#tmp values ('bit3me')

SELECT * FROM #tmp
WHERE pwd = 'bit3me' COLLATE SQL_Latin1_General_CP1_CI_AS

DROP TABLE #tmp
GO

17 February 2006

Crosstab / Pivot queries

I heard that SQL Server 2005 had this really cool new feature: PIVOT

Since I just finished working with one of our lead developers (name omitted to protect the innocent) on a SQL Server 2000 pivot query, I thought I’d take the chance to implement the query using SQL Server 2005’s new PIVOT feature.

First, we need to convert some of our rows into columns (21 in this case). With SQL 2000, we did this using a CASE statement to generate a count for each row that was to be converted to a column.

(SELECT
'AA'
= Count( CASE Action_ID WHEN 'AA' THEN ap_id END),
'AS'
= Count( CASE Action_ID WHEN 'AS' THEN ap_id END),

...
'RZ' = Count( CASE Action_ID WHEN 'RZ' THEN ap_id END),
'XX'
= Count( CASE Action_ID WHEN 'XX' THEN ap_id END)
) AS pivotTable

With SQL 2005, we still need to convert the rows to columns. However, the syntax is somewhat different. Rather than performing a CASE and count on each row, we get to specify the data values we are looking for and it will count them all for us.

PIVOT
(
count
(action_id)
FOR
Action_ID IN(
[AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX] )
)
AS pivotTable

Now that our basic pivot is put together (notice that both queries have an alias of “pivotTable”), we want to count how many occurrences of each unique row we have. We’ll start by selecting ALL of our converted columns and our count. This portion of the query looks the same in both versions of SQL Server.

SELECT [AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX], count(*) AS Frequency

Since we’re doing a count, and we want distinct values, we also have to group by all of our columns. This is another portion of our query that will look the same in SQL 2000 and 2005. We’ll also sort them so we get our most frequently occurring row counts at the top.

GROUP BY [AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX]
ORDER BY Frequency DESC

Now that we have all the pieces of our query put together, let’s tie them up into something useful. In SQL Server 2000, we just need to select our newly created columns from the sub-query containing all the CASE count statements (I’ll add in the base table name and the remainder of our 21 columns, at this point).

SELECT
AA, [AS], CA, CB, CE, CF, CP, CW, CX, CY, CZ, EX, IC, KC, KT, KV, LR, OE, PR, RZ, XX
, count(*) as Frequency
FROM

(
SELECT

'AA' = Count( CASE Action_ID WHEN 'AA' THEN ap_id END),
'AS'
= Count( CASE Action_ID WHEN 'AS' THEN ap_id END),
'CA'
= Count( CASE Action_ID WHEN 'CA' THEN ap_id END),
'CB'
= Count( CASE Action_ID WHEN 'CB' THEN ap_id END),
'CE'
= Count( CASE Action_ID WHEN 'CE' THEN ap_id END),
'CF'
= Count( CASE Action_ID WHEN 'CF' THEN ap_id END),
'CP'
= Count( CASE Action_ID WHEN 'CP' THEN ap_id END),
'CW'
= Count( CASE Action_ID WHEN 'CW' THEN ap_id END),
'CX'
= Count( CASE Action_ID WHEN 'CX' THEN ap_id END),
'CY'
= Count( CASE Action_ID WHEN 'CY' THEN ap_id END),
'CZ'
= Count( CASE Action_ID WHEN 'CZ' THEN ap_id END),
'EX'
= Count( CASE Action_ID WHEN 'EX' THEN ap_id END),
'IC'
= Count( CASE Action_ID WHEN 'IC' THEN ap_id END),
'KC'
= Count( CASE Action_ID WHEN 'KC' THEN ap_id END),
'KT'
= Count( CASE Action_ID WHEN 'KT' THEN ap_id END),
'KV'
= Count( CASE Action_ID WHEN 'KV' THEN ap_id END),
'LR'
= Count( CASE Action_ID WHEN 'LR' THEN ap_id END),
'OE'
= Count( CASE Action_ID WHEN 'OE' THEN ap_id END),
'PR'
= Count( CASE Action_ID WHEN 'PR' THEN ap_id END),
'RZ'
= Count( CASE Action_ID WHEN 'RZ' THEN ap_id END),
'XX'
= Count( CASE Action_ID WHEN 'XX' THEN ap_id END)

FROM
dbo.los_LoanServicingActions
GROUP BY

ap_id
) AS pivotTable

GROUP BY
AA, [AS], CA, CB, CE, CF, CP, CW, CX, CY, CZ, EX, IC, KC, KT, KV, LR, OE, PR, RZ, XX
ORDER BY Frequency DESC

We’ll also add the base table to our SQL 2005 query. Rather than going inside the pivot statement, however, it will be placed inside a sub-query right before the PIVOT clause.

SELECT [AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX], count(*) AS Frequency
FROM(SELECT Action_ID, ap_id FROM los_loanServicingActions) AS p

PIVOT
(
count
(action_id)
FOR
Action_ID IN(
[AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX] )
)
AS pivotTable
GROUP BY [AA], [AS], [CA], [CB], [CE], [CF], [CP], [CW], [CX], [CY], [CZ], [EX], [IC], [KC], [KT], [KV], [LR], [OE], [PR], [RZ], [XX]
ORDER BY Frequency DESC

As you can see, our final queries have the same opening and closing clauses. Be alleviating the CASE count line for each column, SQL 2005 allows us to express our pivot much more concisely. The one awkward thing about the 2005 pivot is enclosing the actual data values inside the IN clause in square brackets as if they were object names.

Well, form and brevity are all very nice but, how do they perform? It turns out, oddly enough (based on all the SQL Server 2005 release hype about performance), that they perform equally well (I pinned the source table in memory to remove disk IO as a factor).

SQL Server 2005:
(143 row(s) affected)
0.280000000 seconds


SQL Server 2000:
(143 row(s) affected)
0.280000000 seconds

So, which style should I use? Performance-wise, it’s a wash. It seems that the new PIVOT query may win in ease of use and reduced typing. One caveat to this is that the database must have the compatibility level set to 90 in order to use the new syntax. If you are developing a query that may have to be run against a database with a compatibility level lower than 90, or on SQL Server 2000 or older, you must continue to use the CASE count syntax.

08 February 2006

SSMS Source Safe integration

Bit of a publishing break but, hey, what's eleven months in the grand scheme of things? I'm now playing with several SQL Server 2005 RTM instances in DEV and one starring in a TEST role until new hardware is in place. We're getting ready to run all of the SQL 2K5 DEV/TEST on a single cluster instead of several separate machines.

At any rate, my current project is still being developed with SQL2K. However, I've begun using SSMS to develop all the non-DTS portions. The source safe integration is VERY NICE. No more check-out/check-in from VSS while editing in Query Analyzer and building the database from the command prompt. I've added my DOS commands for building the database into the SSMS tools menu and I can manage source control, development, and build all from a single interface.

Although the IDE isn't as mature (read "flexible") as VS2005, it looks like it will meet my DB needs quite well.

09 March 2005

Database Snapshots

SQL 2005 introduces database snapshots. You create a point-in-time snapshot that coexists on the server with the original database. It actually uses the same tables until the tables change, at which point, the original data pages (only the affected pages) are copied to the snapshot file. This is implemented through the use of 'sparse files', an artifact of the NTFS. BOL gives a nice overview (ms-help://MS.SQLCC.v9/MS.SQLSVR.v9.en/udb9/html/ f838de94-3ae7-49f7-9dbf-52ed30bcff81.htm if you have SQL 2005 BOL installed).

Based on some recent involvement with versioning test data and maintaining reporting data, I think this is going to be a nice feature. We can now maintain a pristine set of test data, right on the server with the database, and restore to that version after each round of testing. While testing is occurring, we can compare the original data in the snapshot with the data the tests modify. In SQL2K, this can be achieved by restoring backup files and maintaing a second database for comparisons, however, the overhead goes down considerably by having the two entities coexist.

One limitation I came up against was trying to maintain several snapshots for a single database (take a snapshot, make a change, take a snapshot, etc.). When it's time to roll back to a snapshot, only the snapshot to which you want to rollback can exist. You must first drop all the other snapshots for that database. It's nice having serval snapshot points from which to choose, however, it's frustrating to not be able to keep the other snapshot points (other than by taking backups of the source database in conjunction with the snapshots, which is how you get to do the whole snapshot thing in SQL2K, anyway).

Snapshots must be created and dropped using T-SQL. You create a snapshot with the create database statement using the AS SNAPSHOT <snapshot_name> clause. You rollback to a snapshot with the RESTORE DATABASE command using the FROM DATABASE_SNAPSHOT <snapshot_name> clause. DROP DATABASE <snapshot_name> drops the snapshot.

Snapshots can probably also be used for lightweight reporting. Take a month-end (quarter-end, year-end, etc.) snapshot and users have a read-only, point-in-time, picture of the database against which they can do their reporting (snapshot permissions can never be changed so, be sure any reporting specific permissions are in place before the snapshot is taken). I'm guessing you would still have the same old issues of your reporting database fighting for resources with your transactional database (especially since the two would actually be sharing some of the same data pages). I'd say use this technique judiciously.

03 March 2005

Light Auditing Using OUTPUT

Had a brief conversation at work today about the inserted and deleted tables that are available in sql 2000 triggers. The availability of these "tables" for sql 2005 has been expanded with the OUTPUT clause.

Any values that are inserted or deleted as the result of INSERT, UPDATE, or DELETE statements are available through this clause (the values as they exist BEFORE any triggers fire).

This is a handy way to enforce application level auditing without having to worry about maintaining triggers (and disabling them for non-audited, administrative udpates).

An example would really make this more clear. Since I don't want to clutter this up with sample code, I've put some code here(no longer available; 20060313), in case you have a sql 2005 instance on which you would like to run it. If you don't have an instance on which to run it, the query results are included at the end of the file. You can walk through the code to see how they were produced.

The Microsoft SQL Server Management Studio (still couldn't remember the whole thing) doesn't highlight OUTPUT as a keyword. Though it was rather ironic that one of the new features hadn't made it into the code editor, yet. Also noticed that NOCOUNT doesn't highlight like it does in query analyzer. Guess it's just a beta :-)


On a quick, unrelated note, table variables still don't support named constraints. For example,
DECLARE @tableVariable TABLE (pk int identity( 1, 1 )
CONSTRAINT [PK__tableVariable__pk] PRIMARY KEY CLUSTERED )
is not a valid sql statement while
DECLARE @tableVariable TABLE (pk int identity( 1, 1 ) PRIMARY KEY CLUSTERED )
and
CREATE TABLE tableName (pk int identity( 1, 1 )
CONSTRAINT [PK__tableName__pk] PRIMARY KEY CLUSTERED )
both are.

Sorry about the code pollution.

24 February 2005

2K from 2K5

The Microsoft SQL Server Management Studio (had to type it again because I still can't remember its name) can actually manage a SQL Server 2000 instance, as advertised. I tried it out and it seemed to work normally. However, I was unable to connect to the 2K5 instance from the Enterprise Manager (after all these years, I can remember that one, even without typing it out for reference). It gave an error telling me I needed to connect using the MSSMS (consider this a 'synonym' for the new name) or with SMO (SQL Server Management Objects, a scriptable set of objects for managing sql server that live in a DLL somewhere).

Even though I couldn't connect to the 2K5 instance with EM, I was able to connect to it with query analyzer. It's always nice to use a familiar tool when working on a new toy. Speaking of query analyzer, it has shortcut keys you can set up with sql statements. For example, I map the [ctl]+3 key combination to sp_helptext so I can just highight an object name and get the relevant sp_helptext results. I did not see this facility in the MSSMS query window (though you can turn on line numbers, just like basic on a TRS-80; yee-hah!)

Didn't get a chance to do much more playing this time around. The machine I have it installed on switched to Psychic Failover Mode (PFM) and had a network breakdown. It's feeling much better now but, why wait for it to happen in a more permanent manner? I am installing it on another machine where it will live side-by-side with a sql2k instance. I read somewhere that you can do that. If the machine comes back up when it's all said and done, I guess that will be a good sign. It's a dreadfully slow process, the install routine has to spend about 86400 seconds installing the .net 2.0 framework before it actually does anything useful. Microsoft is really keeping up with Sun on this one; .Net will soon have as many versions as Java. As long as we don't get fishing.net and cast.your.net to match java beans and what-not, I guess I can live with it.

I just noticed there are six parentheticals in these three paragraphs (now four paragraphs and seven parentheticals). It appears that staying on topic is not a strong point of SQL2K5. Good thing we can just add more processors and memory.

22 February 2005

Couple of useful new features

We can now INSERT INTO EXEC(...) into a table variable. We are no longer limited to using a temp table for this. Sorry, but, I like this new feature very much. Although temporary table storage is rather sloppy (and temporary tables are more sloppy than table variables), catching the results from xp_cmdshell is one seemingly legitimate use for them.

Also stumbled across the WITH EXECUTE AS clause. I was using some kind of stored procedure wizard thing (IDE folks are sure to love it) to make a stored procedure that provides directory listings using the new INSERT INTO EXEC... functionality. It executes the stored procedure (or queue, or trigger, or function) with the specified permissions. I have no idea what a queue is, beyond the obvious, 'it's a queue'; I'll have to dig that one up. At any rate, declare your procedure as normal...

CREATE PROCEDURE <owner>.<procedure>
@parm0 int
@parm1 int
/* with clause comes after parameter list */
WITH EXECUTE AS < CALLER | SELF | OWNER | 'user_name' >
AS ...

When the procedure is executed, the permissions of the specified executor are used when accessing all required base objects.
AS OWNER, the procedure is executed with the permissions of the procedure owner (dbo is common)
AS SELF executes the procedure with the permissions of the user that created the procedure
AS CALLER executes the procedure with the permissions of the process/user who is calling it
AS 'user_name' can be any valid user; the procedure will be executed with that user's permissions.

This will be handy when ownership chaining causes problems. Instead of the caller having to have permissions on the various base tables (messy bit of work, that is), the procedure can be executed as the owner, or any other use with the appropriate permissions. Let's see how long till there is an escalation exploit for this one :-)


Wow! schemas are no longer directly tied to database users (e.g., a table owner no longer has to be a user). Users can be managed without worrying about the effect on the objects they may have owned. I'll have to read up on this one.