Comma Separated Values with NULL Functions

Comma Separated Values, or values that are delimited by a specific character, are an easy way to pass a list of values from a function or in a dataset.  There are a couple ways to create comma separated values within SQL Server, as a single set which could be useful in a scalar function, or in a dataset which could allow passing additional values in a flat data set.  This blog post is going to focus on building a scalar delimited list using a NULL value function such as ISNULL and COALESCE.   There is a misconception that COALESCE has a mystical power which is needed to build a delimited list, we will take a look at this mystical power and show that the NULL value functions are technically not needed.

Before we dive into building the delimited lists, a quick review of the NULL value functions will be helpful. Both the ISNULL and COALESCE statements can be used to replace NULL values with a different specified value. However the COALSCE statement can evaluate multiple NULL value expressions checks whereas the ISNULL statement has a single evaluation and replacement. Here are a couple links with additional details regarding the COALSCE and ISNULL functions.

So the idea for this blog post is that we take a list of first and last names and return a concatenated, comma separated list. Here is our sample dataset.

FirstName LastName
Bob Roberts
John Johnson
Tom Thompson
Rich Richardson
Will Williams

What we would expect the script to return is a comma separated list of values that looks like this:

Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

Using the following temp table structure and sampling of first and last names gives us the data that we will need for this example.

DROP TABLE IF EXISTS #Person
GO
 
CREATE TABLE #Person (
   FirstName NVARCHAR(50)
 , LastName NVARCHAR(50))
GO
 
INSERT   INTO #Person
         (FirstName, LastName)
VALUES   (N'Bob',N'Roberts')
	   , (N'John',N'Johnson')
	   , (N'Tom',N'Tompson')
	   , (N'Rich',N'Richardson')
	   , (N'Will',N'Williams');
GO

The first step is to build out the string that we will put between the delimited values, in this case we will concatenate the first and last names. Instead of using the addition operator, we could have used the CONCAT function.

SELECT  p.FirstName + ' ' + p.LastName
FROM    #Person p;
(No column name)
Bob Roberts
John Johnson
Tom Thompson
Rich ichardson
Will Williams

As you can see from the result set, we have all of the names concatenated. Now we need to build all of the values into a single result. We can take and store all of the values into a variable, then return the result from the variable

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
Will Williams

The interesting fact here is that only the last value was persisted in the variable. In truth each of the values that were returned in the result set were written to the variable, but each value was overwritten by the next value. We will use this concept by concatenating the variable value with the first and last name values.

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
NULL

A NULL value was returned??? This is because the variable value was never initialized, therefore was set to NULL. When you concatenate a NULL value with any value you will end up with NULL. This is where the COALESCE function comes into play. If we use the COALSCE function to check for the NULL value and properly handle the NULL, we can properly concatenate our values and store them into the variable

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = COALESCE(@FLNameCSV + ', ', '') + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

So as we can see, the magic is not within the COALSCE function it is actually the variable concatenation that is the magic behind building the list of values. Further proof??? Let’s remove the COALSCE and assign a default value to our variable.

DECLARE @FLNameCSV NVARCHAR(500) = '';
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
, Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

We have our delimited list with an extra comma in the front, we can simply use function like SUBSTRING to remove the first few characters.

DECLARE @FLNameCSV NVARCHAR(500) = '';
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  SUBSTRING(@FLNameCSV, 3, LEN(@FLNameCSV));
(No column name)
Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

There we have it, the truth behind comma separated lists and the COALSCE function. In a future blog post, we will take a look at building comma separated lists using the FOR XML clause.

Dynamic T-SQL Pivot

In a prior blog post (Standard T-SQL PIVOT) I showed how we can use a PIVOT query to take distinct values in a column and make those values new columns. However,  data changes and new columns may be needed. Looking at the data from the prior post, we only had three types of candy bars. What happens when a new candy bar shows up in our transactions? Well, we could update our query to use the new column value. If another new candy bar shows up we could update the query again. This process could go on and on and on. I am going to propose an alternative: using a dynamic PIVOT query. In this blog post I am going to show how we can use dynamic SQL to implement a dynamic PIVOT query.

Let’s start out by looking at the original data:

CustomerID ProductName Quantity
1 Snickers 1
1 Butterfinger 1
1 Rolo 2
1 Snickers 1
1 Butterfinger 1
2 Rolo 2

After using a PIVOT query we will see the data as described in the table below:

CustomerIDButterfingerRoloSnickers

CustomerID Butterfinger Rolo Snickers
1 2 2 2
2 NULL 2 NULL

Now update the original data with a few new products:

CustomerID ProductName Quantity
1 Snickers 1
1 Butterfinger 1
1 Rolo 2
1 Snickers 1
1 Butterfinger 1
2 Rolo 2
1 MilkyWay 3
2 Twix 2

If we apply that same query from the prior blog post, listed below, we see the same results as before.

SELECT   [CustomerID], 
         [Butterfinger], [Milkyway], [Rolo], [Snicker]
FROM    #CustomerProducts P 
PIVOT ( SUM(Quantity) FOR P.ProductName IN (
                          [Butterfinger], [Milkyway], [Rolo], [Snicker])
) AS PVT
CustomerID Butterfinger Rolo Snickers
1 2 2 2
2 NULL 2 NULL

But what we really wanted was

CustomerID Butterfinger Milkyway Rolo Snickers Twix
1 2 3 2 2 NULL
2 NULL NULL 2 NULL 2

As I mentioned before, we could simply update our query and this blog post is over. However, we are going to push on and create our dynamic version. Let’s take a look at creating the dynamic portion of the PIVOT query. The key part of the dynamic query is the distinct column list. Getting the distinct columns is just a simple DISTINCT query. To get the comma separated column list we will employ a trick that involves a COALESCE or ISNULL statement.

SELECT @PivotList = COALESCE(@PivotList + ',', '')

Once we have the distinct comma separated list, we can start to build out the dynamic query. We will create an nvarchar variable that we will use to store our dynamic SQL statement, and then we will execute that statement with the sp_executesql function.

Looking at our PIVOT query again, what we will want to do is replace the distinct columns with our comma separated list of values. Then we will take and store the new query into our variable

SELECT   [CustomerID], 
         [Butterfinger], [Milkyway], [Rolo], [Snicker]
FROM    #CustomerProducts P 
PIVOT ( SUM(Quantity) FOR P.ProductName IN (
                          [Butterfinger], [Milkyway], [Rolo], [Snicker])
) AS PVT

Converts to this

SET @Output = 'SELECT  [CustomerID]
	, ' + @PivotList + '
FROM    #CustomerProducts P 
PIVOT ( SUM(Quantity) FOR P.ProductName IN (' + @PivotList + ') ) AS PVT'

Now all that is left is to execute the stored query using the sp_executesql function. Here is the entire Dynamic SQL Pivot Script:

CREATE TABLE #CustomerProducts (
   CustomerID int
 , ProductName nvarchar(50)
 , Quantity int)
 
INSERT   INTO #CustomerProducts
         (CustomerID, ProductName, Quantity)
VALUES   (1, N'Snicker', 1),
         (1, N'Butterfinger', 1),
         (1, N'Rolo', 2),
         (1, N'Milkyway', 1),
         (1, N'Snicker', 1),
         (1, N'Butterfinger', 1),
         (2, N'Rolo', 2),
         (1, N'Milkyway', 3),
         (2, N'Twix', 2);
 
DECLARE @Output nvarchar(max) = N''
 , @PivotList varchar(max)
 
SELECT   @PivotList = COALESCE(@PivotList + ', ', N'') + N'[' + ProductName + N']'
 
FROM     (
          SELECT DISTINCT
                  ProductName
          FROM    #CustomerProducts
         ) AS CustProds;
 
SET @Output = 'SELECT  [CustomerID]
	, ' + @PivotList + '
FROM    #CustomerProducts P 
PIVOT ( SUM(Quantity) FOR P.ProductName IN (' + @PivotList + ') ) AS PVT'
 
EXEC sp_executesql @Output;
 
IF OBJECT_ID('tempdb..#CustomerProducts') IS NOT NULL	
	DROP TABLE #CustomerProducts