Data Sets with JSON (SQL 2016)

With the release of SQL Server 2016 JSON objects support is now available. With the ever growing popularity and versatility of JSON, including JSON support into SQL Server is good for both database and application developers. In a previous post we looked at how to output a JSON object in a dataset using the FOR XML PATH clause and some string concatenation. This process can be rather difficult to read and updating can be a challenge. In this blog post we are going to use the FOR JSON PATH clause to simplify the JSON building process.

A brief background regarding JSON (JavaScript Object Notation). JSON is a lightweight data interchange format similar to XML, however it is based on specific programing languages such as C and Java. It is a data format that is easily understandable for both humans and machines. JSON is machine and language independent, which makes it a great language for data interchange between different systems. For more information regarding JSON please see the following URL https://www.json.org/json-en.html..

In the previous post our goal was to output the student information with a JSON object that described the courses and grades the student received.

{
  "Grades": [{
    "CourseCode": "[Course Code]",
    "CourseName": "[Course Name]",
    "Grade": "[Course Grade]"
  }]
}

In the previous blog post we used some crazy string concatenation to build out the JSON Object. Here is that query, with the string concatenation and using the SUBSTRING function to clean up the excess commas and whitespace.

SELECT s.StudentID
	 , s.FirstName
	 , s.LastName
	 , GradesJSON = '{"Grades":[' + SUBSTRING((
			SELECT ', {"CourseCode":"'+ c.CourseCode + '", "CourseName":"' + c.CourseName + '", "Grade":"' + sc.Grade + '"}' 
			FROM #StudentCourse sc JOIN #Courses c ON c.CourseID = sc.CourseID
			WHERE s.StudentID = sc.StudentID
			FOR XML PATH('')),3,1000) + ']}'
FROM #Students s;

Using the FOR JSON PATH clause instead of FOR XML PATH clause will simplify our syntax and make our code easier to read and understand. We will still need to output the same columns and use the correlated sub query idea to map the student course/grade data to the proper student in the outer query. The names of the columns will become the data elements in JSON object.

SELECT s.StudentID
     , s.FirstName
     , s.LastName
     , GradesJSON = (
           SELECT CourseCode = c.CourseCode
                , CourseName = c.CourseName
                , Grade = sc.Grade
           FROM #StudentCourse sc
           JOIN #Courses c ON c.CourseID = sc.CourseID
           WHERE s.StudentID = sc.StudentID
           FOR JSON PATH
       )
FROM #Students s
WHERE StudentID = 10005;
StudentID FirstName LastName Courses
10005WillWilliams[{“CourseCode”:”ICS140″, “CourseName”:”Introduction to Programming”, “Grade”:”A”}]

The only thing that is missing in the root grades element, so lets add that in.

SELECT s.StudentID
     , s.FirstName
     , s.LastName
     , GradesJSON = (
           SELECT CourseCode = c.CourseCode
                , CourseName = c.CourseName
                , Grade = sc.Grade
           FROM #StudentCourse sc
           JOIN #Courses c ON c.CourseID = sc.CourseID
           WHERE s.StudentID = sc.StudentID
           FOR JSON PATH, ROOT('Grades')
       )
FROM #Students s
WHERE StudentID = 10005;
StudentID FirstName LastName Courses
10005WillWilliams{“Grades”:[{“CourseCode”:”ICS140″, “CourseName”:”Introduction to Programming”, “Grade”:”A”}]}

There we have it, the same dataset structure including the nested JSON objects without all of the crazy string concatenation and SUBSTRING Function usage. Here is the full script:

DROP TABLE IF EXISTS #Students
GO
 
DROP TABLE IF EXISTS #Courses
GO
 
DROP TABLE IF EXISTS #StudentCourse
GO
 
CREATE TABLE #Students (
   StudentID INT PRIMARY KEY NOT NULL
 , FirstName NVARCHAR(50) NOT NULL
 , LastName NVARCHAR(50) NOT NULL
 )
GO
 
CREATE TABLE #Courses (
	CourseID INT PRIMARY KEY NOT NULL
  , CourseCode NVARCHAR(50) NOT NULL
  , CourseName NVARCHAR(50) NOT NULL
)
 
CREATE TABLE #StudentCourse(
	StudentID INT NOT NULL
  , CourseID INT NOT NULL
  , Grade VARCHAR(5) NULL
)
 
INSERT   INTO #Students
         (StudentID, FirstName, LastName)
VALUES   (10001, N'Bob',N'Roberts')
	   , (10002, N'John',N'Johnson')
	   , (10003, N'Tom',N'Tompson')
	   , (10004, N'Rich',N'Richardson')
	   , (10005, N'Will',N'Williams');
GO
 
INSERT INTO #Courses (
    CourseID
  , CourseCode
  , CourseName
)
VALUES (140, N'ICS140', N'Introduction to Programming')
     , (225, N'ICS225', N'Web Programming')
     , (240, N'ICS240', N'Advanced Programming')
     , (310, N'ICS310', N'Database Management');
 
INSERT INTO #StudentCourse (StudentID, CourseID, Grade)
VALUES (10001, 140, 'A')
     , (10001, 225, 'B+')
     , (10001, 240, 'B-')
     , (10001, 310, 'B')
     , (10002, 240, 'C+')
     , (10002, 310, 'A-')
     , (10004, 140, 'A')
     , (10004, 225, 'B')
     , (10005, 140, 'A');
 
 
SELECT s.StudentID
     , s.FirstName
     , s.LastName
     , GradesJSON = (
           SELECT CourseCode = c.CourseCode
                , CourseName = c.CourseName
                , Grade = sc.Grade
           FROM #StudentCourse sc
           JOIN #Courses c ON c.CourseID = sc.CourseID
           WHERE s.StudentID = sc.StudentID
           FOR JSON PATH, ROOT('Grades')
       )
FROM #Students s;

Comma Separated Values using the FOR XML PATH clause

In a previous post we looked at how we can build out a delimited list using the ISNULL and COALESCE functions. This functionality is great if you need to return the delimited list through a function or persist into a variable, however what if you need to return a comma separated list in a result set? This blog post will look at, and describe, how to build out a delimited or comma separated list using FOR XML PATH clause. Using tricks with the FOR XML PATH clause, a correlated subqueries, and un-named columns will give us all the tools needed to return a delimited list within a result set.

So the idea for this blog post is that we take a list of student’s details and include a comma separated list of the course codes that they have registered for. Here is a data diagram that represents our sample dataset.

CSV FOR XML PATH

Here are the populated tables.

StudentID FirstName LastName
10001 Bob Roberts
10002 John Johnson
10003 Tom Thompson
10004 Rich Richardson
10005 Will Williams

310ICS310Database Management

CourseID CourseCode CourseName
140 ICS140 Introduction to Programming
225 ICS225 Web Programming
240 ICS240 Advanced Programming
StudentID CourseID
10001 140
10001 225
10001 240
10001 310
10002 240
10002 310
10004 140
10004 225
10005 140

What we would expect the script to return is a result set with student details including a comma separated list of courses that looks like this:

StudentID FirstName LastName Courses
10001 Bob Roberts ICS140, ICS225, ICS240, ICS310
10002 John Johnson ICS240, ICS310
10003 Tom Tompson  
10004 Rich Richardson ICS140, ICS225
10005 Will Williams ICS140

Using the following temp table structure and sampling of student and course data gives us the data that we will need for this example.

DROP TABLE IF EXISTS #Students
GO
DROP TABLE IF EXISTS #Courses
GO
DROP TABLE IF EXISTS #StudentCourse
GO
 
CREATE TABLE #Students (
   StudentID INT PRIMARY KEY NOT NULL
 , FirstName NVARCHAR(50) NOT NULL
 , LastName NVARCHAR(50) NOT NULL
 )
GO
 
CREATE TABLE #Courses (
    CourseID INT PRIMARY KEY NOT NULL
  , CourseCode NVARCHAR(50) NOT NULL
  , CourseName NVARCHAR(50) NOT NULL
)
 
CREATE TABLE #StudentCourse(
    StudentID INT NOT NULL
  , CourseID INT NOT NULL
)
 
INSERT   INTO #Students (StudentID, FirstName, LastName)
VALUES   (10001, N'Bob',N'Roberts')
       , (10002, N'John',N'Johnson')
       , (10003, N'Tom',N'Tompson')
       , (10004, N'Rich',N'Richardson')
       , (10005, N'Will',N'Williams');
GO
 
INSERT INTO #Courses (
    CourseID
  , CourseCode
  , CourseName
)
VALUES (140, N'ICS140', N'Introduction to Programming')
     , (225, N'ICS225', N'Web Programming')
     , (240, N'ICS240', N'Advanced Programming')
     , (310, N'ICS310', N'Database Management');
 
INSERT INTO #StudentCourse (StudentID, CourseID)
VALUES (10001, 140)
     , (10001, 225)
     , (10001, 240)
     , (10001, 310)
     , (10002, 240)
     , (10002, 310)
     , (10004, 140)
     , (10004, 225)
     , (10005, 140);

We are going to start out by creating a simple query that returns all of the data we need, to generate the output. This will include all data from the student table, all data from the student course weak entity table, and only course code data from the course table.

SELECT s.StudentID, s.FirstName, s.LastName, c.CourseCode
FROM #Students s
LEFT JOIN #StudentCourse sc ON sc.StudentID = s.StudentID
LEFT JOIN #Courses c ON c.CourseID = sc.CourseID
StudentID FirstName LastName CourseCode
10001 Bob Roberts ICS140
10001 Bob Roberts ICS225
10001 Bob Roberts ICS240
10001 Bob Roberts ICS310
10002 John Johnson ICS240
10002 John Johnson ICS310
10003 Tom Tompson  
10004 Rich Richardson ICS140
10004 Rich Richardson ICS225
10005 Will Williams ICS140

Now that we have our dataset, let’s turn our focus on creating the comma separated list of course codes. Here we will implement our firsts bit of trickery. We are going to use the FOR XML clause, however we are going to pass an empty string to the PATH function. This will rename the XML root node as an empty string, thus not returning a root node. You can test this by running both queries, the first query will have a root node of <row> whereas the second query only returns the <CourseCode> nodes.

SELECT c.CourseCode
FROM #Students s
LEFT JOIN #StudentCourse sc ON sc.StudentID = s.StudentID
LEFT JOIN #Courses c ON c.CourseID = sc.CourseID
FOR XML PATH;
 
SELECT c.CourseCode
FROM #Students s
LEFT JOIN #StudentCourse sc ON sc.StudentID = s.StudentID
LEFT JOIN #Courses c ON c.CourseID = sc.CourseID
FOR XML PATH('');

Being that we were able to remove the root node by simply renaming the node, we need to do the same for the column name CourseCode. Here comes some more trickery, if we concatenate a string and do not provide an alias, the column name is undefined and thus is empty. In Management Studio you should see something like this “(No column name)”. This trickery actually works out to have an advantage because we need to concatenate our strings with a comma to create a comma separated list of values. So running the following query will give us all of the values in a comma separated list.

SELECT ', ' + c.CourseCode
FROM #Students s
LEFT JOIN #StudentCourse sc ON sc.StudentID = s.StudentID
LEFT JOIN #Courses c ON c.CourseID = sc.CourseID
FOR XML PATH('');
, ICS140, ICS225, ICS240, ICS310, ICS240, ICS310, ICS140, ICS225, ICS140

So now we have all of the course codes in one big comma separated list, we need to get them into sub lists for each student. If we nest the query with the FOR XML clause in a sub query and reference the student id from the outer query, this will limit the comma separated lists to only the course codes that are linked to the students.

SELECT s.StudentID
	 , s.FirstName
	 , s.LastName
	 , Courses = (
			SELECT ', ' + c.CourseCode
			FROM #StudentCourse sc JOIN #Courses c ON c.CourseID = sc.CourseID
			WHERE s.StudentID = sc.StudentID
			FOR XML PATH(''))
FROM #Students s;

The last piece is a bit of data clean up, using a STUFF or SUBSTRING we can remove the extra comma and white space from the comma separated list. Here is the final query that will return the result with a comma separated list in the result set.

SELECT s.StudentID
	 , s.FirstName
	 , s.LastName
	 , Courses = SUBSTRING((
			SELECT ', ' + c.CourseCode
			FROM #StudentCourse sc JOIN #Courses c ON c.CourseID = sc.CourseID
			WHERE s.StudentID = sc.StudentID
			FOR XML PATH('')),3,1000)
FROM #Students s;

Comma Separated Values with NULL Functions

Comma Separated Values, or values that are delimited by a specific character, are an easy way to pass a list of values from a function or in a dataset.  There are a couple ways to create comma separated values within SQL Server, as a single set which could be useful in a scalar function, or in a dataset which could allow passing additional values in a flat data set.  This blog post is going to focus on building a scalar delimited list using a NULL value function such as ISNULL and COALESCE.   There is a misconception that COALESCE has a mystical power which is needed to build a delimited list, we will take a look at this mystical power and show that the NULL value functions are technically not needed.

Before we dive into building the delimited lists, a quick review of the NULL value functions will be helpful. Both the ISNULL and COALESCE statements can be used to replace NULL values with a different specified value. However the COALSCE statement can evaluate multiple NULL value expressions checks whereas the ISNULL statement has a single evaluation and replacement. Here are a couple links with additional details regarding the COALSCE and ISNULL functions.

So the idea for this blog post is that we take a list of first and last names and return a concatenated, comma separated list. Here is our sample dataset.

FirstName LastName
Bob Roberts
John Johnson
Tom Thompson
Rich Richardson
Will Williams

What we would expect the script to return is a comma separated list of values that looks like this:

Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

Using the following temp table structure and sampling of first and last names gives us the data that we will need for this example.

DROP TABLE IF EXISTS #Person
GO
 
CREATE TABLE #Person (
   FirstName NVARCHAR(50)
 , LastName NVARCHAR(50))
GO
 
INSERT   INTO #Person
         (FirstName, LastName)
VALUES   (N'Bob',N'Roberts')
	   , (N'John',N'Johnson')
	   , (N'Tom',N'Tompson')
	   , (N'Rich',N'Richardson')
	   , (N'Will',N'Williams');
GO

The first step is to build out the string that we will put between the delimited values, in this case we will concatenate the first and last names. Instead of using the addition operator, we could have used the CONCAT function.

SELECT  p.FirstName + ' ' + p.LastName
FROM    #Person p;
(No column name)
Bob Roberts
John Johnson
Tom Thompson
Rich ichardson
Will Williams

As you can see from the result set, we have all of the names concatenated. Now we need to build all of the values into a single result. We can take and store all of the values into a variable, then return the result from the variable

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
Will Williams

The interesting fact here is that only the last value was persisted in the variable. In truth each of the values that were returned in the result set were written to the variable, but each value was overwritten by the next value. We will use this concept by concatenating the variable value with the first and last name values.

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
NULL

A NULL value was returned??? This is because the variable value was never initialized, therefore was set to NULL. When you concatenate a NULL value with any value you will end up with NULL. This is where the COALESCE function comes into play. If we use the COALSCE function to check for the NULL value and properly handle the NULL, we can properly concatenate our values and store them into the variable

DECLARE @FLNameCSV NVARCHAR(500);
 
SELECT  @FLNameCSV = COALESCE(@FLNameCSV + ', ', '') + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

So as we can see, the magic is not within the COALSCE function it is actually the variable concatenation that is the magic behind building the list of values. Further proof??? Let’s remove the COALSCE and assign a default value to our variable.

DECLARE @FLNameCSV NVARCHAR(500) = '';
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  @FLNameCSV;
(No column name)
, Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

We have our delimited list with an extra comma in the front, we can simply use function like SUBSTRING to remove the first few characters.

DECLARE @FLNameCSV NVARCHAR(500) = '';
 
SELECT  @FLNameCSV = @FLNameCSV + ', ' + p.FirstName + ' ' + p.LastName
FROM    #Person p;
 
SELECT  SUBSTRING(@FLNameCSV, 3, LEN(@FLNameCSV));
(No column name)
Bob Roberts, John Johnson, Tom Thompson, Rich Richardson, Will Williams

There we have it, the truth behind comma separated lists and the COALSCE function. In a future blog post, we will take a look at building comma separated lists using the FOR XML clause.