Debugging and Profiling LINQ Queries using Devart LINQ Insight

Language Integrated Query (LINQ) is one of the features that makes .NET developers happy. LINQ opened up the opportunities to uniformly query in-memory objects, data in relational DBs, OData Services and many other data sources. Writing LINQ queries is fun as they reduce the usage of loops in code and keep the code base simple.

There are several ways to write LINQ queries to obtain a given result. As programmers, it is our responsibility to find the best way out of them and use that to make the applications perform better. Also, LINQ queries are hard to debug. It is not possible to debug a LINQ query using immediate window of Visual Studio debugging tools. So, we can’t say if a LINQ query is completely correct unless we write enough unit tests around it. Though unit tests are a good way to verify, nothing beats the ability of debugging the query by changing parameters.

LINQ Insight is a Visual Studio extension from Devart that makes debugging and profiling LINQ queries easier. It also provides profiling data for the LINQ queries and for CRUD operations performed using LINQ to in-memory objects as well as LINQ to remote objects (LINQ to SQL, Entity Framework, NHibernate and others). In this post, we will see a few of the features of this tool.

Debugging LINQ Queries
Once you installed LINQ Insight and restarted Visual Studio, you will see the options to see LINQ Interactive and LINQ Profiler windows under the View menu:

To debug the queries, we need to view the LINQ Interactive window. Select this option to view the window. Say, I have the following generic list of Persons in my application:

var people = new List<Person>() {
    new Person(){Id=1, Name="Ravi", Occupation="Software Professional", City="Hyderabad", Gender='M'},
    new Person(){Id=2, Name="Krishna", Occupation="Student", City="Bangalore", Gender='M'},
    new Person(){Id=3, Name="Suchitra", Occupation="Self Employed", City="Delhi", Gender='F'},
    new Person(){Id=4, Name="Kamesh", Occupation="Business", City="Kolkata", Gender='M'},
    new Person(){Id=5, Name="Rani", Occupation="Govt Employee", City="Hyderabad", Gender='F'}
};

Let’s write a LINQ query to fetch all males or, females from this list using a character parameter. Following is the query:

var gender = 'M';

var males = people.Where(p => p.Gender == gender).Select(p => new { Name=p.Name, Occupation=p.Occupation });

Right click on the query and choose the option “Run LINQ Query”. You will see this query in the LINQ Interactive window and its corresponding results.

As the query accepts a parameter, we can change value of the parameter and test behavior of the query. Hit the Edit Parameters button in the LINQ Interactive window and modify value of the parameters used in the query. On clicking OK, the query is executed and the result is refreshed.

Profiling Calls to Entity Framework
As most of you know, ORMs like Entity Framework convert LINQ Queries and function calls into SQL Queries and statements. The same result can be obtained using different types of queries. It is our responsibility to measure the execution time of all possible approaches of writing a query and choose the best one out of them.

For example, consider a database with two tables: Student and Marks.

The need is to find details of a student and the marks obtained by the student. It can be achieved in two ways. The first way is to find sum of the marks using navigation property on the student object and the other is to query the Marks DbSet using group by clause:

var studMarks1 = context.Students.Where(s => s.StudentId == studentid)
                        .Select(s => new
                        {
                            Name = s.Name,
                            Id = s.StudentId,
                            City = s.City,
                            TotalMarks = s.Marks.Sum(m => m.MarksAwarded),
                            OutOf = s.Marks.Sum(m => m.MaxMarks)
                        });

var studMarks2 = context.Marks.Where(m => m.StudentId == studentid)
                         .GroupBy(sm => sm.StudentId)
                         .Select(marksGroup => new
                         {
                             Name = marksGroup.FirstOrDefault().Student.Name,
                             Id = marksGroup.FirstOrDefault().Student.StudentId,
                             City = marksGroup.FirstOrDefault().Student.City,
                             TotalMarks = marksGroup.Sum(m => m.MarksAwarded),
                             OutOf = marksGroup.Sum(m => m.MarksAwarded)
                         });

Let’s see how much time does each of the above query takes using LINQ Profiler. Go to View menu and choose option to see the LINQ Profiler window. Now debug the application. After both of the queries are executed, visit the profiler window. You will see the data collected by the profiler. Following screenshot shows an instance of profiler’s data after running the above queries:

You can observe how much time each of these queries takes in different instances and with different values of the parameters before concluding that one of them is better than the other one.

As we saw, LINQ Insight adds a value add to the .NET developers. Devart has a number of other tools to help developers as well. Make sure to check them out and use them to make your development experience more enjoyable!

Happy coding!

Exploring Web API OData Query Options

As I mentioned in the post on CRUD operations using Web API, OData has a lot of query options. The full listing of standard query options can be found on the official OData site. In this post, we will explore the query options supported by Web API and we will also see how to use these query options from .NET clients through LINQ queries.

Query support on Web API Data can be enabled in following two ways:

• At the method level, by setting Queryable attribute on the Get method

    [Queryable]
    public IQueryable<Customer> Get()
    {
        .....
    }

• When application starts, by calling EnableQuerySupport method on HttpConfiguration object

    GlobalConfiguration.Configuration.EnableQuerySupport();

To be able to explore more options, we will be using the Northwind database. Create an ADO.NET Entity Data model with following tables:

• Customers
• Orders
• Employee

Let’s make these entities available to the Web API OData endpoint by adding them to an EDM model. Following statements accomplish this:

    var modelBuilder = new ODataConventionModelBuilder();
    modelBuilder.EntitySet<Customer>("Customers");
    modelBuilder.EntitySet<Order>("Orders");
    modelBuilder.EntitySet<Employees>("Employees");
    Microsoft.Data.Edm.IEdmModel model = modelBuilder.GetEdmModel();
    GlobalConfiguration.Configuration.Routes.MapODataRoute("ODataRoute", "odata", model);

Add a Web API controller named CustomersController and modify the code as:

public class CustomersController : ODataController
{
    NorthwindEntities context = new NorthwindEntities();

    public IQueryable<Customer> Get()
    {
        return context.Customers;
    }
}

Now we are all set to test the service. Run the project and change the URL on the browser to:

 localhost:<port-no>/odata

You should be able to see entries for each entity collection we added.



Now that the service is up and running, create a client project and add the service reference of the service just created (You may refer to my post on consuming OData from .NET clients post for setting up the client). Also, create a container object by passing URL of the Web API OData service.

Now that we have service and the client ready, let’s start exploring the different to query the service.

$filter:

All OData query options are query string based. $filter is the most basic and the most powerful of them. As the name suggests, $filter is used to select data out of a collection based on some conditions. The type conditions vary from simple equal to operator to string and date filtering operations.

As we already know, the following URL fetches details of all customers in the Northwind database:

    http://localhost:<port-no>/odata/Customers

Let’s add a condition to fetch details of customers with title Owner. Following URL gets the values for us:

    http://localhost:<port-no>/odata/Customers?$filter=ContactTitle eq 'Owner'

This filter can be applied from .NET client using the LINQ operator Where. Following is the query:

 var filteredCustomers = container.Customers.Where(c => c.ContactTitle == "Owner");

The above query is not fired in-memory. Instead, it generates the above URL by parsing the expressions passed into the LINQ operators. The generated URL can be viewed during debugging. Following screen-shot shows the URL generated by the above query:



As we see, it is same as the URL that we typed manually.

To select all customers with contact Ana at the beginning of their ContactName can be fetched using following URL:

 http://localhost:<port-no>/odata/Customers?$filter=startswith(ContactName,'Ana') eq true

The above query uses the OData function startswith() to compare initial letters of the field value. The corresponding LINQ query is:

    container.Customers.Where(c => c.ContactName.StartsWith("Ana"))

Similarly, endswith() can be used to compare suffix of a string. We can also check if a string is contained in a field using substringof() function.

 http://localhost:<port-no>/odata/Customers?$filter=substringof('ill',CompanyName)%20eq%20true

The corresponding LINQ query is:

    container.Customers.Where(c => c.CompanyName.Contains("ill"))

To get list of all customers with length of their names greater than 10 and less than 20, the URL is:

 http://localhost:<port-no>/odata/Customers?$filter=length(ContactName) gt 10 and length(ContactName) lt 20

You must have already guessed the LINQ query to achieve the same.

    container.Customers.Where(c => c.ContactName.Length > 10 && c.ContactName.Length < 20)

$filter supports querying based on numbers, dates and even checking types. You can get the full listing on the official site for OData.

$orderby:

Result can be ordered based on a field using $orderby operator. Syntax of specifying the operator is same as that of $filter operator. Following URL and the LINQ query fetch details of customers with names of the countries ordered in ascending order:

    http://localhost:<port-no>/odata/Customers?$orderby=Country 

    container.Customers.OrderBy(c => c.Country)

To make the order descending, we just need to specify the keyword desc at the end of the above URL.

$top and $skip

The server collections can be divided into pages from client using the $top and $skip operators. These operators are assigned with numbers that indicate the number of entries to be selected from the top or the number of entries to be skipped from the beginning of the collection.

Following URL and LINQ query select first 10 customers in the Customers table.

    http://localhost:<port-no>/odata/Customers?$top=10

    container.Customer.Take(10)

Syntax of using $skip is similar to that of $top, as $skip also expects a number to be passed in. Both $skip and $top can be used together to fetch data in several chunks. Following URL fetches 10 customers starting with index 40.

    http://localhost:<port-no>/odata/Customers?$skip=40&$top=10

    container.Customers.Skip(40).Take(10)

Paging can be forced from server side as well. If an entity set contains very huge amount of data, the requested client will have to wait for a long time to get the entire response from the server. This can be prevented by sending data in chunks from the server using server side paging. To get next set of values, the client has to send a new request to the server.

Server side paging:

Paging can be enabled on the server using PageSize property of the Queryable attribute. Following snippet applies a page size of 20 while returning a collection of customers:

    [Queryable(PageSize=20)]
    public IQueryable<Customer> Get()
    {
        return context.Customers;
    }

It can also be applied globally across all entity sets by setting the property to configuration object when the application starts, as follows:

    IActionFilter filter = new QueryableAttribute() { PageSize=20 };
    config.EnableQuerySupport(filter);

If you hit the URL to get all customers on your favorite browser after applying above changes, you will be able to see only 20 entries with a link to next set at the end of the result set, as shown in the following screen-shot:



As the URL is passed as a property of the JSON object, it can be easily extracted in any JavaScript client to send request for next set of values. But it is a bit tricky from .NET client. The collection LINQ query written on the client has to be converted to a DataServiceCollection to get the URL of the next request. It is shown below:

    var firstSetOfCustomers = new DataServiceCollection<Customer>(container.Customers);
    var nextSetOfCustomers = new DataServiceCollection<Customer>(container.Execute<Customer>(firstSetOfCustomers.Continuation.NextLinkUri));

Count of records on all pages can be obtained using the query option $inlinecount operator in the above query.

    http://localhost:<port-no>/odata/Customers?$inlinecount=allpages

    var firstSetOfCustomers = new DataServiceCollection<Customer>(container.Customers.IncludeTotalCount());

$expand:

ASP.NET Web API team added support of $expand and $select options in its latest release, which is version 5.0.0-rc1. It is built into Visual Studio 2013 Preview. In Visual Studio 2012, the NuGet package can be upgraded to the pre-release version.

$expand is used to include values of navigation properties of the queried entity set. The interesting part is the results of the navigation properties of the inner entity set can also be included in the result set. Following URL and LINQ query fetch details of Customers, their Orders and the details of the Employee who placed the Order.

    http://localhost:<port-no>/odata/Customers?$expand=Orders/Employee

    container.Customers.Expand("Orders/Employee");

The above query navigates till two levels down the entity set. Depth of the navigation can be controlled using the MaxExpansionDepth property on the Queryable attribute.

    [Queryable(MaxExpansionDepth=1)]
    public IQueryable<Customer> Get()
    {
        return context.Customers;
    }

This configuration can be applied on HttpConfiguration object when the application starts.

    IActionFilter filter = new QueryableAttribute() { MaxExpansionDepth=1 };
    config.EnableQuerySupport(filter);

After applying the above changes, the client cannot navigate to more than one level down the entity set.

$select:

A client can request for projected results based on its need. A list of properties to be fetched can be specified while querying the service to get only those properties. Following are a sample URL and the LINQ query:

    http://localhost:<port-no>/odata/Customers?$select=CustomerID,ContactName

    container.Customers.Select(c => new { ID = c.CustomerID, Name = c.ContactName });

Happy coding!

Basics - LINQ Expressions

LINQ is one of the most useful and most talked features added to .NET framework. It made our lives easier by providing a nice interface to program against list data and helps in writing readable code.

To support querying against in-memory collections, a number of extension methods were added to the interface IEnumerable. Following listing shows signatures of some of these extension methods:

public static long? Sum<TSource>(this IEnumerable<TSource> source, Func<TSource, long?> selector);
public static long Sum<TSource>(this IEnumerable<TSource> source, Func<TSource, long> selector);
public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count);
public static IEnumerable<TSource> TakeWhile<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);
public static IEnumerable<TSource> TakeWhile<TSource>(this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);

We have the interface IQueryable to support querying against remote collections. Any collection that doesn’t reside in the program’s memory is a remote collection. It can be a table in the database, a collection exposed through a service or anything similar. Following listing shows signature of corresponding extension methods defined in IQueryable:

public static long? Sum<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, long?>> selector);
public static long Sum<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, long>> selector);
public static IQueryable<TSource> Take<TSource>(this IQueryable<TSource> source, int count);
public static IQueryable<TSource> TakeWhile<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);
public static IQueryable<TSource> TakeWhile<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, int, bool>> predicate);

Notice the difference in the signatures of methods. Methods of IEnumerable accept Func and predicate, whereas the corresponding methods of IQueryable accept Expression<Func> and Expression<Predicate>.

The difference between Func and Expression<Func> is, Func is a delegate type and Expression<Func> is a data structure. An Expression is represented in the form of a tree. Each node of the tree contains some information which can be inspected. An expression can be created very easily using lambda. An example is shown below:

Expression<Func<int, bool>> compareExpression = (num) => num == 5;

Expression Tree Visualizer is a nice tool to view an expression in the form of a tree while debugging the application. To use it on a computer, one just needs to copy the library into Visual Studio visualizers folder. You can find the instructions in the codeplex site of the sample.

Following screenshot shows tree form of the expression defined above:

A simplified version of the above expression tree can be the following:

As we see, each node contains an expression. In other words, each individual part of an expression is also an expression. These expression types are defined in the namespace System.Linq.Expressions. This means, we can create a lambda expression by hand. Following code snippet demonstrates it:

//Parameter expression for the parameter num
var parameterNumExpr = Expression.Parameter(typeof(int), "num");

//Constant expression for the constant value 5
var constantVal5Expr = Expression.Constant(5);

//Lambda expression that accepts a parameter and compares with a constant
var lambdaCompareExpression = Expression.Lambda<Func<int, bool>>(Expression.MakeBinary(ExpressionType.Equal, parameterNumExpr, constantVal5Expr), parameterNumExpr);

If you view the above created lambdaCompareExpression using Expression Tree Visualizer, you should be able to see the same tree structure as of the previous expression. This expression can be used to query select all 5’s from an array of integers as follows:

int[] array = {2,6,5,1,343,54,5,23,75,46,5,18,3287,328,012,2,5 };
var nums = array.AsQueryable().Where(lambdaCompare);

The collection nums will have 4 5’s after execution of the above statement. As you saw, it takes quite a bit of work to create and use expressions by hand. But it is important to understand how an expression is formed and its components as we use them almost everyday. Following are some of the scenarios where expressions are used:

• Converting LINQ queries to database queries in LINQ to SQL, Entity Framework, NHibernate
• Creating OData URLs in WCF Data Services and Web API
• Fetching data from Windows Azure Table storage
• Configuring entities and their properties in Entity Framework code first fluent API
• Generating HTML input tags with validation attributes in ASP.NET MVC (K. Scott Allen explained this in his blog post on Why All the Lambdas?)
• Features like automatic paging and sorting list data with model binding in ASP.NET 4.5 Web Forms

Expressions are heavily used in .NET framework to provide a friendly interface to the programmers. While writing queries against LINQ to SQL or Entity Framework we feel as if the queries fetch data from in-memory collection. But there is a lot of work done under the covers. Data is extracted from each expression in the query and a SQL query is formed based on the information gathered. This query is fired against the database to fetch data.

Happy coding!