A bug story
I run into a bug today with the way NHibernate dealt with order clauses. In particular, it can only happen if you are:
- Use parameters in the order clause
- Using SQL Server 2005
- Using a limit clause
If you met all three conditions, you would run into a whole host of problems (in particular, NH-1527 and NH-1528). They are all fixed now, and I am writing this post as the build run. The underlying issue is that SQL Server 2005 syntax for paging is broken, badly.
Let us take the this statement:
SELECT THIS_.ID AS ID0_0_, THIS_.AREA AS AREA0_0_, THIS_.PARENT AS PARENT0_0_, THIS_.PARENTAREA AS PARENTAREA0_0_, THIS_.TYPE AS TYPE0_0_, THIS_.NAME AS NAME0_0_ FROM TREENODE THIS_ WHERE THIS_.NAME LIKE ? AND THIS_.ID > ? ORDER BY (SELECT THIS_0_.TYPE AS Y0_ FROM TREENODE THIS_0_ WHERE THIS_0_.TYPE = ?) ASC
And let us say that we want to get a paged view of the data. How can we do it? Here is the code:
SELECT TOP 1000 ID0_0_, AREA0_0_, PARENT0_0_, PARENTAREA0_0_, TYPE0_0_, NAME0_0_ FROM (SELECT ROW_NUMBER() OVER(ORDER BY __HIBERNATE_SORT_EXPR_0__) AS ROW, QUERY.ID0_0_, QUERY.AREA0_0_, QUERY.PARENT0_0_, QUERY.PARENTAREA0_0_, QUERY.TYPE0_0_, QUERY.NAME0_0_, QUERY.__HIBERNATE_SORT_EXPR_0__ FROM (SELECT THIS_.ID AS ID0_0_, THIS_.AREA AS AREA0_0_, THIS_.PARENT AS PARENT0_0_, THIS_.PARENTAREA AS PARENTAREA0_0_, THIS_.TYPE AS TYPE0_0_, THIS_.NAME AS NAME0_0_, (SELECT THIS_0_.TYPE AS Y0_ FROM TREENODE THIS_0_ WHERE THIS_0_.TYPE = ?) AS __HIBERNATE_SORT_EXPR_0__ FROM TREENODE THIS_ WHERE THIS_.NAME LIKE ? AND THIS_.ID > ?) QUERY) PAGE WHERE PAGE.ROW > 10 ORDER BY __HIBERNATE_SORT_EXPR_0__
Yes, in this case, we could use TOP 1000 as well, but that doesn't work if we want pages data that isn't started at the beginning of the data set.
Now, here is an important fact, the question marks that you see? Those are positional parameters. Do you see the bug now?
SQL Server 2005 (and 2008) paging support is broken. I find it hard to believe that a feature that is just a tad less important than SELECT is so broken. Any other database get it right, for crying out load.
Anyway, by now you noticed that when we processed the statement to add the limit clause, we had re-written the structure of the statement and changed the order of the parameters. Tracking that problem down was a pain, just to give an idea, here is a bit of the change that I had to make:
/// <summary> /// We need to know what the position of the parameter was in a query /// before we rearranged the query. /// This is used only by dialects that rearrange the query, unfortunately, /// the MS SQL 2005 dialect have to re shuffle the query (and ruin positional parameter /// support) because the SQL 2005 and 2008 SQL dialects have a completely broken /// support for paging, which is just a tad less important than SELECT. /// See NH-1528 /// </summary> public int? OriginalPositionInQuery;
I fixed the issue, but it is an annoying problem that keep occurring. Paging in SQL Server 2005/8 is broken!
Oh, and just to clarify some things. The ability to use complex expressions for the order by clause using the projection API is fairly new for NHibernate, it is incredibly powerful and really scares me.
Comments
I had never loved mssql paging syntax, and never will.
I've always avoided positional parameters because it fails to fully encapsulate the structure of the SQL statement. Refactoring the SQL has the potential to break the calling code.
Surely NHibernate can be adapted to use named parameters with RDBMSes that support it.
Jason,
This is when using the criteria API, and the criteria API uses positional parameters because that avoids the need to create named parameters.
They are actually translated to named parameters anyway, the bug happened during construction of the SQL by NH, not at the DB level
__This is when using the criteria API, and the criteria API uses positional parameters because that avoids the need to create named parameters.
What do you mean by the 'need to create named parameters' ?
I think it makes more sense to use named parameters (by some GUID?) that are translated into positional for the RDBMSs that do not support named.
Andrey,
This related to the way NHibernate's works internally.
@Ayende: Being the huge NHibernate fan that I am, I must admit here that the problems lay in NH this time.
As Jason and Andrey has pointed out, had the query used named-parameters the 'problem' would have gone away.
as for the 'broken' paging syntax:
the thing is, that this syntax, although not trivial in first look, is actually very smart, and consistent with the way CTEs are used in SQL Server 2k5+. inventing a propriety SQL verbs like "LIMIT" looks cool, but it actually increases complexity.
plus, the SQL syntax you show (which NH is generating) is weird.
there is a redundant nesting level and the TOP is unneeded.
what should be done to a query:
take out the ORDER BY clause and extract it's expression parts.
add the ORDER BY expression parts (without DESC/ASC) to the SELECT clause. number them NH_sort_0, NH_sort_1, etc.
rename the expressions in the ORDER BY clause to match their new designated names
add ROW_NUMBER() OVER(ORDER BY the_order_by ) as ROW
wrap the SELECT with an external select, with the field names from the original select, without the ROW and the injected orders
add "ROW BETWEEN the_first AND the_last" to the WHERE clause
add ORDER BY the_new_order_by
so
SELECT
THIS_.ID AS ID0_0_,
THIS_.AREA AS AREA0_0_,
THIS_.PARENT AS PARENT0_0_,
THIS_.PARENTAREA AS PARENTAREA0_0_,
THIS_.TYPE AS TYPE0_0_,
THIS_.NAME AS NAME0_0_
FROM TREENODE THIS_
WHERE
THIS_.NAME LIKE ?
AND THIS_.ID > ?
ORDER BY (WHADEVER) A
would become
SELECT
ID0_0_,
AREA0_0_,
PARENT0_0_,
PARENTAREA0_0_,
TYPE0_0_,
NAME0_0_
FROM (
SELECT
ROW_NUMBER OVER(ORDER BY (WHATEVER) AS _NH_ROW_,
THIS_.ID AS ID0_0_,
THIS_.AREA AS AREA0_0_,
THIS_.PARENT AS PARENT0_0_,
THIS_.PARENTAREA AS PARENTAREA0_0_,
THIS_.TYPE AS TYPE0_0_,
THIS_.NAME AS NAME0_0_,
(WHATEVER) AS NH__SORT_0
FROM TREENODE THIS_
WHERE
THIS_.NAME LIKE ?
AND THIS_.ID > ?
)
WHERE NH_ROW BETWEEN 10 AND 1000
ORDER BY NH__SORT_0
is the middle query only for not repeating the order-by clause? that's crazy. I'm not sure but I think it will cause an extra table scan - first scan to generate the sort_field, second scan (in the middle query) to order by it.
if you want to avoid the duplication, it's better to use a CTE for that I think.
I'll see if I can spare a little time to look into the SQL dialect and try to make things a wee bit nicer in that aspect
To clarify, is the bug here a problem with NHibernate, or a bug in SQL Server 2005/2008?
The bug was with NHibernate, the problem is that SQL Server syntax sucks
yeah, nasty bug and is not the only one
if you execute an HQL like this
select this.ProductID, this.Discontinued, this.ProductName, this.UnitPrice, this.UnitsInStock, this.UnitsOnOrder, this.ReorderLevel, this.QuantityPerUnit, this.Category.CategoryName, this.Category.CategoryID, this.Supplier.SupplierID, this.Supplier.CompanyName from Product as this order by this.Category.CategoryID asc,this.ProductName asc
on sql server 2005 (Northwind) also using a limit and ordered by more than two fields will generate invalid SQL Server statement like this
SELECT TOP 50 x0_0_, x1_0_, x2_0_, x3_0_, x4_0_, x5_0_, x6_0_, x7_0_, x8_0_, x9_0_, x10_0_, x11_0_ FROM (SELECT ROW_NUMBER() OVER(ORDER BY __hibernate_sort_expr_0__, hibernate_sort_expr_1__) as row, query.x0_0_, query.x1_0_, query.x2_0_, query.x3_0_, query.x4_0_, query.x5_0_, query.x6_0_, query.x7_0_, query.x8_0_, query.x9_0_, query.x10_0_, query.x11_0_, query.__hibernate_sort_expr_0__, query.__hibernate_sort_expr_1 FROM (select product0_.ProductID as x0_0_, product0_.Discontinued as x1_0_, product0_.ProductName as x2_0_, product0_.UnitPrice as x3_0_, product0_.UnitsInStock as x4_0_, product0_.UnitsOnOrder as x5_0_, product0_.ReorderLevel as x6_0_, product0_.QuantityPerUnit as x7_0_, category1_.CategoryName as x8_0_, category1_.CategoryID as x9_0_, product0_.SupplierID as x10_0_, supplier2_.CompanyName as x11_0_, product0_.CategoryID asc as _hibernate_sort_expr_0__, product0_.ProductName as hibernate_sort_expr_1 from dbo.Products product0_, dbo.Categories category1_, dbo.Suppliers supplier2 where product0_.CategoryID=category1_.CategoryID and product0_.SupplierID=supplier2_.SupplierID) query ) page WHERE page.row > 0 ORDER BY __hibernate_sort_expr_0__, hibernate_sort_expr_1
Note the "product0_.CategoryID asc as __hibernate_sort_expr_0__" in the select field list.
this is a new behavior starting R3860, previous the same setup was generating invalid orderby
ORDER BY hibernate_sort_expr_0____hibernate_sort_expr_1
-> concatenating fields in the order by clause
B.
Is this still a problem in the trunk?
If so, please create a JIRA issue
Already did, Jira NH-1533
Is this the same as NH-1350 ?
Ken,
Named parameters would solve the issue, yeah, but that is not really related to the core of the issue, which is how much trouble you have to go in trying to get paging to work.
I don't care that CTE are smart. I want a simple solution to a very common and very simple problem. Smart == complex == hard to use.
The reason for the query being so complex is that we need to handle a lot more scenarios in a generic fashion.
In particular, take a look at http://jira.nhibernate.org/browse/NH-1155 to see one issue that significantly complicated the code.
Comment preview