Slow query - big difference between estimated and actual rows

Question

At my work, we have a problem with a search query.

If I don't include the lookup into Table_39 (see below query), it takes 0 seconds to execute the query. If I include the lookup (..AND (Not Exists(Select 1 From Table_39 ll ..), it takes 4 minutes and 57 seconds.

From the execution plan I can see that there is a huge discrepancy between number of expected rows and actual rows, especially in the index for Table_39 (62.6 estimated vs. 10,956,165), so I tried running UPDATE STATISTICS WITH FULLSCAN, but it didn't make any difference.

I would appreciate any input I can get on how to improve the performance of the query that includes the lookup!

Query:

select [Table_2].Gid
  into #Table_1
  from [Table_2]
where [Table_2].Gid_Line in ('{31445a8f-900b-4d64-b72f-4e2ac10fd6a7}')
union
select '{31445a8f-900b-4d64-b72f-4e2ac10fd6a7}' as Gid

create unique nonclustered index temp_Table1 on #Table_1 (Gid)
go

select top 500 Table_9.Id
 , Table_9.Gid
 , Table_9.name
 , Table_9.TimeEnter
 , Table_9.Prio
 , Table_9.Direction
 , Table_9.status
 , Table_9.CreaterId
 , Table_9.Subject
 , Table_9.BodyType
 , Table_9.Attach
 , Table_9.FromAddr
 , Table_9.ToAddr
 , Table_9.Gid_LineSet
 , Table_9.xml
from Table_9
where Table_9.Gid_LineSet in (
    select Gid
      from #Table_1
      ) 
  and not exists (
   select 1
   from Table_39 ll
   where ll.Id_Obj_To      = Table_9.Id 
    and ll.Id_ObjType_To   = 5 
    and ll.Id_ObjType_From = 3 
    and ll.Id_Obj_From in (
        9602, 10661, 10857, 10858, 10859, 68823
      , 68824, 68825, 68826, 68827, 68828, 68829
      , 68830, 68831, 68832, 68833, 68834, 372513
      )
    )
order by Table_9.Id desc

drop table #Table_1

Execution plan for slow query (with lookup): https://www.dropbox.com/s/rc1i6vvdwwc5hzh/long%20query_Cleaned.sqlplan?dl=0

Execution plan for fast query (no lookup): https://www.dropbox.com/s/bjafn5tiy3uepul/short%20query_Cleaned.sqlplan?dl=0

score 4 · Answer 1 · edited Jun 15 '20 at 09:05

Row goal

The issue with the estimates is most likely due to the row goal introduced by the TOP (500). See:

How (and why) does TOP impact an execution plan?

If you find the estimates are much better without the TOP, or by adding OPTION (QUERYTRACEON 4138) to the query, that would confirm the hypothesis.

Trace flag 4138 is documented in KB 2667211:

A query may take a long time to run if the query optimizer uses the Top operator in SQL Server 2008 R2 or in SQL Server 2012.

You should also enable optimizer fixes with trace flag 4199.

Indexing

Consider creating an index on table 39, keyed on:

Id_Obj_To, Id_ObjType_To, Id_ObjType_From, Id_Obj_From

Ensure Id_Obj_From has a suitable data type (not e.g. a max type).

Predicte ordering

The optimizer does not reorder NOT EXISTS clauses well in your version of SQL Server. If you expect the NOT EXISTS clause to be more selective than the Gid_LineSet predicate, reorder those two clauses in the WHERE part of the query.

score 0 · Answer 2 · answered Dec 15 '16 at 15:09

I have not seen the execution plans, the links don't work.

When you do a where not in Table_39, it will return all rows. Try to put the data from Table_39 into a temp table based on "And ll.Id_Obj_From in(9602,10661,10857,10858,10859,68823,68824,68825,68826,68827,68828,68829,68830,68831,68832,68833,68834,372513)"

It will be a smaller table and could reside in memory instead of written to the tempdb. Only get the columns you need and put them into the temp table.

As an example, I can put 1000 rows into a temp table if they are INT and will reside in memory. But if I put 2 rows of XML at 20 Meg for each row into a temporary table it will be written to the tempdb. This is an exagerated sample, but the point is whether a temp table reside in memory or is written to the tempdb depends on the size of the data.

Slow query - big difference between estimated and actual rows

2 Answers2

Row goal

Indexing

Predicte ordering