-1

I have created a new dataset that has 11,000,000+ rows with 4 pivot tables in MySQL. The tables are not that deep, only 6 - 12 cols per.

My question is I've set up the Lucene Apache SOLR to index the data, and it work's great for searching .. BUT

I haven't noticed a dramatic load time in searches. If I run a raw SQL command, the difference between the two is negligible. At what point is it more beneficial to use a indexing engine vs scripting a raw query yourself?

Now granted I have the ability to set "weight" and all that via SOLR .. But my impression was that I would have dramatically reduced overhead on the server ... Is it just that my dataset isn't large enough / complicated enough to illicit these finding? Other than the obvious advantages being weight, rank and sort/filter -- What is the overhead advantage of using a indexing system such as SOLR?

Zak
  • 113
  • 5

1 Answers1

0

At what point is it more beneficial to use a indexing engine vs scripting a raw query yourself?

An abstracted layer for indexing is never going to be better or more performant than a correctly architected database structure. The purpose of it is instead to be a tool to make it easier for someone who is inexperienced with implementing a correctly architected structure.

I haven't noticed a dramatic load time in searches. If I run a raw SQL command, the difference between the two is negligible.

We'd have to see the table structure, the index definitions, the query you ran, and ideally it's query plan (EXPLAIN ANALYZE) to be able to specifically comment on how efficiently it's operating. But generally if you're just doing straight SELECTs against a table, they'll run relatively well, and when indexed properly should run efficiently.

J.D.
  • 40,776
  • 12
  • 62
  • 141