Questions tagged [partitioning]

Splitting a database table into multiple segments for performance or manageability.

A database table may be split into multiple parts but still queried as a single table. Partitioning allows the parts to be managed independently, stored on separate disk volumes and excluded from query processing operations where they are not needed.

The most common partitioning strategy is known as horizontal partitioning and splits the table into sets of rows based on ranges of values of one or more columns (often called a partition key). The partitioning rule can also be based on hash values or lists for some databases. Horizontal partitioning has a number of benefits:

  • If the query includes a partition key range the optimiser can exclude partitions that are not needed to process the query. This can substantially reduce the work needed to process the query, improving the overall query performance.

  • The partitions can be stored on multiple disk volumes, allowing the storage to be distributed across those volumes. This was a common storage strategy on older systems with limitations on individual disk volume sizes. It can also be used to distribute a table across multiple disk controllers for performance.

  • Individual partitions can be added and removed from the table without the overhead of expensive delete operations. This facilitates managing large tables, such as archiving data based on date.

Horizontal partitioning is often sold as an 'enterprise' feature, only available on more expensive editions of database management systems.

A less commonly used feature is vertical partitioning, which splits individual database records across multiple storage units, with some columns held in one place and others held in a different storage allocation. This is used where a table has a wide column or large set of columns that is only used in certain, uncommonly used queries.

Many database management systems do not support explicit vertical partitioning, although it can happen implicitly with facilities such as BLOB storage.

1086 questions
48
votes
2 answers

How to partition existing table in postgres?

I would like to partition a table with 1M+ rows by date range. How is this commonly done without requiring much downtime or risking losing data? Here are the strategies I am considering, but open to suggestions: The existing table is the master and…
Evan Appleby
  • 1,203
  • 1
  • 10
  • 16
39
votes
1 answer

Index optimization with dates

I have a large table of objects (15M+ row) in PostgreSQL 9.0.8, for which I want to query for outdated field. I want to divide the query by millions, for scalability & concurrency purposes, and I want to fetch all data with the updated_at field with…
37
votes
2 answers

Get all partition names for a table

I want to list all the partitions created by dynamic triggers in PostgreSQL 9.1. I was able to generate a count of partitions using this related answer by Frank Heikens. I have a table foo with an insert trigger that creates foo_1, foo_2 etc.…
Vishnu Kumar
  • 545
  • 1
  • 5
  • 8
33
votes
4 answers

How do I get a list of all the partitioned tables in my database?

How do I get a list of all the partitioned tables in my database? Which system tables/DMVs should I be looking at?
RK Kuppala
  • 2,447
  • 1
  • 22
  • 24
32
votes
3 answers

How Does Table Partitioning Help?

I am having difficulty to grab the idea of pros and cons of table partitioning. I am about to start work on a project which would have 8 tables and one of them will be the main data table which will hold 180-260 million records. As it will be…
Rick James
  • 1,271
  • 3
  • 15
  • 19
27
votes
3 answers

Does the partition key also have to be part of the primary key?

I am partitioning a table based on a column that is not a primary key? I've read some conflicting information today on whether the partition column must be a part of the primary key. My gut says no, but I am not 100% sure. So questions... Must…
AngryHacker
  • 1,961
  • 5
  • 22
  • 33
26
votes
2 answers

How to partition an existing non-partitioned table

I have an existing table with data: dbo.Test (col1,col2,col3....) ON [PRIMARY] I need to change this table to be partitioned like this: dbo.Test(col1,col2,col3....) ON Ps_Date(Col2) How I can I achieve this without dropping and recreating the…
343
  • 363
  • 1
  • 3
  • 5
24
votes
2 answers

SQL Server does not optimize parallel merge join on two equivalently partitioned tables

Apologies in advance for the very detailed question. I have included queries to generate a full data set for reproducing the problem, and I am running SQL Server 2012 on a 32-core machine. However, I do not think this is specific to SQL Server…
Geoff Patterson
  • 8,447
  • 2
  • 28
  • 53
23
votes
2 answers

Many columns vs few tables - performance wise

Yes, I am aware that data normalization should be my priority (as it is). I've got a table with 65 columns storing vehicle data with columns: used_vehicle, color, doors, mileage, price and so forth, in total 65. Now, I can divide that and have a…
23
votes
2 answers

Is it possible to force the optimizer to eliminate irrelevant tables in this partitioned view?

I'm testing different architectures for large tables and one suggestion that I've seen is to use a partitioned view, whereby a large table is broken into a series of smaller, "partitioned" tables. 1, 2, 3, 4 In testing this approach, I've discovered…
swasheck
  • 10,755
  • 5
  • 48
  • 89
22
votes
2 answers

What is the modern way to partition PostgreSQL across machines, when the data is "naturally partitionable"

After several years of dwelling into the "NoSQL" space, now I have a problem that is quite "relational" in its nature. Today I see data stores with quite different eyes than before. Things like Riak have spoiled me in a way that I can no more…
loxs
  • 321
  • 2
  • 5
21
votes
1 answer

Seek and you shall Scan... on partitioned tables

I've read these articles in PCMag by Itzik Ben-Gan: Seek and You Shall Scan Part I: When the Optimizer Doesn't Optimize Seek and You Shall Scan Part II: Ascending Keys I’m currently having a "Grouped Max" problem with all of our partitioned…
21
votes
1 answer

Statistics disappear after incremental update

We have a large partitioned SQL Server database utilizing incremental statistics. All of the indexes are partitioned aligned. When we try to rebuild a partition online by partition all of the statistics disappear after the index is rebuilt. Below…
JasonR
  • 313
  • 1
  • 7
18
votes
4 answers

Can I move rows between partitions by updating the partition key?

I would think that this would be a fairly simply question, but I've actually had a difficult time finding an answer for this. The question: Can you move rows of data within a partitioned table from one partition to another by simply updating the…
Richard
  • 1
  • 8
  • 42
  • 62
18
votes
4 answers

Is it a good idea to Partition a table (in MS SQL) based on date, when there's a clustered index present on ID (INT)

I have a table in MS SQL Server . Table Size: 806 GB Rows : 1.2 billion Index Space : 1.2 GB Table Usage: Logging from the Web Service calls 99.9% is usage is from the logging, developers look rarely into this table in Prod(only when an issue is…
User M
  • 283
  • 2
  • 6
1
2 3
72 73