0

I've created a container named jobs some months ago, and today i see another container named jobs_segments holding several Gigabytes of data.

I've read that it is created to hold the individual parts when doing a segmented SWIFT upload.

Questions are :

  • can or should this container or files inside it be deleted once the segmented upload has finished?
  • is there any option to avoid this second container _segments and work only at the main one, jobs?
CrazyRabbit
  • 669

2 Answers2

0

Containers named SOMETHING_segments are probably the product of a large object upload. The term "large object" is a bit misleading; "segmented objects" would be more descriptive, but this feature is mostly used to store large objects, in particular larger than the object size limit of 5GB.

Since your container is named jobs_segments, I would normally assume that it contains the data of an object named jobs, not a container with this name. However, nothing prevents you from creating a container that has the same name as an object.

To answer the first question: If you delete jobs_segments, you lose the data of the jobs object. Better delete the jobs object rather that only its segments.

Second question: The jobs container is not directly related to jobs_segments.

Having said all this, you can create a container SOMETHING_segments and use it for something else than large objects. In the end, jobs_segments could be an ordinary container.

berndbausch
  • 1,033
0

From the Openstack Swift documentation:

swift command uses a strict convention for its segmented object support. In the above example it will upload all the segments into a second container named test_container_segments. These segments will have names like large_file/1290206778.25/21474836480/00000000, large_file/1290206778.25/21474836480/00000001, etc.

The main benefit for using a separate container is that the main container listings will not be polluted with all the segment names. The reason for using the segment name format of /// is so that an upload of a new file with the same name won’t overwrite the contents of the first until the last moment when the manifest file is updated.

swift will manage these segment files for you, deleting old segments on deletes and overwrites, etc. You can override this behavior with the --leave-segments option if desired; this is useful if you want to have multiple versions of the same large object available.

see https://docs.openstack.org/swift/latest/overview_large_objects.html

Arnold Obdeijn
  • 1