Chunk Size is shown as 1 KB in mongo db logs even it is set to 300 MB

Question

I am running a mongo cluster.
Chunk size set is 300 MB but for today morning it is showing me in logs that chunk size is 1024 Byte. I checked in current op there also it is showing chunks of 1024 byte. I have checked with monos and on all config server chunk size is 300 MB.

Please help me to resolve the issue as it is all of the sudden bringing my shard set up down.

Here is the log from currentOp

 {
                    "opid" : "shard0000:-1945000000",
                    "active" : true,
                    "secs_running" : 0,
                    "microsecs_running" : NumberLong(72072),
                    "op" : "query",
                    "ns" : "DB20150102.locationCount",
                    "query" : {
                            "splitVector" : "DB20150102.locationCount",
                            "keyPattern" : {
                                    "articleId" : 1,
                                    "host" : 1
                            },
                            "min" : {
                                    "articleId" : { "$minKey" : 1 },
                                    "host" : { "$minKey" : 1 }
                            },
                            "max" : {
                                    "articleId" : { "$maxKey" : 1 },
                                    "host" : { "$maxKey" : 1 }
                            },
                            "maxChunkSizeBytes" : 1024,
                            "maxSplitPoints" : 2,
                            "maxChunkObjects" : 250000
                    },
                    "client_s" : "192.168.22.106:55881",
                    "desc" : "conn237027",
                    "threadId" : "0x7c6cc55db700",
                    "connectionId" : 237027,
                    "locks" : {
                            "^ibeat20150102" : "R"
                    },
                    "waitingForLock" : true,
                    "numYields" : 14,
                    "lockStats" : {
                            "timeLockedMicros" : {
                                    "r" : NumberLong(32978),
                                    "w" : NumberLong(0)
                            },
                            "timeAcquiringMicros" : {
                                    "r" : NumberLong(48048),
                                    "w" : NumberLong(0)
                            }
                    }
            },

Entry from setting

{ "_id" : "chunksize", "value" : 250 }

Error in primary shard

2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "", host: "abc.com" }
2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "0", host: "xyz.com" }

I'm seeing this in my mongos log for the same collection

2015-01-02T14:53:58.983+0530 [conn58] warning: splitChunk failed - cmd: { splitChunk: "DB20150102.locationCount", keyPattern: { articleId: 1, host: 1 }, min: { articleId: MinKey, host: MinKey }, max: { articleId: MaxKey, host: MaxKey }, from: "shard0000", splitKeys: [ { articleId: "", host: "abc.com" } ], shardId: "ibeat20150102.locationCount-articleId_MinKeyhost_MinKey", configdb: "192.168.24.192:27017,192.168.24.54:27017,192.168.24.55:27017" } result: { who: { _id: "DB20150102.locationCount", state: 1, who: "ibeatdb61:27017:1420185037:1475849446:conn913:869542099", ts: ObjectId('54a660afdc99ecfb22d83c27'), process: "ibeatdb61:27017:1420185037:1475849446", when: new Date(1420189871037), why: "split-{ articleId: MinKey, host: MinKey }" }, ok: 0.0, errmsg: "the collection's metadata lock is taken" }

I have taken these steps:

1. Stop read an write on servers
2. Stop mongos server
3. Restarted config servers.
4. Restarted mongos server
5. Restarted read and write.

Still the same issue is appearing

My Primary Shard is a replica set configuration is as follows :

1. Primary Server : 512 GB Ram 5 TB Physical memory.
2. Secondary Server 16 GB RAM 5 TB Physical Server.
3. Arbiter 8 GB Ram.

My secondary shard has the same configuration in its replica set.

Answer 1

Please use command sh.status() to confirm how many chunks you have and in wich shard. From given logs, I can see that you (should) have only ONE chunk, so it has not been splitted ever. The error message "the collection's metadata lock is taken" means that there is "lock" in the locks DB. You can see those with

db.getSisterDB('config').locks.find()

Check "when" key, to find that "old" lock document. You can remove it and after things should be better.

Answer 2

How did you change the chunk size? Did you follow this guide?

To recap, one should:

1. connect to mongos
2. switch to config db
3. issue change chunk size command:
   db.settings.save( { _id:"chunksize", value: <sizeInMB> } )

AFAIK,

warning: chunk is larger than 1024 bytes because of key ...

won't bring down a shard, it will only result in a unbalanced cluster.