Adding custom indexes in sharded cluster CSRS(Config Server as replicaSet )?-CodePudding

I am monitoring my ~10TB 5x shards sharding cluster with prometheus/grafana/mongodb_exporter , but I see mongo_exporter is getting number of chunks grouped by shard from the CSRS server and it is taking more then 2min:

 2022-01-18T16:31:48.499 0100 I COMMAND  [conn369096] command config.chunks appName: "mongodb_exporter" command: aggregate { aggregate: "chunks", pipeline: [ { $group: { _id: "$shard", count: { $sum: 1 } } } ], fromMongos: true, cursor: { batchSize: 101 }, lsid: { id: UUID("4b65f5f9-1776-471e-a086-3bee46841edf"), uid: BinData(0, 745F9C5E38C046A7EAF9555E7B0195569974DC916F6DF649D96D26158C1DF113) }, $readPreference: { mode: "secondaryPreferred" }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1642519905, 7), signature: { hash: BinData(0, CA16F59A62FEE55E3A1B73F5A45E450B4EFE5F88), keyId: 7004436913486561282 } }, $client: { driver: { name: "mongo-go-driver", version: "v1.5.3" }, os: { type: "linux", architecture: "amd64" }, platform: "go1.16.5", application: { name: "mongodb_exporter" }, mongos: { host: "mymongos:27019", client: "1.2.3.4:54054", version: "4.0.23" } }, $configServerState: { opTime: { ts: Timestamp(1642519905, 7), t: 120 } }, $db: "config" } planSummary: COLLSCAN keysExamined:0 docsExamined:1620394 cursorExhausted:1 numYields:12684 nreturned:5 reslen:707 locks:{ Global: { acquireCount: { r: 12757 } }, Database: { acquireCount: { r: 12757 } }, Collection: { acquireCount: { r: 12757 } } } storage:{} protocol:op_msg 2464ms

So the questions:

Is it wise to create index in CSRS config.chunks.shard field to speed up this periodical mongodb_exporter aggregation query? Tested index creation and I see the aggregation query pick it up only if it is explicitely adviced via HINT:

CSRS:PRIMARY> db.chunks.createIndex({shard:1})
CSRS:PRIMARY> db.chunks.explain("executionStats").aggregate([ { $group: { _id: "$shard", count: { $sum: 1 } } }],{hint:{"shard":1}} )

                                           "inputStage" : {
                                                 "stage" : "IXSCAN",
                                                 "keyPattern" : {
                                                         "shard" : 1
                                                 },

                              "executionStats" : {
                                 "executionSuccess" : true,
                                 "nReturned" : 1601762,
                                 "executionTimeMillis" : 1377,
                                 "totalKeysExamined" : 1601762,
                                 "totalDocsExamined" : 0



CSRS:PRIMARY> db.chunks.aggregate([ { $group: { _id: "$shard", count: { $sum: 1 } } }],{hint:{"shard":1}} )
{ "_id" : "s4", "count" : 318174 }
{ "_id" : "s3", "count" : 318086 }
{ "_id" : "s0", "count" : 317933 }
{ "_id" : "s1", "count" : 318020 }
{ "_id" : "s2", "count" : 329549 }
CSRS:PRIMARY>

Please, advice if is safe adding custom indices in CSRS config database in general?

Any advice will be highly appreciated ?

CodePudding user response：

Your query covers all collections, however usually it should be per collection, i.e. per namespace. In earlier MongoDB release the documents where like this:

{ 
    "_id": ObjectId("61a7bd45b9a53380435dede7"), 
    "ns": "database.collection", 
    "min": ..., 
    "max": ...., 
    "shard": "shard_02", 
    ...
}

Now in MongoDB 5.0 field ns is replaced by uuid and it looks like this:

{ 
    "_id": ObjectId("61a7bd45b9a53380435dede7"), 
    "uuid": UUID("322758c1-c52f-4ab6-9eb2-c48fc1634ef7"), 
    "min": ..., 
    "max": ...., 
    "shard": "shard_02", 
    ...
}

You should have an unique index on {uuid: 1, shard: 1, min: 1}. You can lookup namespace from config.collections collection.

Perhaps prometheus/grafana does not take this modification into account.

The $group stage can sometimes use an index to find the first document in each group if all of the following criteria are met:

The $group stage is preceded by a $sort stage that sorts the field to group by,
There is an index on the grouped field which matches the sort order and
The only accumulator used in the $group stage is $first.

So, I don't think an index on {shard: 1} will help