GCP how to identify source of millions of calls to servicecontrol.googleapis.com/report

Question

First, I'd like to preface this by saying I did not play a role in setting up this GCP project, so I'm running with limited knowledge on what is configured, but I do know a bit about what applications are running, and what roles they perform.

The known:

There is an API Gateway set up with around 10 cloud run services behind it. All are Java APIs.
Calls to the servicecontrol.googleapis.com/report endpoint have been steadily increasing over time, I think with system usage, to the tune of now being at 2.5m/operations per day. This is causing a noticeable line item on our bills.

Bill screenshot
When I view the API/Service details graphs for Service Control API, rather unhelpfully it says the traffic is coming from "Unspecified" as the credentials. This obviously does not help me narrow down the source.

Unspecified credentials
If I disable the Service Control API, the calls obviously drop to zero, but API Gateway prompts me to enable the Service Control API every time I visit the configuration page, so obviously it's "required" for something.

API Gateway wants me to enable Service Control API

After disabling the Service Control API, I see errors like this in my logs

{ "insertId": "61bd29cad64fcdeb3a72c087c743457e-1@a1",
"jsonPayload": {
  "apiConfig": "//apigateway.googleapis.com/projects/XXXXXXXXXXX/locations/global/apis/adminservice1/configs/my-config-name",
  "httpRequest": {
    "duration": "13ms",
    "responseSize": "39",
    "path": "/v1/services/adminservice1-2flbn13q8pv2z.apigateway.my-project-id.cloud.goog:report",
    "hostname": "servicecontrol.googleapis.com",
    "httpVersion": "HTTP/1.1",
    "status": 403,
    "requestSize": "2236"
  },
  "serviceConfig": "//servicemanagement.googleapis.com/services/adminservice1-2flbn13q8pv2z.apigateway.my-project-id.cloud.goog/configs/my-config-name-0exxy8atd70t3",
  "api": "//apigateway.googleapis.com/projects/XXXXXXXXXXX/locations/global/apis/adminservice1"
},
"resource": {
  "type": "apigateway.googleapis.com/Gateway",
  "labels": {
    "gateway_id": "ezylink-apigateway-preprod",
    "location": "australia-southeast1",
    "resource_container": "projects/XXXXXXXXXXX"
  }
},
"timestamp": "2024-06-26T22:34:01.523107678Z",
"severity": "ERROR",
"logName": "projects/my-project-id/logs/apigateway.googleapis.com%2Fservice_control_queries",
"receiveTimestamp": "2024-06-26T22:34:11.539672849Z"}

The unknown:

Where the heck these calls are coming from! I can't tell if it's code, or configuration, that is causing this.

Any insights would be greatly appreciated.

GCP how to identify source of millions of calls to servicecontrol.googleapis.com/report

0 Answers0