Server error 502 - problems with .net core web app deploy made on GCloud's GKE using OAuth 2.0 Authentication

Question

I have an MVC .net core 6 web application that uses Google's OAuth2.0 authentication and I cannot publish it with Kubernetes.

Project context:

Locally it compiles and runs normally, including creating a docker image and running a container, but to run the docker container locally and have an https port with a valid certificate (I need this for Google's OAuth to work) I need to run the following procedure.

Sequence of commands and procedures to make the local certificate trustworthy (you only have to do this to run in a docker container, running the project directly is not necessary):

create the local certificate pfx: dotnet dev-certs https -ep $env:USERPROFILE\.aspnet\https\Apresentacao.Web.pfx -p pa55w0rd!
Create a UserScretsId configuration line in the application's .csproj: {SomeGuidIdHere}
Enter the certificate password in the local user-secrets: dotnet user-secrets set "Kestrel:Certificates:Development:Password" "pa55w0rd!"
make the certificate 'trusted' dotnet dev-certs https --trust
Run the docker container, but passing through some necessary parameters (run in powershell or bash outside the project folder):

docker run -p 8080:80 -p 8081:443 -e ASPNETCORE\_URLS="https://+;http://+" -e ASPNETCORE\_HTTPS\_PORT="8081" -e ASPNETCORE\_ENVIRONMENT=Development - v $env:APPDATA\microsoft\UserSecrets\\:/root/.microsoft/usersecrets -v $env:USERPROFILE\\.aspnet\https:/root/.aspnet/https/ admin

If I do not perform the above procedure, or execute 'Docker run' without passing the necessary parameters to use the https port and certificate, I receive the following return when running the container:

warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3] Failed to determine the https port for redirect.

My problem is: Locally I found the solution to deal with the docker container, the https port and the ssl certificate for localhost, however I am publishing this same application on Google Cloud using Git Hub actions to run my deployment pipeline and in the cloud I am saving the image docker in a bucket and creating a service with kubernetes. We already have other applications running on this Google cloud infrastructure with Kubernetes and we had no problems exposing the URLs with Ingress and using the certificate provisioned by Google.

I did several tests publishing with different configurations until I suspected that it could be something in the application, I decided to do a test by publishing without the Google authentication configuration and to my surprise the application had health status in Kubernetes and was made available on URL with https correctly. When I include authentication with Google again in the project's Startup.cs, my problem returns in the Kuberntes container and when accessing the URL the return I get is a 502.

Here's how I'm configuring authentication with Google: On Startup.cs

ConfigureServices Method:

public void ConfigureServices(IServiceCollection services)
        {
            //Auth config google
            services.AddAuthentication(options =>
            {
                options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                options.DefaultChallengeScheme = GoogleDefaults.AuthenticationScheme;
            })
            .AddCookie()
            .AddGoogle(options =>
            {
                options.ClientId = "{my_ClientId}";
                options.ClientSecret = "{my_ClientSecret}";
            });
            ...
}

Configure Method:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseExceptionHandler("/Home/PaginaNaoEncontrada");
                app.UseHsts();
            }
            app.Use(async (context, next) =>
            {
                await next();
                if (context.Response.StatusCode == 404)
                {
                    context.Request.Path = "/Home/PaginaNaoEncontrada";
                    await next();
                }
            });
            app.UseHttpsRedirection();
app.UseStaticFiles(new StaticFileOptions
            {
                OnPrepareResponse = o => { o.Context.Response.Headers.Append("Cache-Control", $"public, max-age=10800"); }
            });
            app.UseUnobtrusiveAjax();
            app.UseRouting();
            app.UseAuthorization();
            app.UseAuthentication();            //Important  
}

In my HomeController.cs (where my first request hits when accessing the application for the first time and where I validate that the User is authenticated) the [Authorize] annotation above the Controller is strictly necessary for what I added in Startup to work .cs:

[Authorize]
public class HomeController : Controller
{ ...}

My Action for the Home Index:

[HttpGet]
return View();

LoginGoogle called by Index:

public ActionResult LoginGoogle(){
    return Challenge(new AuthenticationProperties { RedirectUri = "/" }, "Google");}

To publish to Kubernetes I have the following files (I highlight three of them because I consider that the most important settings for the site are in them):

My deploymet looks like this:

apiVersion: apps/v1

My Ingress looks like this:

apiVersion: networking.k8s.io/v1

And my service looks like this:

apiVersion: v1

With the recent changes I made to my .yaml files for Kubernetes I started receiving more logs from the service, similar to what I would receive locally if I ran the 'Docker run' command without parameters to use the local certificate: Latest container logs: https://drive.google.com/file/d/1VcmiBe0LBuMr5B0VC5YivrvZdygZtEf5/view?usp=drive_link

Before these changes (mainly in the deployment file) I was receiving the following return in the logs: Previous logs: https://drive.google.com/file/d/16NoN1fhhmoGk2bZRWTY0oxqFBY3xH4Kj/view?usp=drive_link

Below the logs illustrated above was just an infinite loop of information about ResponseCoockies and the information about the https port was not repeated until a new deployment was made..

How the service responds when accessing the URL when we publish WITH Google authentication configured: https://drive.google.com/file/d/1jDCmZNFPvzRvnq2VeOr7yC79A9Cba1rf/view?usp=drive_link

My question is if I am currently configuring Kubernetes or is something missing for the certificate to be trusted? Why does the deployment happen normally when I remove the Google authentication configuration from my application? (I remove the Startup config and the Controller annotation)

It is important to note that my OAuth Client Id credentials are correct as I was able to test by adding the https localhost URL as authorized, so this would not be the problem either. I added the URL that I am provisioning in Kubernetes to the list of URLs authorized to use the Client Id OAuth, as follows: https://drive.google.com/file/d/1Hzi74jJ6sZ7za5D3rFZcyDQRaIoW4jOS/view?usp=drive_link

Other information.

A screenshot of when I publish the application without Google authentication configuration and it stays UP and the certificate appears to be linked to the URL and secure:

https://drive.google.com/file/d/1RLocACa-TaCsmZo1S74sQoKrzvFgcfyx/view?usp=drive_link

When deploying with Ingress.yaml, it creates a Load balance in the infrastructure and automatically configures a front end endpoint linked to a static-IP:

https://drive.google.com/file/d/1f1LyIzcDqUbEUEpPTh6VQqdwyshMeUVA/view?usp=drive_link

We use the AWS service to purchase and host domains and so that when accessing the URL that I configured, my DNS would be resolved to the IP of my front end, I configured a 'Record name' to be redirected to the IP. This has already been done for other applications and it worked, it even works when we publish this application without authentication...

Below is the registration I made in the AWS hosting zone: https://drive.google.com/file/d/1y8BIeK7RYywP95jPzpCGYkgZ7Lv-w7KG/view?usp=drive_link

And the details of the certificate provisioned by Google and which only became 'Active' when communication with AWS was established due to the mapping I did with the IP, as mentioned above. https://drive.google.com/file/d/1Y9wLH_DDX-93iMY5eNwBN0VcDFbId5I1/view?usp=drive_link

Thank you in advance for your help

score 0 · Accepted Answer · answered Mar 11 '24 at 20:18

We found the solution to the problem described above. Basically, the container within Kubernetes needs that, in addition to the health check end-points, the base url of the application is also up, that is, I could not be trying to add the [Authorize] annotation directly to my HomeController as it was the first to be displayed by the application. I was defining the "default" End-point of my application as follows:

app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllerRoute(
                    name: "default",
                    pattern: "{controller=Home}/{action=Index}");
            });

However, HomeController was completely covered by [Authorize] and the application was being considered unhealth in the kubernetes backend. Therefore, what solved the key problem was creating a Login screen (without applying [Authorize]) to call Google's login and changing the default endpoint for it:

app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllerRoute(
                    name: "default",
                    pattern: "{controller=Login}/{action=Index}/");
            });

So the backend became heath and the application went up, first calling the Login screen and then clicking on "Login with Google" we called Google's OAuth 2.0, here's another problem... the application was redirecting to OAuth using HTTP, but we needed to correct this so that it redirected to HTTPS. We tried all the suggested solutions, but what worked in our case was:

public void ConfigureServices(IServiceCollection services)
        {
 services.Configure<ForwardedHeadersOptions>(options =>
            {
                options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
                // Only loopback proxies are allowed by default. Clear that restriction because forwarders are
                // being enabled by explicit configuration.
                options.KnownNetworks.Clear();
                options.KnownProxies.Clear();
            });
....
}

The Microsoft documentation we used as a basis was: https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-6.0 However this doc. it became confusing for us, we tried all the suggestions and nothing was working in our case... until the Clear settings for KnownNetworks and KnownProxies made our application finally use our redirect for HTPPS.

Server error 502 - problems with .net core web app deploy made on GCloud's GKE using OAuth 2.0 Authentication

1 Answers1