WebHost.CreateDefaultBuilder()
    .ConfigureServices(ConfigureServices)
    .Configure(Configure)
    .ConfigureKestrel(options =>
        options.ConfigureHttpsDefaults(ops => // I don't think these defaults are used when UseHttps is used with a callback.. but I've added them anyway just incase
            ops.ClientCertificateValidation = (certificate2, chain, arg3) => true; // Not called
            ops.ClientCertificateMode = ClientCertificateMode.AllowCertificate;
            ops.OnAuthenticate = OnAuthenticate;
        options.Listen(IPAddress.Any, 8080, listenOptions =>
            listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            listenOptions.UseHttps(async (SslStream stream,
                SslClientHelloInfo clientHelloInfo, object? state, CancellationToken cancellationToken) =>
                var ops = new SslServerAuthenticationOptions
                    ClientCertificateRequired = true
                ops.ServerCertificate = await MyConsulConnectManager.GetLeafCertificate(); // Server certificate loaded dynamically, provided by consul connect
                ops.RemoteCertificateValidationCallback = (sender, certificate, chain, errors) =>
                    Console.WriteLine("EXPECT TO HIT HERE FOR CUSTOM VALIDATION"); // Not called
                    return true;
                ops.CertificateRevocationCheckMode = X509RevocationMode.NoCheck;
                return ops;
            }, null);
    .Build();
I'm using Jwt authentication middleware but I tried adding some debug middleware right after app.UseRouting() (before auth middleware) but the middleware isn't called - I think because the handshake is failing nothing is getting through.
I'm currently looking through the source for Kestrel HttpsConnectionMiddleware but cannot see where the RemoteCertificateValidation callback is added...

      aspnetcore/src/Servers/Kestrel/Core/src/Middleware/HttpsConnectionMiddleware.cs
         Line 320
      8b30d86
SslStream seems to deal with it, so potentially the middleware is failing to add it?

https://github.com/dotnet/runtime/blob/12c4e4cd63e4d8daf2ad4268876a69f8cda606a7/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs#L411
Will keep investigating
          
Update:
If I use a different "UseHttps" now (HttpsConnectionAdapterOptions) and using
ops.ServerCertificateSelector = (context, s) => MyConsulConnectManager.GetLeafCertificate().GetAwaiter().GetResult(); // yuck
ops.ClientCertificateValidation = (certificate2, chain, arg3) =>
                                Console.WriteLine("Want to validate here!");
                                return true;
I can see the remote validation callback has been added to SslStream (wrapped in a callback within HttpsConnectionMiddleware) but the exception is still being thrown and callback isn't called.
So perhaps it's the particular usage of ServerOptionsSelectionCallback that doesn't get the remote validation callback added to sslstream?
However even with the callback present on sslstream, it doesn't work:
I'm now looking at the point the exception is thrown:

https://github.com/dotnet/runtime/blob/12c4e4cd63e4d8daf2ad4268876a69f8cda606a7/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs#L414
I don't think the validation callbacks are called until CompleteHandshake (line 434 above).

So perhaps I need to move this issue over to dotnet runtime?
          
I believe this problem is prior to any authentication takes place - it's during the connection setup. I get the impression that windows refuses mutual TLS when the client certificate isn't signed by a trusted root.
I've got around the problem for now by manually adding any and all CA roots and intermediates to the appropriate X509Store as they get loaded from consul... not a nice solution as it means running my service is making changes to the users certificate store - something I wanted to avoid! But it does work. At runtime, I add the certificates and subsequent connections succeed (and proceed to the remote validation callback and rest of the middleware).
So perhaps it's the particular usage of ServerOptionsSelectionCallback that doesn't get the remote validation callback added to sslstream?
I noticed this when adding the new UseHttps overload that takes the SslServerAuthenticationOptions-returning callback and filed dotnet/runtime#40402. We take the SslServerAuthenticationOptions returned by the UseHttps callback and return it directly without modification to the callback we pass to SslStream.AuthenticateAsServerAsync.
@wfurt Any ideas? I'm not sure what you mean by "negation" in :
I did more testing and the documentation is misleading IMHO. I did testing with 3.1 and the negation fails unless you provide override. With 'ClientCertificateRequired` and client not providing certificate, negations may succeed. I'll try to update docs to make that more clear.
The missing callback was fixed as part of #40110 (since that was pending and touching relevant area)
Should I transfer this issue? At the vary least, there seems to be a lack of parity when passing a RemoteCertificateValidationCallback via the SslStream constructor vs returning it from a ServerOptionsSelectionCallback.
          
@saltxwater I tried creating a simple repro based on the description provided, but I'm not seeing the same issue. See https://github.com/halter73/Test34869/blob/main/Program.cs. This outputs the following which is what I would expect:
C:\dev\halter73\Test34869 main ≡ ❯ dotnet run
Building...
warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Overriding address(es) 'https://localhost:5001, http://localhost:5000'. Binding to endpoints defined in UseKestrel() instead.
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\dev\halter73\Test34869
EXPECT TO HIT HERE FOR CUSTOM VALIDATION
Response: Hello World!
Can you modify my repro to demonstrate the error? I wonder if it has something to do with the certs.
          
As far as the flow, we tell channel to do manual validation so we should not get to place you pointed at because of the validity @saltxwater . I'm wondering if this could fail for some other reasons like ciphers or protocols version.

It may be helpful to get packet capture.
          
@halter73 Thank you for the provided repro attempt.

I've tried running it as provided and I agree that it works.

I then tried changing the hosted (and client) certificates to ones provided to me by Consul and again that works. (Including having removed the Consul CA from my trusted windows certificates.
The only way I've found to reproduce it is by getting the consul ingress gateway (Envoy) running in our cluster to send the request. When envoy tries to connect to your demo Program I get the same win32 failure and the callback isn't called.
I then added the consul CA to my certificate store and the connection from envoy succeeded. I've visually tried to compare the HttpContext between the local httpclient connection and successful remote envoy connection. Both connections seem to be pretty much the same:
Cipher Algorithm Aes256
Cipher Strength 256
Hash Algorithm Sha384
KeyExchangeAlgorithm 44550
NegotiatedCipherSuite TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
But they DO differ in KeyExchangeStrength. Locally it is 384, from envoy it is 255.
I don't know how or why this would affect the server listener behavior but I guess it does suggest that envoy is behaving differently.
I'll try to get envoy running locally (without consul connect) and see if I can provide a reproducible configuration.
@wfurt I just tried installing the SharpPcap package but found the examples a bit overwhelming. Do you know of an example/other package which I could just sit in front of aspnet to collect all packets and forward them between?
          
Use https://www.wireshark.org/#download and set filter to either "port 5001" or "host client".

Get the error, it may provide some explanation. It seems like this is specific to specific client or specific certificate.
          
@wfurt

Thank you for your help with suggesting Wireshark - I've managed to intercept the packets and it revealed a very frustrating realisation that I've neglected to consider.

Bottom line: it's not the server that's failing the TLS handshake - it's the client (envoy)!
Right after the server sends it's Server Hello, the client is replying with Alert (Level: Fatal, Description: Unknown CA).
Given that I was getting success when adding the intermediate and CA to my certificate store I was led to believe it was a hosting problem (and it still sort of is). It turns out that SslStream doesn't send the full certificate chain when the intermediate isn't in the certificate store. (I guess, how could it? The X509Certificate2 I'm using for the server only contains the leaf certificate).

I'm quite surprised that envoy isn't accepting the server certificate given that envoy should have access to the intermediate and CA it loads from consul, so I may also chase this up with hashicorp.
Seems after all this is a duplicate issue:

#10971
I don't know what the outcome of that issue was - it would be extremely useful if it could be followed up and there was a way to provide the full chain for the server certificate, without needing to modify my certificate store.
@halter73 Thank you also for your help with trying to diagnose this issue.
          
Following this I've found SslServerAuthenticationOptions.ServerCertificateContext which you can specify instead of a ServerCertificate, and it includes the option to provide a full chain...

Running it once throws a cryptographic exception (access is denied) while it attempts to look in the Stores.

Running it again works, and subsequent client queries (from envoy) are working, but I realise that SslStream (or some other subsystem) has gone ahead and added those certificates to the Store - something I was trying to avoid.
I've kind of given up with this and will just manage these certificates in the X509Store myself.
          
There is no way how .NET can explicitly pass certificate chain to Schannel. So ServerCertificateContext will populate the intermediate certs store if needed to make sure certificate chain is sent as needed.

BTW if certificate chain is root cause you would see that with Wireshark as well when server sends ServerHello and presents own certificate. RFC says the it should send certificate chain -1 e.g. no root.
  ✔️ Resolution: Answered
  Resolved because the question asked by the original author has been answered.
 label
    Aug 6, 2021
          
This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.
See our Issue Management Policies for more information.
  area-networking
  Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions
    and removed
  area-runtime
  labels
    Jun 2, 2023
    area-networking
  Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions
  This issue describes a behavior which is not expected - a bug.
    feature-kestrel
    ✔️ Resolution: Answered
  Resolved because the question asked by the original author has been answered.
    Status: Resolved