Could not find CSRF cookie XSRF-TOKEN in the request. http://10.14.0.121/jupyter/api/namespaces/kubeflow-user/notebooks
I saw this behavior in the past few days, as well as with a complete fresh installation with the final manifests for the release candidate. This doesn't seem to affect the other web apps as I am able to create experiments and run pipelines.
This is happening with the gateway exposed through a LoadBalancer and http. Possibly related to #5763.
/cc @kubeflow/wg-notebooks-leads
          
This is because by default the CSRF cookie will be set using the Secure attribute.
But in your case I see that you connect via plain http, which prevents the cookie to be set. Here's the relevant section from MDN https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#Secure
Note: Insecure sites (http:) can't set cookies with the Secure attribute (since Chrome 52 and Firefox 52).
The easiest solution will be to use the APP_SECURE_COOKIES=False env variable into each app, which essentially does not set the Secure attribute in the Set-Cookie header.
          
Thanks for the reply. I found the cause and solution quickly after posting the issue, but I believe many more people installing 1.3 will run into this. For this reason I defaulted the Istio ingress gateway to be HTTPS and use the kubeflow-self-signing-issuer to provide a certificate for the gateway in the Argo installation https://github.com/argoflow/argoflow.
          
@thesuperzapper I did by having the ingress gateway use HTTPS. For the Argo deployment I’m using the Kubeflow-self-signing-issuer to generate a very for the ingress gateway to avoid people running into this problem. I’m guessing a lot of people are going to run into this so that is why I hadn’t closed the issue yet.
          
Hi @DavidSpek and everyone,
I'm experiencing the same issue. I installed the Kubeflow dashboard on a bare metal k8s cluster according to v1.3 here: https://github.com/kubeflow/manifests#change-default-user-password
I'm using dex and exposing with MetalLB. When I connect through HTTP with istio-ingressgateway looking like such:

I'm getting the same error when I try to create a notebook, or tensorboard:

Could not find CSRF cookie XSRF-TOKEN in the request. http://130.159.199.64/jupyter/api/namespaces/kubeflow-user/notebooks
Following your advice I tried to patch Istio-ingress to use HTTPS, however I'm not sure how, so I followed guide here: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#secure-with-https
So now my Istio-ingress looks like such:
Also I generated a certificate for MetalLB using cert-manager following guide here: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#expose-with-a-loadbalancer
So yaml looks like this:
However, now when I try to access the Kubeflow dashboard I cannot connect at all.
When I try to manually find the certificates at /etc/istio/ingressgateway-certs there's nothing there.
Could you please give more details on how exactly to patch istio-ingressgateway to use HTTPS?  If I need to provide anything else just let me know.
Thanks.
          
@Dimiftb On first glance everything looks correct. Did you restart the istio ingress-gateway pod after creating the certificate and changing the configuration?

You can see the Istio configuration for my distribution here. The README has instructions how to install Kubeflow with Argo CD which might be helpful and a bit easier to maintain in the long run.
          
I haven't restarted the service (not too sure how), however it seems to be working as when I try to access the dashboard at http://130.159.199.64, I get:

Which means the redirect to https actually works.
kubectl logs istio-ingressgateway-56d9b7fdb-fzjqx  -n istio-system gives
So I'm guessing that might be root of the issue.
          
@Dimiftb you can execute kubectl get pods -n istio-system then do kubectl delete pod istio-ingressgateway-xxx and replace xxx with the value from the first command. For good measure you can do the same for cluster-local-gateway-xxx.
So you need to execute kubectl delete pod istio-ingressgateway-56d9b7fdb-fzjqx -n istio-system.
          
@DavidSpek Okay, I just deleted the pod as you suggested and it's showing as not ready:

kubectl logs istio-ingressgateway-56d9b7fdb-j2bwl -n istio-system

kubectl logs istiod-7486c5469f-gnfpl -n istio-system

Again, it seems that because /etc/istio/ingressgateway-certs/tls.crt is missing, the svc is failing.
I've just followed the guide here: https://github.com/kubeflow/manifests#change-default-user-password and installed Istio by running the following commands:
These are the secrets I have in Istio-system ns:

What am I missing here? How can I generate that certificate?
          
@Dimiftb Try this command:

kustomize build github.com/argoflow/argoflow/kubeflow/common/istio | kubectl apply -f -.

That should get your Istio setup properly with the certificate.
          
@DavidSpek So I removed all Istio resources, then installed with the command you suggested. Unfortunately, no change, apart from the fact I've now got another pod(cluster-localgateway) in Istio-system ns:

I applied the Kubeflow Istio resources after deploying with your command from here:

then edited the kubeflow-gateway as before with HTTPS so it looks like this:

then restarted both the ingress pod and the cluster-gateway pod,
kubectl logs istio-ingressgateway-5c6fbb6c7c-26lwz -n istio-system

kubectl logs istiod-f6bfd6f44-kxh6d -n istio-system

When I check manually, folder /etc/istio doesn't exist.
          
@Dimiftb By executing the commands after what I had sent you you messed up the configuration. I did just realize that the certificate is setup for a more recent version of cert-manager than what is bundled with kubeflow (I'm trying to get the manifest working group to upgrade this). You probably do not need to remove the Istio resources before running the below commands.
Please execute the following commands, and nothing else (all the needed Istio resources are included):

First:

kustomize build github.com/argoflow/argoflow/cert-manager | kubectl apply -f -

Then:

kustomize build github.com/argoflow/argoflow/kubeflow/common/istio | kubectl apply -f -
Then wait and once it's deployed you should see it working.

DO NOT deploy kubeflow-istio-resources or change the gateway after running these commands, they are already setup correctly.
          
@DavidSpek
I followed your advice executing both commands and nothing else after. I've waited for a few mins so far, and the certificate is not getting generated. Ingress pod still showing as not ready:

kubectl logs istiod-f6bfd6f44-qxt75 -n istio-system

kubectl logs istio-ingressgateway-5c6fbb6c7c-fqpmf -n istio-system

I really appreciate you helping me with this. Is it possible to generate the certificate manually somehow?
          
Hey everyone, I ran into this issue too, I used the env.  variable 'APP_SECURE_COOKIES' as seen here in the OCP PR :https://github.com/kubeflow/manifests/blob/276d1badc273d0cde277ed8a93a06b4413f2586f/distributions/stacks/openshift/application/jupyter-web-app/addnoauthflags.yaml

However, is this the solution for this PR, or should we expect a fix for KF 1.3?
          
@nakfour I would not consider this the proper solution. The proper solution is to set the Istio ingress-gateway to use HTTPS. What I did in my PR kubeflow/manifests#1819 and in my Argo distribution is create a certificate for the Istio ingress-gateway using the kubeflow-self-singing-issuer that the applications also use. That way, HTTPS and secure cookies are used by default and people don’t need to know/remember to enable secure cookies when they setup HTTPS. I still think this should be the default to avoid this problem and wasting time debugging all together, but @yanniszark wants to re-evaluate this after the release.
          
@nakfour here's what worked for me:
Firstly, remove cert-manager and Istio-system namespaces.
Secondly install cert-manager and istio by running the following commands from @DavidSpek's answer above:
Finally, wait a couple minutes till tls secret gets deployed you can check with kubectl get secret -n istio-system
Once deployed, delete the istio-ingressgateway pod so it restarts:
kubectl delete pod istio-ingressgateway-xxx -n istio-system
Full credits to @DavidSpek for this solution.
I agree with @DavidSpek that this should be the default, since it's causing problems for everyone and it'll be a pretty easy fix to implement pre-release of 1.3
          
For reference, the first command installs upstream cert-manager 1.3 and the Kubeflow issuer. The second command installs Istio from the manifests repo, but configured it to redirect HTTP to HTTPS and adds the certificate I mentioned above. You can also look at the PR I linked to see the necessary additions you need to implement this in the OpenShift distribution.
          
To anyone bumping in this, please see the new section in the manifests install doc:

https://github.com/kubeflow/manifests/tree/3a2182626f5d9d74ff7150fd481a6072c44cf58b#nodeport--loadbalancer--ingress
It will land in the stable release of 1.3 as well.
          
@DavidSpek @Dimiftb  thanks for providing the solution




    

Followed the steps you mentioned, but with a different git repo
kustomize build github.com/argoflow/argoflow/kubeflow/common/istio | kubectl apply -f -
kustomize build github.com/argoflow/argoflow/cert-manager | kubectl apply -f -
Getting this issue with when I run

k describe po -n istio-system istio-ingressgateway-xxx
Warning  Unhealthy  7s (x146 over 4m57s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
          
Hi @paravatha,
Logs might give you more info regarding what the specific issue is.

k logs istio-ingressgateway-xxx -n istio-system
Also, you can check if the secret is getting generated with

k get secret -n istio-system
@thesuperzapper I did by having the ingress gateway use HTTPS. For the Argo deployment I’m using the Kubeflow-self-signing-issuer to generate a very for the ingress gateway to avoid people running into this problem. I’m guessing a lot of people are going to run into this so that is why I hadn’t closed the issue yet.
@DavidSpek Hi, I'm just trying what you did but there's a problem with the authservice. This error appeared:
time="2021-04-22T06:48:12Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get https://192.168.114.40/dex/.well-known/openid-configuration: x509: certificate signed by unknown authority"
I'm trying to fix it by adding CA_BUNDLE environment variable to authservice but nothing's changed.
          
Hello,
I'm having this problem as well, installing kubeflow via manifests version 1.3.
I'm trying to set the environment variable APP_SECURE_COOKIES to false but it doesn't seem to do anything. I'm adding it to  apps/jupyter/jupyter-web-app/upstream/base/params.env, is that the correct place to set it?
Regards.
          
I think the reason is Jupyter uses Volume web app during creating a new notebook server. So you need to set this APP_SECURE_COOKIES env to false in Volume web app as well.
I listed here where I set it (e.g., Tensorboard Web App, Jupyter Web App, Katib UI, even Central Dashboard):
https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/jupyter/jupyter-web-app/upstream/base/deployment.yaml#L30
https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/volumes-web-app/upstream/base/deployment.yaml#L23
https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/katib/upstream/components/ui/ui.yaml#L32
https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/tensorboard/tensorboards-web-app/upstream/base/deployment.yaml#L23
https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/centraldashboard/upstream/base/deployment.yaml#L42
Note: This is not recommended in production deployment.
          
@jalola Just for general information, the Jupyter Web App does not use the Volumes Web App during the creation of a new notebook server. All the web apps are completely independent. I think the better and easier solution to this problem is to expose the Istio ingressgateway with HTTPS. If you don't have a domain or a proper cluster issuer setup in cert-manager, you can use the kubeflow-self-signing-issuer which is also used by various components and a placeholder domain.
deploy this certificate resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com
Then change the kubeflow-gateway to the following:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
          
Let's not use this issue to provide the instructions and support around enabling HTTPS, since this is highly platform dependent.
To make the app work with plain http, although this is not encouraged, you can set the APP_SECURE_COOKIES: "false" ENV Var in each app. If after this change your app is still not working with http then lets further discuss and try to understand why the apps don't behave as expected.
          
@jalola Thanks, that did it!
@DavidSpek when trying to deploy that certificate got this error:

error: unable to recognize "certificate_test.yaml": no matches for kind "Certificate" in version "cert-manager.io/v1"
I'm guessing this is because Kubeflow installs cert-manager version 0.11.
@kimwnasptd working with http now, yes.
          
@jmdinis That is indeed because the manifests use a very outdated version of cert-manager, which is also not compatible with KFServing so you should probably update it.
You can use this yaml instead in the mean time:
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com
@kimwnasptd While I agree that the web apps not working as expected in HTTP mode needs to be investigated, saying a certificate is highly platform dependent is simply not true. If this were indeed the case, the Kubeflow components would not be able to use cert-manager themselves. Over the past few weeks there have probably been around 30 clusters setup using that certificate manifest, mostly on AWS but also on-prem, while making the ArgoFlow for AWS distribution I started ready for enterprise deployments.

The reason I left this issue open is because I knew many people would run into this problem. It is also objectively less work and better practice to use a dummy certificate and HTTPS than it is to change to insecure cookies for every web app. There is also an infinitely higher chance that somebody forgets to change their manifests back to APP_SECURE_COOKIES: "true" everywhere when moving away from a PoC deployment, than when using a dummy certificate. Whereas forgetting to change the issuerRef for the kubeflow gateway is something people would instantly notice when moving out of the PoC stage.
          
hi @DavidSpek I am facing this issue too. I deployed a kubeflow cluster on Google Kubernetes Engine using the kubeflow v1.3 manifest.

Initially, the istio-ingressway was set to NodePort, I changed that to LoadBalancer and could access the central dashboard. But couldn't create jupyter notebook server due to the error here.
I have tried the solutions above and nothing is working.
When I check the ingressgateway pod, i get this error

 Readiness probe failed: Get "http://10.8.2.12:15021/healthz/ready": dial tcp 10.8.2.12:15021: connect: connection refused
What step can i take to expose my service on HTTPS on GKE.

my cert-manager
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: istio-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: istio-ingressgateway.istio-system.svc
  # Use ipAddresses if your LoadBalancer issues an IP
  ipAddresses:
  - <external-ip>
  # Use dnsNames if your LoadBalancer issues a hostname (eg on AWS)
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: istio-ingressgateway-certs
kubeflow-gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
Thanks
          
Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions.

I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error.




    

certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs
gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
  hanj99, harryleesan, taoyonggang, kezure, Ssuwani, wxyzyu, kosehy, Charlie-L-K, zimmy-upstage, and moey920 reacted with thumbs up emoji
  Charlie-L-K reacted with laugh emoji
  Charlie-L-K reacted with rocket emoji
      All reactions
          
Hello,

I'm sorry but there are a lot of informations here, and I still can't get the dashboard items to work (hitting the CSRF token error), is there some place with the correct updated informations concatenated ?

What is the real fix  for using https correctly ?

I am not a k8s expert otherwise I'm sure I would have fix this, but I don't think kubeflow is intended to be deployed by expert only so ...

Thanks
P.S :
To anyone bumping in this, please see the new section in the manifests install doc:

https://github.com/kubeflow/manifests/tree/3a2182626f5d9d74ff7150fd481a6072c44cf58b#nodeport--loadbalancer--ingress
It will land in the stable release of 1.3 as well.
Just so you know : This is not helpful at all, I see that as a "find your own way for fixing this", kubeflow is aimed to be a "easy to install ML stack" or am i misunderstanding ? If I am not, then this kind of statement does not go into the correct direction, I understand this is not an easy fix to find but if that'd be easy then who'd need something like kubeflow ?

Anyway, thanks for trying
@jalola Just for general information, the Jupyter Web App does not use the Volumes Web App during the creation of a new notebook server. All the web apps are completely independent. I think the better and easier solution to this problem is to expose the Istio ingressgateway with HTTPS. If you don't have a domain or a proper cluster issuer setup in cert-manager, you can use the kubeflow-self-signing-issuer which is also used by various components and a placeholder domain.
deploy this certificate resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com
Then change the kubeflow-gateway to the following:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
Thanks @DavidSpek, but the name of ClusterIssuer in my local cluster is "kubeflow-self-signing-issuer" not "gateways-issuer". Who run into this problem may check it first by "kubectl get ClusterIssuer -n istio-system".
          
Thanks @DavidSpek, @geewynn, and @rky0930.

Because of your suggestion,I solve the issue I faced during running katib random example.
I deployed a kubeflow cluster on on premise k8s using the kubeflow v1.4 manifest.

I am using MetalLB(on premise LoadBalancer) to access ingress-gateway
Below are the steps which I take to expose my service on https.
You can set up the MetalLB configuration using this link.
create gateways-issuer.yaml file.
gateways-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    kind: ClusterIssuer
    name: gateways-issuer
  commonName: istio-ingressgateway.istio-system.svc
  ipAddresses:
  - 192.168.35.100
  - 192.168.35.101
  - 192.168.35.102
apply gateways-issuer.yaml file
kubectl apply -f gateways-issuer.yaml
Then change the kubeflow-gateway to the following:
kubectl edit gateway/kubeflow-gateway -n kubeflow 
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"kubeflow-gateway","namespace":"kubeflow"},"spec":{"selector":{"istio":"ingressgateway"},"servers">
  creationTimestamp: "2021-12-12T14:39:15Z"
  generation: 2
  name: kubeflow-gateway
  namespace: kubeflow
  resourceVersion: "69316"
  selfLink: /apis/networking.istio.io/v1beta1/namespaces/kubeflow/gateways/kubeflow-gateway
  uid: e7bb620a-0aa2-465f-9db2-4765c8eeb535
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
# add below parts
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: kubeflow-ingressgateway-certs
      mode: SIMPLE
After editing kubeflow-gateway gateway, you can access https://192.168.35.100

Also, you can create the notebook without any issues.

          
@kimwnasptd,
Let's not use this issue to provide the instructions and support around enabling HTTPS, since this is highly platform dependent.
To make the app work with plain http, although this is not encouraged, you can set the APP_SECURE_COOKIES: "false" ENV Var in each app. If after this change your app is still not working with http then lets further discuss and try to understand why the apps don't behave as expected.
Your suggestion sounds credible, but where exactly do I have to set APP_SECURE_COOKIES: "false"?

Could you please share screenshot of any of the apps you are talking about?
Thank you.
Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions. I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error.
certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs
gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
Maybe, this comment should be insert into Kubeflow Manifests  installation document as Kubeflow HTTPS setup section or just a example, that more Kubeflow initial user will get a better experience. In fact, the Kubeflow HTTPS setup way  troubled me 2 days, becuase I don't know how to solve  "Could not find CSRF cookie XSRF-TOKEN in the request" problem.
      创建notebook的时候遇到[403] Could not find CSRF cookie XSRF-TOKEN in the request. 公有云模式默认开启https怎么解决呢？
      alauda/kubeflow-chart#10
Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions. I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error.

certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs
gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
Maybe, this comment should be insert into Kubeflow Manifests  installation document as Kubeflow HTTPS setup section or just a example, that more Kubeflow initial user will get a better experience. In fact, the Kubeflow HTTPS setup way troubled me 2 days, becuase I don't know how to solve "Could not find CSRF cookie XSRF-TOKEN in the request" problem.
tks, it works for me!

I installed kubernetes 1.21.13 cluster in my 3 physical servers, and installed kubeflow 1.5.1 according to "https://github.com/kubeflow/manifests/tree/v1.5-branch".