I have been trying to set a blob container's secrets to the databricks cluster level, but using spark.conf.set('property','key') would always set to the session level only. And directly giving the
property key-value
in the Cluster -> Advanced Options -> spark-config is not advisable, whereas giving it as
property dbutils.secrets.get(scope,key)
throws error too.
Can anyone help me out on how to set the container's configuration to a databricks cluster.
I want to access the blob container using databricks without mounting the path to the cluster.(Since I am trying to use a adf copy activity to copy data from databricks table to a SQL DB table and need to use a blob storage as the staging layer where I cant give the mounted path).
The secrets can be Databricks backed or Azure Key Vault backed.
Both can be created by using the databricks-cli tool, and for the Azure Key Vault backed secrets a hidden web ui can be accessed for you Databricks instance [
https://<databricks-instance>#secrets/createScope
].
https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes
Use
dbutils.secrets.get
to access the secret from the notebook.
secret = dbutils.secrets.get(scope = "<scope-name>", key = "<secret-name>")
I tried using dbutils.secrets.get in the spark configuration, but was getting error as "The string is not a valid base64-encoded string".(hopefully since the key value is retrieved as REDACTED instead of its real value)
I actually got a solution for my question,
https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets#--use-a-secret-in-a-spark-configuration-property-or-environment-variable
But unfortunately cluster owner(cluster creator) can only add reference to a secret in the spark configuration or as an environment variable. Its a general cluster that we use throughout, So if there is any option to update the existing cluster owner that would be great. I want to use the same cluster, and cloning the cluster creates one in a different cluster ID, so is it possible to change the cluster owner or any other option to add secrets would also be helpful.
Thanks
Hello @Anonymous ,
The scope of your question is different compared to the original scope of the question asked.
I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.
