Notebook-scoped Python libraries

%pip uninstall -y matplotlib
The -y option is required.
Install a library from a version control system with %pip 
%pip install git+https://github.com/databricks/databricks-cli
You can add parameters to the URL to specify things like the version or git subdirectory. See the VCS support for more information and for examples using other version control systems.
Install a private package with credentials managed by Databricks secrets with %pip 
Pip supports installing packages from private sources with basic authentication, including private version control systems and private package repositories, such as Nexus and Artifactory. Secret management is available via the Databricks Secrets API, which allows you to store authentication tokens and passwords. Use the DBUtils API to access secrets from your notebook.  Note that you can use $variables in magic commands.
To install a package from a private repository, specify the repository URL with the --index-url option to %pip install or add it to the pip config file at ~/.pip/pip.conf.
token = dbutils.secrets.get(scope="scope", key="key")
%pip install --index-url https://<user>:$token@<your-package-repository>.com/<path/to/repo> <package>==<version> --extra-index-url https://pypi.org/simple/
Similarly, you can use secret management with magic commands to install private packages from version control systems.
token = dbutils.secrets.get(scope="scope", key="key")
%pip install git+https://<user>:$token@<gitprovider>.com/<path/to/repo>
Install a package from DBFS with %pip 
You can use %pip to install a private package that has been saved on DBFS.
When you upload a file to DBFS, it automatically renames the file, replacing spaces, periods, and hyphens with underscores. For Python wheel files, pip requires that the name of the file use periods in the version (for example, 0.1.0) and hyphens instead of spaces or underscores, so these filenames are not changed.
%pip install /dbfs/mypackage-0.0.1-py3-none-any.whl
This feature is in Public Preview.
With Databricks Runtime 13.2 and above, you can use %pip to install a private package that has been saved to a volume.
When you upload a file to a volume, it automatically renames the file, replacing spaces, periods, and hyphens with underscores. For Python wheel files, pip requires that the name of the file use periods in the version (for example, 0.1.0) and hyphens instead of spaces or underscores, so these filenames are not changed.
%pip install /Volumes/<catalog>/<schema>/<path-to-library>/mypackage-0.0.1-py3-none-any.whl
Install a package stored as a workspace file with %pip 
With Databricks Runtime 11.3 LTS and above, you can use %pip to install a private package that has been saved as a workspace file.
%pip install /Workspace/<path-to-whl-file>/mypackage-0.0.1-py3-none-any.whl
Any subdirectories in the file path must already exist. If you run %pip freeze > /dbfs/<new-directory>/requirements.txt, the command fails if the directory /dbfs/<new-directory> does not already exist.
Use a requirements file to install libraries 
A requirements file contains a list of packages to be installed using pip. An example of using a requirements file is:
%pip install -r /dbfs/requirements.txt
See Requirements File Format for more information on requirements.txt files.
How large should the driver node be when working with notebook-scoped libraries? 
Using notebook-scoped libraries might result in more traffic to the driver node as it works to keep the environment consistent across executor nodes.
When you use a cluster with 10 or more nodes, Databricks recommends these specs as a minimum requirement for the driver node:
For a 100 node CPU cluster, use i3.8xlarge.
For a 10 node GPU cluster, use p2.xlarge.
For larger clusters, use a larger driver node.
Can I use %sh pip, !pip, or pip? What is the difference? 
%sh and ! execute a shell command in a notebook; the former is a Databricks auxiliary magic command while the latter is a feature of IPython. pip is a shorthand for %pip when automagic is enabled, which is the default in Databricks Python notebooks.
On Databricks Runtime 11.0 and above, %pip, %sh pip, and !pip all install a library as a notebook-scoped Python library. On Databricks Runtime 10.4 LTS and below, Databricks recommends using only %pip or pip to install notebook-scoped libraries. The behavior of %sh pip and !pip is not consistent in Databricks Runtime 10.4 LTS and below.
Known issues 
On Databricks Runtime 10.3 and below, notebook-scoped libraries are incompatible with batch streaming jobs. Databricks recommends using cluster libraries or the IPython kernel instead.