Databricks CLI fundamentals

 

1. pip3 install databricks-cli
2. Check if installed: which databricks
3. Check version: databricks —version
4. databricks configure —token
5. databricks clusters list
6. To edit default configure: vi .databrickscfg
7. Create scope: databricks secrets create-scope  —scope demo
8. Put APP_key into the scope: databricks put —scope demo —key APP_key —string-value some-value
9. To configure password file: 
• vim password.txt (add at the end to remove newline character: set noendofline binary) use :wq to quit editing
• databricks put —scope demo —key password —binary-file password.txt
10. To delete scopes: databricks secrets delete-scope —scope demo
11. To push project to Databricks workspace and load the .whl file to dbfs:

1. To install the .whl file from CLI:
• Get cluster-id: databricks clusters get —cluster-name demo
• To install lib: databricks libraries install —cluster-id your-cluser-id —whl dbfs:/tmp/whl-name.whl
2. To export changes made in Databricks and sync with local and use :git diff weather-wheel.py to see the differences:

1. To import local changes to sync with Databricks(completely overwrite): databricks workspace import -o -l PYTHON weather-notebook.py /cli-demo/weather-notebook
2. Some other interactions with Databricks CLI:
• Start a cluster: databricks clusters start —cluster-id your-cluster-id
• List jobs: databricks jobs list
• Get job detail: databricks jobs get  —job-id job-id-number
• Run a job: databricks jobs run-now —job-id job-id-number
• Get running job detail: databricks runs get-output —run-id id-from-last-step
• To terminate(not delete) a cluster: databricks clusters delete —cluster-id your-cluster-id

To create secrets using Databricks CLI:

- databricks secrets create-scope --scope  your-scope-name

- databricks secrets put --scope your-scope-name  --key username --string-value blabla

- databricks secrets list --scope  your-scope-name

To check secrets in Databricks:

- dbutils.secrets.listScopes()

- dbutils.secrets.list('demo')

- dbutils.secrets.get(scope="demo", key="app_key")

- trick to see the key:

app_key = []

for x in dbutils.secrets.get(scope="demo", key="app_key"):

  if x is not None:

    app_key.append(x)

print("app_key:", ' '.join(app_key))

To create .whl file:

- python -m build