Python - Paramiko- How to SSH and transfer files with python

 You can transfer files from the remote machine to the local or vice versa using SFTP (Secure File Transfer Protocol) and SCP(Secure Copy Protocol).

https://medium.com/@keagileageek/paramiko-how-to-ssh-and-file-transfers-with-python-75766179de73

Databricks - understand your databases/tables

To get the databases you have tables in and how many tables in each database:

This can also be achieved by using Unity Catalog:

%sql

SHOW EXTERNAL LOCATIONS

databases = [ db['databaseName'] for db in spark.sql('show databases').collect()]

tbl_count_list = []

for db_rows in databases :

  print(db_rows)

  final_table_list = []

  table_list = [row['database'] + '.' + row['tableName'] for row in spark.sql(f'show tables in {db_rows}').collect()]

  count = 0 

  for table in table_list:

    try:

      if 'your_data_lake_location' in spark.sql(f'describe detail {table}').collect()[0]['location']:

        count += 1

        final_table_list.append((db_rows, table))

    except:

        print('403 forbidden')

        

  print(final_table_list)  

  tbl_count_list.append((db_rows, count))

  print(tbl_count_list)