How to create logging from Python notebook in Databricks to ADLS Gen2 (without mounting)?

Question

I'm trying to create a logging mechanism inside Databricks Python notebook. Tried using below code to achieve the same -

import logging

def create_logger(name,log_path=None):
  
    logger = logging.getLogger(name)  
    logger.setLevel(logging.DEBUG)
    formatter    = logging.Formatter("%(asctime)s - %(levelname)-8s - %(message)s")  
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(formatter)
    logger.addHandler(stream_handler)
    
    if log_path is not None:        
        file_handler = logging.FileHandler(log_path)
        file_handler.setFormatter(formatter)
        logger.addHandler(file_handler)
        
    return logger

However, whenever I'm trying to call the function as below -

from datetime import date, datetime
current_date = date.today()
current_timestamp = datetime.strftime(datetime.now(),"%Y%m%d%H%M%S")

name = "temp_logs"
log_path = f"abfss://{storageContainer}@{storageAccount}.dfs.core.windows.net/{target_dir}/logs/{current_date}/{name}_{current_timestamp}.txt"

logger = create_logger(name = name,log_path = log_path)

This is giving error as -

[Errno 2] No such file or directory: /databricks/driver/abfss:/temp-ontainer@teststorage.dfs.core.windows.net/test/logs/2021-09-13/temp_logs_20210913101150.txt'

Is there a way to handle this (without using mount point location) ?

Does this answer your question? [Writing log with python logging module in databricks to azure datalake not working](https://stackoverflow.com/questions/55689544/writing-log-with-python-logging-module-in-databricks-to-azure-datalake-not-worki). Just change your FileHandler to BlobStorageRotatingFileHandler — Kafels, Sep 14 '21 at 16:21
@Kafels - Getting error as `module 'logging' has no attribute 'BlobStorageRotatingFileHandler'` — , Sep 16 '21 at 06:02

score 1 · Answer 1 · answered Sep 21 '21 at 13:05

We can try using “BlobStorageRotatingFileHandler” by importing it as below:

from azure_storage_logging.handlers import BlobStorageRotatingFileHandler

Can refer to the python document about azure-storage-logging as it provides functionality to send output from the standard Python logging APIs to Microsoft Azure Storage.

Sample code as below:

import logging
from azure_storage_logging.handlers import TableStorageHandler

# configure the handler and add it to the logger
logger = logging.getLogger('example')
handler = TableStorageHandler(account_name='mystorageaccountname',
                              account_key='mystorageaccountkey',
                              extra_properties=('%(hostname)s',
                                                '%(levelname)s'))
logger.addHandler(handler)

# output log messages
logger.info('info message')
logger.warning('warning message')
logger.error('error message')

Use the above logs to record the error messages.

And for "directory not found" we need to check the path. Please have a look on the documentation

How to create logging from Python notebook in Databricks to ADLS Gen2 (without mounting)?

1 Answers1