4

I'm trying to find an example of Custom Checkpoint Manager in JAVA, that can store checkpoint data in a local folder.

Basically, I'm building a java application that reads data from azure event hub with multiple consumer groups. Previously I was instantiating the EventProcessorHost using storage account connection string and storage container located in azure blobs - which was working fine.

POM entry:

        <dependency>
            <groupId>com.microsoft.azure</groupId>
            <artifactId>azure-eventhubs-eph</artifactId>
            <version>2.4.0</version>
        </dependency>

Java code for instantiating the host:

storageConnectionString="DefaultEndpointsProtocol=https;AccountName=MyAccountName;AccountKey=MyAccountKey;EndpointSuffix=core.windows.net";
storageContainerName="MyContainerName",

EventProcessorHost host = new EventProcessorHost(
                EventProcessorHost.createHostName(hostNamePrefix),
                eventHubName,
                consumerGroupName,
                eventHubConnectionString.toString(),
                storageConnectionString,
                storageContainerName);

Now, the requirement is to use local folder in Azure Databricks cluster (a DBFS:/ path) to store the checkpoint data.

I think I'll have to write a custom checkpoint manager implementing ICheckpointManager. I was able to find an example doing this in SQL database, but I wasn't able to find an example of CheckpointManager storing checkpoint data on a local folder.

Can anyone please help, either give me a link to the example or a code snippet?

0 Answers0