What's the best way to upload an ILP file into QuestDB?

Question

I have a text file in ILP format that I would like to import into QuestDB.

I know QuestDB can ingest data in ILP format, but it seems you can do this only line by line. I have tried using the HTTP endpoint for file ingestion, but it supports loading CSV files only. I am wondering if anyone had to deal with this and how you did it.

For context, I am migrating some data from InfluxDB and I have exported my data (using influxd inspect) as a large file containing all my ILP points. It looks something like this (just several Gigs of it):

diagnostics,device_version=v1.0,driver=Albert,fleet=East,model=F-150,name=truck_1027 current_load=2658 1451612300000000000
diagnostics,device_version=v1.0,driver=Albert,fleet=East,model=F-150,name=truck_1027 current_load=3436 1451612310000000000
readings,driver=Trish,fleet=West,model=H-2,name=truck_972 velocity=89 1451831680000000000

score 0 · Accepted Answer · answered Jan 26 '23 at 16:47

QuestDB does support the ILP protocol. You can just send your ILP points over a socket connection using the TCP port (defaults to 9009).

The official client libraries do exactly that, but they do it in a more convenient way so you don't have to compose the raw message yourself, which could be error prone.

In your case, since you already have valid ILP points, you can just iterate over your file and send via socket. I am showing a basic example using Python:

import socket
import sys

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

def send_utf8(msg):
    print(msg)
    sock.sendall(msg.encode())

if __name__ == '__main__':
    try:
        sock.connect(('localhost', 9009))
        with open("YOUR_FILE") as infile:
            for line in infile:
                # print(line)
                send_utf8(line)
    except socket.error as e:
        sys.stderr.write(f'Got error: {e}')
    sock.close()

What's the best way to upload an ILP file into QuestDB?

1 Answers1

Linked