Expected file to have JSONL format, where every line is a JSON dictionary. openai createFile for fine tune

Question

I created file with name mydata.jsonl and I put on it these lines

    {
        "prompt": "aa",
        "completion": "bb"
    }
    {
        "prompt": "cc",
        "completion": "dd"
    }

then in index.js I did this function

    const { Configuration, OpenAIApi } = require("openai");
    const fs = require("fs");
    
    const configuration = new Configuration({
        apiKey: process.env.OPENAI_API_KEY,
    });
    async function getAiResponse(topic) {
        const openai = new OpenAIApi(configuration);
        const filename = 'example.txt';
        const fileContents = 'This is some example text.';
        const response = await openai.createFile(
            fs.createReadStream("mydata.jsonl"),
            "fine-tune"
        );
        console.log(response)
    }
    getAiResponse("ee");

when I run my code I got error

    Expected file to have JSONL format, where every line is a JSON dictionary. Line 1 is not a dictionary (HINT: line starts with: "{...").

I can't get where is the exactly the error

score 2 · Accepted Answer · answered Apr 05 '23 at 04:10

2

The JSONL format is where the each json object is sperated by a newline characater.

So your output should be:

{ "prompt": "aa", "completion": "bb" }
{ "prompt": "cc", "completion": "dd" }

i.e. one LINE per json object

answered Apr 05 '23 at 04:10

Shane Powell

13,698
2
49
61

score 0 · Answer 2 · answered May 04 '23 at 12:10

You can also get this error if the file you upload is encoded as UTF8 with the Byte Order Mark (BOM) included - despite the JSONL payload being valid.

Their file parser obviously misinterprets the BOM (commonly, hex EF BB BF) as part of the document payload, and decides it's not valid JSONL.

A fast way to check the encoding is to load the file in a good text editor, and it will often tell you. This is from Notepad++, for example:

(Alternatively, just open it in a binary editor, and look for EF BB BF at the start)

See this other answer for how to remove the BOM: How can I remove the BOM from a UTF-8 file?

Expected file to have JSONL format, where every line is a JSON dictionary. openai createFile for fine tune

2 Answers2