Upload binary files using python-gitlab API

Question

I'm tasked with migrating repos to gitlab and I decided to automate the process using python-gitlab. Everything works fine except for binary or considered-binary files like compiled object files ( .o ) or .zip files. (I know that repositories are not place for binaries. I work with what I got and what I'm told to do.)

I'm able to upload them using:

import gitlab

project = gitlab.Gitlab("git_adress", "TOKEN")

bin_content = base64.b64encode(open("my_file.o", 'rb').read() ).decode()

and then:

data = {'branch':'main', 'commit_message':'go away', 'actions':[{'action': 'create', 'file_path': "my_file.o", 'content': bin_content, 'encode' : 'base64'}]}

project.commits.create(data)

Problem is that content of such files inside gitlab repository is something like:

f0VMRgIBAQAAAAAAAAAAAAEAPgABAAAAAAAAAAAAA....

Which is not what I want. If I don't .decode() I get error saying:

TypeError: Object of type bytes is not JSON serializable

Which is expected since I sent file opened in binary mode and encoded with base64.

I'd like to have such files uploaded/stored like when I upload them using web GUI "upload file" option.

Is it possible to achieve this using python-gitlab API ? If so, how?

"*Problem is that content of such files inside gitlab repository is something like...*" So you *don't* want to upload your files as base64-encoded, but yet you explicitly encode it before uploading it...? — esqew, Nov 03 '21 at 17:46
@esqew Yes. It's the only way (so far) to get them (binary files) into repository using python-gitlab API. Thesame "JSON" serialization error occurs if I don't use base64-encoding, when I send just 'rb' file. — Rad226, Nov 03 '21 at 17:54

score 3 · Accepted Answer · answered Nov 03 '21 at 18:46

The problem is that Python's base64.b64encode function will provide you with a bytes object, but REST APIs (specifically, JSON serialization) want strings. Also the argument you want is encoding not encode.

Here's the full example to use:

from base64 import b64encode
import gitlab
GITLAB_HOST = 'https://gitlab.com'
TOKEN = 'YOUR API KEY'
PROJECT_ID = 123 # your project ID
gl = gitlab.Gitlab(GITLAB_HOST, private_token=TOKEN)
project = gl.projects.get(PROJECT_ID)

with open('myfile.o', 'rb') as f:
    bin_content = f.read()
b64_content = b64encode(bin_content).decode('utf-8')
# b64_content must be a string!

f = project.files.create({'file_path': 'my_file.o',
                          'branch': 'main',
                          'content': b64_content,
                          'author_email': 'test@example.com',
                          'author_name': 'yourname',
                          'encoding': 'base64',  # important!
                          'commit_message': 'Create testfile'})

Then in the UI, you will see GitLab has properly recognized the contents as binary, rather than text:

Issue was that I used "encode" instead of "encoding" in data = {} commit, as you said. I have no idea what I was thinking since docs state it clearly: https://python-gitlab.readthedocs.io/en/stable/gl_objects/commits.html Thank you for the solution. — Rad226, Nov 03 '21 at 19:01

Upload binary files using python-gitlab API

1 Answers1