I am using faiss indexflatIP to store vectors related to some words. I also use another list to store words (the vector of the nth element in the list is nth vector in faiss index). I have two questions:
You can do both.
- Is there a better way to relate words to their vectors?
Call index.add_with_ids(vectors, ids)
Some index types support the method add_with_ids, but flat indexes don't.
If you call the method on a flat index, you will receive the error add_with_ids not implemented for this type of index
If you want to use IDs with a flat index, you must use index2 = faiss.IndexIDMap(index)
- Can I update the nth element in the faiss?
If you want to update some encodings, first remove them, then add them again with add_with_ids
If you don't remove the original IDs first, you will have duplicates and search results will be messed up.
To remove an array of IDs, call index.remove_ids(ids_to_replace)
Nota bene: IDs must be of np.int64 type.
You can use the add_with_ids method to add vectors with integer ID values, and I believe this will allow you to update the specific vector too - but you will need to build some sort of added layer of vector-ID mapping and management outside of Faiss because it isn't supported otherwise. I've done this before and it isn't very fun.
If you're open to Faiss alternatives, I'd recommend looking into Pinecone. It manages everything for you so you you just insert your (id, vector) pairs using their upsert method, then to update the vectors you just upsert the new vector with the same ID. It takes maybe 5-10 minutes to set up, this guide explains how.
faiss is only an ann algorithm library, and cannot be used for data persistence and management
There are some open source vector databases on the github, they may be able to help you. such like milvus, vespa, and so on
milvus is the one with the most stars
