Stack Overflow
Stack Overflow

Questions tagged [klepto]

A python library for persistent caching to memory, disk, or database

klepto is a python library for persistent caching to memory, disk, or database.

17 questions
5
votes
1 answer

Python Saving and Editing with Klepto

Okay so my question is pretty specific and I apologize in advance. I'm a new programmer and tried developing on my own from scratch. It was relatively successful only I have one last problem, that I can see. You can view my code here in its…
Mario ER
  • 53
  • 1
  • 7
4
votes
4 answers

Decorators for selective caching / memoization

I am looking for a way of building a decorator @memoize that I can use in functions as follows: @memoize my_function(a, b, c): # Do stuff # result may not always be the same for fixed (a,b,c) return result Then, if I do: result1 =…
Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564
2
votes
0 answers

How to use Klepto package Python

Hellow World! I have a pickle file (containing images in byte form) to open but when I open it with pickle I get this error: Traceback (most recent call last): File "test.py", line 20, in if __name__ == "__main__": main() File…
Stefano Pozzi
  • 529
  • 2
  • 10
  • 26
1
vote
1 answer

Read/Write/Updating object without loading the object to memory

I have been trying out with the Klepto package to Write/Read/Update my object to harddisk, aiming to avoid the "out of memory" issues that I experienced when training my model with my dataset. From my understanding, with the Klepto I could store my…
user15391017
  • 13
  • 2
1
vote
0 answers

Is Klepto (Python module) file_archive supposed to be 10x slower than Pickle?

Paging @mike-mckerns I suppose, but I'd be grateful for answers from anyone with experience with the Klepto module for Python (https://pypi.org/project/klepto/). My situation is that I'm running a simulation which involves generating and logging…
nadafan boy
  • 47
  • 6
1
vote
1 answer

How to set a table name using klepto sqltable_archive in Python 3?

Based on the klepto docs Using: klepto.archives.sqltable_archive('sqlite:///foo.db', cached=False) Successfully creates a database with the default table name of memo However, when table is also…
macourtney7
  • 521
  • 2
  • 10
  • 24
1
vote
1 answer

Keys for klepto dir_archive?

I've created a klepto dir_archive. On subsequent archive access, how can the archive keys be determined without loading the entire archive into memory?
jim vickroy
  • 142
  • 2
  • 9
1
vote
1 answer

How to delete multiple items from Klepto file archive fast?

I'm using Klepto archive to index specs of files in a folder tree. After scanning the tree, I want to quickly remove references to deleted files. But simply removing an item one-by-one from the file archive is extremely slow. Is there a way to sync…
starfish
  • 13
  • 3
1
vote
1 answer

Different tolerance for parameters in klepto cache

I would like to set a tolerance for a certain function argument, but not for others. Let's say I have a function like this: def myFnc(input1,param0 = 0, param1 =1): return "input1:%s [params %s, %s]" %(input1, param0, param1) These are the…
lib
  • 2,918
  • 3
  • 27
  • 53
1
vote
1 answer

How to use LRU caching on disk with klepto?

I am trying to use klepto to do LRU caching. I would like to store the cache to disk, and am currently using klepto's dir_archive option for this. I have written the following code, largely based on the code in the klepto test scripts: def…
user2593669
  • 49
  • 7
0
votes
2 answers

How to save large sklearn RandomForestRegressor model for inference

I trained a Sklearn RandomForestRegressor model on 19GB of training data. I would like to save it to disk in order to use it later for inference. As have been recomended in another stackoverflow questions, I tried the…
kedar
  • 45
  • 7
0
votes
0 answers

Getting KeyError in Klepto when used from local machine

I want to persist my ML model to my local machine. I have followed https://stackoverflow.com/a/29291153/11543494 this answer to store ML model in local machine but when I load persisted ML model from local machine I get Key Error. I have created ML…
Mayur Chawda
  • 16
  • 3
0
votes
1 answer

How to change the name of dictionary in the file when using klepto.archives.file_archive?

After running the following code, I found that the name of the dictionary in file memo.py is memo. I want to change the name of the dictionary in that file. Do anyone know how to do it? >>> init = {'y': 2, 'x': 1, 'z': 3} >>> import klepto >>> cache…
mary
  • 1
  • 1
0
votes
0 answers

How to Persist class Instances using klepto

I'm trying klepto as a replacement for shelve to persist class instances but am seeing a lengthy traceback when attempting to do so. My platform is: python 3.7.3 klepto 0.1.6 The example @ Pickle versus shelve storing large dictionaries in Python…
jim vickroy
  • 142
  • 2
  • 9
0
votes
1 answer

Time-expiring cache in klepto

I'm experimenting with klepto (v. 0.1.5.2) to cache some very heavy and time-consuming computations. In particular, I'm using lru_cache as decorator, coupled with a hdfdir_archive cache archive. I managed to persist cache content on disk using the…
user38320
  • 1
1
2