7

I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk). This function takes a pointer to a foreign memory (where chunk is placed) as an argument. So, I make buffer for a chunk: foreign-buffer. I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:

(let ((foreign-buffer (foreign-alloc :uchar 1024)))
      (stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
    (loop
        for cnt = (read-sequence stream-buffer MY-STREAM)
        while (> cnt 0)
        do
          ;; copy cnt bytes from stream-buffer into foreign-buffer 
          ;; call db-write-chunk with foreign-buffer

L in BLOB is for Large and loop may iterate many times. Besides that, all this code may be wrapped by the external loop (bulk-insert, for example). So, I want to minimize the count of steps in the loop(s) body.

To have this done I need:

  • to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:

    (read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)

  • or to be able to interpret stream-buffer as foreign memory, like this:

    (db-write-chunk (mem-aptr stream-buffer :uchar 0))

Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?

sds
  • 58,617
  • 29
  • 161
  • 278
MrCat
  • 107
  • 2
  • 1
    I think this is highly implementation-dependent. What Lisp are you using? – sds Nov 19 '18 at 20:46
  • 3
    maybe https://github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines – David Hodge Nov 20 '18 at 00:25
  • @sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL. – MrCat Nov 20 '18 at 13:55
  • @DavidHodge, thanks, I've just downloaded it and try to understand how it works – MrCat Nov 20 '18 at 13:58
  • 1
    Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled). – MrCat Nov 20 '18 at 15:45

1 Answers1

1

Like anything else ffi, this is implementation dependent, but cffi has cffi:make-shareable-byte-vector, which is a CL (unsigned-byte 8) array which you can then use with cffi:with-pointer-to-vector-data:

(cffi:defcfun memset :pointer
  (ptr :pointer)
  (val :int)
  (size :int))

(let ((vec (cffi:make-shareable-byte-vector 256)))
  (cffi:with-pointer-to-vector-data (ptr vec)
    (memset ptr 0 (length vec))))

Depending on your use, this might be preferable to static-vectors, because you don't have to remember to free it manually. On SBCL this works by pinning the vector data during with-pointer-to-vector-data.

Wilfredo Velázquez-Rodríguez
  • 132
  • 7