3

Cross posted at https://developer.jboss.org/thread/279735

Suppose we have multiple docker containers (each of which has java webapps, so multiple JVMs essentially), each of which are using the same replicated Infinispan cache either in:

  • Embedded mode using jgroups for discovery
  • Server mode using Docker Hub as Infinispan server, and the clients connecting via hotrod.

In either case, all the cache members/clients have a file-store from which they preload at startup (sample is for server mode, its similar for embedded mode's xml):

Via docker compose, the path inside each container's file store (lets say /var/tmp/server/OUR_CACHE.dat) is bind-mounted to the same file on the docker host.

<paths>
       <path name="cachestore.root" path="/var/tmp"/>
</paths>
...
    <local-cache name="OUR_CACHE">

         <expiration lifespan="-1"/>

         <locking isolation="SERIALIZABLE" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

         <file-store relative-to="cachestore.root" path="server" max-entries="-1" purge="false" passivation="false" preload="true" fetch-state="true"/>

         <memory>

              <binary size="100000000" eviction="MEMORY"/>

         </memory>

    </local-cache>

My question is that - is the persistence mechanism designed so that multiple replicated cache clients can read/write to/from the same file store without any errors?

My understanding is that in replicated mode, the key values will become eventually consistent across all node's memory. But my goal is to ensure that multiple client containers using the same file-store for persistence and preloading should not adversely affect this replication.

If its not advised to share the same file-store .dat file, then what is the best practice to have a GUID in the filepath for each container's path in the .xml file. Each docker container's hostname (which is containerId) is unique, but it won't be known before its deployed, so it won't be easy to populate the infinispan_server.xml file with the value for "path" statically.

Thanks,

_Prateek

PKM
  • 329
  • 4
  • 17

1 Answers1

1

Isn't the point of replicated mode that each node has its own independent copy of the data? I don't fully understand what you're trying to achieve.

To the last point of your question:

If its not advised to share the same file-store .dat file, then what is the best practice to have a GUID in the filepath for each container's path in the .xml file. Each docker container's hostname (which is containerId) is unique, but it won't be known before its deployed, so it won't be easy to populate the infinispan_server.xml file with the value for "path" statically.

Can you put a placeholder in the config xml (e.g.: ${myprop}) and specify it at startup (e.g.: -Dmyprop=hostname01)?

batwad
  • 3,588
  • 1
  • 24
  • 38
  • What I'm hoping to get working is that multiple containers (each of which is a cache client), preload the same data from the file store on disk. At present, the path each of the containers pickup the file store is fixed so all of them will preload the same data correctly. But what I am unable to ascertain if during normal operation, all the containers can persist data to their independent file store whose filepath maps to the same docker host's file path for all containers. Now if I made this path unique somehow like say making the filepath include /$HOSTNAME/.. then at initial deployment – PKM Apr 03 '19 at 09:31
  • (continued) i'd have my file stores at using placeholder as maybe hostname (which is containerId in docker): /path/on/dockerhost/containerId1/filestore/cache.dat and /path/on/dockerhost/containerId2/filestore/cache.dat But after redeploying swarm stack, the unique names might be containerId3 and 4, which need to be mapped via compose to pickup the cache.dat from above container1 or 2. So, my clients might have independent copy of the data replicated in memory, but I need the file stores to be common, so the preloading of all containers finds the same data initially at startup.Thanks – PKM Apr 03 '19 at 09:32
  • Sorry...docker...swarm stack...I don't know about those. I still don't get what you're trying to do. Do you want the containers to be preloaded with a set of data, but have their own stores and allow them to operate independently? – batwad Apr 03 '19 at 10:22
  • In this case (logically speaking) can consider each container as a separate client machine, and I want all of them to preload the same data when started - so the filepath to the filestore they all use, is the same (and is on say, an NFS mounted store). So while it does seem they all can preload the data, I'm not sure if they are writing to that .dat file on the same shared filestore path, without complications. In essence can the file store file alone, be used (read/write) simultaneously by multiple infinispan hotrod clients. – PKM Apr 03 '19 at 11:18
  • What if one clients writes to one of the caches? Should the change be visible everywhere or should it be isolated to that client? – batwad Apr 03 '19 at 12:06
  • If a clients writes to a cache (in this case the single cache server which is centralized), other clients should be able to see the new change since they query the same server. I believe the case wherein each client uses its own local cache (and the task of replication is left to jgroups), *might* be embedded infinispan (afaik http://infinispan.org/docs/stable/user_guide/user_guide.html#clustering) but anyway here i have single cache server and multiple cache clients connecting to it via hotrod. But clients read/write to same shared file store for persistence which I can't find much doc on. – PKM Apr 04 '19 at 02:07
  • But clients talking to different servers should not see the updates that occur on others? They are totally independent apart from starting with the same initial data? I think the simplest thing to do would be to make the preloading of the cache a deployment step and keep each server independent. No need for shared storage, just a simple script will suffice. – batwad Apr 04 '19 at 08:27
  • I have 1 server and n clients. But I'd forgotten that in this mode, there is only one file store (as there's 1 server). its in embedded mode where we have to worry about multiple clients as each client has its own file store. I guess my doubt would apply only to embedded mode. – PKM Apr 04 '19 at 08:37