We have an AlloyDB instance set up with a read pool. Inside our application, we route database queries to either the primary node or the read pool, depending on whether the operation itself is a SELECT or not. This has been working well; however, we have occasionally encountered errors that seem to be the result of changes not being replicated to the read pool. Specifically:
- We insert a record using a connection to the primary node and get the primary key of the inserted record.
- We attempt to fetch the inserted record using the primary key using the read pool.
- The latter query returns 0 rows.
- We can inspect the database after the fact and see the record does in fact exist.
It was my understanding that the replicas would wait until any relevant WAL logs were processed before processing a query, ensuring that their state was always in sync with the primary node. Are there conditions under which the read pool state could be stale or get out of sync with the primary node? We'd like to understand what could explain the behavior we're seeing and what can we do remedy it.