Is this actually broadcast, ie one-to-many transmission? It doesn't sound like it's pure broadcast, because you mention retransmission decisions per receiver. So I'll answer assuming you can send packets to each receiver independently.
You can build a good reliability mechanism by borrowing concepts from TCP. TCP has the most general and well-worn solution - it handles out-of-order reassembly, and can scale to high bandwidths with high latency (using ACK windows), and has some adaptivity to channel conditions.
Depending on what you're doing, that may be overkill. You could, instead, borrow from USB, which also has reliability and disambiguation mechanisms in packet delivery and acknowledgments. But it can only handle one packet outstanding at a time (e.g. window size == 1), and can't handle out-of-order delivery, which become major performance limiters if you have a high-bandwidth or high latency requirement.
In either case, your overall timeout (for signaling a delivery failure to the application layer) should be long that you don't ever expect to hit it during normal operation. TCP implementations, for example, will wait 15+ seconds to give up on delivery and signal a problem to the application layer.
Designing anything beyond basic one-packet-at-a-time delivery requires serious protocol design and QA to get right. The corner cases are hard to hit. If your requirements are non-trivial, you need to hire some solid network protocol engineers, or figure out a way to use an existing solution like TCP!
See also related discussions at What do you use when you need reliable UDP? .