(Disclaimer: I work on Nearby Share)
Nearby Share will always attempt to upgrade to WiFi before sending files larger than 1MB. A grace period of 10sec is given for that upgrade before fallback mechanisms kick in and the file is sent over Bluetooth as a last resort. Even after falling back to Bluetooth, the devices will continue to attempt to upgrade to WiFi in the background, but some failures are unrecoverable and the file will fully send over Bluetooth. Note that urls and very small files will immediately send over Bluetooth.
This upgrade can fail for a multitude of reasons. Most commonly, it's a concurrency issue. The same radio is used for Bluetooth, p2p WiFi, and your normal access point connection and it must be time-shared accordingly. If all 3 want to be on different channels, you will miss messages -- it's a guarantee. If these messages are important, such as the authentication frames when connecting over WiFi Direct, then the connection will fail. If these messages are less important, they may be resent until successfully received, but it will lower throughput for the transfer such that even 5GHz WiFi can look to be as slow as Bluetooth.
Nearby Share tries to avoid this in a few ways. When regulations allow, we will attempt to start the WiFi Direct group on the same channel as the access point. This way, the phone doesn't have to timeshare (although it does have the side effect of colliding with messages from the access point. But generally, both sides will back off a random amount and retransmit, and the loss is less than the loss of multi-channel concurrency). Unfortunately, many countries do have regulations that allow certain (or all) 5GHz channels only indoors -- in those cases, the access point can be set up to utilize it but WiFi Direct cannot.
We also prefer mediums like WiFi Direct over Hotspot, as WiFi Direct commonly uses CTS2SELF frames to advertise "Do not transmit for this duration". Unfortunately it's not as straightforward a message as "I am going off channel, do not try to communicate with me" -- CTS2SELF has the side affect of quieting all communication over the access point even if it wasn't directed to the phone -- but it does get the point across.
We pause any Bluetooth activity we have control over, to reduce the need to timeshare with it. In some severe cases, where the OEM has implemented timesharing that's too aggressive, we will turn off the Bluetooth radio to forcefully interrupt Bluetooth activity, but this is disruptive and usually a bad user experience.
We may try to send the data over your access point itself instead of setting up WiFi Direct. This can avoid the MCC situation described above, but comes at the cost that the data needs to be re-encrypted (because we don't know what other devices are connected over LAN and might be eavesdropping), and it does introduce another hop, as the data needs to go through the AP even if the devices are side-by-side. TDLS helps avoid that last issue, but it has limitations (eg. it won't use 40/80/160MHz bandwidth if the AP isn't set up for it, and it'll send over 2.4GHz if the AP is on 2.4GHz).
Other possible failures include the device getting into a bad state (toggling airplane mode can help here), the Android version being too low (ideally both devices should be R+, as it's almost impossible to fix bugs on older Android OS versions, even though we try), or just bad luck.
