Android data counter discrepancy (sum of apps < total)

Question

I am trying to understand the cause of a data discrepancy I'm noting with only a couple of applications. In both Tidal and Apple music, if I skip through tracks while listening to them, I will immediately get a 10% to 15% data discrepancy between the sum of all apps data usage and the total volume for the device.

Test Case

Performed 36.03MiB through Tidal (skipping tracks) with Data Saver enabled to restrict background data to a minimum.

Android Data Usage Counters (Settings - Connections - Data Usage)

Total

 - Total                             36.03MiB

Apps

 - Tidal                             31 MiB
 - Google Play Services              448.04 KiB
 - My Data Manager                   59.9 KiB
 - com.samsung.accessory.wmmanager   10.94 KiB
 - Google Play Store                 5.27 KiB
 - Android.OS                        5.13 KiB
 - Samsung Push Service              3.59 KiB
 -----------------------------------------------
 - Total of all Apps                 31.52 MiB

Missing Data

 - Total                             4.51 MiB

My Data Manager

For the exact same data test, My Data Manager reports the same values

 - Tidal                             30.8 MiB
 - etc

NetworkStatsManager

In my tracking application I am using NetworkStatsManager.QuerySummaryForDevice for the overall volume, and constructing app totals from NetworkStatsManager.QuerySummary using the bucket uid for the package identifier. This has been confirmed on several Oreo devices.

From this I got:

QuerySummaryForDevice

 - Total                                        37777120 (36.027MiB)

QuerySummary

 - Tidal (com.aspiro.tidal)                     32511933 (31.005MiB)
 - Google Play Services (com.google.uid.shared) 459244 (448.48KiB)
 - My Data Manager (com.mobidia.android.mdm)    61333 (59.90KiB)
 - com.samsung.accessory.wmmanager              11196 (10.933KiB)
 - Google Play Store (com.android.vending)      5397 (5.271KiB)
 - Android.OS (com.uid.system)                  5246 (5.123KiB)
 - Samsung Push Service (com.sec.app.push)      3677 (3.591KiB)
 -------------------------------------------------------------------
 - Total of all Apps                            33058026 (31.527MiB)

Missing Data

 - Total                                        4719094 (4.500MiB)

So, identical results give or take some tiny timing discrepancies in taking / viewing the readings.

As a 4th confirmation, I pulled the /proc/net/xt_qtaguid/iface_stats_all and /proc/net/xt_qtaguid/stats files before and after the test and the figures again all line up within a few bytes of the above methods.

The packet count figures also follow the same pattern with ~3000 missing between total data usage and sum of apps, so I don't think this can be down to packet headers not being tracked in the app counters.

So my questions are:

• Is there any logical reason for this missing data, given every other application tracks correctly leaving a negligible gap between the sum of the apps and the total data?
• Are there any system services that do not get tracked by xt_qtaguid/stats or NetworkStatsManager causing this?

Answer 1

Your first query, I know why its happening the miscalculations.

The Rest API works as the request-response way. You send a request and the response comes and you manipulate the response. But sometimes there is errors like 404 401 or no response etc. When that happens the system can miscalculate the data packets. And sometime the system services are not trackable with the Network manager in android.