I think that you're structuring your data as if you were working in a relational database. Firestore doesn't support foreign keys like SQL databases, so you can't retrieve nested data like SQL. Firestore is a no-SQL database that doesn't have any notion of reference. See Firestore data model.
In Firestore, relationships are usually modeled by storing the id of the document you'd like to "reference". In that sense you might not need to store the 'users' document inside the ‘followers’ but the field 'user_id' would suffice( which you are trying to do). The caveat is that this data layout comes at the price of having to fetch the 'user_id' from ‘followers’ before you can fetch the actual user data. You could read more about data denormalization and modeling relationships in articles link1 , link2 and this other thread.
In Firestore you need to fetch referenced data separately, either you can fetch all users data separately in parallel with followers data and store it in map, or if you don't need users data initially then fetch each users data when needed like when you check details of each followers.