Nested FutureBuilder vs nested calls for lazy loading from database

Question

I need to choose best approach between two approaches that I can follow.

I have a Flutter app that use sqflite to save data, inside the database I have two tables:

Employee:

+-------------+-----------------+------+
| employee_id | employee_name   |dep_id|
+-------------+-----------------+------+
|     e12     | Ada Lovelace    | dep1 |
+-------------+-----------------+------+
|     e22     | Albert Einstein | dep2 |
+-------------+-----------------+------+
|     e82     | Grace Hopper    | dep3 |
+-------------+-----------------+------+

SQL:

CREATE TABLE Employee(
    employee_id TEXT NOT NULL PRIMARY KEY,
    employee_name TEXT NOT NULL ,
    dep_id TEXT,
    FOREIGN KEY(dep_id) REFERENCES Department(dep_id)
    ON DELETE SET NULL
);

Department:

+--------+-----------+-------+
| dep_id | dep_title |dep_num|
+--------+-----------+-------+
| dep1   | Math      | dep1  |
+--------+-----------+-------+
| dep2   | Physics   | dep2  |
+--------+-----------+-------+
| dep3   | Computer  | dep3  |
+--------+-----------+-------+

SQL:

CREATE TABLE Department(
    dep_id TEXT NOT NULL PRIMARY KEY,
    dep_title TEXT NOT NULL ,
    dep_num INTEGER,
);

I need to show a ListGrid of departments that are stored in the Employee table. I should look at Employee table and fetch department id from it, This is easy but after fetching that dep_id I need to make a card from those ids so I need information from Department table. complete inforamtion for thoses id I had fetched from Emplyee table is inside Department table.

There are thousands of rows in each table.

I have a database helper class to connect to the database :

DbHelper is something like this:

Future<List<String>> getDepartmentIds() async{
    'fetch all dep_id from Employee table'
}

Future<Department> getDepartment(String id) async{
    'fetch Department from Department table for a specific id'
}

Future<List<Department>> getEmployeeDepartments() async{
    '''1.fetch all dep_id from Employee table
    2.for each id fetch Department records from Department table'''

    var ids = await getDepartmentIds();
    List<Departments> deps=[];
    ids.forEach((map) async {
        deps.add(await getDepartment(map['dep_id']));
      });
}

There is two approaches:

First One:

Define a function in dbhelper that returns all dep_id from Employee table(getDepartmentIds and another function that returns a department object(model) for that specific id.(getDepartment)
Now I need two FutureBuilder inside each other, one for fetching ids and the other one for fetching department model.

second One:

Define a function that first fetch ids then inside that function each id is maped to department model.(getEmployeeDepartments)
So I need one FutureBuilder .

Which one is better?? should I let FutureBuilders handle it or I should put pressure on dbHelper to habdle it?

If I use the first approach then I have to(as far as I can imagine!) put the the second future call(the one that fetch Department Object(model) based on it's id(getDepartment)) on build function and it's recommended no to do so.

And the problem with second one is that it does a lot of nested call in dbHelper.

I used ListView.builder for performance.

I checked both with some data but couldn't figure out which one is better. I guess it depends both on flutter and sqlite(sqflite).

which one is better or is there any better approach?

score 3 · Accepted Answer · answered May 22 '19 at 21:15

3

Given that I don't see too much code on this example, I'll do a high-level answer on your questions.

Evaluate Approach One

Right off the bat this part sticks out: "returns all dep_id from Employee table"
I would say scratch that, since "return all" is typically never a good solution, especially since you mention your tables have a lot of rows.

Evaluate Approach Two

I'm not sure what the difference in performance this has compared to the first approach, seems also bad for the same reasons. I think this one just changes your UI logic a big is all.

Typical 'Endless' List Approach

You would do a query on the Employees table with a join to the Departments table.
You would implement Pagination on your UI and pass in your values to the query from step one.
At a basic level you'll need these variables: Take, Skip, HasMore
Take: The count # of items to request each query
Skip: The count # of items to skip on the next query, this will be the size of the number of items you currently have in your List in memory driving your UI.
HasMore: You can set this on the response of each query, to let the UI know if there are still more items or not.
As you scroll down the list, when you get to the bottom, you will request more items .

Initially issue a query for example: Take: 10, Skip: 0 Next query when you hit the bottom of the UI: Take: 10, Skip: 10 etc..

Example sql query:

SELECT * 
FROM Employees E
   JOIN Departments D on D.id = E.dept_id
order by E.employee_name  
offset {SKIP#} rows
FETCH NEXT {TAKE#} rows only

Hopefully, this helps, I'm not fully sure what you're trying to do actually - in terms of Code.

answered May 22 '19 at 21:15

Alan Negrete

400
2
11

Thanks @alan-negrete it was very helpful, I read about Pagination here but I didn't know about offset in SQL. – NoobN3rd May 22 '19 at 21:54
1

Hmm. In this case using offset and fetch next would probably work fine as I assume the list is fairly static, but in general I would recommend using the `WHERE >= lastValue` approach instead as that does a better job of avoiding duplicates or skips if items are added/removed from the db between page query requests. – rmtmckenzie May 22 '19 at 21:57
@rmtmckenzie Thanks for your suggestions, I'll try both approaches. – NoobN3rd May 22 '19 at 22:30
1

this approach by @alan-negrete is the most efficient, i like to add to be sure the field(s) at order by (E.employee_name in example) ensure absolute ordering else you may 'loose' a record if you have paging functionality (10, next 10, next 10 and so on). using actual order by field followed by primary key is one standard workaround. – Stefanos Zilellis May 23 '19 at 12:25
@StefanosZilellis Yes in this quesion(https://stackoverflow.com/questions/14468586/efficient-paging-in-sqlite-with-millions-of-records) they recommended also the @ rmtmckenzie recommendation.that question and its answers are also informative. – NoobN3rd May 23 '19 at 18:23
1

@NoobN3rd Yeah good luck! I would recommend making a Generic Widget that handles all this endless scrolling for you executing a callback functions each time it hits the bottom. – Alan Negrete May 29 '19 at 00:41

score 1 · Answer 2 · answered May 22 '19 at 21:54

As far as I can tell, what you're looking to do is get a list of employees with relevant info including department.

If that's the case, then it's tailor made for INNER JOIN. Something like this:

SELECT Employee.*, Department.dep_id, Department.dep_title 
   FROM Employee INNER JOIN Department 
   ON Employee.dep_id = Department.dep_id;

(although you may want to double check that, my SQL is a bit rusty).

This would do what you need in one step. However, there is still the issue of what you're asking which seems to be "Is it more efficient to do many small requests or one big one, and what are the performance ramifications".

The answer to that is a bit specific to Flutter. What's happening when you do a request with SQFLITE, is that it is processing whatever you've passed to it, sending it to java/objc and possibly doing more processing and pushes processing to a backround thread, which then calls to the SQLITE library which does more processing to understand the request, then actually reads the data on the disk to do the operation, then returns back to the java/objc layer, which pushes the response to the UI thread, which in turns responds back to dart.

If that doesn't sound particularly efficient, that's because it isn't =D. If you're doing this a few times (or even a few hundred) it's probably fine, but if you're getting into thousands as you state it might start slowing down.

The alternative you've proposed is to do one large request. You will know better than I whether that is wise; if it's a couple thousand but only ever a couple thousand, and the data you're returning is always going to be relatively small (i.e. just a 10-20 character name and department name), then you'll have say (20+20)*2000 = 8000b = 80kb of data. Even if you assume the overhead will double that size, 160 kb of data shouldn't be enough to faze any relatively recent smartphone (after all that's much smaller than any single photo!).

Now, taking some domain specific knowledge, you could optimize this. For example, if you know the number of departments is much smaller than employees (i.e. < 100 or something), you could skip the entire issue of doing joins, and simply request all departments before this begins and put it in a map (dep_id => dep_title), and then once you've requested employees you could just simply do that lookup from dep_id to dep_title yourself. That way your requests wouldn't have to include the dep_title over and over again.

That being said, you may want to consider paging the employee lookup whether or not you use a join. You'd do this by requesting 100 employees (or whatever number) at a time rather than the entire batch - that way you don't have the overhead of 1000+ calls through the stack, but you also don't have a large block of data all in memory all at once.

SELECT * FROM Employee
WHERE employee_name >= LastValue
ORDER BY employee_name
LIMIT 100;

Unfortunately that doesn't fit in as well with how flutter does lists, so you'd probably need to have something like a 'EmployeeDatabaseManager' that does the actual requests, and your list would call into it to get the data. That's probably beyond the scope of this question though.

Nested FutureBuilder vs nested calls for lazy loading from database

First One:

second One:

2 Answers2

Linked

Related