2

I have phonorgraph object with billions of rows and we are querying it through object set service for example, I want to get all DriverLicences from certain city.

        @Function()
        public getDriverLicences(city: string): ObjectSet<DriverLicences> {
            let drivers = Objects.search().DriverLicences().filter(row => row.city.exactMatch(city));
            return drivers ;
        }

I am facing this error when I am trying query it from slate:

ERROR 400: {"errorCode":"INVALID_ARGUMENT","errorName":"ObjectSet:PagingAboveConfiguredLimitNotAllowed","errorInstanceId":"0000-000","parameters":{}}

I understand that I am probably retrieving more than 100 000 results but I need all the results because of the implemented logic in the front is a complex slate dashboard built by another team that we cannot re-factor.

fmsf
  • 36,317
  • 49
  • 147
  • 195
houssem gharsalli
  • 59
  • 5

1 Answers1

1

The issue here is that, specifically in the Slate <> Function connector, there is a "translation layer" that serializes the contents of the object set and provides a response data structure that materializes the property:value pairs for each object in the set.

This clearly doesn't work for large object sets where throwing so much data into the browser is likely to overwhelm the resources allocated to the tab.

From context it seems like you might be migrating an existing Slate app over to Functions; in the current version, how is the query limiting the number of results returned? It certainly must not be returning several 100 thousand results for further processing on the front end? (And if so, that might be an anti-pattern to consider addressing).

As for options that you could currently explore, you can sort your object set and then specify a smaller limit to return:

Objects.search().DriverLicences().filter(row => row.city.exactMatch(city)).orderBy(date_of_issue).take(100)

You'll find a few more details in the Functions documentation Reference entry on Ontology API: Object Sets in the section on Ordering and limiting.

You can even make a work around for the (current) lack of paging when return an ObjectSet to Slate by using the last value from the property ordered on (i.e. date_of_issue) as a filter in the subsequent request and return the next N objects.

This can work if you need a Slate table or HTML widget that renders on set of results then, on a user action, gets the next page.

Logan Rhyne
  • 581
  • 2
  • 5
  • Thanks Logan, I am following your tips and taking first N values from my object which it's working perfectly now. – houssem gharsalli Jan 31 '22 at 09:39
  • Is there a way to also provide an offset parameter for pagination? – Julio Feb 03 '22 at 14:44
  • There is currently no pagination supported in the Slate <> Functions connection when returning an Object Set. You can work around this if you have your object set ordered, by passing in the value from the last object you have in the front-end and using that to filter the function on the next call. That said, I think proper paging is on the roadmap as it's already supported in the backend. – Logan Rhyne Feb 07 '22 at 10:01