5

In my react based single page application, my page is divided in two panes.

Left Pane: Filter Panel.

Right Pane: Grid (table containing data that passes through applied filters)

In summary, I have an application that looks very similar to amazon.com. By default, when user hits an application's root endpoint (/) in the browser, I fetch last 7 days of data from the server and show it inside the grid.

Filter panel has couple of filters (e.g. time filter to fetch data that falls inside specified time interval, Ids to search data with specific id etc.) and a search button attached in the header of filter panel. Hitting search button makes a post call to a server by giving selected filters inside post form body, server returns back data that matches filters passed and my frontend application displays this data returned back from the server inside grid.

Now, when someone hits search button in the filter panel I want to reflect selected filters in the query parameter of the URL, because it will help me to share these URLs with other users of my website, so that they can see filters I applied and see data inside the grid matching these filters only.

Problem here is, if on search button click, I use http get with query parameters, I will endup breaking application because of limit imposed on URL length by different browsers.

Please suggest me correct solution to create such URLs that will help me to set the selected filters in the filter panel without causing any side effect in my application.

Possible solution: Considering the fact that we cannot directly add plain strings in query parameter because of URL length limitation from different browsers (Note: Specification does not limit the length of an HTTP Get request but different browsers implement their own limitations), we can use something like message digest or hash (convert input of arbitrary length into an output of fixed length) and save it in DB for server to understand the request and serve content back. This is just a thought, I am not sure whether this is an ideal solution to this problem.

Behavior of other heavily used websites:

  • amazon.com, newegg.com -> uses hashed urls.
  • kayak.com -> since they have very well defined keywords, they use short forms like IN for INDIA, BLR for Bangalore etc. and combine this with negation logic to further optimize maximum url length. Not checked but this will ideally break after large selection of filters.
  • flipkart.com -> appends strings directly to query parameters and breaks after limit is breached. verified this.
ThinkGeek
  • 4,749
  • 13
  • 44
  • 91
  • Have you actually experienced the problem? I believe URL length is virtually unlimited in modern browsers. – dostu Oct 08 '17 at 10:28
  • https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers – ThinkGeek Oct 08 '17 at 10:33
  • So the problem with URLs longer than 2000 characters seems to only affect IE <11 and those are not modern browsers. – dostu Oct 08 '17 at 10:39
  • you mean to say creating any length temp URLs should not be a problem? I saw amazon creating hashes and appending these hashes in the URL instead of appending the selected filter in the plain text format. – ThinkGeek Oct 08 '17 at 10:44
  • If the issue you're having is that the request to the server are throwing errors because the query string is too long, you should send the data with the body of your request instead of query parameters. – Cory Danielson Jan 04 '18 at 19:39
  • Cory please read the question properly, I want to get filters details in url so that I can share these urls with others. – ThinkGeek Jan 05 '18 at 02:05
  • can you provide your urls with filters? so that we can analyse the problem. – Kiran Shakya Jan 05 '18 at 08:54

2 Answers2

3

Lets analyse your problem and the solution possible. Problem : You want a URL which has information about the filter applied so that when you share that URL user doesn't land on arbitrary page.

Solutions:

1) Append filter applied with URL. To achieve this you will need to shorten the key of type of filter and the value of filter so that Length of URL don't exceed much for each filter.

Drawback: This is not most reliable solution as the number of filter increase URL length has to increase no other option.

2) Append a unique key of filter applied(hash) with URL. To achieve this you will need to do some changes on server and client both. On client side you will need a encoding algorithm which convert filter applied to unique hash. On server side you will need decoding algorithm which convert unique hash to filter applied. SO now client whenever a URL like this is hit you can make a POST api call which take this hash give you the array of filter applied or on client side only put the logic to convert this hash. Do all this in componentWillMount to avoid any side effect.

I think 2nd solution is scalable and efficient in almost all cases.

cauchy
  • 1,123
  • 1
  • 9
  • 19
  • I also thought of the second solution, issues that needs to be sorted 1. If we use some persistent store to save the mapping between this hash to actual filters, we would ideally want to segregate long-lived "permalinks" from short-lived ephimeral urls, and use that understanding to efficiently expire the short lived hashes. 2. It should be very obvious by looking at the url itself (eg. /temp/filter), as to which urls are short lived and which are permanent. 3. Or should we do something like gitlab? There is no promise that a link will be there unless you click permalink. – ThinkGeek Jan 06 '18 at 09:26
  • This being a very generic problem, I assume there to be lots of articles and open source solutions to this. Do you about any article/open source project describing this in detail? – ThinkGeek Jan 06 '18 at 09:39
  • Lokesh I didn't find any link related to this. For the issue you mentioned 1) No need to use persistent store for that just we can have encrypt and decrypt logic whenever required and pass the hash with some query string like xyx.com/path-to-product?f={hash}. No need to store the mapping – cauchy Jan 08 '18 at 13:04
  • I dont think there is any such encryption algo available. – ThinkGeek Jan 08 '18 at 16:40
  • @LokeshAgrawal was going to comment, but it was getting long. Left an answer to add to the discussion. – Luke Willis Jan 08 '18 at 20:20
3

In response to @cauchy's answer, we need to make a distinction between hashing and encryption.

Hashing

Hashes are by necessity irreversible. In order to map the hash to the specific filter combination, you would either need to

  1. hash each permutation of filters on the server for every request to try matching the requested hash (computationally intensive) or
  2. store a map of hash to filter combination on the server (memory intensive).

For the vast majority of cases, option 1 is going to be too slow. Depending on the number of filters and options, option B may require a sizable map, but it's still your best option.

Encryption

In this scheme, the server would send its public key to the client, then the client could use that to encrypt its filter options. The server would then decrypt the encrypted data with its private key. This is good, but your encrypted data will not be fixed length. So, as more options are selected, you run into the same problem of indeterminate parameter length.

Thus, in order to ensure your URL is short for any number of filters and options, you will need to maintain a mapping of hash->selection on the server.

How should we handle permanent vs temporary links?

You mentioned in your comment above

If we use some persistent store to save the mapping between this hash to actual filters, we would ideally want to segregate long-lived "permalinks" from short-lived ephemeral URLs, and use that understanding to efficiently expire the short-lived hashes.

You likely have a service on the server that handles all of the filters that you support in your application. The trick here is letting that service also manage the hashmap. As more filters and options are added/removed, the service will need to re-hash each permutation of filter selections.

If you need strong support for permalinks, then whenever you remove filters or options, you'll want to maintain the "expired" hashes and change their mapping to point to a reasonable alternative hash.

When do we update hashes in our DB?

There are lots of options, but I would generally prefer build time. If you're using a CI solution like Jenkins, Travis, AWS CodePipeline, etc., then you can add a build step to update your DB. Basically, you're going to...

  1. Keep a persistent record of all the existing supported filters.
  2. On build, check to see if there are any new filters. If so...
    1. Add those filters to the record from step 1.
    2. Hash all new filter permutations (just those that include your new filters) and store those in the hash DB
  3. Check to see if any filters have been removed. If so...
    1. Remove those filters from the record from step 1.
    2. Find all the hashes for permutations that include those filters and either...
      • remove those hashes from the DB (weak permalinks), or
      • Point that hash to a reasonable alternative hash in the DB (strong permalinks)
Luke Willis
  • 8,429
  • 4
  • 46
  • 79
  • Thanks Luke for beautifully explaining/summarizing this. – ThinkGeek Jan 09 '18 at 11:32
  • As I mentioned before, considering this to be a very common requirement for webapps, I am surprised that there is no article on this. – ThinkGeek Jan 09 '18 at 11:33
  • Thanks Luke for adding the explanation @lokesh try the approach.Once done let's right a medium article combinedly on this. – cauchy Jan 09 '18 at 15:04
  • Luke, one more thing here is : I don't want to end-up creating a new hash for the same string(same filters) passed. How can we ensure this in our service? When DB will grow searching and finding an existing entry will come at a cost of search operation. – ThinkGeek Jan 12 '18 at 10:31
  • @LokeshAgrawal see the added section on when to update hashes. – Luke Willis Jan 13 '18 at 01:02
  • @LukeMWillis I am not fully convinced with the approach of storing all filter permutations, instead its better to create hashes with the filters mapping on fly. – ThinkGeek Jan 14 '18 at 15:11
  • @LokeshAgrawal the number of permutations for `n` filters is `n!`. If you only support a couple filters and don't need permalinks, you can do it on the fly. But to guess which filters are requested with a given hash is O(n!) which will not scale well as you add more filter options. – Luke Willis Jan 14 '18 at 16:29
  • Basically, if you have so few filters that reverse engineering the hash is performant enough, you should just not use the hash as your URL won't be too long. – Luke Willis Jan 15 '18 at 16:43