1

I am trying to use some Rust wasm code in Bigquery as UDF, and in order to pass on Java String to Rust code the TextEncoder and TextDecoder would be needed to conveniently doing that. As it mentioned here Passing a JavaScript string to a Rust function compiled to WebAssembly

But when I try out some of my code on BigQuery, I encountered an error saying TextEncoder is not defined.

enter image description here

You can try it out as well with a query like this: https://github.com/liufuyang/rb62-wasm/blob/master/try-3-old.sql

While a working version without using TextEncoder is at https://github.com/liufuyang/rb62-wasm/blob/master/try-3.sql

Fuyang Liu
  • 1,496
  • 13
  • 26

1 Answers1

2

That means the object is not defined.

As an option, bring your own TextEncoder.

For example, take your try-3-old.sql, and then add this line at the end of the JS UDF definition:

return main();
'''
OPTIONS (library="gs://fh-bigquery/js/inexorabletash.encoding.js");

And now it works:

enter image description here

(wondering, what's the goal with rb62?)

Felipe Hoffa
  • 54,922
  • 16
  • 151
  • 325
  • Thank you, I think this is exactly what I need. – Fuyang Liu Feb 06 '20 at 21:24
  • About the goal with rb62: basically it is used internally at Spotify to encode/decode some gid (128bit or 16byte) to 62 based value, as in those examples show. We already have a js UDF library to do this. I am just testing out 1. whether it is possible to use some Rust/wasm function to do the same via js UDF interface. 2. Could the speed be improved or not as we normally have millions of rows to parse. – Fuyang Liu Feb 06 '20 at 21:28
  • What I find so far is that, as it is not convenient to pass JS string (utf16) values to the wasm Rust code environment (where the string is in utf8, also this seems can only be done via copy stuff through the wasm memory), it seems not gaining any performance benefit, specificly when trying to replace some JS string encoding lib to a Rust string encoding lib? – Fuyang Liu Feb 06 '20 at 21:32
  • Please accept answer if useful! And let's start a new question to find a way to improve performance as desired :) – Felipe Hoffa Feb 06 '20 at 21:43
  • Thank you. I will try out some stuff then ask if necessary :) – Fuyang Liu Feb 07 '20 at 08:33
  • Cool! I think the problem with performance in this case is the initialization of wasm, but I have some ideas on how to improve that problem. – Felipe Hoffa Feb 07 '20 at 15:06
  • I have tired the idea of grouping stuff to avoid too much wasm init as you showed in some other answers and blog post. But it still runs slower than a pure JS function. You can see a query example here https://github.com/liufuyang/rb62-wasm/blob/master/try4.sql I was told by Pauna that :"because even though Rust strings are faster than JS strings, it has to do a full copy when converting from a JS string to a Rust string or a Rust string to a JS string so that negates the performance benefits" I think that is probably the case for me. – Fuyang Liu Feb 07 '20 at 16:49