-1

I have the following data:

partial_x_train_features = [
    [b'south pago pago victor mclaglen jon hall frances farmer olympe bradna gene lockhart douglass dumbrille francis ford ben welden abner biberman pedro cordoba rudy robles bobby stone nellie duran james flavin nina campana alfred e green treasure hunt adventure adventure'],
    [b'easy virtue jessica biel ben barnes kristin scott thomas colin firth kimberley nixon katherine parkinson kris marshall christian brassington charlotte riley jim mcmanus pip torrens jeremy hooton joanna bacon maggie hickey georgie glen stephan elliott young englishman marry glamorous american brings home meet parent arrive like blast future blow entrenched british stuffiness window comedy romance'],
    [b'fragments antonin gregori derangere anouk grinberg aurelien recoing niels arestrup yann collette laure duthilleul david assaraf pascal demolon jean baptiste iera richard sammel vincent crouzet fred epaud pascal elso nicolas giraud michael abiteboul gabriel le bomin psychiatrist probe mind traumatized soldier attempt unlock secret drove gentle deeply disturbed world war veteran edge insanity drama war'],
    [b'milka film taboos milka elokuva tabuista irma huntus leena suomu matti turunen eikka lehtonen esa niemela sirkka metsasaari tauno lehtihalmes ulla tapaninen toivo tuomainen hellin auvinen salmi rauni mollberg small finnish lapland community milka innocent year old girl live mother miss dead father prays god love haymaking employ drama'],
    [b'sleeping car david naughton judie aronson kevin mccarthy jeff conaway dani minnick ernestine mercer john carl buechler gary brockette steve lundquist billy stevenson michael scott bicknell david coburn nicole hansen tiffany million robert ruth douglas curtis jason david naughton move abandon train car resurrect vicious ghost landlady dead husband mister near fatal encounter comedy horror']]

partial_x_train_plot = [[b'treasure hunt adventure'],
                        [b'young englishman marry glamorous american brings home meet parent arrive like blast future blow entrenched british stuffiness window'],
                        [b'psychiatrist probe mind traumatized soldier attempt unlock secret drove gentle deeply disturbed world war veteran edge insanity'],
                        [b'small finnish lapland community milka innocent year old girl live mother miss dead father prays god love haymaking employ'],
                        [b'jason david naughton move abandon train car resurrect vicious ghost landlady dead husband mister near fatal encounter']]

partial_x_train_actors_array = [[b'victor mclaglen', b'jon hall', b'frances farmer',
                                 b'olympe bradna', b'gene lockhart', b'douglass dumbrille',
                                 b'francis ford', b'ben welden', b'abner biberman',
                                 b'pedro de cordoba', b'rudy robles', b'bobby stone',
                                 b'nellie duran', b'james flavin', b'nina campana'],
                                [b'jessica biel', b'ben barnes', b'kristin scott thomas',
                                 b'colin firth', b'kimberley nixon', b'katherine parkinson',
                                 b'kris marshall', b'christian brassington', b'charlotte riley',
                                 b'jim mcmanus', b'pip torrens', b'jeremy hooton', b'joanna bacon',
                                 b'maggie hickey', b'georgie glen'],
                                [b'gregori derangere', b'anouk grinberg', b'aurelien recoing',
                                 b'niels arestrup', b'yann collette', b'laure duthilleul',
                                 b'david assaraf', b'pascal demolon', b'jean-baptiste iera',
                                 b'richard sammel', b'vincent crouzet', b'fred epaud',
                                 b'pascal elso', b'nicolas giraud', b'michael abiteboul'],
                                [b'irma huntus', b'leena suomu', b'matti turunen',
                                 b'eikka lehtonen', b'esa niemela', b'sirkka metsasaari',
                                 b'tauno lehtihalmes', b'ulla tapaninen', b'toivo tuomainen',
                                 b'hellin auvinen-salmi'],
                                [b'david naughton', b'judie aronson', b'kevin mccarthy',
                                 b'jeff conaway', b'dani minnick', b'ernestine mercer',
                                 b'john carl buechler', b'gary brockette', b'steve lundquist',
                                 b'billy stevenson', b'michael scott-bicknell', b'david coburn',
                                 b'nicole hansen', b'tiffany million', b'robert ruth']]

partial_x_train_reviews = [
    [b'edward small take director alfred e green cast crew uncommonly attractive brilliant assemblage south sea majority curiously undersung piece location far stylize date goldwyn hurricane admittedly riddle cliche formula package visual technical excellence scarcely matter scene stop heart chiseled adonis jon hall porcelain idol frances farmer outline profile s steam background volcano romantic closeup level defies comparison edward small film typically string frame individual work art say outdid do workhorse composer edward ward song score year prior work universal stun phantom opera'],
    [b'jessica biel probably best know virtuous good girl preacher kid mary camden heaven get tackle classic noel coward role early play easy virtue american interloper english aristocratic family unsettle family matriarch kristin scott thomas noel coward write upper class twit pretension wit keep come kind adopt way adopt oscar wilde george bernard shaw kid grow poverty way talent entertain upper class take coward heart felt modern progressive generally term social trend whittakers easy virtue kind aristocrat anybody like hang party invite noel entertain amelia earhart aviation jessica biel character auto race young widow detroit area course area motor car auto race fresh win monte carlo win young ben barnes heir whittaker estates lot land debt barnes bring biel home family mortify classless american way sense recognize class distinction thing get rid title nobility aristocrats story scott thomas dominate family try desperately estate husband colin firth serve world war horror do probably horror trench war slaughter fact class distinction tend melt combat biel kind like wife rule whittaker roost scandal past threatens disrupt barnes biel marriage form crux story turn fact end really viewer figure eventually happen second film adaption easy virtue silent film direct young alfred hitchcock easy virtue actually premier america london star great american stage actress jane cowl guess coward figure american heroine best american theatergoer british one version easy virtue direct flawlessly stephen elliot fine use period music noel coward cole porter end credit really mock upper class coward tradition play going gets tough tough going believe elliott try say class especially one right stuff course obligatory fox hunt upper class indulge oscar wilde say unspeakable uneatable chance younger generation expose noel coward worth see'],
    [b'saw night eurocine event movie european country show day european city hear le bomin barely hear derangere la chambre des officiers fortunately surprise discover great talent unknown large audience derangere absolutely astonish play character antonin verset victim post wwi trauma live trouble scene endure month war cast excellent great work cinematography offer really nice shot great landscape stun face edit really subtile bit memory make sense story minute movie show real chill ww archive action flick like sensitive psychologic movie really think absolutely recommend les fragments d antonin let le bomin'],
    [b'rauni mollberg earth sinful song favorite foreign film establish director major talent film festival circuit get amazing followup milka base work novelist timo mukka till worthy major dvd exposure unlike kaurismaki bros follow double handedly create tongue cheek deadpan finnish film style fan world mollberg commit naturalistic approach film overflow nature life lust earthiness find scandi cinema mainly work famous talent swede vilgot sjoman curious yellow fame director film tabu title imply mollberg effort quite effective sidestep fully treat screen theme incest making adult character father figure real blood relate daddy applies usual merely step father gimmick use countless time american movie incest work matti turunen kristus perkele translate christ devil really common law step dad underage milka beautiful offbeat fashion young girl portray shot irma huntus bring screen sexiness bergman harriet andersson decade earlier create international success summer monika sawdust tinsel imagine actress milka role shame do pursue act career afterward completing strong line leena suomu earth mother type confines act narrow emotional range prove solid rock crucial role bookended spectacularly beautiful shot birch wood winter virtually black white visually color presence milka film quickly develop nature theme presence strange click beak bird talisman early scene milka handyman turunen frolicking naked lake emerge oh natural sex play year old milka man result tastefully shoot intimacy imply ejaculation set trouble come religious aspect remote farm community heavily stress especially enjoy motif spiritual guidance cantor malmstrom quality anti stereotypical play eikka lehtonen instead rigid cruel turn care milka illegitimate baby bear strong romance turunen stud continue service mom woman neighborhood present utterly natural viewer position watch ethnographic exercise moralistic tale powerful technique milka frequently speak directly camera viewer forceful monologue bear crisp sound record sound nature include rain constant motif make milka engross experience view film subtitle knowledge finnish lapp recall best silent era classic direction strong convey dramatic content theme way transcend language kudos mollberg talented cinematographer job work remain obscurity ripe rediscovery'],
    [b'wonder horror film write woody allen wannabe come like check imaginatively direct typical enjoyable haunt place premise solid makeup effect good job major flaw dialogue overload cheeky wisecrack witticisms sample want scary shopping ex wife hit mark deliver inappropriate moment hero battle evil ghost']]

partial_y_train = [[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                   [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],
                   [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0],
                   [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                   [0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]]  # multilabel classification

And I want to transform them into Tensors with the tf.Dataset.from_tensor_slices() method like below:

partial_x_train_features_tensor=tf.data.Dataset.from_tensor_slices((partial_x_train_features, partial_y_train))
partial_x_train_plot_tensor=tf.data.Dataset.from_tensor_slices((partial_x_train_plot, partial_y_train))
partial_x_train_reviews_tensor=tf.data.Dataset.from_tensor_slices((partial_x_train_reviews, partial_y_train))
partial_x_train_actors_array=tf.data.Dataset.from_tensor_slices((partial_x_train_actors_array, partial_y_train))

But I get the following error:

ValueError: Can't convert non-rectangular Python sequence to Tensor

I know that actors are not equally sized arrays but searching on a couple of similar questions (i.e. question1, question2) couldn't resolve my problem.

Please also follow my colab notebook if you want to replicate the issue and please write in the comments if I missed any duplicate question.

Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
NikSp
  • 1,262
  • 2
  • 19
  • 42

2 Answers2

3

You will need to turn these strings into vectors, and pad them to equal length. I'll show you an example with just partial_x_train_actors_array:

import tensorflow as tf

partial_x_train_actors_array = [b'victor mclaglen', b'jon hall', b'frances farmer', 
                                b'olympe bradna', b'gene lockhart', b'douglass dumbrille',
                                b'francis ford', b'ben welden', b'abner biberman',
                                b'pedro de cordoba', b'rudy robles', b'bobby stone',
                                b'nellie duran', b'james flavin', b'nina campana']

tok = tf.keras.preprocessing.text.Tokenizer(char_level=True)
tok.fit_on_texts(partial_x_train_actors_array)
seq = tok.texts_to_sequences(partial_x_train_actors_array)

This is what seq looks like:

[[20, 10, 11, 16, 7, 4, 5, 12, 11, 6, 1, 17, 6, 2, 3],
 [21, 7, 3, 5, 22, 1, 6, 6],
 [14, 4, 1, 3, 11, 2, 13, 5, 14, 1, 4, 12, 2, 4],
 [7, 6, 18, 12, 19, 2, 5, 8, 4, 1, 9, 3, 1],
 [17, 2, 3, 2, 5, 6, 7, 11, 28, 22, 1, 4, 16],
 [9, 7, 15, 17, 6, 1, 13, 13, 5, 9, 15, 12, 8, 4, 10, 6, 6, 2],
 [14, 4, 1, 3, 11, 10, 13, 5, 14, 7, 4, 9],
 [8, 2, 3, 5, 29, 2, 6, 9, 2, 3],
 [1, 8, 3, 2, 4, 5, 8, 10, 8, 2, 4, 12, 1, 3],
 [19, 2, 9, 4, 7, 5, 9, 2, 5, 11, 7, 4, 9, 7, 8, 1],
 [4, 15, 9, 18, 5, 4, 7, 8, 6, 2, 13],
 [8, 7, 8, 8, 18, 5, 13, 16, 7, 3, 2],
 [3, 2, 6, 6, 10, 2, 5, 9, 15, 4, 1, 3],
 [21, 1, 12, 2, 13, 5, 14, 6, 1, 20, 10, 3],
 [3, 10, 3, 1, 5, 11, 1, 12, 19, 1, 3, 1]]

Then, pad the sequences to equal length:

padded = tf.keras.preprocessing.sequence.pad_sequences(seq)

array([[ 0,  0,  0, 20, 10, 11, 16,  7,  4,  5, 12, 11,  6,  1, 17,  6,  2,  3],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0, 21,  7,  3,  5, 22,  1,  6,  6],
       [ 0,  0,  0,  0, 14,  4,  1,  3, 11,  2, 13,  5, 14,  1,  4, 12,  2,  4],
       [ 0,  0,  0,  0,  0,  7,  6, 18, 12, 19,  2,  5,  8,  4,  1,  9,  3,  1],
       [ 0,  0,  0,  0,  0, 17,  2,  3,  2,  5,  6,  7, 11, 28, 22,  1,  4, 16],
       [ 9,  7, 15, 17,  6,  1, 13, 13,  5,  9, 15, 12,  8,  4, 10,  6,  6,  2],
       [ 0,  0,  0,  0,  0,  0, 14,  4,  1,  3, 11, 10, 13,  5, 14,  7,  4,  9],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  8,  2,  3,  5, 29,  2,  6,  9,  2,  3],
       [ 0,  0,  0,  0,  1,  8,  3,  2,  4,  5,  8, 10,  8,  2,  4, 12,  1,  3],
       [ 0,  0, 19,  2,  9,  4,  7,  5,  9,  2,  5, 11,  7,  4,  9,  7,  8,  1],
       [ 0,  0,  0,  0,  0,  0,  0,  4, 15,  9, 18,  5,  4,  7,  8,  6,  2, 13],
       [ 0,  0,  0,  0,  0,  0,  0,  8,  7,  8,  8, 18,  5, 13, 16,  7,  3,  2],
       [ 0,  0,  0,  0,  0,  0,  3,  2,  6,  6, 10,  2,  5,  9, 15,  4,  1,  3],
       [ 0,  0,  0,  0,  0,  0, 21,  1, 12,  2, 13,  5, 14,  6,  1, 20, 10,  3],
       [ 0,  0,  0,  0,  0,  0,  3, 10,  3,  1,  5, 11,  1, 12, 19,  1,  3,  1]])

And finally:

ds = tf.data.Dataset.from_tensor_slices(padded)
next(iter(ds))

<tf.Tensor: shape=(18,), dtype=int32, numpy=
array([ 0,  0,  0, 20, 10, 11, 16,  7,  4,  5, 12, 11,  6,  1, 17,  6,  2,
        3])>

If, for any reason, you need all your inputs (not just partial_x_train_actors_array) to have the same padded shape, you can use the maxlen argument.

Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • Thanks a lot for the answer. However, even though I know the approach of the Tokenizer and padded sequences, I would prefer to stick with the from_tensor_slices() approach. And I think you did that on your edit. – NikSp Jul 21 '20 at 12:50
  • what do you mean? you can combine both (see update) – Nicolas Gervais Jul 21 '20 at 12:51
  • 1
    Note that this will encode the strings on a character level. Do let me know if you want me to edit the solution so one name would be one number. – Nicolas Gervais Jul 21 '20 at 13:09
  • I have already done this but if you want add it as an edit :)...because on character level is not current for a whole name with fullname + last name – NikSp Jul 21 '20 at 13:13
  • Great! So you have everything you need? Don't hesitate to let me know if I can improve my answer in any way – Nicolas Gervais Jul 21 '20 at 13:18
  • Yeah it seems that your answer will suffice but I want to test it first because I have 4 inputs as you see. So 4 inputs as Tensors and then fit them on a neural network...Can you check my colab notebook I have attached above? It seems that I get an error when I try to fit my data – NikSp Jul 21 '20 at 13:28
  • 1
    I see your update but it would be outside of the scope of _this_ question. You're trying to pass multiple inputs to a Sequential model which is problematic. If your issue with the equal shapes and from_tensor_slices is solved, I would mark this question as solved and ask another question for the multiple input (or read the tutorials for the Functional API etc) – Nicolas Gervais Jul 21 '20 at 13:40
1

Elements of one of the data arrays (i.e. partial_x_train_actors_array) have different length along the second dimension (that's why the error complains of not having a rectangular shape). Therefore, you should either make them have the same size (e.g. by padding or truncating), or instead use the RaggedTensor structure (doc, guide) to be able to store and process it:

partial_x_train_actors_array = tf.ragged.constant(...)

This latter approach is especially useful and efficient in cases where you want to have the data as-it-is and perform custom or complex processing on it using tf.data.Dataset API (e.g. inside map method).

today
  • 32,602
  • 8
  • 95
  • 115
  • Thanks for your answer. Could you please give me an example based on the RaggedTensor and the array of actors in my example.? It would help me a lot. Thank you in advance. – NikSp Jul 21 '20 at 13:16
  • 1
    @NikSp What kind of example you are looking for? It depends on what you want to achieve with this data (and probably that's out-of-scope for this question). But anyways if you want to be able to use and process non-rectangular data in a `tf.data.Dataset` instance (without any preprocessing outside `tf.data.Dataset` API) then the `RaggedTensor` is the solution, AFIK. – today Jul 21 '20 at 13:21
  • My goal is to train a neural network saved on Tensorflow Hub, Can you please check my colab notebook attached? – NikSp Jul 21 '20 at 13:28
  • Today, please check my update to see my final goal. :) Hope it helps you. – NikSp Jul 21 '20 at 13:37
  • 1
    @NikSp Please always first read the relevant docs. The tf-hub model you are using expects "a batch of sentences in a 1-D tensor of strings as input." ([source](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1)) So it takes a string sentence as input and gives a 20-dim embedding vector. So you should preserve your training data in the format of string (even if you perform any preprocessing on it) and also adheres to input format of the model: **a batch of 1D sentences in string format**. I think you should read more about Keras model, `fit` and `Dataset` API to understand them better. – today Jul 21 '20 at 13:50
  • Thank you for the response! – NikSp Jul 21 '20 at 13:57
  • @NikSp And note that the answer provided by Nicolas, although correct for some use cases, does not work for this specific tf-hub model; that's because it's integer-encoding the strings which is not expected with this model. To emphasis again: to use that specific tf-hub model, you should provide **raw strings as input**. Good luck! – today Jul 21 '20 at 14:12
  • I understand your concern about the input of the specific TF model. But in any case the question was about transforming a list of arrays into a tensor. You are right about the end result that it cannot fit on the specific neural model but the answer of Nicolas is still valid I guess. In any case I am glad that you raised the issue. – NikSp Jul 21 '20 at 14:31