Stack Overflow
Stack Overflow

Questions tagged [bigdl]

BigDL is a distributed deep learning library for Apache Spark. Please add the tag for the language you are using; BigDL supports both scala and python.

BigDL is a distributed deep learning library for Apache Spark. With BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

More information can be found at the BigDL project website: https://bigdl-project.github.io/

13 questions
3
votes
1 answer

How to get the SparkSession to find added python files

After running pip install BigDL==0.8.0, running from bigdl.util.common import * from python completed without issue. However, with either of the following SparkSessions: spark = (SparkSession.builder.master('yarn') .appName('test') …
Clay
  • 2,584
  • 1
  • 28
  • 63
1
vote
0 answers

Loss of TransE gets plateau at the margin value with BigDL library

I have been trying to use optimizer(SGD, Adagrad) from BigDL library on TransE with scala. My current implementation works with mini batch in sequential way. I followed this example to optimize the embeddings(as Tensors) without creating a layered…
TTR
  • 11
  • 1
1
vote
0 answers

BigDL: get hidden layer's output from auto-encoder model

After training an autoencoder with one hidden layer using BigDL and Spark, I want to extract the hidden layer output with the "encoded" hidden layer of size 128. How can I do it?
Mario Raw
  • 41
  • 6
1
vote
0 answers

convert pickle (.pck) file into spark data frame using python

Hello! Dear members i want to train model using Bigdl, i have data set of Medical images in the form of pickle object files (,pck).that pickle file is a 3D image(3D array) i have tried to convert this into spark datafram by using BigDl python API …
Hassan Shabbir
  • 11
  • 1
1
vote
1 answer

AssertionError: Multiple .dist-info directories on Data Science Experience

In a Python 3.5 notebook, backed by an Apache Spark service, I had installed BigDL 0.2 using pip. When removing that installation and trying to install version 0.3 of BigDL, I get this error: (linebreaks added for readability) AssertionError:…
Roland Weber
  • 1,865
  • 2
  • 17
  • 27
1
vote
1 answer

mean() got an unexpected keyword argument 'dtype'!

I am trying to implement image classification using Intel Bigdl. It is using mnist dataset for classification. Since, I don't want to use the mnist dataset I wrote the alternative approach to it as below: Image Utils.py from StringIO import…
programmingtech
  • 438
  • 1
  • 8
  • 21
0
votes
0 answers

Exception: Java gateway process exited before sending its port number

I created a docken container from this bigdl image. However wheb I tried to start nn_context: sc = init_nncontext("...") spark= SparkContext(sc) I got this error: Exception: Java gateway process exited before sending its port…
user
  • 33
  • 5
0
votes
1 answer

BigDL docker container error: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe

I created a docker container from this BigDL image. when I tried to collect the predictions using collect() this error occurs: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. PS: the java…
user
  • 33
  • 5
0
votes
1 answer

Error "java.lang.UnsatisfiedLinkError:__strncpy_chk" when running Spark on kubernetes

I am trying to run DL job on Spark on kubernetes in cluster mode, and getting error java.lang.UnsatisfiedLinkError:__strncpy_chk: symbol not found. This is the complete error stack. Exception in thread "main" java.lang.UnsatisfiedLinkError: …
Shailesh Singh
  • 56
  • 2
0
votes
1 answer

BigDL : Saving a model throws Exception: "This is a deprecated method. Please use saveModel instead."

I am using BigDL + Analytics-Zoo for doing Anomaly detection on time-series data. When I try to save the model using the .save() method after training, I am getting the Exception: "This is a deprecated method. Please use saveModel instead." …
Remis Haroon - رامز
  • 3,304
  • 4
  • 34
  • 62
0
votes
2 answers

Simple PySpark BigDL test: Optimizer fails

Running BigDL example at: https://bigdl-project.github.io/0.4.0/#ProgrammingGuide/optimization/ in PySpark local node: from bigdl.nn.layer import Linear from bigdl.util.common import * from bigdl.nn.criterion import MSECriterion from…
dokondr
  • 3,389
  • 12
  • 38
  • 62
0
votes
1 answer

Running BigDL Text Classifier fails

When I run BigDL (https://bigdl-project.github.io/0.4.0/) Text Classifier example (https://github.com/intel-analytics/BigDL/tree/master/pyspark/bigdl/models/textclassifier) with single node PySpark I get the following error. Any ideas how to solve…
dokondr
  • 3,389
  • 12
  • 38
  • 62
0
votes
2 answers

Install BigDL in Data Science Experience on Cloud

I would like to use Intel BigDL in notebooks on Data Science Experience on Cloud. How can I install it?
Roland Weber
  • 1,865
  • 2
  • 17
  • 27