1

According to Can't install due to dependency on numpy #289, TenforFlow Transform (tft) supports Python 3.9 and there is no limitation for Mac OS on Apple silicon stated in TensorFlow Transform github.

However, it is not working for me, as you can see from the below log:

$ OPENBLAS="$(brew --prefix openblas)" pip install tensorflow_transform
...
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.12.0-py3-none-any.whl (439 kB)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /Users/takoika/venv/ml/lib/python3.9/site-packages (from tensorflow_transform) (1.3.0)
Collecting apache-beam[gcp]<3,>=2.41
  Using cached apache-beam-2.45.0.zip (3.0 MB)
  Preparing metadata (setup.py) ... done
Collecting tensorflow-metadata<1.13.0,>=1.12.0
  Using cached tensorflow_metadata-1.12.0-py3-none-any.whl (52 kB)
Collecting pyarrow<7,>=6
  Using cached pyarrow-6.0.1-cp39-cp39-macosx_11_0_arm64.whl (13.8 MB)
Requirement already satisfied: protobuf<4,>=3.13 in /Users/takoika/venv/ml/lib/python3.9/site-packages (from tensorflow_transform) (3.19.6)
Requirement already satisfied: pydot<2,>=1.2 in /Users/takoika/venv/ml/lib/python3.9/site-packages (from tensorflow_transform) (1.4.2)
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.11.0-py3-none-any.whl (446 kB)
  Using cached tensorflow_transform-1.10.1-py3-none-any.whl (439 kB)
  Using cached tensorflow_transform-1.10.0-py3-none-any.whl (439 kB)
  Using cached tensorflow_transform-1.9.0-py3-none-any.whl (436 kB)
  Using cached tensorflow_transform-1.8.0-py3-none-any.whl (435 kB)
Collecting pyarrow<6,>=1
  Using cached pyarrow-5.0.0-cp39-cp39-macosx_11_0_arm64.whl (12.3 MB)
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.7.0-py3-none-any.whl (433 kB)
Collecting tensorflow-metadata<1.8.0,>=1.7.0
  Using cached tensorflow_metadata-1.7.0-py3-none-any.whl (48 kB)
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.6.1-py3-none-any.whl (427 kB)
  Using cached tensorflow_transform-1.6.0-py3-none-any.whl (427 kB)
Collecting tensorflow-metadata<1.7.0,>=1.6.0
  Using cached tensorflow_metadata-1.6.0-py3-none-any.whl (48 kB)
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.5.0-py3-none-any.whl (422 kB)
Collecting absl-py<0.13,>=0.9
  Using cached absl_py-0.12.0-py3-none-any.whl (129 kB)
Collecting numpy<1.20,>=1.16
  Using cached numpy-1.19.5.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting tensorflow_transform
  Using cached tensorflow_transform-1.4.1-py3-none-any.whl (413 kB)
  Using cached tensorflow_transform-1.4.0-py3-none-any.whl (413 kB)
  Using cached tensorflow_transform-1.3.0-py3-none-any.whl (407 kB)
  Using cached tensorflow_transform-1.2.0-py3-none-any.whl (406 kB)
  Using cached tensorflow_transform-1.1.1-py3-none-any.whl (401 kB)
  Downloading tensorflow_transform-1.1.0-py3-none-any.whl (401 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.2/401.2 kB 6.9 MB/s eta 0:00:00
  Downloading tensorflow_transform-1.0.0-py3-none-any.whl (402 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 402.9/402.9 kB 11.2 MB/s eta 0:00:00
Requirement already satisfied: six<2,>=1.12 in /Users/takoika/venv/ml/lib/python3.9/site-packages (from tensorflow_transform) (1.16.0)
  Downloading tensorflow_transform-0.30.0-py3-none-any.whl (398 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 398.7/398.7 kB 7.2 MB/s eta 0:00:00
Collecting tensorflow-metadata<0.31,>=0.30
  Downloading tensorflow_metadata-0.30.0-py3-none-any.whl (47 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.8/47.8 kB 3.6 MB/s eta 0:00:00
Collecting pyarrow<3,>=1
  Using cached pyarrow-2.0.0.tar.gz (58.9 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [6536 lines of output]
      Ignoring numpy: markers 'python_version < "3.7"' don't match your environment
      Collecting cython>=0.29
        Using cached Cython-0.29.33-py2.py3-none-any.whl (987 kB)
      Collecting numpy==1.16.0
        Using cached numpy-1.16.0.zip (5.1 MB)
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting setuptools
        Using cached setuptools-67.4.0-py3-none-any.whl (1.1 MB)
      Collecting setuptools_scm
        Using cached setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
      Collecting wheel
        Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
      Collecting typing-extensions
        Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
      Collecting packaging>=20.0
        Using cached packaging-23.0-py3-none-any.whl (42 kB)
      Collecting tomli>=1.0.0
        Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
      Building wheels for collected packages: numpy
        Building wheel for numpy (setup.py): started
        Building wheel for numpy (setup.py): finished with status 'error'
        error: subprocess-exited-with-error
      
        × python setup.py bdist_wheel did not run successfully.
        │ exit code: 1
        ╰─> [6114 lines of output]
            Running from numpy source directory.
            /private/var/folders/_4/8v285hqs45xfzk0l1nlr3yq40000gn/T/pip-install-mqqkj7nw/numpy_fb4228ef9b4f4de5a81a9e9f13289377/numpy/distutils/misc_util.py:476: SyntaxWarning: "is" with a literal. Did you mean "=="?

...
error: Command "gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/takoika/venv/ml/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/common -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/npymath -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/common -Ibuild/src.macosx-10.9-universal2-3.9/numpy/core/src/npymath -c numpy/random/mtrand/mtrand.c -o build/temp.macosx-10.9-universal2-3.9/numpy/random/mtrand/mtrand.o -MMD -MF build/temp.macosx-10.9-universal2-3.9/numpy/random/mtrand/mtrand.o.d" failed with exit status 1
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: legacy-install-failure
      
      × Encountered error while trying to install package.
      ╰─> numpy
      
      note: This is an issue with the package mentioned above, not pip.
      hint: See above for output from the failure.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

My environment is the following:

Mac M2 Ventura 13.0.1
Python 3.9.13
numpy 1.24.1
TensorFlow version: 2.10.0
tensorflow-metadata  1.11.0

What could be causing this problem? Is Tensorflow Transform completely unsupported on Apple silicon? Are there any workarounds for this problem?

Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
mon
  • 18,789
  • 22
  • 112
  • 205

1 Answers1

1

Conclusion

TensorFlow Transform does not work on Apple Silicon as of now. See Update section at the bottom.

TensorFlow Transform requires TFX Basic Shared Libraries(tfx-bsl) as listed in TensorFlow Transform.

tensorflow-transform apache-beam[gcp] pyarrow tensorflow tensorflow-metadata tfx-bsl
GitHub master 2.41.0 6.0.0 nightly (2.x) 1.12.0 1.12.0
1.12.0 2.41.0 6.0.0 2.11 1.12.0 1.12.0

Apparently tfx-bsl does not support Apple Silicon yet as Any plans to support Apple silicon devices like M1 Macs? #48 is still open and people are asking for Apple Silicon support.

Because tfx-bsl is the basis for TFX, this will be a generic issue of TFX available for Apple Silicon platform.

TFX Basic Shared Libraries (tfx_bsl) contains libraries shared by many TensorFlow eXtended (TFX) components.

Findings

Certain installation below appeared to work but it installed version 0.12.0 which only works with Tensorflow 1.x (1.14) because it requires tensorflow.contrib that has been removed in 2.x.

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 1
----> 1 import tensorflow_transform as tft

File ~/venv/tf/lib/python3.9/site-packages/tensorflow_transform/__init__.py:19
     15 # GOOGLE-INITIALIZATION
     16 
     17 # pylint: disable=wildcard-import
     18 from tensorflow_transform import coders
---> 19 from tensorflow_transform.analyzers import *
     20 from tensorflow_transform.api import apply_function
     21 from tensorflow_transform.inspect_preprocessing_fn import *

File ~/venv/tf/lib/python3.9/site-packages/tensorflow_transform/analyzers.py:44
     40 from tensorflow_transform import nodes
     41 from tensorflow_transform import tf_utils
---> 44 from tensorflow.contrib.boosted_trees.python.ops import gen_quantile_ops
     45 from tensorflow.contrib.boosted_trees.python.ops import quantile_ops
     46 from tensorflow.python.ops import resources

ModuleNotFoundError: No module named 'tensorflow.contrib'

Installation attempted

Not conclusive but apparently numpy, pyarrow==6.0.0, and tensorflow-transform need to be installed with OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 based on the test below.

Does work

It works, however it installs TensorFlow Transform 0.12 that does not work with TensorFlow 2.x, hence not worthwhile.

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 python3 -m pip install --no-cache-dir \
  setuptools \
  wheel \
  Cython \
  pyarrow==6.0.0 \
  numpy \
  tensorflow-transform

Do not work

pip install  \
  setuptools \
  wheel \
  Cython \
  pyarrow==6.0.0

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 python3 -m pip install --no-cache-dir \
  tensorflow-transform

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 python3 -m pip install --no-cache-dir \
  setuptools \
  wheel \
  Cython \
  numpy \
  tensorflow-transform

pip install  \
  setuptools \
  wheel \
  Cython \
  numpy \
  pyarrow==6.0.0

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 python3 -m pip install --no-cache-dir \
  tensorflow-transform

OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=13.0.1 python3 -m pip install --no-cache-dir \
  setuptools \
  wheel \
  Cython \
  pyarrow \
  numpy \
  tensorflow-transform

References

Update

Tensorflow Transform team confirmed Apple Silicon is not supported yet in TFX and its modules and there is no ETA as of now.

Apple Silicon support for tensorflow-transform not available as tfx-bsl does not support it? #298 As far as I know TFX and it's supported libraries like tensorflow-transform does not support for Apple Mac M1 and M2 at the moment and We are currently working on Apple Mac M1, M2 support, and will have an update in the fairly near future. In the meantime, Other option include using a VM. We understand that option is not ideal.

We don’t have a firm ETA yet. As similar issues are already being tracked #1864 , #48 and TF Forum thread so I would request you to please follow the same issues or TF-Forum thread to get latest updates on this issue and close this issue. Thank You!

See TensorFlow Forum where Google answers on the Apple silicon topic.

TensorFlow Forum - want to use the tfx 1.6.1 on Mac(M1 Max)

mon
  • 18,789
  • 22
  • 112
  • 205