1

I am trying to monkey patch a Python package before using conda pack to package up all of the packages for deployment.

The script sets up conda:

conda install -y --channel conda-forge conda-pack
conda create -y --name venv python=3.7
conda install -y --name venv --file requirements.txt

Then it monkey patches the library:

sed --in-place \
  's/CFUNCTYPE(c_int)(lambda: None)/# CCCFUNCTYPE(c_int)(lambda: None)/g' \
  /opt/conda/envs/venv/lib/python3.7/ctypes/__init__.py

Then it packages everything up for deployment:

conda pack --name venv --output "$BUILD_DIR/runtime.tar.gz"

So the weird thing is that when I copy the file directly into the build folder:

cp /opt/conda/envs/venv/lib/python3.7/ctypes/__init__.py "$BUILD_DIR"

The monkey-patched file is there.

However, when I extract $BUILD_DIR/runtime.tar.gz, the file is in its original form.

The other weird behavior is that when I manually run these steps, the monkey patched file is in $BUILD_DIR/runtime.tar.gz.

There's quite a bit of containers around, so I thought that maybe conda is using some catched tarball instead so I tried to add this into the script:

conda clean --tarballs

But that still didn't fix the problem.

I also tried to use conda pack's explicit path option, but it didn't work either:

conda pack --prefix /opt/conda/envs/venv --output "$BUILD_DIR/runtime.explicit.tar.gz"

Does conda pack pull the files from another location aside from: /opt/conda/envs/venv/lib/python3.7/site-packages

This doesn't explain why doing things manually would work, but maybe it'll point me to a new rock to look under.

Thank you for your time

Here is the entire script:

#!/usr/bin/env bash
#
# Bundle this project into a Rapid Deployment Archive (RDA)

set -ex

###########################
# Pre-reqs and pre-checks #
###########################

if [ ! -x "$(command -v conda)" ]; then
    echo "conda is required to run this script" >&2
    exit 1
fi

if [ -n "$CI_PROJECT_DIR" ]; then
    DIR="$CI_PROJECT_DIR"
else
    DIR="$(cd "$(dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd)/.."
fi

DIST_DIR="$DIR/dist"
rm -rvf "$DIST_DIR"/*.zip
BUILD_DIR="$(mktemp -d)"

#####################################
# Build client Angular UI component #
#####################################

CLIENT_DIR="$DIR/client"
mkdir -pv "$BUILD_DIR/client"
pushd "$CLIENT_DIR" || exit 1
npm install
npm run build:default
cp -R "$CLIENT_DIR/dist/template-angular-ts-master/"* "$BUILD_DIR/client"
popd

#######################################
# Build server Python/flask component #
#######################################

# Packages are installed to /opt/conda/envs/$VENV_NAME/lib/python$PYTHON_VERSION/site-packages
if [ -n "$SSL_NO_VERIFY" ]; then
    conda config --set ssl_verify false
fi
conda install -y --channel conda-forge conda-pack
conda create -y --name venv python=3.7
# Conda does not support -r spec or --file within a file
cp requirements/prod.txt requirements/prod.txt.noflag
sed -i '/^-r/d' requirements/prod.txt.noflag
conda install -y --name venv --file requirements/_base.txt --file requirements/prod.txt.noflag
sed --in-place \
  's/CFUNCTYPE(c_int)(lambda: None)/# CCCFUNCTYPE(c_int)(lambda: None)/g' \
  /opt/conda/envs/venv/lib/python3.7/ctypes/__init__.py
rm -f requirements/prod.txt.noflag
conda clean --tarballs
conda pack --name venv --output "$BUILD_DIR/runtime.tar.gz"

# Junk within pyapp that might be present if not building in CI
if [ -z "$CI_PROJECT_DIR" ]; then
    find "$DIR" -name '*.pyc' -type f -delete
    find "$DIR" -name '.DS_Store' -type f -delete
    find "$DIR" -name '__pycache__' -type d -delete
fi

# Copy this project's stuff into build dir
cp -v -R "$DIR/config" "$DIR/pyapp" "$BUILD_DIR"
cp -v "$DIR"/rda/* "$BUILD_DIR"
cp -v setup.{py,cfg} pyproject.toml "$BUILD_DIR"
cp -v "$DIR"/scripts/{start-server.sh,wsgi.py} "$BUILD_DIR"

cp -v /opt/conda/envs/venv/lib/python3.7/ctypes/__init__.py "$BUILD_DIR"

# Try to extract the version and appKey if we have jq
if [ -x "$(command -v jq)" ]; then
    VERSION="-$(jq -j '.version' rda/rda.manifest)"
    appKey="$(jq -j .appKey rda/rda.manifest)"
else
    VERSION=''
    appKey="$(grep --color=never -oP 'appKey":\s+"\K[^"]+' rda/rda.manifest)"
fi
if [ -z "$appKey" ]; then
    appKey="my.webapp.ng-py"
fi

# Bundle into RDA ZIP
mkdir -pv "$DIST_DIR"
pushd "$BUILD_DIR"
zip -q -9 -r "$DIST_DIR/${appKey}${VERSION}.rda.zip" *
popd
rm -rf "$BUILD_DIR"
ls -1 -l -F "$DIST_DIR"/*.zip

conda clean -afy
Zhao Li
  • 4,936
  • 8
  • 33
  • 51
  • 1
    If I recall correctly from last time I looked into it, I thought I had found that it uses the package cache (e.g., `/opt/conda/pkgs`) to source the builds. If the file you're patching isn't hardlinked to there (many are, but not all), then it would still be pulling in an unpatched version. However, if you edit the cached package files, you risk triggering a checksum mismatch, i.e., Conda will flag the package as corrupted. Anyway, this is all off-the-cuff - I could be misremembering or conda-pack may have changed its strategy. – merv Nov 24 '21 at 06:05
  • Thanks @merv that sounds like a reasonable explanation to what I'm experiencing. If you write up your comment as an answer, I'll accept it. – Zhao Li Nov 24 '21 at 06:44

1 Answers1

0

I wasn't able to get the monkey patching to work, but I was able to figure out that ctypes is not part of numpy and rather is part of Python's standard library. So conda pack could very well treat Python standard libraries a bit differently.

So I gave up on monkey patching and found out that upgrading my Python version fixed the underlying issue.

Thanks

Zhao Li
  • 4,936
  • 8
  • 33
  • 51