DDEV multisite setup with Acquia pull

Question

I've just gotten DDEV setup and I have multisite working by manually running ddev import-db --target-db=[db-name]. It's working just fine but I would like to figure out how to get database pulls from Acquia to work where I can specify the site to pull from.

I have this script working but is there a way to do this with DDEV commands that would be a little cleaner?

First I modified acquia.yaml to this:

environment_variables:
  project_id: mysite.dev
  uri: mysite.com
  db_name: mysite_us

#uri: mysite.ca
#db_name: mysite_canada

#uri: mysite.co.uk
#db_name: mysite_unitedkingdom

# etc etc

db_pull_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null
    acli remote:drush -n ${project_id} -- sql-dump --extra-dump=--no-tablespaces --uri=${uri}  >${db_name}.sql

Then I wrote the following script which i call like:

./ddev-refresh-db.sh mysite_us mysite.com

#!/bin/bash

site="$1"
uri="$2"

ddev pull acquia
ddev import-db --target-db=${site} --src=.ddev/.downloads/${site}.sql
ddev drush --uri=${uri} cr

However this still requires us to change the site and URI in the acquia.yaml file before running this command.

Is there a way to pass a variable through to ddev pull acquia ? And also a way to mimic what this script is doing with a real DDEV command?

Answer 1

Here's a more complete answer for Acquia multisite pull, pulling all sites. As of DDEV v1.18.0, the ddev pull itself really isn't robust enough to pull multiple sites, because it assumes one database and one set of files. This works where @kelly howard's answer in https://stackoverflow.com/a/68553116/215713 is inadequate. (In her example, she pulls just one of the multisites, and it works great for that situation.)

But here we'll put all the logic in a DDEV custom command and pull all databases and files for any named site, so ddev acquiapull <sitename>

Place this file in the project as .ddev/commands/web/acquiapull

#!/bin/bash

###
### This DDEV custom command pulls database and files from
### Acquia DEV environment in a Drupal multisite setup.
###
### Usage: `ddev acquiapull [ --skip-db ] [ --skip-files ] <site1> <site2>`
### Example: `ddev acquiapull subsite1`
###
### This assumes that each subsite has its own database (named for the site)
### and that each subsite has its own files in sites/<sitename>/files
###
### To use it, set up the needed ACQUIA_API_KEY and ACQUIA_API_SECRET
### in global or project config, just as described in
### https://ddev.readthedocs.io/en/stable/users/providers/acquia/
###

acquia_project_id=projectid.dev
tmpdir=/tmp  #inside web container

set -eu -o pipefail

function show_help() {
  sed -n 's/^###//p' ${0}
}

while :; do
  case ${1:-} in
  -h | -\? | --help)
    show_help
    exit
    ;;
  -y|--yes)
    SKIP_CONFIRMATION=true
    ;;
  --skip-files)
    SKIP_FILES=true
    ;;
  --skip-db)
    SKIP_DB=true
    ;;
  --) # End of all options.
    shift
    break
    ;;
  -?*)
    printf 'WARN: Unknown option (ignored): %s\n' "$1" >&2
    ;;
  *) # Default case: No more options, so break out of the loop.
    break ;;
  esac

  shift
done


# Map sitename to database name
function target_db_name() {
  site_name=$1
  echo $site_name
}

# Map sitename to files dir
function target_files_dir() {
  site_name=$1
  echo "sites/${site_name}/files"
}

# Get the files from upstream and load them.
function files_pull() {
  set -x   # You can enable bash debugging output by uncommenting
  set -eu -o pipefail
  site_name=$1
  files_dir=$(target_files_dir $1)
  mkdir -p ${DDEV_DOCROOT}/${files_dir}/
  echo "Using drush rsync to update files for ${site_name}..."
  drush rsync --alias-path=~/.drush -q -y -r ${DDEV_DOCROOT} --verbose @${acquia_project_id}:${files_dir}/ ${DDEV_DOCROOT}/${files_dir}/
}

# Get the db from upstream and load it
function db_pull() {
  set -x   # You can enable bash debugging output by uncommenting
  set -eu -o pipefail
  site_name=$1
  target_db=$(target_db_name ${site_name})
  rm -rf ${tmpdir}/*.sql ${tmpdir}/*.sql.gz
  echo "Downloading ${site_name} database..."
  acli pull:db -n --on-demand --no-import -- ${acquia_project_id} ${site_name}
  mv ${tmpdir}/*.sql.gz ${tmpdir}/${site_name}.sql.gz
  echo "Loading ${site_name} into database '${target_db}'..."
  mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS ${target_db}; GRANT ALL ON ${target_db}.* TO 'db'@'%'"
  gunzip ${tmpdir}/${site_name}.sql.gz
  mysql -uroot -proot ${target_db} <${tmpdir}/${site_name}.sql
  drush -r root --uri=${site_name} cr
}

# Handle initial authentication via Acquia secrets and ssh
function authenticate() {
  if [ -z "${ACQUIA_API_KEY:-}" ] || [ -z "${ACQUIA_API_SECRET:-}" ]; then echo "Please make sure you have set ACQUIA_API_KEY and ACQUIA_API_SECRET in your project or global config" && exit 1; fi
  if ! command -v drush >/dev/null; then echo "Please make sure your project contains drush, ddev composer require drush/drush" && exit 1; fi
  ssh-add -l >/dev/null || (echo "Please 'ddev auth ssh' before running this command." && exit 1)
  acli auth:login -n --key="${ACQUIA_API_KEY}" --secret="${ACQUIA_API_SECRET}"
  acli remote:aliases:download -n >/dev/null
}

# Main script
authenticate || (printf "Failed to authenticate" && exit $?)

if [ $# -eq 0 ]; then
  show_help
  exit 1
fi

if [ "${SKIP_CONFIRMATION:-}" != "true" ]; then
  echo "This will overwrite your database and files for sites $*. OK?"
  select yn in "Yes" "No"; do
      case $yn in
          Yes ) break;;
          No ) exit;;
      esac
  done
fi

for subsite in $*; do
  echo "Pulling subsite: $subsite"
  if [ "${SKIP_DB:-}" != "true" ]; then
    db_pull ${subsite} || (printf "Failed to pull db for ${subsite}" && exit $?)
  else
    echo "Skipping db pull for ${subsite}"
  fi

  if [ "${SKIP_FILES:-}" != "true" ]; then
    files_pull ${subsite} || (printf "Failed to pull files for ${subsite}" && exit $?)
  else
    echo "Skipping files pull for ${subsite}"
  fi
done

Answer 2

Thanks to the guidance from @rfay I set up a set of files in .ddev/providers for each country. Each one is structured like this:

environment_variables:
  uri: mysite.be
  db_name: belgium

auth_command:
  command: |
   <no changes>

db_pull_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null
    acli remote:drush -n ${ACQUIA_PROJECT_ID} -- sql-dump --extra-dump=--no-tablespaces --uri=${uri}  >${db_name}.sql

Then I created a custom command in .ddev/commands/host that has the contents of my script. There are more cases in the real script to cover all the countries.

#!/usr/bin/env bash

## Description: Refresh a database from Acquia and run post-db commands
## Usage: refresh-db [dbname]
## Example: "ddev refresh-db france"

site="$1"

case $site in
  canada)
    uri="mysite.ca"
    ;;
  australia)
    uri="mysite.com.au"
    ;;
  belgium)
    uri="mysite.be"
    ;;
  brazil)
    uri="mysite.com.br"
    ;;
  *)
    site="db"
    uri="mysite.com"
    ;;
esac

ddev pull ${site} -y 2>/dev/null # suppress pull failed message since it really didn't
ddev import-db --target-db=${site} --src=${DDEV_APPROOT}/.ddev/.downloads/${site}.sql
ddev drush --uri=${uri} cr
ddev drush --uri=${uri} -y pmu simplesamlphp_auth
ddev drush --uri=${uri} -y config-set system.performance css.preprocess 0
ddev drush --uri=${uri} -y config-set system.performance js.preprocess 0

I tried to handle the db import during the db_pull_command as suggested but I couldn't get past database permission errors for importing a DB that I had not already imported using ddev import-db. However with the custom command I can also incorporate the post-db-import steps that normally would only run against the default DB if done through config.yaml.

The other change I made was to move the project ID into the web environment settings in global_config.yaml file. This way if we want to change the environment we want to pull from, we just make an edit to the project ID there and don't have to edit the provider files.

I'm not experienced with contributing back to open source projects but if this can be helpful to others I'd love to work with someone to do that pull request on the documentation or wherever it belongs.

Answer 3

I'm going to go ahead and answer in general, but you can add a full answer when you get this sorted out. (I don't have access to an Acquia multisite.)

You're on the right track, but you can do all of this in the pull script. The problem you're having is that ddev just assumes a single database, and you have multiple.

Here's a strategy for your acquia.yaml:

1. Create all the databases. You can use mysql -e "CREATE DATABASE IF NOT EXISTS <dbname>;, use several lines or a for loop.
2. Pull all the databases. You can do this with separate acli lines, or use a for loop.
3. Import the databases that aren't the primary db using the mysql command. mysql <dbname> < <dbname.sql Again, this can be a few lines or a for loop. (You can also just import the primary db and it will just be re-imported by ddev, no harm done if it's not large.)

Thanks for the great question, and I hope you'll give a full answer here. Your answer could also be incorporated into https://ddev.readthedocs.io/en/stable/users/providers/acquia/ - you could do a PR there by clicking the pencil link at the upper right.