A faster route to deploying static content in Magento 2? Dev to Live etc?

Question

This is my environment. Please note this is also set in the relevant development modes and production modes.

Dev:
https://ar.dev.loc/
https://en.dev.loc/

Live:
https://ar.site.com/
https://en.site.com/

I am using a multi store setup with Arabic and English and everything is working well including building modules and template building.

However, if I make a change to any less file or JS file (despite using grunt less or grunt watch) I have to run the following commands on my development environment everything single time to see them on my local machine.

$ rm -rf var/cache var/page_cache var/view_preprocessed pub/static
$ mkdir pub/static
$ bin/magento setup:static-content:deploy
$ bin/magento setup:static-content:deploy ar_SA
$ grunt exec less // sometimes I leave this do this
$ grunt // I swap between these

This takes a long time to do this process everytime. It is frustrating as I am fast coder and like to see CSS and Less immediately on the site and not wait around.

The quick approach what our team is doing is actually making changes in pub/static and then we ship these to less and app/design etc and then do the process above and then git.

Live server is pretty much the same. Git pull and then maintenance mode (madness on a live ecom site! Who build M2?? Then we run commands above - downtime of 45mins)

Surely there must be a quicker way for our deployment, development and team to work better and to see changes faster without downtime!

Even Magento 2 official documentation says your LIVE site needs to go into maintenance and downtime mode to publish content - this is not an option for us. The CTO are not happy. Simply absurd.

Related questions with people asking about faster development same issues:

Css changes reflect only after deploy command in magento2

Magento 2 static cache

Changes to CSS and JavaScript applies only after deploying static content

So I want to collate all the issues and resolve this.

Answer 1

yes it can be faster by following steps :

1. Change in your js/css file in app directory
2. Now Delete the file form pub/static folder

you can find the file in pub static by following command

find pub/static -iname yourjsfile.js

3. remove only that file in which you have changes from pub/static folder
4. make sure that pub/static.htaccess file & pub/static.php file is there
5. assign file write permissions to pub folder
6. now hit the URL in browser .htaccess will directly deploy missing js file

.htaccess in pub/static send the missing files to pub/static.php with parameter resource and pub/static.php file creates an StaticResource for the file if available and deploy that.

Note: this only works with apache mod_rewrite

For Nginx you need to configure the same via nginx.conf.sample

Answer 2

Since I use -j (--jobs) option to fork a new process, I reduce the static-content:deploy time from more than 10 minutes to less than one minute. See Magento/Deploy/Process/Queue.php

php bin/magento setup:static-content:deploy -j[JOBS_AMOUNT]

--jobs option enable parallel processing using the specified number of jobs. The default is 4. To cause the task to run in one process (for example, if your system does not support process forking), use --jobs 1. // see: devdocs.magento.com

This option can be used only if pcntl is enabled.

---

pcntl

No external libraries are needed to build this extension.

Process Control support in PHP is not enabled by default.

You have to compile the CGI or CLI version of PHP with --enable-pcntl configuration option when compiling PHP to enable Process Control support.

Note: Currently, this module will not function on non-Unix platforms (Windows).

Note: that pcntl_fork will not work if PHP is being run as an Apache module, in which case this function will not exist!

Answer 3

I haven't touched setup:static-content:deploy in awhile. After getting fed up with the slow deployment process in Magento 2, I devised my own that allows me to make a change in a CSS or JS file and have it reflected almost immediately on the site. Here's the magic:

cd /path/to/docroot/app/design/frontend/$VENDOR/$THEME
find * -type f \( -name "*.css" -o -name "*.js" \) -cmin -10 | xargs -I {} bash -c 'dest=/path/to/docroot/pub/static/frontend/$VENDOR/$THEME/*/$(echo {} | sed -E "s/(web\/|\/[^/]+$)//g"); echo Deploying {} to $dest ...; mkdir -p $dest && cp {} $_'
cd - 1> /dev/null

Let's break it down...

1. cd /path/to/docroot/app/design/frontend/$VENDOR/$THEME - change to the active theme directory (replace $VENDOR and $THEME with your own values)
2. find * -type f \( -name "*.css" -o -name "*.js" \) -cmin -10 - find all CSS and JS files that were changed in any way within the last 10 minutes inside the theme directory (* gets rid of the leading ./ in the results); feel free to change any of the parameters to fit your needs
3. xargs -I {} bash -c - for each changed file, execute the commands in #4-6 (the relative path to the changed file is stored in {})
4. dest=/path/to/docroot/pub/static/frontend/$VENDOR/$THEME/*/$(echo {} | sed -E "s/(web\/|\/[^/]+$)//g") - set the deployment destination path
   • the * glob takes care of matching whatever locale you’re in
   • $(...) spawns a subshell to extract only the part of the source path that we need to append to the destination path (web directory level doesn't exist under pub)
5. echo Deploying {} to $dest ... - log the activity to the console so you know which files are being deployed
6. mkdir -p $dest && cp {} $_ - create the destination directory structure; if and only if the directory structure is successfully created, copy the changed file to its deployment destination under pub ($_ points to the last argument of the previous command, mkdir, which is $dest)
7. cd - 1> /dev/null - change to the previous directory so you can continue where you left off

---

You can throw all this into an alias to reduce it to one command, but remember to escape the single quotes:

alias update='cd /path/to/docroot/app/design/frontend/$VENDOR/$THEME; find * -type f \( -name "*.css" -o -name "*.js" \) -cmin -10 | xargs -I {} bash -c '"'"'dest=/path/to/docroot/pub/static/frontend/$VENDOR/$THEME/*/$(echo {} | sed -E "s/(web\/|\/[^/]+$)//g"); echo Deploying {} to $dest ...; mkdir -p $dest && cp {} $_'"'"'; cd - 1> /dev/null'

If you use varnish, then you should add a command to restart varnish at the end of the alias (e.g., sudo service varnish restart) or you'll likely still be hitting cached static assets.

If you're too lazy to type update every time you make a change, then you can put it in a cron or integrate it into a grunt monitor task or equivalent.

Answer 4

General comment:

You must not do "grunt"ing here as you do setup:static-content:deploy. You can generate the static content for en_US (that is w/o the parameter) and ar_SA in parallel (bash allows you to do that quite easily).

If you miss HTTPS (secure) resources (maybe that was the reason you used grunt additionally?), ensure the environment variable "HTTPS" is set (e.g. HTTPS=ON) for the process that compiles static content (ref).

---

Apart from that, yes it takes time. What has come to attention is that the PHP less compilation takes quite some time. One idea that came to my mind when I heard about that from another developer was to cache less compilation in a local less cache and only re-compile if a file actually changes. Maybe you can try that as well.

I'm not yet responsible with hacking a PoC together for it but think it should be a good test for the claim that the less compilation is the bottleneck.

I also can heavily suggest to run a build server that compiles on git push automatically to lower the burden.

Answer 5

In our company we manage Magento2 deploys by using Capistrano, there is also a specific taks set for Magento 2 that works very well.

With capistrano you configure your webserver root to point to a symlink that points to a "release" folder. When you start a deploy all the compiling (di, static contents, ecc) is made on a separated folder so your online website isn't affected and remains online. At the end of the procedure, and only if there are no errors, the symlink is switched to the new release.

In this way there downtime is usually zero or reduced to a couple of seconds.

Capistrano also provides a basic "rollback" feature that allows to quickly revert to an old release if something goes wrong.

Answer 6

I am not familiar with magneto but this may work for you. When I update my sites I follow those steps:

Suppose the site is located under site directory.

$ cp -r site site-update

# update the site in site-update directory

$ mv site site-old && mv site-update site

This way my sites are updated without any downtime.

Answer 7

I have also noticed that if you have css and js signing on, it seems to be get wacked out if you run setup upgrade, it then breaks the version and it waks out the css/js until you do a rebuild the static content.

I have no idea how they think this should work for people in production. Just about anything can get it out of wack and you have to rebuild.

I have had a bit of luck with 1. turn css/js signing on 2. then you can run static content deploy, seems to keep the ur pretty intack 3. flush cache (then adds the new version12312312 to the static urls)

But seriously, no way to be rock solid with deployments.

TheBlackBenzKid very curios where you ended up a year later, did you dump magento?

Answer 8

I have come with a solution for this problem.

You have to publish the content of the store you want by

php bin/magento setup:static-content:deploy [lang(en_US)] -t [vendor]/[theme]

I won't change because magento cache, so just delete it manually

rm -rf pub/static/_*/*

php bin/magento cache:flush; php bin/magento cache:clean;

The refresh your page and done now it takes new changes when it regenerates de page.

This steps are mandatory for changes in phtml and static content, layout changes require cache flush, and changes on controllers are a pain in the arsh because you need to use di:compile this one really make you leave from production.