This is not available to do this. For Changed Date field:
Change Date
The date and time when a work item was modified.
Reference name=System.ChangedDate, Data type=DateTime.
Since using data time type, it's not able to auto exclude weekends, this is by designed for now.
State Change DateThe date and time when the value of the State field changed.
DateTime
= , <> , > , < , >= , <= , =[Field], <>[Field], >[Field], <[Field], >=[Field], <=[Field], In, Not In, Was Ever Macros: @StartOfDay, @StartOfWeek, @StartOfMonth, @StartOfYear, and @Today; each of these
macros can be specified with a +/- n interger.
There has also been a related feature request:
Add option to exclude weekends when setting up Styling rules using the 'Changed Date' field rule criteria
https://developercommunity.visualstudio.com/idea/376310/add-option-to-exclude-weekends-when-setting-up-a-s.html
You could vote up it and our PM will kindly review it. Unfortunately, we do not have any perfect solution/workaround, you may have to manually change/update the colored task based on those tasks which get started on Friday.