Decompiled EPiServer source code and found that when forceBinFolderScan attribute is set to true makes EPiServer framework to scan each dll in bin folder. It iterates through all dlls and loads them into AppDomain.
If this attribute is set to false, it doesn't scan bin folder, but relies on ASP.NET assembly loading.
After assemblies are loaded in AppDomain from bin (or not loaded if attribute is false) it gets all assemblies from AppDomain and then filters them based on add/remove configuration. So add/remove tags do not affect bin folder scanning.
Actually this behavior and reason for such behavior is described in EPiServer documentation: MEF - The Discovery Mechanism (see note at the bottom of paragraph). Author just didn't mentioned that it is controlled by forceBinFolderScan attribute.
UPDATE
Here is an example which describes that behavior.
Assume that we have two assemblies: MyPlugins.dll - which contains some plugins and NoPlugins.dll. Also assume that those DLLs are not loaded by ASP.NET at application start.
We have this configuration which tells not to force scan bin folder and include all assemblies except NoPlugins.dll:
<scanAssembly forceBinFolderScan="false">
<add assembly="*" />
<remove assembly="NoPlugins.dll" />
</scanAssembly>
With such configuration MyPlugins.dll and NoPlugins.dll is not loaded in AppDomain and can't be scanned. So no plugins will be available.
If we set forceBinFolderScan="true":
<scanAssembly forceBinFolderScan="true">
<add assembly="*" />
<remove assembly="NoPlugins.dll" />
</scanAssembly>
With such configuration both DLLs are explicitly loaded from bin folder into AppDomain. Then filtering occurs and NoPlugins.dll is removed from module/plugin scanning collection. As a result plugins from MyPlugins.dll will be loaded.
UPDATE 2
Tested different configurations to see if optimizations help, but seems that filtering assemblies gives even worse results.
My site has quite a lot of assemblies, so it takes a lot of time to start up. I tried 3 different scenarios.
1:
<scanAssembly forceBinFolderScan="false">
<add assembly="*" />
<!-- Remove assemblies generated by EPiOptimizer -->
<remove assembly="..." />
</scanAssembly>
2:
<scanAssembly forceBinFolderScan="false">
<add assembly="*" />
</scanAssembly>
3:
<scanAssembly forceBinFolderScan="true">
<add assembly="*" />
</scanAssembly>
I performed simple tests 3 times each - did iisreset and in Chrome refreshed page and checked timeline (I know it is not perfect test) and here are results:
1: 1.3 min, 1 min, 1.3 min
2: 1 min, 57 s, 1 min
3: 57 s, 57 s, 57 s
I tried these tests else several times in different order, but results were same. So it is not reasonable to make optimizations by adding tag. Seems that it is more expensive to filter out assemblies which are loaded in AppDomain then to scan all of them. The strange thing is that forcing bin folder scan gives better results, but probably it is my test issue.