Build specific solution files from the repo

Question

I have a git repository in Azure repo and have multiple ASP.net solutions in different folders. I want to setup a build pipeline where it only builds specific solution and ignores the rest. e.g. in the picture i am trying to build solution file only from Folder1 and Folder4 and not from Folder2 and Folder3.

Here is how the yaml pipeline looks like:

trigger:
  branches:
    include:
      - main
      - feature/*       

variables:
      solution: '**/*.sln'
      buildPlatform: 'Any CPU'
      buildConfiguration: 'Release'
      artifactName: 'drop'
      vmImageName: 'windows-latest'

stages:
- stage: Build
  jobs:  
  - job: Build
    pool:
      name: Azure Pipelines
      vmImage: $(vmImageName)
      demands:
      - msbuild
      - visualstudio
      - vstest    

    steps:
    - task: NuGetToolInstaller@1
      displayName: Use NuGet      

    - task: NuGetCommand@2
      displayName: NuGet restore
      inputs:
        restoreSolution: $(solution)          

    - task: VSBuild@1
      displayName: Build solution
      inputs:
        solution: $(solution)          
        msbuildArgs: /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"
        platform: $(buildPlatform)
        configuration: $(buildConfiguration)    

    - task: VSTest@2
      displayName: Test Assemblies
      inputs:
        testAssemblyVer2: |
          **\$(buildConfiguration)\*test*.dll
          !**\obj\**
        platform: $(buildPlatform)
        configuration: $(buildConfiguration)

    - task: PublishBuildArtifacts@1

I tried to skip those folders like this during build but the build pipeline errors out with this error message: ##[error]Exception calling "GetDirectoryName" with "1" argument(s): "The path is not of a legal form."

- task: VSBuild@1
      displayName: Build solution
      inputs:
        solution: |
          $(solution)
          !**\Folder2\**
          !**\Folder3\**
        msbuildArgs: /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"
        platform: $(buildPlatform)
        configuration: $(buildConfiguration)

Any suggestion if this can be done, other than creating pipeline for each solution for moving each solution to its own git repo?