31

After publishing a .Net Core RC1 application, commands specified in the project.json had corresponding .cmd files created for them which could be executed after deployment (e.g. web.cmd and ef.cmd). In my case, I want to run the following Entity Framework command on my deployment target:

dotnet ef database update -c MyContext

This works fine when I run this from the folder containing the source code, however after publishing, it doesn't appear to find the command within the compiled DLLs. My understanding of the change in commands with RC2 is that 'tools' can be compiled as standalone applications named dotnet-*.dll and can be executed via the CLI. How can the Entity Framework Core tools be exposed as executable DLLs in the published output?

FYI, my build/deployment workflow is as follows:

TeamCity

dotnet restore => dotnet build => dotnet test => dotnet publish

Octopus Deploy

Upload Package => EF Update Database => etc

TallMcPaul
  • 586
  • 1
  • 9
  • 18
  • 1
    No. Why? You only need the tools to create the migration files and manually migrate it. Just call `await _context.Database.MigrateAsync();` somewhere during update. – Tseng Jun 01 '16 at 07:48
  • 2
    @Tseng, Do you mean during startup? I'd like to complete schema changes prior to that if possible. I guess an alternative would be a small standalone console app which I can call during deployment to make the changes prior to taking the app online – TallMcPaul Jun 01 '16 at 08:04
  • 1
    Why would you want do it? Doing it during the startup is perfectly fine. If the migration fails, all changes will be rolled back anyways. Cause if you change it prior to deploying, your current app fails to work in the timeframe before the deployment completes. Also, there is no way back anymore once the schema is updated. If you migrate it in-app, then it may fail after deployment but you can just redeploy the old application and continue using it, while upgrading before deployment makes this step pretty much impossible – Tseng Jun 01 '16 at 08:32
  • 1
    Fair enough, I can definitely see advantages to migrating at runtime. I also had concerns about allowing my runtime DB user account to make schema changes, but I'm already using a separate context for migrations only which can be set up with a more powerful conn string. Thanks! – TallMcPaul Jun 01 '16 at 08:44
  • 1
    Has anyone found a solution to this yet? – Piguy Nov 23 '16 at 14:27
  • 1
    The problem with migrating at runtime in my case is... the user the exe runs under may just have read/write privieges on the database server and not have DB create privileges. – andrew pate Jan 07 '20 at 11:46
  • A workround may be to create a related a small console app project (executed during deployment) which simply calls _context.Database.MigrateAsync(). Just spotted this solution is already suggested by Major – andrew pate Jan 07 '20 at 11:58

6 Answers6

34

Unfortunately EF Core migration s*cks, a lot... I have seen tons of solutions for this but lets do a list of them. So here is what you can do to run and deploy EF migrations without Visual Studio. None of the below is perfect solution all have some caveats:

  1. Use EF Core Tools here is a link to the official MS site which explains how to install and use it.
  • Pros: MS official tool. Supported in all version of .NET Core.
  • Cons: It seems like the successor of EF6 "Migrate.exe". But it is not! Currently it is not possible to use this tool without the actual source code (.csproj). Which is not really a good fit for Live/Prod deployments. Usually you don't have C# projects on your DB Server.
  1. dotnet exec I have tried to make sense of the huge amount of poorly documented parameters. And failed to run migrations until found this script. The name suggest .NET core 2.1 but I have used it with 3.0 and worked. UPDATE: did not run it with .NET 5
  • Pros: It can be used like EF6 "migrate.exe". Finally migration works without the source code. And most probably this is the only way to do migration from script and using an Assembly.
  • Cons: Very hard to set up the script, and easy to miss one parameter. Not really documented solution and might change .NET Core version to version. Also most probably you will need to change your code as well. You have to implement IDesignTimeDbContextFactory<DbContext> interface in order to make it work. Also make sure you have EF.dll and Microsoft.EntityFrameworkCore.Design.dll on your deploy server. The linked script is looking for those in numerous folders. Best is to copy it during build from your .nuget folders to your artifact. Sounds complicated, yes it is... But linked script helps a lot.
  1. Add EF migration to your Startup.cs or any point where your code start running and has access to a DBContext. Use dbContext.Database.Migrate();
  • Pros: Migrations happens automatically every time and nothing else had to be done.
  • Cons: Migrations happens automatically every time... The problem you might don't want to that happen. Also it will run on every App start. So your startup time will be very bad.
  1. Custom app. It is similar to the previous solution (point 3.). So you use .NET code to run migration. But instead of putting it into your app you should create a small console app and call migrate in that one. You have to build this app and put into the Artifact to run it during the deployment.
  • Pros: No script involved. You can call it any time in your deployment pipeline. So your real app startup time not suffers from it.
  • Cons: You have to maintain, build and package an application just to do EF Core migrations.
  1. If you are using Azure Devops for deploy, you can use extension like this. Or just search Azure Devops Marketplace for one.
  • Pros: it should work :) Haven't tried any of them and don't know what they do. (I'm pretty sure they are also using 'dotnet exec' point 2.)
  • Cons: Not everyone can have access to Live/Prod from Azure Devops.
  1. Generate SQL script: If none of the above works you can generate a migration SQL and run it later. Run EF tool with "script" param: dotnet ef migrations script --output <pathAndFile>.sql --context <DbContextName> --idempotent. The output is an SQL file which can be executed manually or by a script in CI/CD pipeline.
  • Pros: it is perfect solution if you don't have access or schema change rights to prod DB only DBAs. You can provide DBAs with a "safe and secure" SQL file to run...
  • Cons: very important to emphasize this solution must run in working dir where your .csproj file is. So it requires source code! Also you have to change your code a bit. Need to implement IDesignTimeDbContextFactory<DbContext>.

UPDATE: In .NET 5 there is some improvements. It is now easier to implement and make use of IDesignTimeDbContextFactory but most importantly Microsoft fixed this bug. Now it is possible to pass an SQL connection string as args. So if you implemented IDesignTimeDbContextFactory<T> it is simple to use it with .NET CLI and EF tool:

dotnet ef database update --context <DbContextName> --project "**/<ProjectName>.csproj" -- "<SQL connection will be passed into args[0]>"

Also important to emphasize this works only with .NET 5 and requires source code as well! You can also use it with Option 6 (generate SQL script).

Second annoying issue once implemented IDesignTimeDbContextFactory<T> this will be discovered by ALL ef commands (even commands run from Visual Studio during development). If you require SQL connection string from args[0] you have to pass it in during development migrations add or for any other ef command!

Sorry the list got very long. But hope it helps.

JohnLBevan
  • 22,735
  • 13
  • 96
  • 178
Major
  • 5,948
  • 2
  • 45
  • 60
26

I ended up in the same problem on a project but for several reasons I don't want migrations to run automatically on application boot.

To solve it I updated Program.cs to take two arguments (full code is listed below)

  • --ef-migrate, to apply all pending migrations, and
  • --ef-migrate-check, to validate if all migrations have been applied

If arguments are present then the EF actions are applied and the program exits, otherwise the web application is launched.

Please note that it depends on the Microsoft.Extensions.CommandLineUtils package to ease the command line parsing.

For octopus deploy one can then publish the package twice to seperate locations - one for running migrations and the other for webhosting. In our case, we added a "post deploy powershell script" with the content

$env:ASPNETCORE_ENVIRONMENT="#{Octopus.Environment.Name}"
dotnet example-app.dll --ef-migrate

In a docker context it would work perfectly too

docker run -it "example-app-container" dotnet example-app.dll --ef-migrate

Full Program.cs excluding namespace and usings:

//Remember to run: dotnet add package Microsoft.Extensions.CommandLineUtils
public class Program
{
    public static void Main(string[] args)
    {
        var commandLineApplication = new CommandLineApplication(false);
        var doMigrate = commandLineApplication.Option(
            "--ef-migrate",
            "Apply entity framework migrations and exit",
            CommandOptionType.NoValue);
        var verifyMigrate = commandLineApplication.Option(
            "--ef-migrate-check",
            "Check the status of entity framework migrations",
            CommandOptionType.NoValue);
        commandLineApplication.HelpOption("-? | -h | --help");
        commandLineApplication.OnExecute(() =>
        {
            ExecuteApp(args, doMigrate, verifyMigrate);
            return 0;
        });
        commandLineApplication.Execute(args);
    }

    private static void ExecuteApp(string[] args, CommandOption doMigrate, CommandOption verifyMigrate)
    {
        Console.WriteLine("Loading web host");
        //
        // Please note that this webHostBuilder below is from an older 
        // dotnet core version. Newer dotnet cores have a simplified version
        // Use that instead and just take the command line parsing stuff with you
        var webHost = new WebHostBuilder()
            .UseKestrel()
            .UseContentRoot(Directory.GetCurrentDirectory())
            .UseIISIntegration()
            .UseStartup<Startup>()
            .Build();

        if (verifyMigrate.HasValue() && doMigrate.HasValue())
        {
            Console.WriteLine("ef-migrate and ef-migrate-check are mutually exclusive, select one, and try again");
            Environment.Exit(2);
        }

        if (verifyMigrate.HasValue())
        {
            Console.WriteLine("Validating status of Entity Framework migrations");
            using (var serviceScope = webHost.Services.GetRequiredService<IServiceScopeFactory>().CreateScope())
            { 
                using (var context = serviceScope.ServiceProvider.GetService<DatabaseContext>())
                {
                    var pendingMigrations = context.Database.GetPendingMigrations();
                    var migrations = pendingMigrations as IList<string> ?? pendingMigrations.ToList();
                    if (!migrations.Any())
                    {
                        Console.WriteLine("No pending migratons");
                        Environment.Exit(0);
                    }

                    Console.WriteLine("Pending migratons {0}", migrations.Count());
                    foreach (var migration in migrations)
                    {
                        Console.WriteLine($"\t{migration}");
                    }

                    Environment.Exit(3);
                }
            }
        }

        if (doMigrate.HasValue())
        {
            Console.WriteLine("Applyting Entity Framework migrations");
            using (var serviceScope = webHost.Services.GetRequiredService<IServiceScopeFactory>().CreateScope())
            {
                using (var context = serviceScope.ServiceProvider.GetService<DatabaseContext>())
                {
                    context.Database.Migrate();
                    Console.WriteLine("All done, closing app");
                    Environment.Exit(0);
                }
            }
        }

        // no flags provided, so just run the webhost
        webHost.Run();
    }
}

Community
  • 1
  • 1
nover
  • 2,259
  • 1
  • 27
  • 27
  • 1
    Thanks @nover, this really helped me out. It would be nice if the `dotnet database update` command would just take a compiled dll, hopefully in the future. – Matthew Aug 12 '18 at 23:19
  • I had to add using Microsoft.Extensions.DependencyInjection; to avoid The non-generic method 'IServiceProvider.GetService(Type)' cannot be used with type arguments As per https://stackoverflow.com/questions/54216939/the-non-generic-method-iserviceprovider-getservicetype-cannot-be-used-with-t – mattbloke Nov 15 '19 at 13:05
  • As DbContext is ScopedService, one should first use using (var serviceScope = webHost.Services.GetRequiredService().CreateScope()), then var context = serviceScope.ServiceProvider.GetService() – Lapenkov Vladimir Nov 11 '20 at 06:42
  • You're absolutely right, I'll update the example above – nover Nov 12 '20 at 07:01
3

There is an extremely useful post with a solution to this problem here.

It worked for me (I had to tweak the commands a little bit, but it gave me a good basis to start).

In sum: you can replicate the dotnet ef database update command by passing the ef.dll (e.g. directly from your nuget folder (or from somewhere else if you don’t have nuget, since you are on a prod machine..)) with your .dll containing the migrations with some additional parameters (see below) to dotnet.exe (or the linux equivalent).

For completeness here is the .cmd (also from the blogpost!)

set EfMigrationsNamespace=%1
set EfMigrationsDllName=%1.dll
set EfMigrationsDllDepsJson=%1.deps.json
set DllDir=%cd%
set PathToNuGetPackages=%USERPROFILE%\.nuget\packages
set PathToEfDll=%PathToNuGetPackages%\microsoft.entityframeworkcore.tools.dotnet\1.0.0\tools\netcoreapp1.0\ef.dll

dotnet exec --depsfile .\%EfMigrationsDllDepsJson% --additionalprobingpath %PathToNuGetPackages% %PathToEfDll% database update --assembly .\%EfMigrationsDllName% --startup-assembly .\%EfMigrationsDllName% --project-dir . --content-root %DllDir% --data-dir %DllDir% --verbose --root-namespace %EfMigrationsNamespace%

(A bash version if this cmd is in the blogpost)

Btw. this approach was also mentioned in many github issues: https://github.com/aspnet/EntityFramework.Docs/issues/328 https://github.com/aspnet/EntityFramework.Docs/issues/180

ps: I found this in the blog of Ben Day, so all credit goes to Ben!

gregkalapos
  • 3,529
  • 2
  • 19
  • 35
3

A lot of time has passed since the original question and nowadays there's a better way to handle ef migrations in CI/CD scenarios.

There's an official tool called 'bundles'.

All you have to do is to build a bundle

dotnet ef migrations bundle

Later on, you can execute the bundle passing the target db connection string

.\efbundle.exe --connection 'Data Source=(local)\MSSQLSERVER;Initial Catalog=Blogging;User ID=myUsername;Password=myPassword'

Official documentation

  • 1
    Good shout, this new EF feature solves my two initial reservations about deploying schema changes during app startup: 1) The separation of deployment steps from runtime 2) Avoiding the need for the runtime user to have permission to update the DB schema. – TallMcPaul Jan 16 '23 at 11:31
0

Within my context I have this hack from here

    // Hack added so EntityFrameworkCore\Add-Migration initial works
public class ApplicationContextDbFactory : IDesignTimeDbContextFactory<MyContext>
{
    MyContext IDesignTimeDbContextFactory<MyContext>.CreateDbContext(string[] args)
    {
        IConfigurationRoot configuration = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("appsettings.json")
            .Build();

        var optionsBuilder = new DbContextOptionsBuilder<MyContext>();
        optionsBuilder.UseSqlServer(configuration.GetConnectionString("StreamCheckpointContextDB"));
        return new MyContext(optionsBuilder.Options);
    }
}

I made my Program.Main read like this...

    public static void Main(string[] args)
    {
        if (args.Contains("JustMigrateMe"))
        {
            IDesignTimeDbContextFactory<MyContext> factory = new ApplicationContextDbFactory();
            var ctx = factory.CreateDbContext(new string[0]);
            ctx.Database.Migrate();
            ctx.Dispose();
            return;
        }

        // Other stuff
    }
}

So to apply migrations I simply call the exe with the added argument.

andrew pate
  • 3,833
  • 36
  • 28
0

For EF Core 3.1 I succeeded with the following line ran in the release files folder. Of course the path to MyUser can be tweaked with

dotnet exec --depsfile ProjectName.deps.json --runtimeconfig ProjectName.runtimeconfig.json C:\Users\MyUser.nuget\packages\dotnet-ef\3.1.9\tools\netcoreapp3.1\any\tools\netcoreapp2.0\any\ef.dll database update --context MyContext --assembly Project.dll --verbose