Invalid input is removed from input after compiling in Angular directive

Question

I have a directive that adds custom and native validations to inputs according to a list of validations in a model. Values of the validations can change according to input elsewhere in the form (e.g. enddate should not be before the startdate).

function questionValidations($compile, $parse) {
    var directive = {
        restrict: 'A',
        link: linker
    }

    return directive;

    function linker(scope, elem, attrs) {
        var validations = $parse(attrs.questionValidations)(scope);
        setValidations();

        function setValidations() {
            if (!validations || validations.length < 1) {
                return;
            }

            // Logic that adds/removes attributes following the parsed validations above                                           
            // Additionally a service is called to determine if 
            // the validation value should be watched. 
            // Watch calls setValidations.

            // Compile if there are new or removed attributes
            if (new || removed) {
                var newElem = angular.element(elem[0]);
                elem.replaceWith(newElem);
                $compile(newElem)(scope);
            }
        }
    }
}

The attributes are added/removed and validations seems to work in a way, but all invalid input is removed. I tried assigning the ngModel in an isolated scope following this post, but that didn't change the behavior.

When I use the following code for compilation the validations work as desired (not sure why though)

         $compile(elem)(scope,
            function(clone) {
                elem.after(clone);
                elem.remove();
          });

However, if setValidations is called by one of the watches, it only compiles the element the first time. For the startdate example this means that for the first date entered the enddate input gets compiled just fine, but when I change the input nothing happens (logging the attributes before calling the compile does show updated values). The compile method in the original directive does recompile on model changes, but all invalid input gets removed.

edit: added Plunker

Looks like the debounce on the input has something to do with it. Unfortunately, I have a bunch of code running on my ng-change that I need the debounce for.

edit2: values are no longer removed when I add allowInvalid: true to my ng-model-options. I'm not really interested in invalid values, so I would prefer not using this.