I had a similar issue when rotating around the z-axis. Setting shouldRasterize = YES prevented the jagged edges however it came at a performance cost. In my case I was re-using the views (and its layers) and keeping the shouldRasterize = YES was slowing things down.
The solution was, to turn off rasterization right after I didn't need it anymore. However since animation runs on another thread, there was no way of knowing when the animation was complete...until I found out about an extremely useful CATransaction method. This is an actual code that I used and it should illustrate its use:
// Create a key frame animation
CAKeyframeAnimation *wiggle = [CAKeyframeAnimation animationWithKeyPath:@"transform"];
NSInteger frequency = 5; // Higher value for faster vibration
NSInteger amplitude = 25; // Higher value for lower amplitude
// Create the values it will pass through
NSMutableArray *valuesArray = [[NSMutableArray alloc] init];
NSInteger direction = 1;
[valuesArray addObject:@0.0];
for (NSInteger i = frequency; i > 0; i--, direction *= -1) {
[valuesArray addObject:@((direction * M_PI_4 * (CGFloat)i / (CGFloat)amplitude))];
}
[valuesArray addObject:@0.0];
[wiggle setValues:valuesArray];
// Set the duration
[wiggle setAdditive:YES];
[wiggle setValueFunction:[CAValueFunction functionWithName:kCAValueFunctionRotateZ]];
[wiggle setDuration:0.6];
// Turn on rasterization to prevent jagged edges (anti-aliasing issues)
viewToRotate.layer.shouldRasterize = YES;
// ************ Important step **************
// Very usefull method. Block returns after ALL animations have completed.
[CATransaction setCompletionBlock:^{
viewToRotate.layer.shouldRasterize = NO;
}];
// Animate the layer
[viewToRotate.layer addAnimation:wiggle forKey:@"wiggleAnimation"];
worked like a charm for me.
I have not tried using this with implicit animations (i.e. animations that happen due to value change in animatable property for a non-view associated layer), however I would expect it to work as long as the CATransaction method is called before the property change, just as a guarantee the block is given to CATransaction before an animation starts.