Detecting rectangle-circle intersections and finding intersection points
You can decompose the problem into two smaller problems. There will be an intersection between the rectangle and the circle if:
- The circle lies entirely within the rectangle, OR
- Any of the 4 sides of the rectangle intersect the circle
The first case is easy, just check if the center-point of the circle is inside the rectangle and greater than or equal to radius distance away from each side. In this case there is an intersection, and the intersection arc is the entire circle. There are between zero and four intersection points: one for each edge if that edge is exactly radius away from the center-point and zero if it is greater than radius: done.
For the second case use four line-segment-to-circle intersection tests, one for each side of the rectangle, which will tell you if there is an intersection and what the intersection points are. There can be zero, one or two of them for each side.
Finding intersection arcs
This applies to the second case only. Once you have the intersection points, you can figure out the intersection arcs (in terms of start and end angle) by looking at pairs of points, one where a rectangle edge enters the circle and one where an edge (possibly not the same edge) leaves it, according to an arbitrary winding direction going around the edges of the rectangle. Then for each point in each pair take the x and y differences between the intersection point and the circle center-point and work out the angle using atan2(). These are the start and end angles of the intersection arc and there may be up to four such arcs in total.
However, first you have to exclude a special case. If there is only 1 intersection point then the circle is "kissing" the rectangle: one of the sides is exactly radius away from the circle center-point and they are just touching, and there is no intersection arc.
Testing a rectangle against arcs instead of a circle
If your circle is already segmented into pre-defined arcs before you do the test, then you can check which arc each intersection point belongs to by comparing its angle (calculated with atan2()) to the start and end angles of each pre-defined arc. Similarly if you only want to test a rectangle against a single arc instead of an entire circle, check that the angle of a given intersection point is within the start-end range of the arc and if not then ignore it.