Let's quote the AngularJS documentation :
From this chapter:
What are Directives?
At a high level, directives are markers on a DOM element (such as an attribute, element name, comment or CSS class) that tell AngularJS's HTML compiler ($compile) to attach a specified behavior to that DOM element or even transform the DOM element and its children.
From that chapter:
Use controllers to:
Do not use controllers to:
Manipulate DOM — Controllers should contain only business logic. Putting any presentation logic into Controllers significantly affects its testability. Angular has databinding for most cases and directives to encapsulate manual DOM manipulation.
Format input — Use angular form controls instead.
Filter output — Use angular filters instead.
Share code or state across controllers — Use angular services instead.
Manage the life-cycle of other components (for example, to create service instances).
You've got everything there, almost.
The subtle point is that because AngularJS works on top of Html pages and js code alone (as opposed to a more traditional client/server setup), the view and the "backend" usually supported by the server are all embedded in the same document, and the directive, as a facility to manipulate the DOM, is also a core functionality to bootstrap the application, so it must be tempting to just hardcode the application directly there. And this is what is happening already in the framework: specific directives are devoted for controllers.
In the MVC architecture, the controller helps bridging the views and the models, and centralise functionalities revolving around a group of models and their data. It also helps delineating what belongs to pure UI from what is business logic: changing your UI should not impact that logic, unless you plan to deliver new functionalities. But again, here these patterns overlap, because of the platform constraints.
The decisive argument is probably the following: if you did not use the builtin controller modules and directives, you would have to derive new directives for binding UI entry points to business logic, and in the process you would recreate objects following the controller pattern to hold code behind these entry points.
AngularJS makes it easy for you and gives you that out of the box.
Or maybe you would come up with an alternative?