non-convex is a sub-set of optimisation problems domain, where a utility / penalty function does not meet a condition of convexity
In optimization, the problems for which an assumption of convex function holds over convex sets, simplifies the optimization search process in some sense, right by using the underlying convex-property, which causes the search to become, in it's way, "easier" than the general case.
non-convex problem domains do not enjoy such "ease" - for example, having both local and global minima, having inflections and similar "un-easy" properties, that may confuse or trap convex-solvers on non-convex domains.