Solving systems of linear equations where variables are integers.
An integer program is a linear-programming problem where, in addition to the linear inequalities constraints, the variables are also constraint to be integers only.
While linear-programming can be solved in polynomail time, integer-programming is NP-hard.
A good solver for such optimization problems is lp_solve. See also GLPK (http://www.gnu.org/software/glpk/) and CBC (https://projects.coin-or.org/Cbc)
However, for large or hard instances you may need a commercial solver such as CPLEX or Gurobi.
For more information see wikipedia.