To get the three coordinates or vertices of the triangle you can use some trigonometry and the player's yaw
. According to your image, one vertex of the triangle is the player's Location
. The interior angle
of the triangle at that point and a length
, which defines how long the two sides of the triangle protruding from the player are, define the entire triangle. When these two values are defined, we can find the coordinates of the two remaining vertices of the triangle. Here is a rough sketch:

An example of how to get the Location
s of the two vertices:
double angle = 90; // An example angle of 90 degrees
double length = 5; // An example length of 5
Location v1 = player.getLocation(); // The first vertex of the triangle, the player's location
double yaw = v1.getYaw(); // The player's yaw (in degrees) between 0º and 360º
World world = player.getWorld(); // The player's world
Location v2 = new Location(world, (-Math.sin(Math.toRadians(yaw - (angle / 2))) * distance) + v1.getX(), 0, (Math.cos(Math.toRadians(yaw - (angle / 2))) * distance) + v1.getZ()); // The second vertex
Location v3 = new Location(world, (-Math.sin(Math.toRadians(yaw + (angle / 2))) * distance) + v1.getX(), 0, (Math.cos(Math.toRadians(yaw + (angle / 2))) * distance) + v1.getZ()); // The third vertex
The negative sine of the angle in radians is used for the X coordinate while the cosine of the angle in radians is used for the Z coordinate.
I used some particles to visualize the lines of the triangle in Minecraft, this one has a length of 5 and an angle of 90 degrees:

One method you can use to check whether a point is inside a two dimensional triangle is posted as an answer to this question and uses barycentric coordinates. Here is an example method adapted for Bukkit/Spigot.
// v1, v2 and v3 are the vertices of the triangle, in no specific order
// The Y coordinates are ignored (two dimensional)
public static boolean isLocationInTriangle(Location location, Location v1, Location v2, Location v3) {
// Bunch of math...
double a = 0.5 * (-v2.getZ() * v3.getX() + v1.getZ() * (-v2.getX() + v3.getX()) + v1.getX() * (v2.getZ() - v3.getZ()) + v2.getX() * v3.getZ());
int sign = a < 0 ? -1 : 1;
double s = (v1.getZ() * v3.getX() - v1.getX() * v3.getZ() + (v3.getZ() - v1.getZ()) * location.getX() + (v1.getX() - v3.getX()) * location.getZ()) * sign;
double t = (v1.getX() * v2.getZ() - v1.getZ() * v2.getX() + (v1.getZ() - v2.getZ()) * location.getX() + (v2.getX() - v1.getX()) * location.getZ()) * sign;
return s > 0 && t > 0 && (s + t) < 2 * a * sign;
}
I did some testing with this and it seems to work fairly well.
You could use this in your method like so:
List<Entity> nearbyEntities = player.getNearbyEntities(8, 2, 8); // Get entities within 16x4x16 box centered around player
for (Entity entity : nearbyEntities) { // For each Entity found
if (entity instanceof LivingEntity) { // If the Entity is an instance of a LivingEntity
LivingEntity living = (LivingEntity) entity; // Cast
if (isLocationInTriangle(living.getLocation(), player.getLocation(), v2, v3)) { // If the LivingEntity is inside the triangle
living.damage(6); // Damage the entity (just as an example)
}
}
}
Note that even though the IsLocationInTriangle
method ignores the Y coordinate or height (since it's a two dimensional check), the getNearbyEntities
method looks for entities inside a box, and in this example that box has a height of 4 blocks, with the center being at the player's location. Therefore entities more than 2 blocks above or below the player would not be checked.
Alternatively, once you have have the three coordinates of the triangle v1
, v2
and v3
, you could of course also use them to create a JavaFX Polygon
like Joop mentioned and use its contains
method to check whether an entity is inside the triangle instead of using the barycentric coordinates method.