As pointed out, your join condition doesn't make sense, you are joining an employee number to a department number, which wouldn't match (or would, but wouldn't achieve the logic you desire).
SELECT
d.name
, AVG(e_man.salary) avg_man
, AVG(e_woman.salary) avg_woman
FROM department d
LEFT OUTER JOIN employee e_man ON e_man.deptNo = d.deptNo AND e_man.sex = 'm'
LEFT OUTER JOIN employee e_woman ON e_woman.deptNo = d.deptNo AND e_woman.sex = 'f'
GROUP BY d.name
Here you join twice, once to male rows and once to female and calculate the averages based on them. OUTER joins are used in case there aren't any men/women.
Note: my queries will return you a single row per department with male and female averages as 2 columns. Alternate solutions are to return you 0-2 rows per department, with separate rows per sex and listing the relevant salary per row/sex.
EDIT: Alternative sub-query solution which people seem very unhappy with. In my experience sometimes sub-queries are the right answer and sometimes a join is the right answer, and you would often use whichever fits your logical aim/representation best as long as it doesn't sacrifice performance. The optimiser for the RDBMS you are using will often rewrite it anyway. As always, best to check the Plan and see what is going on.
SELECT
d.name
, (
SELECT AVG(e1.salary)
FROM employee e1
WHERE e1.sex = 'm'
AND e1.deptNo = d.deptNo
) avg_man
, (
SELECT AVG(e2.salary)
FROM employee e2
WHERE e2.sex = 'f'
AND e2.deptNo = d.deptNo
) avg_woman
FROM department d