Difficult to tell without a reprex, but it sounds like your data is something like this:
library(ggplot2)
library(dplyr)
set.seed(69)
df <- data.frame(factor_var = factor(rep(1:6, 100)),
independent_var = rnorm(600),
response_var = rnorm(600))
And your first plot is wrong because it looks like this:
df %>%
ggplot(aes(x = factor_var, y = response_var)) +
geom_point() +
stat_smooth() +
facet_wrap(~factor_var)
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

You would be happier with something like this:
df %>%
filter(factor_var == 1) %>%
ggplot(aes(x = independent_var, y = response_var)) +
geom_point() +
stat_smooth()
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

But you want a separate version for each of the factor variables, and you want them all on the same page. In that case, you put the independent variable on the x axis, the dependent variable on the y axis, and facet by the factor variable:
df %>%
ggplot(aes(x = independent_var, y = response_var)) +
geom_point() +
stat_smooth() +
facet_wrap(~factor_var)
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Created on 2020-07-14 by the reprex package (v0.3.0)