ggplot facets: show annotated text in selected facets

Question

I want to create a 2 by 2 faceted plot with a vertical line shared by the four facets. However, because the facets on top have the same date information as the facets at the bottom, I only want to have the vline annotated twice: in this case in the two facets at the bottom.

I looked a.o. here, which does not work for me. (In addition I have my doubts whether this is still valid code, today.) I also looked here. I also looked up how to influence the font size in geom_text: according to the help pages this is size. In the case below it doesn't work out well.

This is my code:

library(ggplot2)
library(tidyr)

my_df <- read.table(header = TRUE, text = 
                      "Date AM_PM First_Second Systolic Diastolic Pulse 
                    01/12/2017 AM 1 134 83 68 
                    01/12/2017 PM 1 129 84 76
                    02/12/2017 AM 1 144 88 56
                    02/12/2017 AM 2 148 93 65
                    02/12/2017 PM 1 131 85 59
                    02/12/2017 PM 2 129 83 58
                    03/12/2017 AM 1 153 90 62
                    03/12/2017 AM 2 143 92 59
                    03/12/2017 PM 1 139 89 56
                    03/12/2017 PM 2 141 86 56
                    04/12/2017 AM 1 140 87 58
                    04/12/2017 AM 2 135 85 55
                    04/12/2017 PM 1 140 89 67
                    04/12/2017 PM 2 128 88 69
                    05/12/2017 AM 1 134 99 67
                    05/12/2017 AM 2 128 90 63
                    05/12/2017 PM 1 136 88 63
                    05/12/2017 PM 2 123 83 61
                    ")

# setting the classes right
my_df$Date <- as.Date(as.character(my_df$Date), format = "%d/%m/%Y")
my_df$First_Second <- as.factor(my_df$First_Second)

# to tidy format
my_df2 <- gather(data = my_df, key = Measure, value = Value, 
                  -c(Date, AM_PM, First_Second), factor_key = TRUE)

# Measures in 1 facet, facets split over AM_PM and First_Second
## add anntotations column for geom_text
my_df2$Annotations <- rep("", 54)
my_df2$Annotations[c(4,6)] <- "Start"

p2 <- ggplot(data = my_df2) +
  ggtitle("Blood Pressure and Pulse as a function of AM/PM,\n Repetition, and date") +
  geom_line(aes(x = Date, y = Value, col= Measure, group = Measure), size = 1.)  +
  geom_point(aes(x = Date, y = Value, col= Measure, group = Measure), size= 1.5) +
  facet_grid(First_Second ~ AM_PM) +
  geom_vline(aes(xintercept = as.Date("2017/12/02")), linetype = "dashed", 
             colour = "darkgray") + 
  theme(axis.text.x=element_text(angle = -90)) 

p2

yields this graph:

This is the basic plot from which I start. Now we try to annotate it.

p2 + annotate(geom="text", x = as.Date("2017/12/02"), y= 110, label="start", size= 3) 

yielding this plot:

This plot has the problem that the annotation occurs 4 times, while we only want it in the bottom parts of the graph.

Now we use geom_text which will use the "Annotations" column in our dataframe, in line with this SO Question. Be carefull, the column added to the dataframe must be present when you create "p2", the first time (that is why we added the column supra)

p2 + geom_text(aes(x=as.Date("2017/12/02"), y=100, label = Annotations, size = .6))

yielding this plot:

Yes, we succeeded in getting the annotation only in the bottom two parts of the graph. But the font is too big ( ... and ugly) and when we try to correct it with size, two things are interesting: (1) the font size is not changed (although you would expect that from the help pages) and (2) a legend is added.

I have been clicking around a lot and have been unable to solve this after hours and hours. Any help would be appreciated.