If you want a border around the tab (not the label), you have to use this:
.tab-pane > .tab-header-area > .headers-region > .tab {
-fx-background-color: red;
-fx-padding: 20px;
-fx-border-color: black;
-fx-border-width: 1px;
}

If you want to manipulate the tab-container (where the label is in) itself you need this:
.tab-pane > .tab-header-area > .headers-region > .tab > .tab-container{
-fx-border-color: black;
-fx-border-width: 1px;
}
.tab-pane > .tab-header-area > .headers-region > .tab {
-fx-padding: 20px;
-fx-background-color: red;
}

UPDATE
Default for a selected tab is that:
.tab-pane:focused > .tab-header-area > .headers-region > .tab:selected .focus-indicator {
-fx-border-width: 1, 1;
-fx-border-color: -fx-focus-color, -fx-faint-focus-color;
-fx-border-insets: -4 -4 -6 -5, -2 -2 -5 -3;
-fx-border-radius: 2, 1; /* looks sharper if outer border has a tighter radius (2 instead of 3) */
}
And this it how it goes:
.tab-pane > .tab-header-area > .headers-region > .tab {
-fx-padding: 20px;
-fx-background-color: red;
}
.tab-pane > .tab-header-area > .headers-region > .tab:selected {
-fx-padding: 20px;
-fx-background-color: red;
-fx-border-width: 1px;
-fx-border-color: black;
}
.tab-pane > .tab-header-area > .headers-region >.tab:selected .focus-indicator{
-fx-border-width: 0px;
}

Look at the modena.css (default JavaFX stylesheet) file for info on things to change.
Font size will not change dynamic, you have to take care of font size with a listener on size/width/height property of the tab (in relation to font size).
And there are a lot of pseudo tags like .tab:selected .tab:top etc. So be aware of this kind of things if you want the default behavior only with new design.
And finally have a look at css selectors, you missed the descending selectors ('>'): http://www.w3schools.com/cssref/sel_element_gt.asp