Wrap your rotated tool items in a Group, then the in-built layout of toolbar will know that the rotation is a permanent one which should be taken into account for layout calculations and not just a temporary thing which might be used for animations. Read the javadoc for Group, where it talks about layout bounds calculations to better understand this.

<?xml version="1.0" encoding="UTF-8"?>
<?import javafx.scene.*?>
<?import javafx.scene.control.*?>
<?import java.lang.*?>
<?import javafx.scene.layout.*?>
<HBox maxHeight="-Infinity" maxWidth="-Infinity" minHeight="-Infinity" minWidth="-Infinity" prefHeight="200.0" prefWidth="100.0" style="-fx-background-color: cornsilk;" xmlns="http://javafx.com/javafx/8" xmlns:fx="http://javafx.com/fxml/1">
<children>
<ToolBar orientation="VERTICAL" style="-fx-base: palegreen;">
<items>
<Group>
<children>
<Button rotate="-90.0" style="-fx-base: gold;" text="Project" />
</children>
</Group>
<Group>
<children>
<Button rotate="-90.0" style="-fx-base: khaki;" text="Structure" />
</children>
</Group>
</items>
</ToolBar>
</children>
</HBox>
Update 24th April 2017
The above solution is fine as far as it goes, but does suffer from an issue in that the buttons in the toolbar misalign when they receive focus.
What a group does is size itself based upon its contents. When the size of the contents changes, the size of the group also changes. When a button or control gets focus in JavaFX it gets a focus ring around the control. The display for the focus ring is defined in CSS and contains negative values for background inset display. The result is that, when a control is focused, it is slightly larger than when it is not focused. Normally, when you use a standard layout pane, this is not an issue as the layout pane will just ignore the background insets for layout purposes. A group however takes the full size into account and does not ignore the focus ring. The result is that a group that consists of only a control will change in size slightly when it is focused or unfocused. This presents an issue with the solution above because, when a button becomes focused, it will get slightly larger and the shift in layout in the toolbar, which is not ideal.
The solution to the focus shift problem in the above code is to just rotate the entire ToolBar within a group rather than rotating each button within a group per button. This works fine, but then presents some other issues such as the ToolBar not taking up the entire available area at the left side of the scene (due to wrapping it in a group removing the dynamic layout properties of the ToolBar). To get around this, a binding in code can be used to size the ToolBar to the available area of its parent layout container.
So we end up with the slightly more verbose solution below:
skinsample/toolbar.fxml
<?xml version="1.0" encoding="UTF-8"?>
<?import javafx.scene.control.ToggleButton?>
<?import javafx.scene.control.ToolBar?>
<?import javafx.scene.Group?>
<?import javafx.scene.layout.BorderPane?>
<?import javafx.scene.layout.HBox?>
<?import javafx.scene.layout.Pane?>
<?import javafx.scene.control.ToggleGroup?>
<BorderPane fx:id="border" prefHeight="200.0" prefWidth="100.0" xmlns="http://javafx.com/javafx/8.0.65" xmlns:fx="http://javafx.com/fxml/1" fx:controller="skinsample.VerticalToolbarController">
<left>
<Group>
<fx:define>
<ToggleGroup fx:id="selectedTool"/>
</fx:define>
<ToolBar fx:id="toolbar" rotate="-90.0" style="-fx-base: palegreen;">
<Pane HBox.hgrow="ALWAYS" />
<ToggleButton style="-fx-base: khaki;" text="Structure" toggleGroup="${selectedTool}"/>
<ToggleButton style="-fx-base: gold;" text="Project" toggleGroup="${selectedTool}" selected="true"/>
</ToolBar>
</Group>
</left>
</BorderPane>
skinsample/VerticalToolbarController.java
package skinsample;
import javafx.beans.binding.Bindings;
import javafx.fxml.FXML;
import javafx.scene.control.ToolBar;
import javafx.scene.layout.BorderPane;
public class VerticalToolbarController {
@FXML
private BorderPane border;
@FXML
private ToolBar toolbar;
public void initialize() {
toolbar.minWidthProperty().bind(Bindings.max(border.heightProperty(), toolbar.prefWidthProperty()));
}
}
skinsample/ToolDisplayApp.java
package skinsample;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Scene;
import javafx.stage.Stage;
public class ToolDisplayApp extends Application {
@Override
public void start(Stage stage) throws Exception {
FXMLLoader loader = new FXMLLoader(getClass().getResource("toolbar.fxml"));
Scene scene = new Scene(loader.load());
stage.setScene(scene);
stage.show();
}
public static void main(String[] args) {
launch(args);
}
}
Notes:
- This solution also demonstrates use of ToggleButtons rather than standard buttons within the ToolBar.
We also eliminate the default overflow behavior of the ToolBar (as it seems a little annoying in the vertical toolbar situation), using:
toolbar.minWidthProperty().bind(Bindings.max(border.heightProperty(), toolbar.prefWidthProperty()));
If you want to retain the overflow behavior, then use:
toolbar.prefWidthProperty().bind(border.heightProperty());
An alternate solution to the focus issue (using CSS to remove the focus ring entirely), is presented in: