Set label.StyledSettings.Style
false, then it will follow the Fontstyle
settings.

Here a sample code to toggle StyledSettings.Style
with in code (although I don't remember that I've ever played back and forth with these. For me it's more a one time setup at start).
procedure TForm6.Button9Click(Sender: TObject);
begin
if TStyledSetting.Style in Label3.StyledSettings then
Label3.StyledSettings := Label3.StyledSettings - [TStyledSetting.Style]
else
Label3.StyledSettings := Label3.StyledSettings + [TStyledSetting.Style]
end;
And to toggle the TextSettings.Font.Style
procedure TForm6.Button8Click(Sender: TObject);
begin
if TFontStyle.fsBold in Label3.TextSettings.Font.Style then
Label3.TextSettings.Font.Style := Label3.TextSettings.Font.Style - [TFontStyle.fsBold]
else
Label3.TextSettings.Font.Style := Label3.TextSettings.Font.Style + [TFontStyle.fsBold];
end;