Absolutely can and do use ES6 W/O babel. All major browsers support the vast majority of features natively (see CanIUse.com), in fact, the only major feature not supported is the import/export of modules.
For these, you still have to manually import your modules in the correct order using script tags in the HTML. Client-side only (Node needs the modules).
However, be aware that this is fine for dev but in production, you'll need to concatenate and minimize all the JS into one module anyway so using a Babel/Webpack or Babel/Browserify style toolchain may ultimately be where you end up.