css masonry layout without horizontal scrollbar

Question

I'm trying to achieve masonry for my panels. This is how it looks currently:

I'd like it to fill the gaps whenever it's possible. I've tried using flexbox but it behave like there was 1 row and after adding more panels it just made them smaller. I also tried using column-countproperty but my panels were separated (panel heading from panel body. blue and white) unless there were as many elements in a row as there were columns (i.e. column-count: 4looked good only if number of panels was multiplicity of 4). Here's how my panel code looks:

<div class="container-fluid">
  <div class="col-md-3 ng-cloak" ng-repeat="card in vm.cards" style="margin-bottom: 10px;">
    <div class="panel panel-info" style="min-width:235px;">
      <div class="panel-heading clearfix">
        <h4 class="panel-title pull-left">{{ card.question }}</h4>
        <div class="btn-group pull-right">
          <a href="#" class="btn btn-sm btn-primary">Manage</a>
        </div>
      </div>
      <div class="panel-body">
        {{ card.answer }}
      </div>
    </div>
  </div>
</div>

Here's the exact page in my repository: Github Reposiory

Also here's the working page: http://188.166.160.66/App/Demo

I'd like to achieve this with css, but if it's impossible I'll give js a shot. If more screenshots are required I can provide them.

@Edit: I want to prevent my site from having horizontal scrolling.