CSS3 grid items won't stretch vertically/in column. How to let the items span automatically?

Question

I am trying to build a CSS grid for a put in order schedule, including horizontally spanning elements. It should appear in one block, but the column items won't stretch vertically.

In Detail: The order of the div elements has to be strictly maintained, also the 6 div elements with class "sem" have always to be the title cell of each column.

In order to align it to one block the grid items have to vertically span over 1 or 2 cells, filling the remaining space - automatically.

The number of rows respectively the number of div-elements after each div with class "sem" will be variable in a range of 2-8. So arranging the grid items by telling them exactly where to start/end or span is not an option.

Now, it looks like this (picture):

I want to make it look like this (picture):

Here is the html:

<div class="grid">
    <!-- 1. COLUMN -->
    <div class="sem sem1">1. COLUMN</div>
    <div>1.1</div>
    <div>1.2</div>
    <div class="span2sem">1.3</div>
    <!-- 2. COLUMN -->
    <div class="sem sem2">2. COLUMN</div>
    <div>2.1</div>
    <div>2.2</div>
    <div>2.3</div>
    <!-- 3. COLUMN -->
    <div class="sem sem3">3. COLUMN</div>
    <div>3.1</div>
    <div>3.2</div>
    <div>3.3</div>
    <div class="span2sem">3.4</div>
    <div class="span2sem">3.5</div>
    <div class="span2sem">3.6</div>
    <!-- 4. COLUMN -->
    <div class="sem sem4">4. COLUMN</div>
    <div>4.1</div>
    <div>4.2</div>
    <div>4.3</div>
    <!-- 5. COLUMN -->
    <div class="sem sem5">5. COLUMN</div>
    <div>5.1</div>
    <div>5.2</div>
    <div class="span2sem">5.3</div>
    <div class="span2sem">5.4</div>
    <!-- 6. COLUMN -->
    <div class="sem sem6">6. COLUMN</div>
    <div>6.1</div>
    <div>6.2</div>
</div>

And here the CSS:

    .grid {
        display: grid;

        grid-template: repeat(auto-fit, minmax(auto, 2fr)) /  repeat(6, 1fr);

        grid-auto-flow: column dense;

        /*align-items: stretch;
        align-content: stretch;
        justify-content: stretch;
        justify-items: stretch;*/

        grid-gap: 4px;
        height: 420px;
    }

    .grid>div {
        background-color: aquamarine;
        border: 1px solid cornflowerblue;
        color: cornflowerblue;
        font-family: sans-serif;
        padding: 3px;

        /*display: flex;
        flex-direction: column;
        flex: 1;
        align-self: stretch;
        justify-self: stretch;*/
    }

    .span2sem {
        grid-column-end: span 2 !important;
    }

    .sem {
        grid-row: 1;
    }

    .sem1,
    .sem1+div,
    .sem1+div+div,
    .sem1+div+div+div {
        grid-column-start: 1;
    }

    .sem2,
    .sem2+div,
    .sem2+div+div,
    .sem2+div+div+div {
        grid-column-start: 2;
    }

    .sem3,
    .sem3+div,
    .sem3+div+div,
    .sem3+div+div+div,
    .sem3+div+div+div+div,
    .sem3+div+div+div+div+div,
    .sem3+div+div+div+div+div+div {
        grid-column-start: 3;
    }

    .sem4,
    .sem4+div,
    .sem4+div+div,
    .sem4+div+div+div {
        grid-column-start: 4;
    }

    .sem5,
    .sem5+div,
    .sem5+div+div,
    .sem5+div+div+div,
    .sem5+div+div+div+div {
        grid-column-start: 5;
    }

    .sem6,
    .sem6+div,
    .sem6+div+div,
    .sem6+div+div+div {
        grid-column-start: 6;
    }

(here is also a codepen)

To maintain the order and to keep every item in the right column, I had to tell each div in which column to start, so the items kind of wrap. I haven't tried to work with "order" instead of telling in which column to start. Do you think that would be better?

I've also tried to combine grid with flex, but it didn't work.

I would love to write grid-row: repeat(auto-fit, minmax(span 1 / span 2)) or minmax(1fr, 2fr), but that didn't work also.

Am I missing something?