Boost MultiIndexContainer, how to do delayed indexing for fast insertion?

Question

I used below MultiIndexContainer

typedef multi_index_container<PositionSummary*,
        indexed_by<
                ordered_unique<
                        composite_key<PositionSummary, const_mem_fun<PositionSummary, int, &PositionSummary::positiondate>,
                                const_mem_fun<PositionSummary, const std::string&, &PositionSummary::accountid>,
                                const_mem_fun<PositionSummary, const std::string&, &PositionSummary::instid> > >,
                ordered_unique<
                        composite_key<PositionSummary, const_mem_fun<PositionSummary, int, &PositionSummary::positiondate>,
                                const_mem_fun<PositionSummary, const std::string&, &PositionSummary::instid>,
                                const_mem_fun<PositionSummary, const std::string&, &PositionSummary::accountid> > > 
                    > > PositionSummaryContainer;

And I inserted 10000 insts*36 accounts*100 days=36 million records

//Begin testing of the multiIndexContainter
std::cout << "Begin inserting data from array into the multiIndexContainter" << std::endl;
timer.reset();
timer.begin();
for (int i = 0; i < numOfDays_; i++)
{
    for (int j = 0; j < accountSize_; j++)
    {
        for (int k = 0; k < instSize_; k++)
        {
            PositionSummary* ps = psArray_[(i * accountSize_ + j) * instSize_ + k];
            uniqueIndex.insert(ps);
        }
    }

}

printMemoryUsage();
timer.end();
std::cout << "Time take is " << timer.getInterval() << std::endl;

And I found the speed of insertion is a little bit slow, about 20K+ records per second... Is there anyway to enhance this insertion speed? My data was in Oracle, properly indexed, so there should be no danger of corrupted data structure. I knew that in oracle you can first load then build index to save time, can I do the same with MultiIndexContainer, if there is a way? By the way, the parallel query speed is quite satisfactory, querying all the 36 m records on a 4 cpu(8kernal) machine takes only 2.8 seconds, code as below

#pragma omp parallel for collapse(2)
    for (int i = 0; i < numOfDays_; i++)
    {

        for (int j = 0; j < accountSize_; j++)
        {
            const int& date = dates_[i];
            const std::string& accountID = accountIDs_[j];
            for (int k = 0; k < instSize_; k++)
            {
                const std::string& instID = instIDs_[i];
                PositionSummaryContainer::iterator it = uniqueIndex.find(boost::make_tuple(date, accountID, instID));
                if (it != uniqueIndex.end())
                {
#pragma omp atomic
                    sum2 += (*it)->marketvalue();
                }
            }
            //std::cout << "accountID: " << accountID << std::endl;
        }

    }

if `psArray_` is already suitably organized (perhaps it's even memory-mapped?) you can perhaps just query on it directly — sehe, Feb 15 '15 at 12:59

score 1 · Answer 1 · answered Feb 19 '15 at 10:42

Ah, I think I found the key. It was so simple.

psContainer_.insert(&psArray_[0], &psArray_[accountSize_ * instSize_ * numOfDays_]);

Now the insertion speed is reduced from 53 seconds to 5.29 seconds. Ten times improvement. I think boost authors knew they did some batch insertion using this way, just I didn't notice.

Boost is a great lib, but the document is not so thorough.

score 0 · Answer 2 · edited May 23 '17 at 10:24

As far as I am aware this is not a feature.
^{(Is using a map where value is std::shared_ptr a good design choice for having multi-indexed lists of classes?)}

I must say that having the two equivalent composite indexes seems like a code smell. Seems to me you should have fewer, or more separate, indices.
storing the PositionSummary by pointer could be a smell too; it is prone to killing the memory locality
with the usage pattern shown, using multi-index container is overkill. You could (should) just use e.g. boost::flat_set (basically a vector<PositionSummary> order by the primary index that is being used in your usage example). See
- boost::flat_map and its performance compared to map and unordered_map
you could use a persistent container in any of the above scenarios (use Boost Interprocess managed_shared_memory/managed_mapped_file segment allocators. See many of my answers for examples). This would completely/largely eliminate load times
you could use #pragma omp parallel for reduction(+:sum2), which would greatly reduce cache invalidation on the sum2 variable
it looks an awful lot like you're on-the-fly consolidating measures on two orthogonal dimensions (date/account). This should remind you of "pivot tables" and hence of OLAP (cube) reporting databases.

Instead of "exporting" the bulk data into your proprietary C++ application you could export it to an OLAP analysis/reporting engine (Business Objects (ROLAP), MS Analysis Server (OLAP), Oracle Essbase¹) and use their excellently optimized reporting and (pre)consolidation features.

There is a dedicated query language: MDX for OLAP databases which you would just be able to use much like you would use e.g. OCI from C++

¹ former Arbor Essbase -> Hyperion Essbase -> Oracle Essbase

I've expanded the answer considerably with some more and less lateral ideas — sehe, Feb 15 '15 at 11:27
Thank you for your thorough answer. The actual project is much complicated than this code, and this code is supposed to demonstrate the issue. But I am doing multidimensional queries and aggregations(can be quite complicated and may need data that is not in the position summary). — Michael, Feb 16 '15 at 07:18
Okay, that's good. I'm still a bit surprised at the overlapping indices (you will realise that this is the prime area to be optimizing your insertion times). Memory mapping a binary file would seem to be your best bet to get rid of the load time, then ([example](http://stackoverflow.com/questions/27328835/want-to-efficiently-overcome-mismatch-between-key-types-in-a-map-in-boost-interp/27330042#27330042)). And it still reminds me of too many "NIH-syndrome reporting applications". Really OLAP is quite powerful and often underrated by developers. — sehe, Feb 16 '15 at 07:24
I developed Essbase Applications many years ago for financial accounting and financial management for petrochemical industry, but the current case is in the area of financial engineering in financial industry, the aggregation logic can be very complicated, such as for Bond Future Option, is Delta*holding*contractsize*underlying marketvalue, and would differ for each instrument type. And the data volume is not that big and the speed requirement is important. — Michael, Feb 16 '15 at 07:39
@Michael We did a lot of similar stuffs :) Anyhoops, good to know you're on the right track here. Good luck. — sehe, Feb 16 '15 at 10:42

Boost MultiIndexContainer, how to do delayed indexing for fast insertion?

2 Answers2