I am writing an event-based backtesting framework in python. My main issue is to have fast indexing/slicing abilities as I have to frequently accessing data of certain periods, certain assets and certain fields. The data format look like this

a typical indexing could be to query columns of [open, close, volume] of root symbol futures ['a', 'm', 'cu'] between time '2020-01-10 14:00:00' and '2020-03-10 15:00:00'.

My first attemp was to use the pandas Multiindex dataframe. The code looks easy and clean but the indexing speed is slow. It took on average 2~5ms using the pandas .loc. This is unacceptable for the backtesting purposes as I need to do those kinds of indexing like a 100000 times.

So I wonder if there is any suggested alternatives for those kinds of purposes. Like using numpy and to manually find the location. Or some other column-oriented data format like the apache arrow. Thanks.

CodePudding user response：