Pandas/Python append or concat dataframes being created in a for loop-CodePudding

I have a process which I am able to loop through for values held in a list but it overwrites the final dataframe with each loop and I would like to append or concat the result of the loops into one dataframe. For example given below I can see 'dataframe' will populate initially with result of 'blah1', then when process finishes it has the result of 'blah2'

listtoloop = ['blah1','blah2']

for name in listtoloop:


    some process happens here resulting in

        dataframe = result of above process

CodePudding user response：

The typical pattern used for this is to create a list of DataFrames, and only at the end of the loop, concatenate them into a single DataFrame. This is usually much faster than appending new rows to the DataFrame after each step, as you are not constructing a new DataFrame on every iteration.

Something like this should work:

listtoloop = ['blah1','blah2']
dfs = []
for name in listtoloop:
    # some process happens here resulting in
    # dataframe = result of above process
    dfs.append(dataframe)
final = pd.concat(dfs, ignore_index=True)

CodePudding user response：

Put your results in a list and then append the list to the df making sure the list is in the same order as the df

listtoloop = ['blah1','blah2']
df = pd.DataFrame(columns="A","B")
for name in listtoloop:
    ## processes here

    to_append = [5, 6]
    df_length = len(df)
    df.loc[df_length] = to_append

CodePudding user response：

data_you_need=pd.DataFrame()  
listtoloop = ['blah1','blah2']  
for name in listtoloop:     
    ##some process happens here resulting in          
    ##dataframe = result of above process    
    data_you_need=data_you_need.append(dataframe,ignore_index=True)