Using pd.concat instead of df.append with pandas 1.4-CodePudding

I am using df.append() in my code to append the percentage change across dataframe columns. With df.append() being depreciated in pandas 1.4, I am trying to use pd.concat but I am not able to replicate the output.

So here is what I have now:

import numpy as np
import pandas as pd

df = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
                         "bar", "bar", "bar", "bar"],
                   "B": ["one", "one", "one", "two", "two",
                         "one", "one", "two", "two"],
                   "C": ["small", "large", "large", "small",
                         "small", "large", "small", "small",
                         "large"],
                   "D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
                   "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]})

table = pd.pivot_table(df, values='D', index=['A', 'B'],
                    columns=['C'], aggfunc=np.sum, fill_value=0, margins=True)

table.append(
    (table
     .iloc[-1]
     .pct_change(periods=1, fill_method=None)
     .fillna('')
     .apply(lambda x: '{:.1%}'.format(x) if x else '')
    )
)

The output is:

    C   large   small   All
A   B           
bar one 4   5   9
two 7   6   13
foo one 4   1   5
two 0   6   6
All     15  18  33
            20.0%   83.3%

which is what I am after, but I am getting the depreciated warning,

FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.table.append()

I changed my code to use pd.concat(), as follow:

pd.concat([table,
    (table
     .iloc[-1]
     .pct_change(periods=1, fill_method=None)
     .fillna('')
     .apply(lambda x: '{:.1%}'.format(x) if x else '')
    )],
)

now I am getting:

    large   small   All 0
(bar, one)  4.0 5.0 9.0 NaN
(bar, two)  7.0 6.0 13.0    NaN
(foo, one)  4.0 1.0 5.0 NaN
(foo, two)  0.0 6.0 6.0 NaN
(All, ) 15.0    18.0    33.0    NaN
large   NaN NaN NaN 
small   NaN NaN NaN 20.0%
All NaN NaN NaN 83.3%

which is not what I expected - note the percentage changes (20% and 83.3%) compared to the output from append. Any input would be appreciated.

CodePudding user response：

Use pd.concat like this. Convert your inner command to df using Series.to_frame and then transpose it using df.T:

In [74]: pd.concat([table, table.iloc[-1].pct_change(periods=1, fill_method=None).fillna('').apply(lambda x: '{:.1%}'.format(x) if x else '').to_frame().T])
Out[74]: 
C       large  small    All
A   B                      
bar one     4      5      9
    two     7      6     13
foo one     4      1      5
    two     0      6      6
All        15     18     33
               20.0%  83.3%