I have got an application/octect-stream with a application/zip as body in requests.Response object returned from an api call with a csv file inside it. I am trying to read the csv file to pandas without writing to the disk, if possible. And if I want to write the zip file to a path as a zip file, how can I do that?

resp = requests.get(url, headers=headers)
resp.raise_for_status()
csv_obj = zlib.decompress(resp.content, wbits=zlib.MAX_WBITS|32)
print(type(csv_obj))
export_file = pd.read_csv(csv_obj)
export_file.to_csv('./Test_export.csv')

CodePudding user response：

Updated version

# step 1: it turns out pandas can read zipped csv files even from urls!
some_dataframe = pandas.read_csv(url)

If pandas can't figure it out by itself there are some parameters you can try to massage.

# step 1: it turns out pandas can read zipped csv files even from urls!
some_dataframe = pandas.read_csv(zip_filename, compression='zip', header=0) # etc..

Previous version

I will leave the previous version of my answer below for reference.

# step 1: downloading the zip file
zip_filename = 'response.zip'
with open(zip_filename, 'wb') as zip_file:
  for chunk in response.iter_content(chunk_size=255): 
      if chunk:
        zip_file.write(chunk)

# step 2: turns out pandas can read zipped csv files!
some_dataframe = pandas.read_csv(zip_filename)

CodePudding user response：

import pandas as pd
import io
import zipfile

resp = requests.get(url, headers=headers, stream=True)
resp.raise_for_status()
zfile = zipfile.ZipFile(io.BytesIO(resp.content))
# I only had one file, so calling zfile.namelist
export_file = pd.read_csv(zfile.open(f'{zfile.namelist()[-1]}'))