UTF-8 error while adding data to postgresql with python-CodePudding

import io
import csv
import datetime 
import psycopg2
import glob

conn = psycopg2.connect("host= 127.0.0.1 dbname=postgres user=postgres password=123 client_encoding='utf8' ")

f = io.StringIO()
w = csv.writer(f)


f.seek(0)
cursor = conn.cursor()

cursor.copy_expert(f""" COPY ama3 FROM 'C:/abc/asd.csv'  (FORMAT CSV)""", f)

conn.commit()

ERROR: CharacterNotInRepertoire: ERROR: Invalid byte sequence for language encoding "UTF8":

The csv file contains letters containing "ş,i,ö,Ğ,İ". I think the error is caused by this, but I don't want to change them.

CodePudding user response：

You are not adding the data from ptyhon, you are adding it directly from the filesystem using PostgreSQL's copy command. Make sure your csv file is UTF-8 encoded.

While in linux/mac you can easily do a:

$ file /abc/asd.csv

In Windows is not that straightforward (I am assuming you are using Windows).

In windows, you could:

Open up your file using regular old vanilla Notepad that comes with Windows. It will show you the encoding of the file when you click Save As....

If it's not in UTF-8, convert it and try running your script again.