Unicode and API [Feb 24]

:raising_hand_man: Pierric asked

hi ! I’m new to NocoDB. I’ve prototyped a small N8N chain to play around, and it was working on a development deploy of NocoDB, powered by SQLite. But since I want to be able to edit table structure flexibly in the future, I’ve now deployed MariaDB and recreated my NocoDB container accordingly. Now the exact same N8N process fails at the NocoDB step with an “incorrect string value” error from MariaDB, apparently because the strings I’m inserting have some unicode characters. How can I overcome this? (I would like to keep the Unicode characters if possible) The field that fails is defined as SingleLineText.

:raising_hand_man: Pierric replied

I’m noticing that when I add text directly to a table from the GUI, unicode characters are accepted. But when it’s over the API, I get the mentioned error.

:raising_hand_man: Pierric replied

Additionally, it seems that MariaDB is using UTF-8 by default, and entries created directly from the GUI are stored correct as UTF-8. It’s just through the API that I’m getting this error 400 due to that MariaDB error.

:raising_hand_man: Pierric replied

if that is useful, I am using linuxserver.io/mariadb on Arm7, together with the official nocodb container.

:raising_hand_man: Pierric replied

I’m starting to think this is on the N8N side

:person_tipping_hand: o1lab replied

The table create API are not cross compatible with databases

:raising_hand_man: Pierric replied

Hmmm… I’m not sure I understand what you mean. Just to be clear after moving from sqlite to mariadb i didn’t reuse / migrate anything in the dB side. I created everything from scratch again.

:person_tipping_hand: o1lab replied

I reread the description, so it looks like it has to do with probably encoding or escaping those unicode? See what is the difference when compared to GUI and API

:raising_hand_man: Pierric replied

that’s where I’m at a loss. I can’t really capture the API call going out from n8n (though if I have time and can’t solve this I might write a little Flask app to simulate the nocodb API and output the n8n request… happy to hear about better solutions if they exist!). But from the n8n source code it looks like the request should be good, and when I do a call to nocodb API directly from the console with CURL, I can insert utf-8 without problem. So I don’t see where the difference can be :frowning_with_open_mouth:

:person_tipping_hand: o1lab replied

Hard to tell

:person_tipping_hand: o1lab replied

You can log in n8n I think

:person_tipping_hand: o1lab replied

Like if it’s source code node I mean

:raising_hand_man: Pierric replied

hmm … I did write a little “listener” in python and the request looks fine … here’s what the python looks like:

```python

import http.server as SimpleHTTPServer

import socketserver as SocketServer

import logging

PORT = 9876

class GetHandler(

SimpleHTTPServer.SimpleHTTPRequestHandler

):

def do_POST(self):

print(self.headers)

content_len = int(self.headers.get(‘content-length’, 0))

post_body = self.rfile.read(content_len)

print(post_body)

with open(‘output.txt’,‘wb’) as f:

f.write(post_body)

def do_GET(self):

logging.error(self.headers)

SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)

Handler = GetHandler

httpd = SocketServer.TCPServer((“”, PORT), Handler)

httpd.serve_forever()

```

and here’s the output.txt file:

```text

[{“gmailid”:“17f259931d268600”},{“gmailid”:“17f258573645e9e3”},{“gmailid”:“17f22604a66e97b3”,“subject”:“:circus_tent: “Les films doivent être des dessins qui prennent vie””},{“gmailid”:“17f2245118cccf93”},{“gmailid”:“17f222b4395f3ac4”}]

```

:raising_hand_man: Pierric replied

(I modified my n8n nodes to not return personal data but still return some unicode)

:raising_hand_man: Pierric replied

it prints on screen (through the print command before I write to file) as:

```text

b’[{“gmailid”:“17f259931d268600”},{“gmailid”:“17f258573645e9e3”},{“gmailid”:“17f22604a66e97b3”,“subject”:“\xf0\x9f\x8e\xaa \xe2\x80\x9cLes films doivent \xc3\xaatre des dessins qui prennent vie\xe2\x80\x9d”},{“gmailid”:“17f2245118cccf93”},{“gmailid”:“17f222b4395f3ac4”}]’

```

:raising_hand_man: Pierric replied

headers are as such:

```text

Accept: application/json

xc-auth: kljlk

Content-Type: application/json

User-Agent: axios/0.21.4

Content-Length: 232

Host: pi4:9876

Connection: close

```

:raising_hand_man: Pierric replied

but that seems to break mariadb when the exact same request goes to nocodb

:raising_hand_man: Pierric replied

now I’m at a loss again. I’ve tried using Curl queries again with that same text, copy/pasted directly, and nocodb returns an error again, even though with other unicode characters previously it didn’t.

:raising_hand_man: Pierric replied

ok, I got something! it depends on the actual range of the unicode character. :circus_tent: breaks nocodb+mariadb, no matter the method (curl or n8n) whereas :rewind: works all the time.

:raising_hand_man: Pierric replied

i’m not super savvy about unicode, but I’d say both are supposed to be utf-8?