From andrew at digital-domain.net  Thu Jan  5 15:42:09 2023
From: andrew at digital-domain.net (Andrew Clayton)
Date: Thu, 5 Jan 2023 15:42:09 +0000
Subject: UnicodeEncodeError in Python application
In-Reply-To: <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
Message-ID: <20230105154209.13b8b2a6@kappa.digital-domain.net>

On Sun, 1 Jan 2023 18:37:29 +0100
Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:

> Hello,
> 
> this patch didn't work in our Netbox container.

OK. Here's another (replacement) one to try...

<patch>
diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c
index bdb04579..3c03252d 100644
--- a/src/python/nxt_python.c
+++ b/src/python/nxt_python.c
@@ -75,8 +75,13 @@ static nxt_python_proto_t    nxt_py_proto;
 static nxt_int_t
 nxt_python3_init_config(nxt_int_t pep405)
 {
-    PyStatus  status;
-    PyConfig  config;
+    PyStatus     status;
+    PyConfig     config;
+    PyPreConfig  preconfig;
+
+    PyPreConfig_InitIsolatedConfig(&preconfig);
+    preconfig.utf8_mode = 1;
+    Py_PreInitialize(&preconfig);
 
     PyConfig_InitIsolatedConfig(&config);
 
</patch>

Cheers,
Andrew

From tobias.genannt at kappa-velorum.net  Thu Jan  5 16:01:01 2023
From: tobias.genannt at kappa-velorum.net (Tobias Genannt)
Date: Thu, 5 Jan 2023 17:01:01 +0100
Subject: UnicodeEncodeError in Python application
In-Reply-To: <20230105154209.13b8b2a6@kappa.digital-domain.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
Message-ID: <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>

Hi Andrew,

this one fixed it for out Netbox image.

Greetings,
Tobias

On 05/01/2023 16:42, Andrew Clayton wrote:
> On Sun, 1 Jan 2023 18:37:29 +0100
> Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:
> 
>> Hello,
>>
>> this patch didn't work in our Netbox container.
> 
> OK. Here's another (replacement) one to try...
> 
> <patch>
> diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c
> index bdb04579..3c03252d 100644
> --- a/src/python/nxt_python.c
> +++ b/src/python/nxt_python.c
> @@ -75,8 +75,13 @@ static nxt_python_proto_t    nxt_py_proto;
>   static nxt_int_t
>   nxt_python3_init_config(nxt_int_t pep405)
>   {
> -    PyStatus  status;
> -    PyConfig  config;
> +    PyStatus     status;
> +    PyConfig     config;
> +    PyPreConfig  preconfig;
> +
> +    PyPreConfig_InitIsolatedConfig(&preconfig);
> +    preconfig.utf8_mode = 1;
> +    Py_PreInitialize(&preconfig);
>   
>       PyConfig_InitIsolatedConfig(&config);
>   
> </patch>
> 
> Cheers,
> Andrew

From andrew at digital-domain.net  Thu Jan  5 17:20:05 2023
From: andrew at digital-domain.net (Andrew Clayton)
Date: Thu, 5 Jan 2023 17:20:05 +0000
Subject: UnicodeEncodeError in Python application
In-Reply-To: <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
 <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
Message-ID: <20230105172005.00f25477@kappa.digital-domain.net>

On Thu, 5 Jan 2023 17:01:01 +0100
Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:

> Hi Andrew,

Hi Tobias,

> this one fixed it for out Netbox image.

Thanks for testing!

Still need to do some digging to see what exactly is happening here
and why there is a difference if we go through the new PyConfig or
not... I'm also not sure if we can unilaterally just enable UTF-8
support.

But at least I now have a pretty specific bit to look at.

Cheers,
Andrew

From andrew at digital-domain.net  Thu Jan  5 20:41:03 2023
From: andrew at digital-domain.net (Andrew Clayton)
Date: Thu, 5 Jan 2023 20:41:03 +0000
Subject: UnicodeEncodeError in Python application
In-Reply-To: <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
 <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
Message-ID: <20230105204103.2fd819b6@kappa.digital-domain.net>

On Thu, 5 Jan 2023 17:01:01 +0100
Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:

> Hi Andrew,

Hi Tobias,

> > +    PyPreConfig_InitIsolatedConfig(&preconfig);
> > +    preconfig.utf8_mode = 1;

Could you check that setting preconfig.utf8_mode to -1 still works for
you?

This means that it will use the locale settings, specifically LC_CTYPE,
to determine whether to use UTF-8 or not.

It will use UTF-8 if LC_CTYPE is either: C, POSIX or some specific
UTF-8 locale, so looking at your list of locales it should be OK.

(You can either edit the patch and re-apply it, or just edit
src/python/nxt_python.c directly)

Cheers,
Andrew

From tobias.genannt at kappa-velorum.net  Fri Jan  6 09:04:11 2023
From: tobias.genannt at kappa-velorum.net (Tobias Genannt)
Date: Fri, 6 Jan 2023 10:04:11 +0100
Subject: UnicodeEncodeError in Python application
In-Reply-To: <20230105204103.2fd819b6@kappa.digital-domain.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
 <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
 <20230105204103.2fd819b6@kappa.digital-domain.net>
Message-ID: <0d83ee7b-7ccd-4464-07f1-0d3bf56fb20d@kappa-velorum.net>

Hello Andrew,

setting preconfig.utf8_mode to -1 also works for my container. Locale is 
set like this:

LANG=C.utf8
LANGUAGE=
LC_CTYPE="C.utf8"
LC_NUMERIC="C.utf8"
LC_TIME="C.utf8"
LC_COLLATE="C.utf8"
LC_MONETARY="C.utf8"
LC_MESSAGES="C.utf8"
LC_PAPER="C.utf8"
LC_NAME="C.utf8"
LC_ADDRESS="C.utf8"
LC_TELEPHONE="C.utf8"
LC_MEASUREMENT="C.utf8"
LC_IDENTIFICATION="C.utf8"
LC_ALL=

Greetings,
Tobias

On 05/01/2023 21:41, Andrew Clayton wrote:
> On Thu, 5 Jan 2023 17:01:01 +0100
> Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:
> 
>> Hi Andrew,
> 
> Hi Tobias,
> 
>>> +    PyPreConfig_InitIsolatedConfig(&preconfig);
>>> +    preconfig.utf8_mode = 1;
> 
> Could you check that setting preconfig.utf8_mode to -1 still works for
> you?
> 
> This means that it will use the locale settings, specifically LC_CTYPE,
> to determine whether to use UTF-8 or not.
> 
> It will use UTF-8 if LC_CTYPE is either: C, POSIX or some specific
> UTF-8 locale, so looking at your list of locales it should be OK.
> 
> (You can either edit the patch and re-apply it, or just edit
> src/python/nxt_python.c directly)
> 
> Cheers,
> Andrew

From andrew at digital-domain.net  Fri Jan  6 13:06:47 2023
From: andrew at digital-domain.net (Andrew Clayton)
Date: Fri, 6 Jan 2023 13:06:47 +0000
Subject: UnicodeEncodeError in Python application
In-Reply-To: <0d83ee7b-7ccd-4464-07f1-0d3bf56fb20d@kappa-velorum.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
 <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
 <20230105204103.2fd819b6@kappa.digital-domain.net>
 <0d83ee7b-7ccd-4464-07f1-0d3bf56fb20d@kappa-velorum.net>
Message-ID: <20230106130647.5c0e2eb0@kappa.digital-domain.net>

On Fri, 6 Jan 2023 10:04:11 +0100
Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:

> Hello Andrew,

Hi Tobias,
 
> setting preconfig.utf8_mode to -1 also works for my container. Locale is 
> set like this:

Great!, thanks for checking! This looks like the way to go...

> LANG=C.utf8
> LANGUAGE=
> LC_CTYPE="C.utf8"

Yeah, that does the trick...

Cheers,
Andrew

From andrew at digital-domain.net  Thu Jan 12 18:51:24 2023
From: andrew at digital-domain.net (Andrew Clayton)
Date: Thu, 12 Jan 2023 18:51:24 +0000
Subject: UnicodeEncodeError in Python application
In-Reply-To: <20230106130647.5c0e2eb0@kappa.digital-domain.net>
References: <0ea768ea-d611-b894-d4aa-e6b58574449f@kappa-velorum.net>
 <20221221144147.222362e1@kappa.digital-domain.net>
 <8033b370-fd32-8e8a-fcde-f0745cb7af06@kappa-velorum.net>
 <20221222011923.00abaee6@kappa.digital-domain.net>
 <20221230154623.64959ec9@kappa.digital-domain.net>
 <566f4cde-fe33-4937-0ec2-0d3b16d10a52@kappa-velorum.net>
 <20230105154209.13b8b2a6@kappa.digital-domain.net>
 <a65a95e4-bb69-b225-02c4-05ecf5094ce0@kappa-velorum.net>
 <20230105204103.2fd819b6@kappa.digital-domain.net>
 <0d83ee7b-7ccd-4464-07f1-0d3bf56fb20d@kappa-velorum.net>
 <20230106130647.5c0e2eb0@kappa.digital-domain.net>
Message-ID: <20230112185124.5c41061f@kappa.digital-domain.net>

On Fri, 6 Jan 2023 13:06:47 +0000
Andrew Clayton <andrew at digital-domain.net> wrote:

> On Fri, 6 Jan 2023 10:04:11 +0100
> Tobias Genannt <tobias.genannt at kappa-velorum.net> wrote:

[...]

> > setting preconfig.utf8_mode to -1 also works for my container. Locale is 
> > set like this:  
> 
> Great!, thanks for checking! This looks like the way to go...

This fix has been merged to master
<https://github.com/nginx/unit/commit/f3d05bba525c048d8ca905831a7729d7f90c94bc>

Cheers,
Andrew