This is not a PoSH issue. It is a pure PKI 101 (cert implementation) issue.
Self-signed certificates will always be considered untrusted in most cases, because there is no way to validate it, no public registered body for it and no public CRL (Certificate Revocation List / Authority) associated with it.
You cannot create a PKI cert for a remote location on your local machine. You must create the cert on the remote location, or buy a public cert and install it on the remote location certificate store. The public and private key must reside on the remote server / site. For any server / site, the certificate must be registered / issued to that server then manually assigned to a site (FTP/s, HTTP/s).
Then, you download the public cert from the destination and install that on your local machine. Normally installed to the local machine store. If you cannot download that public certificate and certificate chain using a browser, by clicking on the lock, after visiting the site, then you must request that the destination server/site owner send you the public cert for you to install locally. Again, normally installed to the local machine store.
I know your post is about a Filezilla server (Full Disclosure: I've never seen on used one), but the approach as shown in the articles below on setting up FTP over SSL on IIS should be similar.
FTP over SSL
The element specifies the FTP over Secure Sockets Layer (SSL)
settings for the FTP service; FTP over SSL was first introduced for
IIS 7 in FTP 7.0.
Unlike using HTTP over SSL, which requires a separate port and
connection for secure (HTTPS) communication, secure FTP communication
occurs on the same port as non-secure communication. FTP 7 supports
two different forms of FTP over SSL:
https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/ftpserver/security/ssl
https://learn.microsoft.com/en-us/iis/publish/using-the-ftp-service/using-ftp-over-ssl-in-iis-7
Update to find the FileZilla SSL guidance
Install a SSL certificate on FileZilla FTP Server
https://www.tbs-certificates.co.uk/FAQ/en/FileZilla_FTP_Server.html
Installing a certificate on an OpenSSL-based server is really similar
than doing so on Apache: Install an Apache certificate, except that
the instructions indicating the path to th files are not the same!
for FTP FileZilla server, via the interface: FileZilla Server
Option -> SSL/TLS settings:
•import the private key (.key file generated along with the CSR) in
"Private key file".
•import the certificate and the certification chain in the same file:
1) on your certificate status page, download the "file.cer" file and
the certification chain "chain-xxx.txt" 2) concatenate those two
files into one 3) import the file in "Certificate file"
How to connect FTP over SSL/TLS in FileZilla?
Create Site
Go to File >> Site Manager >> New Site.
Following are the required details to fill up.
• Host: Enter Hostname(i.e. ftp.yourdomain.com) or IP address which we
have sent in Welcome e-mail. • Port: 21 (Default FTP port is 21, you
can also keep it blank). • Protocol: FTP - File Transfer Protocol. •
Encryption: Select Required explicit FTP over TLS from dropdown list.
• Logon Type: Select Normal from the dropdown list. • User: Your FTP
username. • Password: Your FTP Password.
https://manage.accuwebhosting.com/knowledgebase/761/How-to-connect-FTP-over-SSLorTLS-in-FileZilla.html
The FileZilla wiki also talks to how to do the SSL implementation.