I don't know if there is a specific tool that does exactly what you need (file encoding detection inside SQL Server).
However, you can write one that does the job inside SQL Server as per your requirements. The file encoding detection can easily be done in C# and IMHO a more elegant solution than trying to write one in SQL, if possible at all.
So you should write a C# function that does that and then host it inside SQL Server for execution.
Here is a post that explains how to detect any encoding in a file
Effective way to find any file's Encoding
And here is a blog post that explains how to register a C# method and execute it from SQL Server
http://www.codemahek.com/blog/execute-c-methods-from-sql-server
There should be enough information here for you to do what you want.
Welcome to Stackoverflow!