To me it doesn't seem reasonable to compress a string using UTF-8 as the destination... It looks like just looking for trouble. I think it would be better to lose some compression and using plain 7-bit ASCII as the destination if over-the-wire size is important.
If the storage limit is based on UTF-16 characters then a large safe subset could be looked for if you care about escaping or UTF-16 compliance or you could just try to use each char as 0..65535 if everything else involved (e.g. databases) don't have problems.
Most software layers should have no problems with that (ab)use but note that in UTF-16 range 0xD800-0xDFFF is reserved for a special use (surrogate pairs) so some combinations are formally "encoding errors" and could in theory be stopped or distorted.
In a toy 4 KB JavaScript demo I wrote for fun I used an encoding for the result of compression that stores four binary bytes into five chars chosen from a subset of ASCII of 85 chars that is clean for embedding in a JavaScript string (85^5 is slightly more than (2^8)^4, but still fits in the precision of JavaScript integers). This makes compressed data safe for example for JSON without need of any escaping.
In code the following builds the list of 85 "safe" characters:
let cset = "";
for (let i=35; i<35+85+1; i++) {
if (i !== 92) cset += String.fromCharCode(i);
}
Then to encode 4 bytes (b0, b1, b2 and b3 each from 0...255) into 5 characters the code is:
// First convert to 0...4294967295
let x = ((b0*256 + b1)*256 + b2)*256 + b3;
// Then convert to base 85
let result = "";
for (let i=0; i<5; i++) {
let x2 = Math.floor(x / 85);
result += cset[x - x2*85];
x = x2;
}
To decode you do the reverse, i.e. compute x from the base-85 number and then extract the 4 base-256 digits (i.e. the bytes).
NOTE: in the torus code I used a slightly different charset, instead of skipping 92 \ I replaced it with 126 ~. For who is interested the full decompression code is
// There are two Huffman-encoded code streams
// T - single chars (0..127) and sequence lengths (128...255)
// A - high bits of relative addresses of sequence (0..255)
//
// Expansion algorithm is:
// 1) Read a code X from T
// 2) If it's a char (X < 128) then add to output
// 3) otherwise (X>=128) read sequence address ADDR from stream A (high bits)
// and from input (low bits) and copy X-128 bytes from ADDR bytes "ago"
//
let Z = 5831; // expanded size
let i = 0, // source ptr
a = 0, // current bits accumulator
n = 0; // number of available bits in a
// Read a single bit
let b = function(){
if (!n) {
// There are no more bits available in the accumulator, read a new chunk:
// 5 ASCII escape-safe chars will be transformed in 4 8-bit binary bytes
// (like BASE64, just a bit more dense)
a = 0;
let w = 5;
while (w--) {
let y = s.charCodeAt(i+w); // get next char
a = a*85 + (y > 125 ? 92 : y) - 35; // extract base-85 "digit" (note, uses ~ instead of \ that requires quoting)
}
n = 32; // we got 32 bits in a
i += 5; // we consumed 5 characters from source
}
return (a >> --n) & 1; // extract a single bit
};
// Read a code of z bits by concatenating bits coming from b()
let v = function(z){
return (--z ? v(z) : 0)*2+b();
};
// Read an Huffman (sub-)tree: a bit will tell if we need to
// read a two sub-trees or a leaf
let h = function(){
return b() ? [h(), h()] : v(8);
};
// Read A and T Huffman trees
let A = h(), T = h();
// Extract a code given a node:
// if the node is an array (intermediate node) then we need to read a bit
// from the input binary stream to decide which way to go down the tree,
// if it's a number then we just return the value.
// `n.map` is truthy for arrays and falsy for numbers.
let d = function(n){
return n.map ? d(n[b()]) : n;
};
let S=""; // Output
// While we're not done
while(S.length<Z){
// Extract a code from T
x = d(T);
if (x < 128) {
// This is a single character, copy to output
S += String.fromCharCode(x);
} else {
// This is a sequence of x-128 bytes, get address and copy it
// Note: high 8 bits are from the Huffman tree A and 8 low bits
// are instead directly form the bit stream as they're basically
// noise and there's nothing to gain by trying to compress them.
S += S.substr(S.length-(d(A)<<8)-v(8), x-128)
};
}
(note that I dind't test this reformatted/commented version, typos may be present)
