1

I need some help to get this working the way I wan't.

I am trying to convert the easylist.txt to a specific Json Format and import to my Untangle Firewall.

Bellow is the PHP script that i am running to convert it. The problem is that there is a limit of about 2000 lines during the import. So i need it to create a Export file for each 2000 line. and do as many files as needed.

Any help in this matter would be assume.

Thanks WebFooL

<?php
$Content = "./easylist.txt"; 
$lines = file($Content); 
$vJsonFileName = "AD-Blocker". ".json"; 
$badcharacters = array("#",'"', "'", "[", "]", "\n", "\t"); 
header("Content-type: application/json"); 
header("Content-Disposition: attachment; filename=$vJsonFileName");
header('Cache-Control: public');
echo "["; 
foreach($lines as $str){ 
$cleanstr = str_replace($badcharacters, "", $str);
echo '{"enabled":true,"string":"'.$cleanstr.'","javaClass":"com.untangle.uvm.node.GenericRule"},'; 
}
echo "]"; 
?>
WebFooL
  • 53
  • 5
  • Why dont you try `for` loop instead of `foreach` ? – safarov Mar 24 '12 at 08:46
  • I don't understand your problem. Why not put a counter in your for cycle store the lines in an array and when you hit 2000 write them to a file and empty the array. Please give more details if this is not what you want. – slash197 Mar 24 '12 at 08:49
  • It was a few years ago that i work with PHP or programming for that matter so i am a bit "rusty". Both sounds like possible ways. – WebFooL Mar 24 '12 at 08:54

2 Answers2

0

Assuming that you can just arbitrarily split the input data (i.e. that entries don't span multiple lines and each line is "atomic") then you can just extract the number of lines you need

$Content = "./easylist.txt"; 
$lines = file($Content); 
$linesSplit = array_chunk( $lines, 2000 );

BTW, do you really need to remove the "bad characters"? Have you investigated using json_encode?

liquorvicar
  • 6,081
  • 1
  • 16
  • 21
  • Will test this and get back to you. Thanks.. As for the bad characters they need to be removed do to the import script. But it might be that i am removing a few extra that aren't needed to be removed. – WebFooL Mar 24 '12 at 08:57
  • Ahh looking to the changes and i think I miss led you in the first post. I need to get the first 2000 line in to Exportfile1.json Secound 2000 lines in to ExportFile2.json and so on and the last file should print as many there are left. – WebFooL Mar 24 '12 at 09:00
  • @liquarvicar, Thanks for the feedback.. Your help was is what is moving me forward right now but i am haivng a issue with having it "export" multiple files so i will have to look at it. Again thanks. – WebFooL Mar 24 '12 at 09:29
  • @WebFooL Can you not just loop through the resulting array and save each array to a separate file? It looks like your script is outputing a file to the browser? That won't work with multiple files. If you need to output them that way see Andy Gee's answer. – liquorvicar Mar 24 '12 at 10:27
0

You can't download multiple files so the following will create multiple files. There's a post here about zipping the files: How to [recursively] Zip a directory in PHP?

<?php
$Content = "./easylist.txt"; 
$lines = file($Content); 
$vJsonFileName = "AD-Blocker.zip"; 
$badcharacters = array("#",'"', "'", "[", "]", "\n", "\t"); 
$f = 1;
foreach($lines as $str)
{
    $c++;
    if($c==2000)
    {
        $f++;
    }
    $cleanstr = str_replace($badcharacters, "", $str);
    $json[$f] = '{"enabled":true,"string":"'.$cleanstr.'","javaClass":"com.untangle.uvm.node.GenericRule"},'; 
}

foreach($json as $f => $arr)
{
    file_put_contents('file'.$f,'['.implode('',$arr).']');
}


//zip up all the files here to $vJsonFileName


header("Content-type: application/json"); 
header("Content-Disposition: attachment; filename=$vJsonFileName");
header('Cache-Control: public');
?>
Community
  • 1
  • 1
Andy Gee
  • 3,149
  • 2
  • 29
  • 44
  • Thanks will have to look if i should write them all in to one file but just separate lines or have to do some thing with the zip file. – WebFooL Mar 24 '12 at 09:16
  • doensn't scale well - you store the files full content in memory twice, plus overhead. Would be better to process the file line by line (`fopen()` + `fgets()`) – Kaii Mar 24 '12 at 11:42
  • Kaii - Admittedly it's not very scalable at all. I wouldn't imagine that's a problem though as it's just for importing firewall rules and not likely to be used by many people simultaneously very large. If the file is large or memory is an issue `fopen()` + `fgets()` would be a much better solution. Then again I wouldn't want to clog up my firewall with that many rules. – Andy Gee Mar 24 '12 at 22:54