Question on files

[ktsirig]

Hi,
I am using PHP to make a website related to biology. In this site I use PHP code to communicate with external programms (not in PHP, but some biological packages). These programms write their output in temporary files, which I then read using fread() or file_get_contents() functions.
My problem is that these functions seem to have a restriction as to the size of the file that they can read. Is there any other way I read big files (for files close to 20-30MBs for example)?
Please note that there is no other way for me to do this because these programs, by default, write their output to a text file, so I can't use PHP's system commands and read the output on the fly. The output file is first created and then I read it into a string and parse it according to what I need.

[grahame]

I've handled files up to 2 GBytes in size with PHP ... but there are a number of issues to consider.

1. The size of the PHP "footprint" in memory - when you have a huge data file, you cannot simply read it all in with file or file_get_contents (nor with fread, trying to read the file all at once). Rather, you need to iterate through the file in blocks - typically a line at a time, but I've also worked in 100k bloacks.

2. You are very likely to hit the fierce time limit that PHP imposes to stop a program that's looping infinitely from hogging the server for too long. You can solve that one by increasing the time limit.

3. Browsers will also time out (and users of your page get bored too) if you're not able to give a response quite quickly. Options to solve this include sending out a holding page / update periodically and (the way I did it) running my PHP analysis of the huge data file as a command line program rather than through the browser.

-- Graham

[King Kovifor]

Originally Posted by ktsirig

Hi,
I am using PHP to make a website related to biology. In this site I use PHP code to communicate with external programms (not in PHP, but some biological packages). These programms write their output in temporary files, which I then read using fread() or file_get_contents() functions.
My problem is that these functions seem to have a restriction as to the size of the file that they can read. Is there any other way I read big files (for files close to 20-30MBs for example)?
Please note that there is no other way for me to do this because these programs, by default, write their output to a text file, so I can't use PHP's system commands and read the output on the fly. The output file is first created and then I read it into a string and parse it according to what I need.

Would the include() function work better in this case? If your trying to insert it, wouldn't that be better.

[Tim356]

I think he's manipulating the data within those text files, not just including them on his page.