[SOLVED] Is this a suitable 'search CSV' function?

[Aso]

I've written a very basic search CSV function that finds the line with the matching 'searched' area code and prints out the corresponding area name (Brighton, Tonbridge etc.)

The CSV file is a downloaded list of postcodes in the format

Area Code, Area Name on each line (and it's quite large with 2,821 rows)

So is this 'efficient' enough to search such a file, or is there a more effective (or quicker) way?

PHP: Select all

<?php
function searchPostcode($areaCode) {
$file = fopen("postcodes.csv", "rb");
    while (!feof($file) ) {
    $line = fgets($file);
    $parts = explode(',', $line);
        if (in_array($areaCode,$parts)) {
        print 'Area Code: '.$parts[0].' Name: '.$parts[1];
        }
    }
fclose($file);
}
?>

Many thanks

[hschmitz]

Originally Posted by aso186

So is this 'efficient' enough to search such a file, or is there a more effective (or quicker) way?

This clearly depends on your application.

Reading the file line by line is probably a bit slow. You could try reading the entire file into an array using the file function and then iterating through the array.

If you really need to speed things up, you should consider storing the postcodes in a database instead of in a file.

[c010depunkk]

Yup, like hschmitz said, the most efficient way would be to import the post codes into a database.

If you want to stick with files, then I have to disagree with hschmitz. I think line by line is faster and better programming practice. Let PHP handle reading the file with a stream and pointer. This way, when you find the post code you are searching for on the 10th line, you can just delete the file resource and continue. It's unnecessary to load the whole file into an array ... you're just filling up your server RAM and re-inventing the wheel - fopen, fgets, feof and fclose are optimized for doing exactly this....

[hschmitz]

Originally Posted by c010depunkk

I think line by line is faster and better programming practice. Let PHP handle reading the file with a stream and pointer.

In terms of cleaner programming, I agree that reading the file line by line is better practice.

Concerning speed, my recollection is that IO functions normally slow down the code. It is normally a lot faster to read a big block of data into memory once and then performing operations on it. Multiple calls to fgets will tend to slow the system down.

This is my experience with a number of different programming languages. I haven't tested this with PHP, but I will do this first thing tomorrow. I'll let you know my results.

[Aso]

Hey All

Thanks for the feedback . I ran a few primitive tests and the times weren't bad for line by line, plus fair point c010depunkk regarding 10th line etc., so I stuck with .

I agree about the db but I just don't have that facility with this hosting package, so it's the next best thing really.

Cheers

Alex

[c010depunkk]

Originally Posted by hschmitz

I'll let you know my results.

Please do. I'd be interested....

[hschmitz]

Ok, I've run some benchmarks and here are my results.


I've tested three different file sizes: The small file contains 300 records, the medium file contains 3000 records and the large file contains 30000 records. The data in the files was generated randomly. Each line consisted of a three letter "postcode"followed by a comma followed by a 15 letter name.

Each file was searched 10 times to get some kind of average value. I have taken the time used by the function call using the microtime function.

I found that, especially for the line by line reading, the time needed for the first run was significantly longer than for the following runs. This is probably because the system caches the file after the first read.

So here are the numbers (times are in seconds):

Code: Select all

First Read:

        line         file
short:    0.014614     0.001286
medium: 0.049765     0.011119
long:   0.121069     0.109896


FollowingReads (avg over 9 runs):

        line         file
short:  0.00161      0.00171    
medium: 0.015044     0.029462
long:   0.211312     0.195516

Judging from these results, I have to admit that is doesn't really matter
which method is chosen in terms of speed. If the search is performed often the data will be cached. If the search is performed only once in a while, it doesn't matter if it is a bit slower. Let the system judge which data to buffer in the memory.

[c010depunkk]

Hey, kool!!!

Thanks a lot. Really interesting!!!

[alexgeek]

That is quite interesting, I didn't know about the microtime function. Thanks!