Using standard web logs

eon201 · #1 (**permalink**) Oct 30th, 2007, 09:54

Hi,

Im looking to build a php file upon my server that checks my server web logs (iis server) and displays them graphically on another page, refreshing every hour.

Has anyone played with this before or come across scripts that allow you to do this?? Any help would be greatly appreciatted, as my experience with web logs is limited!

Thanks in advance. eon201

eon201 · #2 (**permalink**) Oct 30th, 2007, 10:17

I have found a free perl script call weblog ( http://awsd.com/scripts/weblog/ ), its a perl script and seems to do the things I want it to. But how compatible are perl and php. Im thinking they work well together as php seems very very similar... Does anyone have any experience with this???

Thanks. eon201

Rakuli · #3 (**permalink**) Oct 30th, 2007, 11:48

Perl is generally the best bet for these kinds of things..

Is there any particular reason you would like to use it? There are some great log packages out there... These can do much more than you could easily write with PHP

Awstats (my favourite)..and available for IIS
Webalizer

eon201 · #4 (**permalink**) Oct 30th, 2007, 12:32

Hi again Rakuli,

What I am trying to do is to take the raw log files upon my server read them with php , calculate the values, and display information such as;

Total page views per day
No. of unique visitors
Top 5 viewed pages
Overall number of page views
What percentage of users are using certain browsers
What percentage of users are using certain os's

And include visual and audible alerts from the javascript phone validating system you helped me create.

Then print() the stats into a live page.

I know that this seems ridiculous. And many people will say 'why dont you use google analytics', but i just cant. My boss just does not want it done that way. He wants it done so that we can display it on a plasma in the office and it just refreshes itself. That way we know when an alert hits for the phone callback validation. This is the only way he wants it done

So far I can get my php to read the .txt log files . But what I really want it to do is skip the first 4 lines of code, then I need to seperate each piece of data into a variable... Is this possible? I cant seem to find any literature on how to do this..

Thanks. eon201

Rakuli · #5 (**permalink**) Oct 30th, 2007, 12:47

Can you show me an example of the raw log for your server? Then I can show you some code of how to break it up.

Cheers,

eon201 · #6 (**permalink**) Oct 30th, 2007, 12:51

quick question.... Is it safe to post that kind of information on a forum?? Can it be used in a malicious way by someone????

thanks eon201.

Rakuli · #7 (**permalink**) Oct 30th, 2007, 12:58

Just paste a few lines and replace the text -- it's just so I can see the format..

Change anything that may identify your server

I'm not out to do you any harm lol.

eon201 · #8 (**permalink**) Oct 30th, 2007, 13:04

OK so im using this code so far in my php...

Code: Select all

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="refresh" content="60" />
<title>Untitled Document</title>
</head>
<body>
<p>This should print the text file</p>
<?php
$filename = 'file.txt';
$fp = fopen($filename, "r");
$contents = fread($fp, filesize($filename));

print($contents);
fclose($fp);
?>
</body>
</html>

and my raw server log looks like this (any ip's have been * out)

Code: Select all

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-10-10 12:28:14
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) sc-status sc-win32-status sc-bytes cs-bytes time-taken 
2007-10-10 12:28:14 W3SVC10019 **.***.***.*** GET /site/familycruising/familyaccommodation/familysuites - ** - **.*.**.*** HTTP/1.0 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) - - *** * **** *** ***

There are obviously more lines than that but the first 4 are generics put in by the server, whilst any lines after the 4th one are the ones I wish to format.

Thanks, eon201

ps. You would have thought I could have had something easier to do in my first php challenge! lol

Rakuli · #9 (**permalink**) Oct 30th, 2007, 13:34

Okay, so this should be simple enough...

here is a quick script I made for you -- it is commented so you can see what is happening.

PHP: Select all


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

<meta http-equiv="refresh" content="60" />

<title>Untitled Document</title>

</head>

<body>

<p>This should print the text file</p>

<?php

$filename = 'file.txt';

$fp = fopen($filename, "r");



// make sure that the file opening was successful

if ($fp)

{

    // now cycle through one line at a time until we reach the end of the file (feof() checks end of file)

    while (!feof($fp))

    {

        $contents = fgets($fp);

        // discard commented links

        if (strpos($fp, '#') === 0)

            continue;

            

        // Now we'll create the variables to output

        // list will take the keys of an array and read them into the variables as named, the raw output is separated by spaces so we will use explode to split the string into and array

        // on the spaces..... Then you can just echo as required

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes, $timeTaken) = explode(' ', $content);

        

        // now just echo to the page howvere you want

        

        echo $date, ' - ', $time, '<br />'; // etc etc 

        

        

    }

}



fclose($fp);

?>



<script type="text/javascript">

// set the page to reload every 15 minutes

setTimeout('window.location.href = '<?php echo $_SERVER['php_self']; ?>', 900000);

</script>

</body>

</html>

eon201 · #10 (**permalink**) Oct 30th, 2007, 13:48

lol. im a noob. Thanks for that, really appreciatte it...

But why am I getting errors such as:

Code: Select all

Notice:  Undefined variable:  content in D:\home\Pla\monitor.php on line 26

and..

Code: Select all

Notice:  Undefined offset:  18 in D:\home\Pla\monitor\monitor.php on line 26

Is this becuase I have not echo'd the variables yet??

Rakuli · #11 (**permalink**) Oct 30th, 2007, 13:51

My bad sorry,

Replace

explode(' ', $content);

with

explode(' ', $contents);

eon201 · #12 (**permalink**) Oct 30th, 2007, 13:56

hmmm.. ok now its giving me the errors of 'Notice: Undefined offset' on the same line that we just edited.

And it is echo'ing data from the first four lines.

Rakuli · #13 (**permalink**) Oct 30th, 2007, 14:03

Hmm, the log file says there are 19 different fields but the actual sample log has only 18.....

try replacing

PHP: Select all


// on the spaces..... Then you can just echo as required

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes, $timeTaken) = explode(' ', $content);

with

PHP: Select all


// on the spaces..... Then you can just echo as required

        $logBits  = explode(' ', $content);

        // bit sloppy but oh well :(

        // COunt how many things there are and list accordingly

        if (count($logBits) == 19)

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes, $timeTaken) = $logBits;

        else

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes) = $logBits;

eon201 · #14 (**permalink**) Oct 30th, 2007, 14:15

your going to end up hating me rakuli! lol.

Still erroring...

Code: Select all

        $logBits  = explode(' ', $content);

        // bit sloppy but oh well :(

        // COunt how many things there are and list accordingly

        if (count($logBits) == 19)

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes, $timeTaken) = $logBits;

        else

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes) = $logBits;

This time the errors are on the first line of THIS code (Notice: Undefined variable) and the last line (Notice: Undefined offset).

Rakuli · #15 (**permalink**) Oct 30th, 2007, 14:17

Gosh! I did it again

$logBits = explode(' ', $content);

should be

$logBits = explode(' ', $contents);

eon201 · #16 (**permalink**) Oct 30th, 2007, 14:28

ok getting there slowly, but it's still erroring (Notice: Undefined offset) on this line....

Code: Select all

        list($date, $time, $sSitename, $sIp, $csMethod, $csUriStem, $csUriQuery, $sPort, $csUserName, $cIp, $csVersion, $csUserAgent, $csCookie, $csReferer, $scStatus, $scWin32Status, $scBytes, $csBytes) = $logBits;

And it is showing those first four lines of code from the txt again?? ( #Software: - Microsoft #Version: - 1.0 #Date: - 2007-10-10 #Fields: - date 2007-10-10 - 12:28:14 2007-10-10 - 12:30:11) Weird!

Rakuli · #17 (**permalink**) Oct 30th, 2007, 14:37

That list() call should really be working....

O well, scrap that!...

PHP: Select all


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

<meta http-equiv="refresh" content="60" />

<title>Untitled Document</title>

</head>

<body>

<p>This should print the text file</p>

<?php

$filename = 'file.txt';

$fp = fopen($filename, "r");



// make sure that the file opening was successful

if ($fp)

{

    // now cycle through one line at a time until we reach the end of the file (feof() checks end of file)

    while (!feof($fp))

    {

        $contents = fgets($fp);

        // discard commented links

        if (strpos($contents, '#') === 0)

            continue;

            

        // Now we'll create the variables to output

        // list will take the keys of an array and read them into the variables as named, the raw output is separated by spaces so we will use explode to split the string into and array

        // on the spaces..... Then you can just echo as required

        $logBits  = explode(' ', $contents);

        

        // cylcle though the bits of the log

        

        echo '<style="border: 1px; margin: 1em; padding: 1em">';

        

        foreach ($logBits as $bits)

        {

            echo $bits, '<br />';

        }

                                echo '</div>';

        

        

        

    }

}



fclose($fp);

?>



<script type="text/javascript">

// set the page to reload every 15 minutes

setTimeout('window.location.href = '<?php echo $_SERVER['php_self']; ?>', 900000);

</script>

</body>

</html>

That ^^ Should definitely work..

eon201 · #18 (**permalink**) Oct 30th, 2007, 14:54

yay! lol

no errors!

At this piont no that the $contents has explode(d) how can I call each part of the data seperatly?

Thanks again Rakuli. eon201

Rakuli · #19 (**permalink**) Oct 30th, 2007, 15:02

Hooray!

You can create an array with the names of each piece of data.. then write that to the screen at the same time...

Something like :

PHP: Select all


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

<meta http-equiv="refresh" content="60" />

<title>Untitled Document</title>

</head>

<body>

<p>This should print the text file</p>

<?php

$filename = 'file.txt';

$fp = fopen($filename, "r");





// an array of the possible bits



$bitNames = array('Date', 'Time', 'S-Sitename', 'CS-Method', 'CS-URI-Stem', 'CS-URI-Query', 'S-Port', 'CS-Username', 'C-IP', 'CS-Version', 'CS(User-Agent)', 'CS(Cookie)', 'CS(Referer)', 'SC-Status', 'SC-Win32-Status', 'SC-Bytes', 'Time-Taken');

// make sure that the file opening was successful

if ($fp)

{

    // now cycle through one line at a time until we reach the end of the file (feof() checks end of file)

    while (!feof($fp))

    {

        $contents = fgets($fp);

        // discard commented links

        if (strpos($contents, '#') === 0)

            continue;

            

        // Now we'll create the variables to output

        // list will take the keys of an array and read them into the variables as named, the raw output is separated by spaces so we will use explode to split the string into and array

        // on the spaces..... Then you can just echo as required

        $logBits  = explode(' ', $contents);

        

        // cylcle though the bits of the log

        

        echo '<style="border: 1px; margin: 1em; padding: 1em">';

        

        foreach ($logBits as $index => $bits)

        {

            echo '<strong>', $bitNames[$index], '</strong> : ', $bits, '<br />';

        }

		Webforumz.com > Main Forums > Program Your Website > PHP Forum
Using standard web logs