20 Data Log File

In this example, we’ll be using the text file logfile.txt located in the data/ folder of the book’s github repository:

https://raw.githubusercontent.com/gastonstat/r4strings/master/data/logfile.txt

This file is a server log file that contains the recorded events taking place in a web server. The content of the file is in a special format known as common log format. According to wikipedia:

“The Common Log Format is a standardized text file format used by web servers when generating server log files.”

Here’s an example of a log record; the text should be in one line of code, but I’ve split it into 2 lines for readibility purposes:

pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:25 -0700] 
"GET /accesswatch/accesswatch-1.33/ HTTP/1.0" 200 1004
  • A "-" in a field indicates missing data.
  • pd9049dac.dip.t-dialin.net is the IP address of the client (remote host) which made the request to the server.
  • [01/May/2001:01:51:25 -0700] is the date, time, and time zone that the request was received, by default in strftime format%d/%b/%Y:%H:%M:%S %z.
  • "GET /accesswatch/accesswatch-1.33/ HTTP/1.0" is the request line from the client.
  • The method GET, /accesswatch/accesswatch-1.33/ is the resource requested, and HTTP/1.0 is the HTTP protocol.
  • 200 is the HTTP status code returned to the client.
    • 2xx is a successful response
    • 3xx a redirection
    • 4xx a client error, and
    • 5xx a server error
  • 1004 is the size of the object returned to the client, measured in bytes.

If you want to download a copy of the text file to your working directory (from within R) run the following code:

# download file
github <- "https://raw.githubusercontent.com/gastonstat/r4strings"
textfile <- "/master/data/logfile.txt"
download.file(url = paste0(github, textfile), destfile = "logfile.txt")

20.1 Reading the text file

The first step involves reading the data in R. How can you do this? One option is with the readLines() function which reads any text file into a character vector:

# one option is to read in the content with 'readLines()'
logs <- readLines('logfile.txt')

Let’s take a peek at the content of the vector logs:

# take a peek at the contents in logs
head(logs)
#> [1] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:25 -0700] \"GET /accesswatch/accesswatch-1.33/ HTTP/1.0\" 200 1004"              
#> [2] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:26 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"  
#> [3] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:26 -0700] \"GET /sa.inside.jpg HTTP/1.0\" 304 -"                                 
#> [4] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:19 -0700] \"GET /accesswatch/accesswatch-1.33/ HTTP/1.0\" 200 7791"              
#> [5] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:20 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"  
#> [6] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:20 -0700] \"GET /accesswatch/accesswatch-1.33/img/blueblock.gif HTTP/1.0\" 304 -"

Because the file contains 26033 lines (or elements), let’s get a subset by taking a random sample of size 50:

# subset a sample of lines
set.seed(98765)
s <- sample(1:length(logs), size = 50)
sublogs <- logs[s]

20.1.1 JPG File Requests

To begin our regex experiments, let’s try to find out “how many requests involved a JPG file?”. One way to answer the previous question is by counting the number of lines containing the pattern "jpg". We can use grep() to match or detect this pattern:

# matching "jpg" (which lements)
grep("jpg", sublogs)
#>  [1]  2  6  8 10 11 14 26 34 35 36 40 41 45

# showing value of matches
grep("jpg", sublogs, value = TRUE)
#>  [1] "hj.a11.betware.com - - [29/May/2001:02:15:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 200 18646"                     
#>  [2] "port194.ds1-gl.adsl.cybercity.dk - - [29/May/2001:15:33:04 -0700] \"GET /testing/images/archi_jsp.jpg HTTP/1.1\" 200 21211"           
#>  [3] "pd9049e9b.dip.t-dialin.net - - [06/May/2001:12:45:35 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"     
#>  [4] "pd9049e94.dip.t-dialin.net - - [04/May/2001:04:12:53 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 200 108154"
#>  [5] "edslink9.eds.com - - [23/May/2001:02:22:12 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.0\" 200 34301"                               
#>  [6] "bloodymary.vebis.de - - [22/May/2001:01:15:28 -0700] \"GET /testing/images/scope.jpg HTTP/1.0\" 200 32117"                            
#>  [7] "line210-137.iplan.com.ar - - [28/May/2001:14:49:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 304 -"                   
#>  [8] "aannecy-101-2-1-40.abo.wanadoo.fr - - [31/May/2001:08:43:33 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.1\" 200 34301"              
#>  [9] "rr-2s01.inf.fh-rhein-sieg.de - - [23/May/2001:06:40:00 -0700] \"GET /xp-r/img17.jpg HTTP/1.0\" 304 -"                                 
#> [10] "pd9541d23.dip.t-dialin.net - - [29/May/2001:07:08:26 -0700] \"GET /sa.inside.jpg HTTP/1.1\" 304 -"                                    
#> [11] "rose.ap.dregis.com - - [29/May/2001:05:09:28 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"                             
#> [12] "208.36.196.8 - - [23/May/2001:22:39:40 -0700] \"GET /testing/images/archi.jpg HTTP/1.1\" 200 18531"                                   
#> [13] "hlch0enk.htc.com - - [22/May/2001:11:01:53 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"

We can try to be more specific by defining a pattern ".jpg" in which the . corresponds to the literal dot character. To match the dot, we need to escape it with "\\.":

# we could try to be more precise and match ".jpg"
grep("\\.jpg ", sublogs, value = TRUE)
#>  [1] "hj.a11.betware.com - - [29/May/2001:02:15:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 200 18646"                     
#>  [2] "port194.ds1-gl.adsl.cybercity.dk - - [29/May/2001:15:33:04 -0700] \"GET /testing/images/archi_jsp.jpg HTTP/1.1\" 200 21211"           
#>  [3] "pd9049e9b.dip.t-dialin.net - - [06/May/2001:12:45:35 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"     
#>  [4] "pd9049e94.dip.t-dialin.net - - [04/May/2001:04:12:53 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 200 108154"
#>  [5] "edslink9.eds.com - - [23/May/2001:02:22:12 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.0\" 200 34301"                               
#>  [6] "bloodymary.vebis.de - - [22/May/2001:01:15:28 -0700] \"GET /testing/images/scope.jpg HTTP/1.0\" 200 32117"                            
#>  [7] "line210-137.iplan.com.ar - - [28/May/2001:14:49:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 304 -"                   
#>  [8] "aannecy-101-2-1-40.abo.wanadoo.fr - - [31/May/2001:08:43:33 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.1\" 200 34301"              
#>  [9] "rr-2s01.inf.fh-rhein-sieg.de - - [23/May/2001:06:40:00 -0700] \"GET /xp-r/img17.jpg HTTP/1.0\" 304 -"                                 
#> [10] "pd9541d23.dip.t-dialin.net - - [29/May/2001:07:08:26 -0700] \"GET /sa.inside.jpg HTTP/1.1\" 304 -"                                    
#> [11] "rose.ap.dregis.com - - [29/May/2001:05:09:28 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"                             
#> [12] "208.36.196.8 - - [23/May/2001:22:39:40 -0700] \"GET /testing/images/archi.jpg HTTP/1.1\" 200 18531"                                   
#> [13] "hlch0enk.htc.com - - [22/May/2001:11:01:53 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"

A similar output of grep() can be obtained with str_detect(), which allows you to detect what elements contain a match to the specified pattern:

# matching "jpg" (which lements)
str_detect(string = sublogs, pattern = "\\.jpg")
#>  [1] FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE
#> [13] FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [25] FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
#> [37] FALSE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE
#> [49] FALSE FALSE

We can do the same for PNG extensions (or for GIF or ICO):

# matching "png" (which lements)
str_detect(string = sublogs, pattern = "\\.png")
#>  [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [13] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [49] FALSE  TRUE

20.1.2 Extracting file extensions

Another common task when working with regular expressions has to do with pattern extraction. For this purposes, we can use str_extract():

# extracting "jpg" (which lements)
str_extract(string = sublogs, pattern = "\\.jpg")
#>  [1] NA     ".jpg" NA     NA     NA     ".jpg" NA     ".jpg" NA     ".jpg"
#> [11] ".jpg" NA     NA     ".jpg" NA     NA     NA     NA     NA     NA    
#> [21] NA     NA     NA     NA     NA     ".jpg" NA     NA     NA     NA    
#> [31] NA     NA     NA     ".jpg" ".jpg" ".jpg" NA     NA     NA     ".jpg"
#> [41] ".jpg" NA     NA     NA     ".jpg" NA     NA     NA     NA     NA

str_extract() actually let us confirm that we are matching the desired patterns. Notice that when there is no match, str_extract() returns a missing value NA.

20.1.3 Image files

Now let’s try to detect all types of image files: JPG, PNG, GIF, ICO

# looking for image file extensions
jpgs <- str_detect(logs, pattern = "\\.jpg ")
sum(jpgs)
#> [1] 5509

pngs <- str_detect(logs, pattern = "\\.png ")
sum(pngs)
#> [1] 1374

gifs <- str_detect(logs, pattern = "\\.gif")
sum(gifs)
#> [1] 8818

icos <- str_detect(logs, pattern = "\\.ico ")
sum(icos)
#> [1] 100

20.1.4 How to match image files with one regex pattern?

We can use character sets to define a more generic pattern. For instance, to match "jpg" or "png", we could join three character sets: "[jp][pn][g]". The first set [jp] looks for j or p, the second set [pn] looks for p or n, and the third set simply looks for g.

# matching "jpg" or "png"
jpg_png_lines <- str_detect(sublogs, "[jp][pn][g]")
sum(jpg_png_lines)
#> [1] 15

Including the dot, we can use: "\\.[jp][pn][g]"

# matching "jpg" or "png"
jpg_png_lines <- str_detect(sublogs, "\\.[jp][pn][g]")
sum(jpg_png_lines)
#> [1] 15

We could generalize the pattern to include the GIF and ICO extensions:

# matching "jpg" or "png" or "gif"
image_lines1 <- str_detect(sublogs, "[jpgi][pnic][gfo]")
sum(image_lines1)
#> [1] 44

To confirm that we are actually matching jpg, png, gif and ico, let’s use str_extract()

# are we correctly extracting image file extensions?
str_extract(sublogs, "[jpgi][pnic][gfo]")
#>  [1] "ing" "ing" NA    "ing" "ing" "ing" "ing" "jpg" "ing" "jpg" "ing" "ing"
#> [13] NA    "ing" "ing" "ing" "ing" "ing" "ing" "gif" "ing" "ico" "ing" "ing"
#> [25] "ing" "ing" "ing" NA    "pco" "ing" "gif" "ing" NA    "ing" "inf" "jpg"
#> [37] "gif" "ing" "ing" "ing" "ing" "gif" "ing" "ing" "ing" "ing" NA    NA   
#> [49] "gif" "ing"

The previous pattern does not really work as expected: note that we are matching the patterns formed by "ing" and "inf" which do not correspond to image file extensions.

An alternative way to detect JPG and PNG is by grouping patterns inside parentheses, and separating them with the metacharacter "|" which means OR:

# detecting .jpg OR .png
jpg_png <- str_detect(sublogs, "\\.jpg|\\.png")
sum(jpg_png)
#> [1] 15

Here’s how to detect all the extension in one single pattern:

# matching "jpg" or "png" or "gif" or "ico"
image_lines <- str_detect(sublogs, "\\.jpg|\\.png|\\.gif|\\.ico")
sum(image_lines)
#> [1] 28

To make sure our regex operation is successful, let’s see the output of str_extract():

images_output <- str_extract(sublogs, "\\.jpg|\\.png|\\.gif|\\.ico")
images_output
#>  [1] NA     ".jpg" NA     NA     NA     ".jpg" ".gif" ".jpg" NA     ".jpg"
#> [11] ".jpg" NA     NA     ".jpg" NA     ".png" NA     NA     NA     ".gif"
#> [21] ".gif" ".gif" NA     NA     ".gif" ".jpg" NA     NA     ".gif" NA    
#> [31] ".gif" ".gif" NA     ".jpg" ".jpg" ".jpg" ".gif" NA     NA     ".jpg"
#> [41] ".jpg" ".gif" NA     ".gif" ".jpg" ".gif" NA     NA     ".gif" ".png"

There’s some repetition with the dot character; we can modify our previous pattern by placing the dot "\\." at the beginning:

images_output <- str_extract(sublogs, "\\.jpg|png|gif|ico")
images_output
#>  [1] NA     ".jpg" NA     NA     NA     ".jpg" "gif"  ".jpg" NA     ".jpg"
#> [11] ".jpg" NA     NA     ".jpg" NA     "png"  NA     NA     NA     "gif" 
#> [21] "gif"  "ico"  NA     NA     "gif"  ".jpg" NA     NA     "gif"  NA    
#> [31] "gif"  "gif"  NA     ".jpg" ".jpg" ".jpg" "gif"  NA     NA     ".jpg"
#> [41] ".jpg" "gif"  NA     "gif"  ".jpg" "gif"  NA     NA     "gif"  "png"

Notice that the dot only appears next to ".jpg" but not with the other type of extensions. What we need to do is group the file extensions by surrounding them with parentheses:

images_output <- str_extract(sublogs, "\\.(jpg|png|gif|ico)")
images_output
#>  [1] NA     ".jpg" NA     NA     NA     ".jpg" ".gif" ".jpg" NA     ".jpg"
#> [11] ".jpg" NA     NA     ".jpg" NA     ".png" NA     NA     NA     ".gif"
#> [21] ".gif" ".gif" NA     NA     ".gif" ".jpg" NA     NA     ".gif" NA    
#> [31] ".gif" ".gif" NA     ".jpg" ".jpg" ".jpg" ".gif" NA     NA     ".jpg"
#> [41] ".jpg" ".gif" NA     ".gif" ".jpg" ".gif" NA     NA     ".gif" ".png"

Now let’s apply the pattern on the entire log file, to count the number of files of each type:

# frequencies
img_extensions <- str_extract(logs, "\\.(jpg|png|gif|ico)")
table(img_extensions)
#> img_extensions
#> .gif .ico .jpg .png 
#> 8818  100 5509 1374

Make a donation

If you find this resource useful, please consider making a one-time donation in any amount. Your support really matters.