# download file
<- "https://raw.githubusercontent.com/gastonstat/strings-data"
github <- "/main/data/logfile.txt"
textfile download.file(url = paste0(github, textfile), destfile = "logfile.txt")
20 Data Log File
In this example, we’ll be using the text file logfile.txt
located in the data/
folder of the book’s github repository:
https://raw.githubusercontent.com/gastonstat/strings-data/main/data/logfile.txt
This file is a server log file that contains the recorded events taking place in a web server. The content of the file is in a special format known as common log format.
https://en.wikipedia.org/wiki/Common_Log_Format
According to wikipedia:
“The Common Log Format is a standardized text file format used by web servers when generating server log files.”
Here’s an example of a log record; the text should be in one line of code, but I’ve split it into 2 lines for readibility purposes:
pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:25 -0700]
"GET /accesswatch/accesswatch-1.33/ HTTP/1.0" 200 1004
- A
"-"
in a field indicates missing data. pd9049dac.dip.t-dialin.net
is the IP address of the client (remote host) which made the request to the server.[01/May/2001:01:51:25 -0700]
is the date, time, and time zone that the request was received, by default in strftime format%d/%b/%Y:%H:%M:%S %z
."GET /accesswatch/accesswatch-1.33/ HTTP/1.0"
is the request line from the client.- The method
GET, /accesswatch/accesswatch-1.33/
is the resource requested, andHTTP/1.0
is the HTTP protocol. 200
is the HTTP status code returned to the client.2xx
is a successful response3xx
a redirection4xx
a client error, and5xx
a server error
1004
is the size of the object returned to the client, measured in bytes.
If you want to download a copy of the text file to your working directory (from within R) run the following code:
20.1 Reading the text file
The first step involves reading the data in R. How can you do this? One option is with the readLines()
function which reads any text file into a character vector:
# one option is to read in the content with 'readLines()'
<- readLines('logfile.txt') logs
Let’s take a peek at the content of the vector logs
by running head(logs)
. We display the output of the first three lines below::
# take a peek at the contents in logs
head(logs)
[1] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:25 -0700] \"GET /accesswatch/accesswatch-1.33/ HTTP/1.0\" 200 1004"
[2] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:26 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"
[3] "pd9049dac.dip.t-dialin.net - - [01/May/2001:01:51:26 -0700] \"GET /sa.inside.jpg HTTP/1.0\" 304 -"
[4] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:19 -0700] \"GET /accesswatch/accesswatch-1.33/ HTTP/1.0\" 200 7791"
[5] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:20 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"
[6] "pd9049dac.dip.t-dialin.net - - [01/May/2001:02:20:20 -0700] \"GET /accesswatch/accesswatch-1.33/img/blueblock.gif HTTP/1.0\" 304 -"
Because the file contains 26033 lines (or elements), let’s get a subset by taking a random sample of size 50:
# subset a sample of lines
set.seed(98765)
<- sample(1:length(logs), size = 50)
s <- logs[s] sublogs
20.1.1 JPG File Requests
To begin our regex experiments, let’s try to find out “how many requests involved a JPG file?”. One way to answer the previous question is by counting the number of lines containing the pattern "jpg"
. We can use grep()
to match or detect this pattern:
# matching "jpg" (which lements)
grep("jpg", sublogs)
[1] 2 6 8 10 11 14 26 34 35 36 40 41 45
# showing value of matches
grep("jpg", sublogs, value = TRUE)
[1] "hj.a11.betware.com - - [29/May/2001:02:15:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 200 18646"
[2] "port194.ds1-gl.adsl.cybercity.dk - - [29/May/2001:15:33:04 -0700] \"GET /testing/images/archi_jsp.jpg HTTP/1.1\" 200 21211"
[3] "pd9049e9b.dip.t-dialin.net - - [06/May/2001:12:45:35 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"
[4] "pd9049e94.dip.t-dialin.net - - [04/May/2001:04:12:53 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 200 108154"
[5] "edslink9.eds.com - - [23/May/2001:02:22:12 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.0\" 200 34301"
[6] "bloodymary.vebis.de - - [22/May/2001:01:15:28 -0700] \"GET /testing/images/scope.jpg HTTP/1.0\" 200 32117"
[7] "line210-137.iplan.com.ar - - [28/May/2001:14:49:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 304 -"
[8] "aannecy-101-2-1-40.abo.wanadoo.fr - - [31/May/2001:08:43:33 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.1\" 200 34301"
[9] "rr-2s01.inf.fh-rhein-sieg.de - - [23/May/2001:06:40:00 -0700] \"GET /xp-r/img17.jpg HTTP/1.0\" 304 -"
[10] "pd9541d23.dip.t-dialin.net - - [29/May/2001:07:08:26 -0700] \"GET /sa.inside.jpg HTTP/1.1\" 304 -"
[11] "rose.ap.dregis.com - - [29/May/2001:05:09:28 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"
[12] "208.36.196.8 - - [23/May/2001:22:39:40 -0700] \"GET /testing/images/archi.jpg HTTP/1.1\" 200 18531"
[13] "hlch0enk.htc.com - - [22/May/2001:11:01:53 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"
We can try to be more specific by defining a pattern ".jpg"
in which the .
corresponds to the literal dot character. To match the dot, we need to escape it with "\\."
:
# we could try to be more precise and match ".jpg"
grep("\\.jpg ", sublogs, value = TRUE)
[1] "hj.a11.betware.com - - [29/May/2001:02:15:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 200 18646"
[2] "port194.ds1-gl.adsl.cybercity.dk - - [29/May/2001:15:33:04 -0700] \"GET /testing/images/archi_jsp.jpg HTTP/1.1\" 200 21211"
[3] "pd9049e9b.dip.t-dialin.net - - [06/May/2001:12:45:35 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 304 -"
[4] "pd9049e94.dip.t-dialin.net - - [04/May/2001:04:12:53 -0700] \"GET /accesswatch/accesswatch-1.33/img/allifou.jpg HTTP/1.0\" 200 108154"
[5] "edslink9.eds.com - - [23/May/2001:02:22:12 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.0\" 200 34301"
[6] "bloodymary.vebis.de - - [22/May/2001:01:15:28 -0700] \"GET /testing/images/scope.jpg HTTP/1.0\" 200 32117"
[7] "line210-137.iplan.com.ar - - [28/May/2001:14:49:26 -0700] \"GET /testing/images/archi_servlet.jpg HTTP/1.1\" 304 -"
[8] "aannecy-101-2-1-40.abo.wanadoo.fr - - [31/May/2001:08:43:33 -0700] \"GET /testing/images/pshtk.jpg HTTP/1.1\" 200 34301"
[9] "rr-2s01.inf.fh-rhein-sieg.de - - [23/May/2001:06:40:00 -0700] \"GET /xp-r/img17.jpg HTTP/1.0\" 304 -"
[10] "pd9541d23.dip.t-dialin.net - - [29/May/2001:07:08:26 -0700] \"GET /sa.inside.jpg HTTP/1.1\" 304 -"
[11] "rose.ap.dregis.com - - [29/May/2001:05:09:28 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"
[12] "208.36.196.8 - - [23/May/2001:22:39:40 -0700] \"GET /testing/images/archi.jpg HTTP/1.1\" 200 18531"
[13] "hlch0enk.htc.com - - [22/May/2001:11:01:53 -0700] \"GET /testing/images/archi.jpg HTTP/1.0\" 200 18531"
A similar output of grep()
can be obtained with str_detect()
, which allows you to detect what elements contain a match to the specified pattern:
# matching "jpg" (which lements)
str_detect(string = sublogs, pattern = "\\.jpg")
[1] FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE
[13] FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[25] FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE
[37] FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE
[49] FALSE FALSE
We can do the same for PNG extensions (or for GIF or ICO):
# matching "png" (which lements)
str_detect(string = sublogs, pattern = "\\.png")
[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[13] FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[49] FALSE TRUE
20.1.2 Extracting file extensions
Another common task when working with regular expressions has to do with pattern extraction. For this purposes, we can use str_extract()
:
# extracting "jpg" (which lements)
str_extract(string = sublogs, pattern = "\\.jpg")
[1] NA ".jpg" NA NA NA ".jpg" NA ".jpg" NA ".jpg"
[11] ".jpg" NA NA ".jpg" NA NA NA NA NA NA
[21] NA NA NA NA NA ".jpg" NA NA NA NA
[31] NA NA NA ".jpg" ".jpg" ".jpg" NA NA NA ".jpg"
[41] ".jpg" NA NA NA ".jpg" NA NA NA NA NA
str_extract()
actually let us confirm that we are matching the desired patterns. Notice that when there is no match, str_extract()
returns a missing value NA
.
20.1.3 Image files
Now let’s try to detect all types of image files: JPG, PNG, GIF, ICO
# looking for image file extensions
<- str_detect(logs, pattern = "\\.jpg ")
jpgs sum(jpgs)
[1] 5509
<- str_detect(logs, pattern = "\\.png ")
pngs sum(pngs)
[1] 1374
<- str_detect(logs, pattern = "\\.gif")
gifs sum(gifs)
[1] 8818
<- str_detect(logs, pattern = "\\.ico ")
icos sum(icos)
[1] 100
20.1.4 How to match image files with one regex pattern?
We can use character sets to define a more generic pattern. For instance, to match "jpg"
or "png"
, we could join three character sets: "[jp][pn][g]"
. The first set [jp]
looks for j
or p
, the second set [pn]
looks for p
or n
, and the third set simply looks for g
.
# matching "jpg" or "png"
<- str_detect(sublogs, "[jp][pn][g]")
jpg_png_lines sum(jpg_png_lines)
[1] 15
Including the dot, we can use: "\\.[jp][pn][g]"
# matching "jpg" or "png"
<- str_detect(sublogs, "\\.[jp][pn][g]")
jpg_png_lines sum(jpg_png_lines)
[1] 15
We could generalize the pattern to include the GIF and ICO extensions:
# matching "jpg" or "png" or "gif"
<- str_detect(sublogs, "[jpgi][pnic][gfo]")
image_lines1 sum(image_lines1)
[1] 44
To confirm that we are actually matching jpg
, png
, gif
and ico
, let’s use str_extract()
# are we correctly extracting image file extensions?
str_extract(sublogs, "[jpgi][pnic][gfo]")
[1] "ing" "ing" NA "ing" "ing" "ing" "ing" "jpg" "ing" "jpg" "ing" "ing"
[13] NA "ing" "ing" "ing" "ing" "ing" "ing" "gif" "ing" "ico" "ing" "ing"
[25] "ing" "ing" "ing" NA "pco" "ing" "gif" "ing" NA "ing" "inf" "jpg"
[37] "gif" "ing" "ing" "ing" "ing" "gif" "ing" "ing" "ing" "ing" NA NA
[49] "gif" "ing"
The previous pattern does not really work as expected: note that we are matching the patterns formed by "ing"
and "inf"
which do not correspond to image file extensions.
An alternative way to detect JPG and PNG is by grouping patterns inside parentheses, and separating them with the metacharacter "|"
which means OR:
# detecting .jpg OR .png
<- str_detect(sublogs, "\\.jpg|\\.png")
jpg_png sum(jpg_png)
[1] 15
Here’s how to detect all the extension in one single pattern:
# matching "jpg" or "png" or "gif" or "ico"
<- str_detect(sublogs, "\\.jpg|\\.png|\\.gif|\\.ico")
image_lines sum(image_lines)
[1] 28
To make sure our regex operation is successful, let’s see the output of str_extract()
:
<- str_extract(sublogs, "\\.jpg|\\.png|\\.gif|\\.ico")
images_output images_output
[1] NA ".jpg" NA NA NA ".jpg" ".gif" ".jpg" NA ".jpg"
[11] ".jpg" NA NA ".jpg" NA ".png" NA NA NA ".gif"
[21] ".gif" ".gif" NA NA ".gif" ".jpg" NA NA ".gif" NA
[31] ".gif" ".gif" NA ".jpg" ".jpg" ".jpg" ".gif" NA NA ".jpg"
[41] ".jpg" ".gif" NA ".gif" ".jpg" ".gif" NA NA ".gif" ".png"
There’s some repetition with the dot character; we can modify our previous pattern by placing the dot "\\."
at the beginning:
<- str_extract(sublogs, "\\.jpg|png|gif|ico")
images_output images_output
[1] NA ".jpg" NA NA NA ".jpg" "gif" ".jpg" NA ".jpg"
[11] ".jpg" NA NA ".jpg" NA "png" NA NA NA "gif"
[21] "gif" "ico" NA NA "gif" ".jpg" NA NA "gif" NA
[31] "gif" "gif" NA ".jpg" ".jpg" ".jpg" "gif" NA NA ".jpg"
[41] ".jpg" "gif" NA "gif" ".jpg" "gif" NA NA "gif" "png"
Notice that the dot only appears next to ".jpg"
but not with the other type of extensions. What we need to do is group the file extensions by surrounding them with parentheses:
<- str_extract(sublogs, "\\.(jpg|png|gif|ico)")
images_output images_output
[1] NA ".jpg" NA NA NA ".jpg" ".gif" ".jpg" NA ".jpg"
[11] ".jpg" NA NA ".jpg" NA ".png" NA NA NA ".gif"
[21] ".gif" ".gif" NA NA ".gif" ".jpg" NA NA ".gif" NA
[31] ".gif" ".gif" NA ".jpg" ".jpg" ".jpg" ".gif" NA NA ".jpg"
[41] ".jpg" ".gif" NA ".gif" ".jpg" ".gif" NA NA ".gif" ".png"
Now let’s apply the pattern on the entire log file, to count the number of files of each type:
# frequencies
<- str_extract(logs, "\\.(jpg|png|gif|ico)")
img_extensions table(img_extensions)
img_extensions
.gif .ico .jpg .png
8818 100 5509 1374