Importance of regular expressions on the web

1. REGULAR EXPRESSIONS, EXTRAORDINARY POWER
UNSL
2013
Burdisso Sergio - sergio.burdisso@gmail.com

2.  I have 20 min to cover all about using REs on
theW3

3.  HTTP
 Internet bots
 Web Crawler
 Web Scraping

4. HyperText Transfer Protocol

6. WWW (The Web)
Web Browser
Request
Response
HTTP
HTTP

7.  Application layer protocol
 HTTP is the protocol to exchange or transfer hypertext
Http documentation: http://www.w3.org/Protocols/rfc2616/rfc2616.html
sequences of characters

8.  HTTP Response example
Header
Body

9. EXTRAORDINARY POWER

10.  FirstThings First…
Regular ExpressionsAre
Awesome!
 Gather text
 Replace /Transform text
 Search /Validate text

11.  POSIX regular expressions (standard)
▪ ^. [ ] [^ ] (0) * {m,n} ? +|$
 regex.h
 pattern = "(d{1,3}).(d{1,3}).(d{1,3}).(d{1,3})"
 regcomp(regex_t *regex, pattern, cflags);
 regex.re_nsub = 4 //Number of parenthesized subexpressions
 regexec(regex, text, pmatch[])
 pmatch[nsub].rm_so, pmatch[nsub].rm_eo <= 255

12.  Making use of RE to parse HTTP responses headers

13. Great! Now we’re able to parse the http response headers… so what?
-We can properly process the response body!
Ah I see! … and what would I do that for?
-Let me show you!

14. Just like spiders on the web!

15. Regular Expressions cartoon from xkcd
Web Scraping
(we will see!)

16.  Internet bots (web robots,WWW robots or
bots) are software applications that run
automated tasks over the Internet
 A Web crawler is an Internet bot that
systematically browses theWorld Wide Web,
typically for the purpose ofWeb indexing
 Web scraping is a computer software technique
of extracting information from websites

17.  A Web Crawler Starts with a list of URLs to visit. As the
crawler visits these URLs, it identifies all
the hyperlinks in the page and adds them to the list of
URLs to visit
hyperlinks0 = getAllLexemes(rsp.Body, "href="((http:)?//([^/rn]*))?(/?[^"]*)"");
hyperlinks1= getAllLexemes(rsp.Body, "src="((http:)?//([^/rn]*))?(/?[^"]*)"");

18.  Web Scraping: A simple yet powerful approach to
extract information from web pages can be based on
regular expression matching facilities of programming
languages (for instance C++, Perl or Python)

19. Regular Expressions cartoon from xkcd
WebScraping wScraping (8, "http://emails.com/victim");
wScraping.findAll(
"^(?n:(?<address1>(d{1,5}( 1/[234])?(x20[A-Z]([a-z])+)+ )|(P.O. Box
d{1,5}))s{1,2}(?i:(?<address2>(((APT|B
LDG|DEPT|FL|HNGR|LOT|PIER|RM|S(LIP|PC|T(E|OP))|TRLR|UNIT)x20w{
1,5})|(BSMT|FRNT|LBBY|LOWR|OFC|PH|REAR|SIDE|UPPR).?)s{1,2})?)(?<
city>[A-Z]([a-z])+(.?)(x20[A-Z]([a-z])+){0,2}),
x20(?<state>A[LKSZRAP]|C[AOT]|D[EC]|F[LM]|G[AU]|HI|I[ADL
N]|K[SY]|LA|M[ADEHINOPST]|N[CDEHJMVY]|O[HKR]|P[ARW]|RI|S[CD]
|T[NX]|UT|V[AIT]|W[AIVY])x20(?<zipcode>(?!0{5})d{5}(-d {4})?))$"
);
We’ve saved the day!

20. Everybody stand back!
We know regular expressions
The end
Thank you for your patience!