Web Scraping: Applications and Tools

WEBSCRAPING,APPLICATIONSANDTOOLS
EuropeanPublicSectorInformationPlatform
TopicReportNo.2015/10
WebScraping:ApplicationsandTools
Author:OsmarCastrillo-Fernández
Published:December2015
ePSIplatform Topic Report No. 2015 / 10 , December 2015
1
WEBSCRAPING,APPLICATIONSANDTOOLS
TableofContents
TableofContents......................................................................................................................2
Keywords...................................................................................................................................3
Abstract/ExecutiveSummary...................................................................................................3
1 Introduction............................................................................................................................4
2 Whatiswebscraping?............................................................................................................6
ThenecessitytoscrapewebsitesandPDFdocuments........................................................6
TheAPImana........................................................................................................................7
3 Webscrapingtools.................................................................................................................8
PartialtoolsforPDFextraction.............................................................................................8
PartialtoolstoextractfromHTML........................................................................................9
Completetools....................................................................................................................12
Confrontingthethreetools.................................................................................................25
Othertools..........................................................................................................................26
4 Decisionmap........................................................................................................................27
5 Scrapingandlegislation........................................................................................................28
6 Conclusionsandrecommendations......................................................................................29
AbouttheAuthor....................................................................................................................30
Copyrightinformation.............................................................................................................31
ePSIplatform Topic Report No. 2015 / 10 , December 2015
2
WEBSCRAPING,APPLICATIONSANDTOOLS
Keywords
web scraping, data extracting, web content extracting, data mining, data harvester, crawler,
opengovernmentdata
Abstract/ExecutiveSummary
Internetisthevastestinformationanddatasourceeverbuiltbymankind.However,itisahuge
collectionofheterogeneousandpoorlystructureddata,difficulttocollectinamanualwayand
complicatedtouseinautomatedprocesses.
Overthelastyears,techniquesandtoolshavesurged,allowingdatacollectionandconversion
tostructureddatatobemanagedbyB2CandB2Bsystems.Thisarticleoffersanintroduction
to web scraping techniques and some of the most popular and novel techniques for data
extractionandreuseincomplexprocesses.
Thepossibilitiestotakebenefitofsuchdataaremany,includingareaslikeOpenGovernment
Data, Big Data, Business Intelligence, aggregators and comparators, development of new
applicationsandmashups,amongothers.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
3
WEBSCRAPING,APPLICATIONSANDTOOLS
1 Introduction
Every single day, several petabytes of information are published via the Internet in various
formats,suchasHTML,PDF,CSVorXML.Curiously,HTMLisnotnecessarilythemostcommon
format to publish content. For instance, 70% of the content indexed by Google is extracted
from PDF documents. This is an additional obstacle for the different roles involved in data
extraction from multiple sources. Journalists, researchers, data analysts, sales agents or
software developers are some examples of professionals typically using the copy-and-paste
techniquetogetinformationinaspecificformatandexportittoaspreadsheet,anexecutive
reportortosomedataexchangeformatsuchasXML,JSONoranyoftheseveralvocabularies
availablebasedonthem.
Focusing on data acquisition from HTML (i.e, a web document), this article explains the
mechanismsandtoolsthatmayhelpustominimizethetediousdutiesofdataextractionfrom
theInternet.
With regards to the commitment of transparency and data openness that public
administrations have assumed over the last decade, scraping and crawling are techniques
that may be useful. Whereas current information systems in use by public administrations
ePSIplatform Topic Report No. 2015 / 10 , December 2015
4
WEBSCRAPING,APPLICATIONSANDTOOLS
considerthesetechniques,arelevantamountofwebsites,contentmanagementsystemsand
ECMs(EnterpriseContentManagement)donot.Exchangingthesesystemsfornewonesimply
considerable economic efforts. Scraping tools provide an alternative to exchanging which
minimizes such efforts. Open Government Data and Transparency Policies should take
advantageofthisopportunity.
extraction
transformation
reuse
ePSIplatform Topic Report No. 2015 / 10 , December 2015
5
WEBSCRAPING,APPLICATIONSANDTOOLS
2 Whatiswebscraping?
One of the many definitions of this concept, and the favourite one for the author of this
document,is:
Awebscrapingtoolisatechnologysolutiontoextractdatafromwebsites,inaquick,efficientand
automatedmanner,offeringdatainamorestructuredandeasiertouseformat,eitherforB2Borfor
B2Cprocesses.
Scrapingprocessesmaybewrittenindifferentprogramminglanguages.Themostpopularare
Java,Python,RubyorNode.Asitisobvious,expertprogrammersarerequiredtodevelopand
evolve them, and even to use them. Nonetheless, some software companies have designed
differenttoolsthatenableotherpeopletousescrapingtechniquesbymeansofattractiveand
powerfuluserinterfaces.
WebscrapingtoolsarealsoreferredasWebDataExtractors,DataHarvesters,CrawlingTools
orWebContentMiningTools.
ThenecessitytoscrapewebsitesandPDFdocuments
Asalreadystated,approximately70%oftheinformationgeneratedintheInternetisavailable
in PDF documents, an unstructured and hard to handle format. However, a web page has a
structuredformat(HTMLcode),althoughinanon-reusableway.
PDFscrapingisnottheobjectoftheanalysisofthisarticle,althoughitistruethatsometools
exist to extract information, mainly related to data tables. This enormous amount of
informationpublishedbutcaptiveofthiskindofformatisusuallycalled“thetyrannyofPDF”.
SometoolsthatarepresentedinlatersectionsofthisdocumentcanreadPDFdocumentsand
returninformationinastructuredformat,althoughinabasicandrudimentaryway.
Following with the main scope of this document (HTML documents), its structured nature
multipliesthepossibilitiesopenbyscrapingtechniques.Webscrapingtechniquesandscraping
tools rely in the structure and properties of the HTML language. This facilitates the task of
scrapingfortoolsandrobots,stoppinghumansfromtheboringrepetitiveanderrorproneduty
ofmanualdataretrieval.Finally,thesetoolsofferdatainfriendlyformatsforlaterprocessing
andintegration:JSON,XML,CSV,XLSoRSS.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
6
WEBSCRAPING,APPLICATIONSANDTOOLS
TheAPImana
Data collection in a handcrafted way is truly inefficient: search, copy and paste data in a
spreadsheet to later process them. This is a tedious, annoying and tiresome process.
Therefore, it makes much more sense to automate this process. Scraping allows this kind of
automation, as the majority of the available tools provide an API to facilitate access to the
contentgenerated.AnAPI(ApplicationProgrammingInterface)isamechanismtoconnecttwo
applications,allowingthemtosharedata.ScrapingtoolsfacilitateaURL,anInternetaddressas
thoseyoumaynoticeintheaddressbarofawebbrowser,givingaccesstothedatascraped.In
some cases, APIs are not only limited to a URL. They can also be programmed in any way to
modify the final result of the scraping process. This feature is absolutely useful for B2B
integrationprocesses,enablingsubsequentapplicationsandservicesbasedonsuchdata.
Therefore,itisrecommendedthatagoodscrapingtoolprovidesasimpleandprogrammable
API.TheconceptofAPIisveryrelevantinthistopic.Occasionally,thistermisalsoknownas
EndPoint.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
7
WEBSCRAPING,APPLICATIONSANDTOOLS
3 Webscrapingtools
Once that the basic concepts related to scraping have been commented, this documents
focusesontoolscurrentlyavailableinthemarket.Basicinformationabouttheirfunctionality
andhowtheyworkisprovided,avoidingdeeptechnicaldetailsasmuchaspossible.
Firstly,wesuggestaninitialtoolsegmentation:
1. Partialtools.Thesearetypicallypluginstothird-partysoftware.Theyusuallyfocuson
a specific scraping technique(forinstance,HTMLtables).TheydonotprovideanAPI
forB2Bintegration.
2. Complete tools. This segment includes the latest tools in the market and some older
tools, created as a general scraping service. They offer features such as powerful
graphical user interface, visual scraping utility, SaaS and/or desktop licensing models,
querycachingandrecording,APIsorreportingandauditdashboards.
PartialtoolsforPDFextraction
This segment includes these tools oriented to read PDF documents and extract all or part of
theinformationcontained.
WithinthistypemayincludethoseorientedexclusivelytoopenPDFsandextractallorpartof
itsinformation.Thefigurebelowprovidesabriefdescriptionofsomeofthesetools.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
8
WEBSCRAPING,APPLICATIONSANDTOOLS
PDFtoExcelOnLine
Zamzar
www.pdftoexcelonline.com
Works
online.
Allows
CometDocs
PDFTables
Tabula
www.zamzar.com
www.cometdocs.com
pdftables.com
tabula.technology
Works
Works
Works online. Only
Desktop
tool.
converts
Focused
on
online.
online.
several input and output
Large amount of
Converts
formats, including Word,
input and output
Word,
Excel, PowerPoint, PDF).
formats.
PowerPoint.
Theresultingfileissentvia
resultingfileissent
based
email.Freetouse.
via email. Free to
storage. Application
use.
versions for Desktop,
limited) plus paid
available.
Smartphone
accounts
with
Oriented
licensing
software
The
PDF
Excel
to
existing
and
tables in a PDF
extraction
Cloud-
document to Excel.
tables
document
API available. Free
documents.
account
Source
and
Tablet. Free account
various
with limited features.
terms.
(very
of
in
PDF
code
to
developers.
API for document
translation.
On-line
X
X
X
X
Off-line
X
SaaSFreeService
X
X
X
X
SaaSPaidService
X
X
Document
word,excel,powerpoint,
formats
pdf
many
API
Orientedto
developers
Sourcecode
available
ConvertsPDFto
Word,Exceland
Convertsexisting
Onlyextracts
tablesintoPDF
datafromtables
powerpoint
documentstoExcel
intoPDF
format
documents
X
X
X
X
PartialtoolstoextractfromHTML
Regarding techniques and tools for web content (HTML documents) partial scraping, some
examplesoftoolsarecommentedbelow.
GoogleSpreadsheetsandtheIMPORTHTMLformula
This is a simple solution, but sufficiently effective to extract data from an HTML table to a
GoogleSpreadsheetsdocument.Theactualformatoftheformulais:
=IMPORTHTML(“URL”;”table”;N)
Testing the IMPORTHTML formula is fairly simple, as long as value N is known. This value
represents the order of the table in the list of tables available in the HTML code available in
ePSIplatform Topic Report No. 2015 / 10 , December 2015
9
WEBSCRAPING,APPLICATIONSANDTOOLS
addressURL.Anexampleofusebasedintheformatis:
=IMPORTHTML("http://www.euskalmet.euskadi.net/s075853x/es/meteorologia/app/predmun_o.apl?muni=";"table";1)
TableCapture,aGoogleChromeextension
TableCaptureisanextensiontothepopularGoogleChromewebbrowser.Itenablesusersto
copythecontentoftablesincludedinawebpageinaneasymanner.Theextensionisavailable
fromtheaforementionedbrowser,typingthefollowingURLinitsaddressbar:
https://chrome.google.com/webstore/detail/table-capture/iebpjdmgckacbodjpijphcplhebcmeop
Once installed, if Chrome detects tables in the web page currently rendered, a red icon is
showntotherightoftheaddressbar.Clickingonitshowsalistingofthetableddetectedand
twocontrols.Thefirstonecopiescontenttotheclipboard,andthesecondoneopenaGoogle
Docsdocumenttolaterpastethecontentofthetable.Anexampleisshowninthefollowing
figure.
ExampleofuseofTableCapture
ePSIplatform Topic Report No. 2015 / 10 , December 2015
10
WEBSCRAPING,APPLICATIONSANDTOOLS
TabletoClipboard,Firefoxadd-on
Asinthepreviouscase,theFirefoxwebbrowseralsosupportsadd-ons(analogoustoChrome’s
extensions) to extract data from HTML tables. An example is Table2Clipboard, which can be
downloaded
and
installed
from
https://addons.mozilla.org/es/firefox/addon/dafizilla-
table2clipboard/?src=userprofile
Inthiscase,acontextmenushowinguponaright-clickonatableallowscopyingituporjust
the clicked row or column. This is a mechanism quite useful and less intrusive that offers an
interestingfunctionalityinmanycases.
ExampleofuseofTabletoClipboard
ePSIplatform Topic Report No. 2015 / 10 , December 2015
11
WEBSCRAPING,APPLICATIONSANDTOOLS
Completetools
Overthelastyears,asetofcompanies,mayofthemstart-ups,haverealizedthatthemarket
wasdemandingtoolstoextract,store,processandrenderdataviaAPIs.Thesoftwareindustry
moves fast and in many directions and a good web scraper can help in application
development, Public Administration transparency, Big Data processes, data analysis, online
marketing, digital reputation, information reuse or content comparers and aggregators,
amongothertypicalscenarios.
But,whatshouldatoolofthiskindincludeatleasttobeconsideredasaseriousalternative?In
theopinionoftheauthor:
•
Apowerfulandfriendlygraphicaluserinterface.
•
Aneasy-to-useAPItolinkandintegratedata.
•
Avisualaccesstowebsitestoperformdatapicking.
•
Datacachingandstorage.
•
Alogicalorganizationandmanagementofthequeriesusedtoextractdata.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
12
WEBSCRAPING,APPLICATIONSANDTOOLS
Import.io
CompanylocatedinLondon,specializedindataminingandInternetdatatransformationtousersto
exploitinformation.
Websiteàimport.ioandenterprise.import.io
MottoàInstantlyturnwebpagesintodata
Indubitably, this is one of the reference tools in the market. It may be used in four different
ways.Thefirstone(namedMagic,thatcanbeclassifiedasbasic)istheaccesstoimport.ioin
ordertotypetheaddressofthewebsiteonwhichwewanttoperformscraping.Theresultis
shown in an attractive visual tabular format. The main con is that the process is not
configurableatall.
The second way of use is named Extractor. It is the most common usage of import.io
technology:download,installandexecuteinyourowncomputer.Thistoolisacustomizedweb
browser available for Windows, OS X and Linux. This way requires some previous skills using
softwaretoolsandsometimetolearnhowtousethetool.However,“picking”isofferedina
quite reasonable manner –although open to improvement, at the same time. “Picking” is
performedbyclickingonthepartsofthescrapedwebsitethatwewanttoextract,inasimple
andvisualway.Thisisafeaturethatanywebscrapingtoolsmustincludethesedays.
Oncethatquerieshavebeencreated,outputformatsareonlytwo:JSONandCSV.Queriesmay
also be combined in order to page results (“Bulk Extract”) or aggregate them (“URLs from
anotherAPI”).ItisalsorelevanttonotethatuserswillhaveaRESTfulAPIEndPointtoaccess
the data source –which is a mandatory feature in any relevant complete scraping tool
nowadays.
The import.io application requires a simple user registration process. With a username and
password, the application can be used for free, with a set of basic features that may be
sufficientforsmalldeveloperteamswithoutcomplexscrapingrequirements.Forcompaniesor
professionals demanding more flexibilityand backend resources, contact with import.io sales
teamisrequired.
Thethirdandfourthwaysofuseprovidemorevaluetothetool.TheyaretheCrawlerandthe
Connector,respectively.TheCrawlertriestofollowallthelinksinthedocumentindicatedvia
ePSIplatform Topic Report No. 2015 / 10 , December 2015
13
WEBSCRAPING,APPLICATIONSANDTOOLS
its URL and it allows information extraction based on the picking process carried out in the
initialdocument.Inthetestscarriedouttowritethisarticle,wehavenotmanagedtofinishall
this process, as it seems to keep working all the time without producing any results. The
Connector permits to record a browsing script to reach the web document from which to
extractinformation.Thisapproachisveryinteresting,forinstance,ifthedatatobescrapedare
theresultofasearch.
Insummary,import.ioisatoolwithinterestingfunctionalityfree to use,withahigh level of
maturity,anattractiveandmoderngraphicaluserinterface,supportingcloudstorageofthe
queries,whichdemandslocalinstallationtotakefulladvantageofitsfeatures.
Strengths
Weaknesses
VisualInterface
Desktopinstallation
BlogandDocumentation
Limitedamountofoutputdocumentformats
Allows pagination, aggregation, crawling and script
Learningcurve
recording
StrenghtsvsWeaknessesforimport.io
Viewofthebasicuseofimport.io(availableathttps://import.io)
ePSIplatform Topic Report No. 2015 / 10 , December 2015
14
WEBSCRAPING,APPLICATIONSANDTOOLS
import.iodesktopapplication
Screentoselectdataextractionmodeinimport.io
ePSIplatform Topic Report No. 2015 / 10 , December 2015
15
WEBSCRAPING,APPLICATIONSANDTOOLS
Kimonolabs
KimonoLabsisacompanylocatedinMountainView,California.TheyofferaSaaStoolaftertheuseof
aChromeextension.Itallowsextractingdatafromwebdocumentsinanintuitiveandeasyway.They
provideresultsinJSON,CSVandRSSformats.
Websiteàwww.kimonolobas.com
MottoàTurnwebsitesintostructuredAPIsfromyourbrowserinseconds
Kimono Labs is another key player in the field of web scraping. It uses a strategy similar to
import.io, using a web browser. In their case, they offer a Chrome extension, rather than
embeddingawebbrowserinanativedesktopapplication.Therefore,thefirststeptousethis
toolistoinstallGoogleChromeandthenthisextension.Registrationisoptional.Itisaquick
andsimpleprocessthatisrecommended.
After installation and registration, we can use Chrome to reach a web document with
interesting information. In this moment, we click the icon installed by the Kimono extension
andthepickingprocessstarts.Thisprocessprovideshelptotheuseratfirstexecutionwitha
visualandattractiveformat.Itsgraphicaluserinterfaceisreallypolishedandthetoolresults
veryfriendly.
InitialhelpviewofferedbyKimono
ePSIplatform Topic Report No. 2015 / 10 , December 2015
16
WEBSCRAPING,APPLICATIONSANDTOOLS
TostartworkingwithKimono,theusermustcreatea“ball”intheupperendofthescreen.By
default,aballisalreadycreated.Thevariousballscreatedareshownwithadifferentcolour.
Afterwards,sectionsofthedocumentmaybeselectedtoextractdataan,subsequently,they
are highlighted with the colour of the associated ball. At the same time, the ball shows the
number of elements that match the selection. Balls may be used to select different zones of
the same document, although their purpose is to refer zones with certain semantic
consistency.
For instance, in the web site of a newspaper, we might create a ball named “Title” and then
selectaheadlineinawebpage.Then,thetoolhighlightsoneortwoadditionalheadlinesas
interesting. After selecting a new headline, the tool starts highlighting another 20 interesting
elements in the web page. We may notice that there are some headlines which are not
highlightedbythetool.Weselectoneofthemandnowover60headlinesarehighlighted.This
process may be repeated until the tool highlights all the headlines after selecting a small
amount of them. This process is known as “selector overload” and is available in several
scrapingtools.
Oncethatalltheheadlineshavebeenselected,wecantrydoingthesamewiththeopening
paragraph of each news item: create a ball, click on the text area of an opening paragraph,
thenanotherone,andgoonwiththeprocessuntilhavingallthedesiredinformationreadyfor
extraction.
Although the idea is really good, our tests have found that the process is somehow
bothersome.Sometimes,boxhighlightinginwebpagesdoesnotworkwellwith,forinstance,
problemsintextswhicharelinksatthesametime.
Oncethatthepickingprocessisfinishedbyclickingonthe“Done”button,wecannameour
new API and parameterize its temporal programming. With this, the system may execute the
API every 15 minutes, hour, day, etc. and store the results in the cloud. Whenever the user
calls the API by accessing the associated URL, Kimono does not access the target site but
returnsthemostrecentdatastoredinitscache.Thiscachingmechanismishighlyusefulbut
notexclusiveofKimono.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
17
WEBSCRAPING,APPLICATIONSANDTOOLS
Exampleofhighlightinginthepickingprocessofkimono
The query management console, available at www.kimonolabs.com, provides access to the
newly created API and various controls and panels to read data and configure how they are
obtained.Thisincludesaninterestingfeature,whichareemailalertstobereceivedwhendata
changeinthetargetsite.
There is an additional interesting option named “Mobile App” that integrates the content of
thecreatedAPIinaviewresemblingamobileapplication,allowingsomestylingconfiguration.
However, the view generated by this option is a web document accessible by the URL
announced, aimed to be rendered in a mobile browser. Unluckily, the name of the option
misleads users and does not generate a mobile application to be published in any mobile
applicationstore.Still,itmaybeausefuloptionforrapidprototyping.
The console menu also offers the “Combine APIs” option. Initially, it may look like an
aggregator, assembling the data obtained from several heterogeneous APIs in a single API.
Nevertheless,helpinformationinthisoptionindicatesthattheaggregatedAPIsmusthavethe
same exact name of data collections. The conclusion is that this option is useful to paginate
information,butnottoaggregate.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
18
WEBSCRAPING,APPLICATIONSANDTOOLS
Managementconsoleofkimono
In summary, kimono is a free tool, with a high level of maturity, a very good graphical user
interface,providingcloudstorageforqueries,requiringChromebrowserandtheirextension–
bothoftheminstalledlocally.
Strengths
Weaknesses
Visualinterface
Chromebrowserdependency
Documentation
Doesnowallowaggregation
Picker
Weakmobileappoption
StrenghtsvsWeaknessesforkimono
ePSIplatform Topic Report No. 2015 / 10 , December 2015
19
WEBSCRAPING,APPLICATIONSANDTOOLS
myTrama
myTramaisawebcrawlingtooldevelopedbyVitesiaMobileSolutions,acompanylocatedinGijón,
Spain.myTramaallowsanyusertoextractdatalocatedindifferentInternetsitesandobtainthemin
anorderedandstructuredwayinvariousformats(JSON,XML,CSVandPDF).
Websitesàwww.mytrama.comandwww.vitesia.com
MottoàDataisthenewoil
myTrama is a new web crawling tool positioned as a clear competitor to those previously
commented.ItisapurelySaaSservice,thusavoidingtheneedforuserstoinstallanysoftware
nortodependonaspecificwebbrowser.myTramaworksonChrome,Firefox,InternetExplorer
andSafari.Itisavailableathttps://www.mytrama.com.
A general analysis of this tool suggests that myTrama takes the best ideas of import.io and
kimono. It presents information in a graphical user interface, perhaps not so good but more
compactandwiththelookandfeelofaprojectmanagementtool.Someofthefeatureswhich
seemmoreinterestinginthistoolarecommentedbelow:
•
Main view is organized in a way similar to an email client, with 3 vertical zones: 1)
folders,2)queries,and3)querydetail.Itisefficientandfriendly.
•
Besides JSON, XML and CSV, the classical structured formats for B2B integrations, it
adds PDF for quick reporting and sends results in an easily viewable and printable
format.
•
ItincludesaquerylanguagenamedTrama-WQL(quitesimilartoSQL),whichissimple
to use while powerful. It is useful when visual picking is not sufficient, providing a
programmaticmannertodefinethepickingprocess.Documentationofthislanguageis
availableinthetoolasamenuoption.
•
The “Audit” menu option gives access to a compact control panel with information
abouttherequestscurrentlybeingmadetoeachoftheAPIs(EndPoints).
•
Thepickeriscompletelyintegrated.Itisnotnecessaryanytypeofadditionalsoftware.
Itissimilartotheapproachusedinkimono,althoughituses“boxes”insteadof“balls”.
Asubtledifferentiationisthatamagicwandreplacesthedefaultmousepointerwhen
pickingisavailable.Inaddition,thepickingprocessmaybestoppedbyright-clickingon
theareabeingpicked.
•
myTrama permits grouping boxes within boxes, although only one level of grouping
ePSIplatform Topic Report No. 2015 / 10 , December 2015
20
WEBSCRAPING,APPLICATIONSANDTOOLS
andonlyagroupwithqueryareallowed.Thisisaveryusefulfeatureinordertohave
results properly grouped. Hopefully, the development team will improve this feature
soontoprovideuserswithmoreflexibility.
•
Queryconfigurationallowsupdatefrequencywithagranularityofminutes,from0to
9999999.Zeromeansrealtime(thisis,accessingthetargetsiteuponeachrequestto
the EndPoint). For any other value, information is obtained from the cache –as in
kimono.
•
APIs may be programmed using parameters sent via GET and POST requests.
Unfortunately,thedevteamhasnotpublishedsufficientdocumentationrelatedtothis
feature.Forexample,itispossibletousetheURLofanAPIandoverwritetheFROM
parameter(theURLreferencingthetargetdocument)inrealtime.Itisalsopossibleto
passparametersviaGETandPOSTinthesameAPI.Additionally,thereisaservicethat
allowstheexecutionofaTrama-WQLsentencewithoutanyquerycreatedinthetool.
As these are not very well documented features, the best choice is to contact the
peopleatVitesia.
•
Paging queries and aggregation of heterogeneous queries are supported in a fairly
simpleandcomfortableway.
•
Forthosepreferringthebrowserextensionwayofscraping,aChromeextensionisalso
available.Thismechanismallowsuserstobrowsesitesandstartthescrapingprocess
byclickingonthebuttoninstalledbytheextension.Thispluginisnotyetpublishedbut
canberequestedtoVitesia.
•
PDFisnotonlyaformatavailableasoutput,butalsoasinput.Therefore,aURLmay
reference only HTML documents but also PDF. For instance, users will be able to
extractinformationfromPDFsandgenerateJSONdocumentsthatfeedadatabasefor
later information analysis. The business hypothesis to support this is based on the
evidenceinitiallycommentedattheintroductionofthisarticlethatstatedthat70%of
thecontentpublishedintheInternetiscontainedinPDFdocuments.Vitesiaconsider
thatthismaybeadifferentiatingfeaturebetweenmyTramaandtheircompetitors.
•
APIs preserve session. This allows chaining calls to queries in myTrama and fulfil
businessprocesses,suchassearches,step-basedforms(wizards)oraccessinformation
availablebehindaloginmechanism.
•
Itisavailableintwolanguages:EnglishandSpanish.
•
Access to this platform is based on invitation. Users remain active for 30 days. Later
contactwiththedevteamisrequiredinordertomovetoastableuser.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
21
WEBSCRAPING,APPLICATIONSANDTOOLS
Amongallthetoolsanalysed,myTramaseemstobethemostcompleteandcompact,although
its user interface is one step behind kimono and import.io. For users with software
development skills, myTrama seems to be the best choice –although requiring direct contact
withVitesia.
InitialscreenofmyTrama
QuerymanagementconsoleinmyTrama
ePSIplatform Topic Report No. 2015 / 10 , December 2015
22
WEBSCRAPING,APPLICATIONSANDTOOLS
PickingprocessinmyTrama
DashboardscreeninmyTrama
In summary, myTrama is a tool solely offered as a SaaS service, very complete to carry our
scraping processes, with cloud storage and that may be operated with any web browser. Its
major weakness is the lack of documentation of many differentiating issues relevant to
developersinterestedintakingadvantageofscrapingprocesses.
Strengths
Weaknesses
TheTrama-WQLlanguage
Limiteddocumentation
Dashboards
Freelicenseonlyfor30days
Picker
Moreorientedtodevelopers
SessionpreservationbetweenAPIrequests
StrengthsvsweaknessesformyTrama
ePSIplatform Topic Report No. 2015 / 10 , December 2015
23
WEBSCRAPING,APPLICATIONSANDTOOLS
ePSIplatform Topic Report No. 2015 / 10 , December 2015
24
WEBSCRAPING,APPLICATIONSANDTOOLS
Confrontingthethreetools
X
SaaSmodel
(requires
installationof
X
others
features
operation
distribution
chromeextension)
X
Desktopinstaller
(Windows,OSX
X
X
(required)
(optional)
andLinux)
Chromeextension
Freelicense
X
X
X
Cross-browsercompatibility
Ownbrowser
OnlyChrome
X
Valuationofuserinterface
Good
Verygood
Good
APIscreationsimplicity
X
X
X
Visualpicking
X
X
X
Cachingandstorage
X
X
X
Queryorganization
X
X
X
Ownquerylanguage
TramaWQL
Statisticsandauditdashboards
X
PDFextraction
X
Outputformats
JSON,CSV
JSON,CSV,RSS
APIscreationsimplicity
X
X
X
SessionpreservationbetweenAPIinvocations
X
Automaticcrawling
?
Maturitylevel
High
High
Medium/High
Complexofuse
Medium/High
Medium
Medium
Cloudstorage
X
X
X
Querypagination
X
X
X
Queryaggregation
X
X
Realtimedata
?
?
X
Orientationtosoftwaredevelopment
Medium/Low
Medium/Low
Medium
GoogleDocsintegration
X
X
Levelofdocumentation
High
High
Low
JSON,CSV,XML,
PDF
ePSIplatform Topic Report No. 2015 / 10 , December 2015
25
WEBSCRAPING,APPLICATIONSANDTOOLS
Othertools
Thisarticleanalysesthreetoolsinthecategoryofcompletetoolsbutmanyotherexist.Forthe
sake of brevity, and for those readers interested in this kind of tools, some other interesting
tools,utilitiesorframeworkspermittingwebscrapingarelistedbelow.
Mozenda
QuBole
ScraperWiki
Scrapy
Scrapinghub
ParseHub
UbotStudio5
Scraper
ApacheNutch
(Chrome OutwitHub
Plugin)
Fminer.com
80legs
ContentGrabber
CloudScrape
Webhose.io
UIPath
Winautomation
VisualWebRipper
AddTolt
AgentCommunity
AllinOneStats
Automation
Clarabridge
DarcyRipper
DataIntegration
Anywhere
Enterprise
DataCrops
Dataddo
Diffbot
EasyWebExtract
Espion
Feedity
Ficstar
Web
Grabber
PDFCollector
PDF
Plain
ForNova Big Data HeliumScraper
KapowKatalyst
Platform
Text
RedCritter
Scrape.it
SolidConverter
TextfromPDF
Trapeze
UnitMiner
WebDataMiner
Web
Extractor
Spinn3r
Web
Extractor
SyncFirstStandard
Content Web
Data
Extraction
Robots WebHarvy
Scraping
If you are not convinced to use any of the three recommended tools, Mozenda or ParseHub
maybeinterestingalternatives.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
26
WEBSCRAPING,APPLICATIONSANDTOOLS
4 Decisionmap
The following diagram may be helpful in the process of deciding which tool meets which
scrapingrequirements.Obviously,thediagramcouldbemorecomplex,asmorequestionsmay
beaskedinadecisionprocess.However,theriseofcomplexityinthefiguresuggestskeepingit
simple,butsufficientlyillustrative.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
27
WEBSCRAPING,APPLICATIONSANDTOOLS
5 Scrapingandlegislation
Thequestiontobemadeissimple:iswebscrapinglegal?
Theanswerisnotassimple.Firstly,weneedtoknowthespecificlegislationofeachcountry
concerned.TheUnitedStatesofAmericaaremorepermissivethanEurope.InSpain,wherethe
authorlivesandworks,lawsarefairlymorerestrictive.However,therearevariousstatements
(includingonefromtheSupremeCourtofSpain),wherewebscrapingisconsideredlegalunder
specificconditions.
Forexample,usersmustbecarefulwithlegalconditionsincertainwebsites.Iftheircontentis
proprietaryandorientedtocommercialpurposes,scrapingmightbeanillegalactivity.Author
rights are another legal characteristic of special interest before scraping the information of a
website.Forinstance,newsfromdigitalnewspaperscannotbeextractedtobepublishedina
blogorapplicationwithoutstatingthesource.
Asexpected,allthisgeneratesdoubts,soitisagoodpracticetoconsultlawyersspecializedin
digitalcontent,intellectualproperty,unfaircompetitionandlicensing.Inanycase,ifdataare
clearlyopen,youwillnotneedtoworry.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
28
WEBSCRAPING,APPLICATIONSANDTOOLS
6 Conclusionsandrecommendations
Webscrapingisafamiliartermthathasgainedimportancebecauseoftheneedto“free”data
storedinPDFdocumentsorwebpages.Manyprofessionalsandresearchersneedthedatain
order to process it, analyse it and extract meaningful results. On the other hand, people
dealing with B2B use cases need to access data from multiple sources to integrate it in new
applicationsthatprovideaddedvalueandinnovation.
Themarketdemandsholisticwebscrapingsolutions,thatencompasscloudstorageandease
to build interoperable APIs. In this article we have analysed and compared 3 web scraping
solutions: import.io, Kimono and myTrama. These solutions differ in their implementation
details but share more common ground than believed: a visual picker, JSON results, data
caching,backgroundrobotstogatherdata,programmaticAPIs,etc.
Anexperiencedteam,usingthesesolutions,candevelopservicessuchascontentaggregators,
ranking tools, mobile apps, data monitoring systems, reputation management applications,
businessintelligencesolutions,BigData,etc.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
29
WEBSCRAPING,APPLICATIONSANDTOOLS
AbouttheAuthor
OsmarCastrilloFernándezhasmorethan15yearsofexperienceintheITindustry.Heholdsa
Bachelor’sdegreeinComputerSciencefromtheUniversityofOviedo.Hismainexpertisefields
arewebdevelopment,service-orientedarchitecturesandpublicadministration.
InApril2004hestartedworkingforCTICFoundationintheR&Dteamwhatwasinchargeof
developinganewJ2EEframeworkforthePrincipalityofAsturias(FWPA).TheFWPAwasakey
elementinthesuccessofthemodelofE-GovernmentinthePrincipalityofAsturiasandhelped
to simplify and homogenise the development of new government-related applications and
services.
HewasateacherinthefirstandsecondeditionsoftheJ2EE-FWPAcourseforITprofessionals
inAsturias.HealsotaughtseveralothercoursesrelatedtoUML2.0andn-tierarchitectures.
InOctober2012hefoundedandbecomeCTOofVitesiaMobileSolutions,aCompanydevoted
to extracting and analysing published data on the Internet. At the start of 2013 he started
leadingtheTRAMAproject,atechnologytofacilitatewebscraping,thatlaterbecamethetool
myTrama.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
30
WEBSCRAPING,APPLICATIONSANDTOOLS
Copyrightinformation
© 2015 European PSI Platform – This document and all material therein has been compiled
with great care. However, the author, editor and/or publisher and/or any party within the
European PSI Platform or its predecessor projects the ePSIplus Network project or ePSINet
consortiumcannotbeheldliableinanywayfortheconsequencesofusingthecontentofthis
document and/or any material referenced therein. This report has been published under the
auspicesoftheEuropeanPublicSectorinformationPlatform.
The report may be reproduced providing acknowledgement is made to the European Public
SectorInformation(PSI)Platform.
ePSIplatform Topic Report No. 2015 / 10 , December 2015
31