For this, a combination of Web archives together with a distributed infrastructure supporting extended analytical tools is a necessary tool. With such an infrastructure, large-scale measurements, topological information and trends at Internet scale can be brought to researchers and information professional’s scrutiny.

Internet Memory developed a new infrastructure with the ambition to reach “Web-scale” in terms of Web documents acquisition (billions of resources crawled per week) and computable data storage (Petabytes of data). This platform, partly supported by several EU projects among which LAWA (Longitudinal Analytics of Web Archive data) includes:

- A new crawler, entirely implemented in Erlang to support the retrieval of billions of pages in days. Thanks to its innovative frontier and seen-URL data structure, it sustains throughput for weeks while enabling Web-scale exploration.
- A new Web Archive repository for content and metadata based on HBase. It offers a perfect storage layer for Web archives as it is functionally isomorphic to WARC, but abstracts lots of the underlying data management (replication, index creation etc) while exposing analytical friendly APIs.
- Filters and extractors to distil relevant information and create processing chain in a distributed execution environment.

This presentation will offer an overview of this platform and discuss the next steps of its development.
International Internet Preservation Consortium (IIPC) 2012 General Assembly
Library of Congress, Washington DC
Tuesday May 1, 2012, 2:30 pm -3:45 pm (Members only)
Presented by Leïla Medjkoune

Mignify: A Big Data Refinery Built on HBase

HBasecon 2012
Tuesday, May 22, 2012, 2:20pm – 3:00pm, InterContinental San Francisco Hotel
Presented by Stanislav Barton

This platform is partly supported by several EU projects among which LAWA (Longitudinal Analytics of Web Archive data).

Mignify is a platform for collecting, storing and analyzing Big Data harvested from the web. It aims at providing an easy access to focused and structured information extracted from Web data flows. It consists of a distributed crawler, a resource-oriented storage based on HDFS and HBase, and an extraction framework that produces filtered, enriched, and aggregated data from large document collections, including the temporal aspect. The whole system is deployed in an innovative hardware architecture comprising of a high number of small (low-consumption) nodes. This talk will tackle the decisions made along the design and development of the platform, both under a technical and functional perspective. It will introduce the cloud infrastructure, the LTE-like ingestion of the crawler output into HBase/HDFS, and the triggering mechanism of analytics based on a declarative filter/extraction specification. The design choices will be illustrated with a pilot application targeting Daily Web Monitoring in the context of a national domain.

HBasecon 2012 is the first industry conference for Apache HBase users, contributors, administrators and application developers and we are glad to present

At the College de France, Chair of Information Technology and Digital Sciences

Information technology has revolutionized our lives. Computers are traditionally seen as computing machines, although their main purpose is now to manage data. This course will cover essential aspects of data management, including its close relationship with mathematical logic and complexity theory. The Web can be seen as a huge distributed database: its most exciting aspects will also be studied, such as its scale or the challenges of distributed computing and the Semantic Web.

Wednesday, March 28th, from 10.00 to 12.00 am: Semantic Web, Open Data and Web Archiving

Serge Abiteboul opens the conference with a lecture about the Semantic Web and invites François Bancilhon, Director of DataPublica to talk and Julien Masanès, Director of the Internet Memory Foundation to talk about Open Data and Web archiving.

Feel free to join!

Address:
Amphithéâtre Maurice Halbwachs
Collège de France
11, place Marcelin Berthelot
75231 Paris Cedex 05
France

The naive approach that consists in getting all the required data at the Client in order to apply locally some processing should be limited in a distributed setting to trivial tasks operating on a tiny subset. There are two fundamentals reasons for that. First, this generates a lot of network exchanges, consuming without necessity a lot of resources and sometimes leading to unacceptable response time. Second, centralizing all the information then processing it, simply misses all the advantages brought by a powerful cluster of hundreds or even thousands machines. The lesson is simply:

It is fair to acknowledge that server-side computation is not limited to Hadoop-like frameworks, but is also possible with relational systems, in the form of “transactional SQL” languages - e.g., PL/SQL - which are executed in the server. Still, moving the computation to the data server, instead of moving the data to the client computing, becomes of the essence in a BigData context. The principle is most often termed data locality.

Server-side filtering

Let us examine one important feature of HBase which helps us to put this principle in action: filters. For concreteness, consider the canonical example of a program which scans a collection of Web documents and applies some analysis method to RSS feeds (typical of the daily tasks operated at Internet Memory). The algorithm is trivial: we need to access each document, check whether it is indeed a RSS one, and run the analytics. We can implement this algorithm as a program running at a Client node, using a scanner on some HBase table and a filter (on documents’ type) on the client side. The program retrieves the documents from the distributed system (input data flows), and locally performs the computation.

The disadvantage is obvious. For very large data sets, the computing and storage resources of the Client machine are likely to become quickly overwhelmed, creating a bottleneck. Most of the time will be spent by exchanging documents that do not participate to the result. HBase filters enable a different scenario, illustrated in the Figure below, when the selection of RSS documents occurs at each server. This is likely (on our example) to drastically limit communications by applying local data processing as much as possible. In such a setting, the Client plays the role of a coordinator sending pieces of code to each server, initiating, and possibly coordinating a fully decentralized computation.

Filters in HBase

Filters in HBase are objects implementing the “Filter” interface, equipped with a Boolean “FilterRow()” method which tells whether a row passes or not the filter. The semantics is that rows are filtered out by a filter, which means that you rather tell the rows that you want to ignore (just the opposite of what you are used to express if you are familiar with the SQL “WHERE” clause). Scanners can incorporate filters with the “setFilter()” method.

This means, among others consequences, that you can use filters for MapReduce jobs that take their inputs from a HBase scanner. We will not cover the full Filter functionality in this post (rather look at the HBase Wiki or the Javadoc) but briefly touch a few of it meain features.

HBase comes with a list of pre-defined filters, including:

 
• “RowFilter”: data filtering based on row key values;
 
• “FamilyFilter”: allows to filter out some families on their names;
 
• “QualifierFilter”: allows to filter out some qualifiers on their names;
 
• “ValueFilter”: allows to filter out some qualifiers on their values;
 
• “TimestampsFilter”: allows to filter out rows based on a list of timestamps.
 

As suggested but these examples, filters are much more powerful than a simple all-or-nothing semantics applied to HBase rows. You can choose to filter out a row as whole based on some qualifier value, but also filter out some part of a row, namely a family and/or a column (qualifier) in a family, etc. In other words, you can apply filters to the schema (family and column names) as well as on the values, a recurring features of semi-stuctured data models.

Compare with the well-known SQL world. When you express a SELECT-FROM-WHERE query, you restrict the number or rows (with the “WHERE” clause) and the number of columns for each row (with the “SELECT” clause). Filters in HBase let you do both: fully ignore some rows, and for those rows that pass, restrict the family, columns, or timestamps. This must be related to the underlying motivation: limit as much as possible the network bandwidth used to communicate withe the client application.

There exists many other filters, some of which implementing utilities likes pagination. Again refer to the documentation. 

Combining filters: “FilterList”

Even though HBase comes with a lot of filter types, their expressive power would remain limited without the ability to combine them with Boolean connectors. This is the purpose of the “FilterList” class. A “FilterList” is defined by a connector (“and”, “or”) and a list of component filters (which can be of any type). The main constructor is:

Since a filter list is itself a “Filter” instance, we can build hierarchies of filters representing nested Boolean combinations, and gaining the ability to obtain arbitrarily complex filters.

Building custom filters

Finally, it is worth mentioning that you can write your own filters, in case the existing ones would not be sufficient. This amounts to write a Java class subclassing the “FilterBase” abstract class, implementing a few abstract methods which operate, in the Region servers, on the local rows. The downside of custom filters is that they require a dissemination in the cluster prior to their execution, which makes the set-up of the whole system a bit more complicated.

Summary: let HBase do the data selection job for you!

The bottom line of what precedes is: do not ever overload in your application with the burden of row filtering! HBase can do it for you in a much more effective way, at scale. And this encourages us to thing as if our computing machinery is no longer our laptop, but a whole set of servers interconnected with a bandwidth. Consider the resources and limitations of such a system, particularly regarding the network bandwidth, and adopt the measures that avoid to overwhelm these resources. HBase filters (and other features to be covered next) are just built for that.

Few months ago, the Cultural Heritage Agency of the Netherlands (RCE) contacted us to expose its situation:

Inside Installations Project, Preservation and Presentation of Installation Art, was a research project (2004-2007) into the management and conservation of installations and was supported by the European Commission’s Culture 2000 programme.
Rapid obsolescence of media technologies, interactivity and, for instance, the site specific character of many installations are a challenge for prevailing views about long-term conservation, documentation and presentation. Thirty complex installations (many multimedia) were re-installed, investigated and documented.
By sharing their experience partners worked together to develop guidelines for conservation, re-installation and documentation of installation art.

The Cultural Heritage Agency of the Netherlands was the coordinator of the project, which was co-organised by:
- Tate, London;
- Restaurierungszentrum, Düsseldorf;
- Stedelijk Museum for Modern Art (S.M.A.K.), Ghent;
- Museo Nacional Centro de Arte Reina Sofia, Madrid
- and the Foundation for the Conservation of Modern Art (SBMK) in The Netherlands.

In this framework, they developed a high content website (Online Version). (Archived Version)

See archived version captured in February 2012.

Results of such Web archiving initiatives

* Websites are preserved and therefore they might remain a part of the cultural heritage for decades.
* They are publicly available online.
* This solution is less expensive than maintaining websites that are not any more updated.

Web archiving as an efficient solution to offer a second life to your project websites!

Internet Memory proposes solutions to archive and preserve high quality websites such are research projects’ websites thanks to its automated Web archiving platform, ArchivetheNet.

Basics

Many video formats are found on the Web, including Windows Media (.wmv), RealMedia (.rm), Quicktime (.mov), MPEG, Adobe Flash (.flv), etc. In order to display a video, we need a player, which can be incorporated in the Web browser. The player depends on the specific video format, but most browsers are able to detect the format and choose the appropriate player. Firefox for instance comes with a lot of plugins, which can be quickly integrated in the presence of a specific video to display it content.

There are basically two ways to play a video. The simplest one is a two-steps process: first the whole file is downloaded from the Web server to the user’s computer, and then displayed by the player running the local copy. It has the disadvantage that the download step may take a while is the file is big (hundreds of megabytes are not uncommon). The second one uses (true) streaming: the video file is split into fragments which are sent from the Web server to the player, giving the illusion of a continuous stream. From the user point of view, it looks as if a window is swept over the video content, saving the need of a full initial download of the whole file.

Obviously, streaming is a more involved method because it requires a strong coordination between the components involved in the process, namely the player, the Web server, and the file system from which the video is retrieved. We examine this technical issue in the context of a Hadoop system where files are stored in HDFS, a file system dedicated to large distributed storage.

File seeking with HDFS

At explained above, streaming requires a strong coordination between the Web server and the file system. The former produces requests to access chunks of the video file (think to what happens when the user suddenly requires a move to a specific part of the video), whereas the later must be able to seek in the file to position the cursor at a specific location. When using HDFS, enabling such a close cooperation turns out to be a problem because HDFS can in principle only be accessed through a Hadoop client, which the standard Apache server is not. We investigated two possible solutions: Hoop, the Hadoop web server, and Apache/FUSE.

Hoop (see http:///cloudera.github.com/hoop/) is an HTTP-HDFS-Connector. It allows the HDFS file system to be accessed via HTTP. A working local prototype has been developed using JW Player and a large video file. Streaming works, but seeking in an unbuffered part results in the playback stopping. It seems that the Hoop API does not support seeking in a file, so we had to give up this approach.

The second solution is based on HDFS/FUSE. FUSE (File System in User Space) is an API that captures the file system operations and allows to implement them with ad-hoc functions running in the the user’s processus space (thereby saving the need to change the operating system kernel, a tricky and dangerous option). FUSE is provided in Hadoop as a component named “Mountable HDFS” (see http://wiki.apache.org/hadoop/MountableHDFS). It lets the standard file system user or program see the HDFS name space as a locally mounted directory. All file system operations, including directory browsing, file opening and content access, are enabled over HDFS content through the FUSE interface.

Apache server configuration

It remained to configure Apache to access the mounted FUSE system and load content from video files. How this is done depends on the video format. At the moment, we tested and validated .mp4 files and Flash video files. For the first format we use H264 Streaming Module (see http://h264.code-shop.com/trac), an Apache plugin, which enables adaptive streaming. For FLV we used pseudo-stream module for Apache named “mod_flv”. Both behave nicely and go along with the mountable HDFS without problem.

Conclusion

The solution based on Apache + Mountable HDFS (FUSE) turned out to be both reliable, functionally adequate (seeking is well supported) and efficient. The architecture is simple and easy to set up, and allows to combine the benefits of HDFS for very large repositories and standard Web server streaming solutions. Although we chose to adopt Apache plugins in our current service, nothing keeps you from using a more powerful streaming server since the FUSE approach (virtually) moves all the HDFS content in the standard file system scope.

Hoop remains a potential option for the future, but it appeared not mature enough when we tested it, at least for the complex operations (seeking at a specific offset in a file) required by video streaming.

LivingKnowledge Project

The LivingKnowledge project (LK) enhances the state of the art of retrieving information from the Web by formalizing the notions of bias and diversity, creating tools that analyze, summarize and visualize bias in textual and image documents and finally, by creating applications that exploit this technology.

LivingKnowledge Testbed

The testbed integrates the following components, all of which contribute to diversity and bias aware search:
- document collections chosen to reflect a diversity of document types and content,
- image and text analysis tools supporting the analysis of diversity in text and image documents,
- indexing and search tools supporting the bias and diversity aware search including novel visualization methods,

The testbed processing starts with document collections that are available upon request from the Internet Memory Foundation, including 280 News sites and 750 blogs.
Furthermore, the testbed supports a number of collection formats allowing users to incorporate their own collections.

Hands-On session with over 30 participants (Symposium on Bias and Diversity) was held during the 8th International Summer School on Information Retrieval (ESSIR), which tooks place in Koblenz (Germany) in August/September 2011.

More info

Living Knowledge Project
SourceForge
Diversity Engine
Symposium on Bias and Diversity in IR (ESSIR 2011)

As PC-Chair and Organizer, Internet Memory Foundation informs you that the submission deadline for paper is fixed to February 24, 2012.

Objectives

The objective of this workshop is to provide a venue for researchers of all domains (IE/IR, Web mining etc.) where the temporal dimension opens up an entirely new range of challenges and possibilities. The workshops ambition is to help shaping a community of interest on the research challenges and possibilities resulting from the introduction of the time dimension in Web analysis.

TempWeb focuses on temporal data analysis along the time dimension for Web data that has been collected over extended time periods. A major challenge in this regard is the sheer size of the data it exposes and the ability to make sense of it in a useful and meaningful manner for its users. Web scale data analytics therefore needs to develop infrastructures and extended analytical tools to make sense of these.

Workshop topics

• Web scale data analytics
• Temporal Web analytics
• Distributed data analytics
• Web science
• Web dynamics
• Data quality metrics
• Web spam
• Knowledge evolution on the Web
• Systematic exploitation of Web archives
• Large scale data storage
• Large scale data processing
• Data aggregation
• Web trends
• Topic mining
• Terminology evolution
• Community detection and evolution

Important Dates

• Paper submission deadline: February 24, 2012
• Notification of acceptance: March 5, 2012
• Camera ready copy deadline: March 16, 2012
• Workshop: April 17, 2012

Please post your submission (up to 8 pages) using the ACM template:
http://www.acm.org/sigs/publications/proceedings-templates
at:
https://www.easychair.org/account/signin.cgi?conf=tempweb2012

Note that the workshop proceedings will be published in ACM DL (ISBN 978-1-4503-1188-5)

Support

This workshop is organized with the support of the EU 7th Framework ICT STREP on Longitudinal Analytics of Web Archive data (LAWA) under contract no. 258105.

Workshop Officials

Chair:

PC-Chairs and Organizers:

Ricardo Baeza-­Yates (Yahoo! Research, Spain)
Julien Masanès (Internet Memory Foundation, France and Netherlands)
Marc Spaniol (Max Planck Institute for Informatics, Germany)

Program Committee:

Eytan Adar (University of Michigan, USA)
Omar Alonso (Microsoft Bing, USA)
Srikanta Bedathur (IIIT-Delhi, India)
Andras Benczur (Hungarian Academy of Science)
Klaus Berberich (Max Planck Institute for Informatics, Germany)
Roi Blanco (Yahoo! Research, Spain)
Adam Jatowt (Kyoto University, Japan)
Scott Kirkpatrick (Hebrew University Jerusalem, Israel)
Christian König (Microsoft Research, USA)
Frank McCown (Harding University, USA)
Michael Nelson (Old Dominion University, USA)
Nikos Ntarmos (University of Patras, Greece)
Kjetil Norvag (Norwegian University of Science and Technology, Norway)
Philippe Rigaux (Internet Memory Foundation, France and Netherlands)
Thomas Risse (L3S Research Center, Germany)
Pierre Senellart (Télécom ParisTech, France)
Torsten Suel (NYU Polytechnic, USA)
Masashi Toyoda (Tokyo University, Japan)
Peter Triantafillou (University of Patras, Greece)
Michalis Vazirgiannis (Athens University of Economics and Business & École Polytechnique)
Gerhard Weikum (Max Planck Institute for Informatics, Germany)

The Internet has become one of the most productive media for information and news. Thus, there’s an absolute need to preserve web content and promote Web archiving at large scale. This idea begins to be one of the great challenges of the Web.
Media are already interested in the subject, and France24, the French international news channel, is broadcasting a video reportage on web harvesting in France (due to the French legal deposit), on Web archiving in general and on giving access to these Web archive collections.

This video shows a rapid overview of French initiatives and Web archiving technologies thanks to the participation of the National Library of France, the National Audiovisual Institute of France and the Internet Memory Foundation (interview of Julien Masanès by Natalia Gallois in our offices in Paris).

To view the video and discover the challenges of Web archiving click here (in French).
TV Show: “Web News”, News seen on the Web and about the web.

Beside Internet Memory Foundation are involved:
- Deutsches Forschungszentrum für Künstliche Intelligenz GmbH(Germany) as Coordinator,
- The University of Sheffield (United Kingdom),
- Ontotext AD (Bulgaria),
- University of Southampton (UK),
- Eurokleis S.R.L. (Italy),
- Sora Ogris & Hofinger GmbH (Austria)
- and Hardik Fintrade Pvt Ltd. (India).

This project aims at delivering innovative, portable open-source real-time methods for cross-lingual mining and summarization of large-scale stream media.

IMF will contribute to the Platform for Real Time Media collection, Analysis and storage by :
- providing scalable infrastructure to partners, with support for integration and experiment.
- designing and developing an application-aware crawler mechanism for social media.

For more information on TrendMiner, please visit the Project website (under construction).

Was this your first participation in the FIAT?

France Lasfargues: Personally, yes. But, it is not the first participation of Internet Memory Foundation, which is an associate member of IFTA. Last year, Chloe Martin, Business Developer at the Foundation, presented a poster based on our web archiving platform, Archivethe.net (ATN) and issues related to the collection and access videos that are broadcasted on the Web.

Is it easy for Internet Memory to participate in this international conference?

FL : To make this conference, we must first answer the call for participation that occurs at least three months before. It is then decided how we can present our activities, the participants we would like to integrate and form of presentation (poster, workshop, plenary session, ...). We submit our proposal and expect a return from IFTA. So, we decided to focus on issues that involve the expectations and needs of audiovisual archives and our skills and areas of development. The shape of the workshop seemed most appropriate, also to open a space for dialogue and exchange with the audience.

This brings us precisely to talk in more detail the reason for the presence of the Internet Memory at the IFTA.

FL : Our goal is simple : to communicate about the need of web archiving for audiovisual archives and, thereby, to share our expertise in this area. Internet Memory wants to drive projects, motivate institutions to get engaged in web archiving projects, now, in order to stop the loss of relevant content and high added value.

What is the angle chosen by Internet Memory for this workshop?

FL : The workshop was mainly an opportunity to invite audiovisual archives to share their current experience and problems in terms of web archiving and make up to date with capture and access solutions that we have developed. We must say that we have strong arguments on the matter. This gave us the opportunity to communicate on all of our projects LIWA, LK, LAWA, SCAPE, especially, ARCOMEM, which are a European large-scale projects and an excellent reference to show the extent of our technologies and skills. In detail, and because attendees were audiovisual archives, we have focused on the technical challenges of capturing video in web sites (LIWA). Equally important, the social web and the challenges it poses for archivists (ARCOMEM). We have also talked about various tools that we develop (including Application Aware Crawling, API Crawls, etc ...) to solve archiving problems and improve the archival collection.

How many participants were present to this international conference? Was your workshop appreciated?

The conference brought over 300 audiovisual archivists.
As for the workshop that we held, the workshop room was full with over than 120 participants. I admit that we did not expect such a success. Last year, the workshop on web archiving had mobilized at most 40 people! Moreover, the organizers of the conference highlighted our “score of attendance”. This proves that much more archivists are interested in web archiving and audiovisual archives. Internet Memory services could be developed in the near future and we are always ready to repeat the experience at the IFTA.

This workshop will present our platform Archivethe.net (AtN) and several concrete use cases, which will contribute to raise awareness and interest of audiovisual archivists.
1/ SWR use case: Rock am Ring 2011
2/ Multimedia Web archiving following recently finalized Living Web Archives project (LiWA).

Hope you will be many to attend it!

Find all details on FIAT website

The map structure: representing data with key/value pairs

{  
   'url': 'http://internetmemory.org', type: 'text/html', content: 'my document content' 
}

• there is no schema that constrains the list of keys (unlike relational table where the schema is fixed and uniform for all rows),
• the value may itself be some complex structure.

An HBase "table" is a multi-map structure

• a first one, called "column" in HBase terminology (an unfortunate choice, since this is hardly related to the column relational concept),
• a second "timestamp" (each map is named after its key).

The full picture: rows and tables

1. a "table" is actually a map where each row is a value, and the key is chosen by the table designer.
2. we already explained that the structure of a "row" has little to do with the flat representation of relational row.
3. regarding data manipulation, the nature of a "table" implies that two basic operations are available: put(key, row) and get(key): row. Nothing comparable to SQL here!

References

• Understanding HBase and BigTable
• The HBase Wiki page

The FIRE Research Workshop will be held on May 16th 2011 in Budapest (Hungary). This event is part of the Future Internet Week 2011. LAWA is proud to give a presentation in the Future Internet, Living Labs and Web analytics session.

Presentation Abstract
Organizations like the Internet Archive have been capturing Web contents over decades. This time-versioned content is a gold mine for analysts, focusing on longitudinal studies. An application example is tracking and analyzing a politician’s public appearances over a decade. The LAWA project develops methods and tools for time-travel indexing and querying, entity detection and tracking along the time axis, and advanced analyses and knowledge discovery. For scalability, we pursue Hadoop-based distributed computations. We also prepare reference data and will provide analytics services. We will offer a user workshop in late 2011 to disseminate these opportunities and explore interesting use cases.

LK partners are present at FET 2011 in Budapest, Hungary in May 4th-6th, 2011

Looking for a search engine to find images or web pages about David Beckham arranged in terms of the various clubs he has played for?
A 15 minute demonstration of LivingKnowledge showcases technologies enabling bias-aware, diversity-aware and evolution-aware information access, including diversity-aware search within texts and images, analysis of future predictions as well as fact-and-opinion extraction.

What is it about?
The SCAPE project will develop scalable services for planning and execution of institutional preservation strategies on an open source platform that orchestrates semi-automated workflows for large-scale, heterogeneous collections of complex digital objects. SCAPE will enhance the state of the art of digital preservation in three ways: by developing infrastructure and tools for scalable preservation actions; by providing a framework for automated, quality-assured preservation workflows and by integrating these components with a policy-based preservation planning and watch system. These concrete project results will be validated within three large-scale Testbeds from diverse application areas.

SCAPE also has a Twitter account: @SCAPEproject
Tweets with the hashtag #SCAPEproject (or pointed to @SCAPEproject) will be re-tweeted and appear on the website’s Twitter feed.