DBChEx: Interactive Exploration of

Data and Schema Change

Tobias Bleifuß

Hasso Plattner Institute

University of Potsdam

tobias.b[email protected]

Leon Bornemann

Hasso Plattner Institute

University of Potsdam

leon.bor[email protected]

f[email protected]

Divesh Srivastava

AT&T Labs – Research

div[email protected]

exploring static datasets, dynamic datasets that change over

time still lack effective exploration support.

To address this shortcoming, we present our innovative

tool Database Change Explorer (DBChEx) that enables ex-

ploration of data and schema change through a set of ex-

ploration primitives. Users gain valuable insights into data

generation processes and data or schema evolution over time

by a mix of serendipity and guided investigation. The tool is

a server-client application with a web front-end and an un-

derlying database that stores the history of changes in the

opment of numerous methods and systems to manage and

document changes to data. However, only recently the drop-

ping prices of hard drive storage rendered it possible to now

also keep all or at least a large portion of historical data

for analysis of its changes. The nature of changes reveal

rich insights that cannot be found in a static version of the

dataset and can serve many purposes, such as summariza-

tion, compression, future prediction, provenance, or pattern

discovery.

Imagine a data scientist Alice who has gathered a number

January 13-16, 2019 , Asilomar, California, USA.

a rough understanding of the current state of the dataset.

While analyzing the dataset, a set of questions arose that

only the history of changes to the dataset can answer. For

example: Am I right to assume that attribute X is constant?

How old are the entities in table Y? When were they last

updated? Do the changes to entities of table Z correlate?

However, answering those questions using existing tools is a

tedious task, especially if the historic dumps originate from

different systems and use different legacy schemata.

e.g., defining different namespaces in the same database, she

would still store highly redundant data.

The queries to actually identify the change would also be

non-trivial and probably inefficient, as one would have to

join a large number of different relations. Due to the com-

plexity and poor performance, Alice quickly loses interest

and she is less creative than she could be: she writes ad-hoc

queries, executes them, waits for their completion, reconsid-

ers her assumptions, writes ad-hoc queries, . . . If she did

not face all of those issues, but had a suitable tool for ex-

how to recognize, explore, and evaluate change in data and

schema. We have published a comprehensive vision paper

on change exploration [2], which touches on many areas that

aim, among others, to capture, explore, categorize, and pre-

dict changes in databases. We have also taken initial steps to

make parts of this vision a reality. For example, we track ob-

jects in databases over time to understand their changes at

a finer level. In addition, for an overview of the large num-

ber of changes, we developed a framework to map change

histories to time series and cluster those time series [3]. We

implementing as part of our project “J

J

anus”

.

In this paper, we present an innovative tool that imple-

ments several aspects of this vision to support Alice in her

exploration of change: the Database Change Exploration

(DBChEx) tool. DBChEx allows Alice to interactively ex-

plore change data. The goal is to provide an easy, interactive

way to obtain a high-level understanding of how a dataset

changes over time. By physically implementing exploration

primitives and operators from our vision, DBChEx allows

In the next section we describe the general ideas behind

change exploration with DBChEx. Then follows a more de-

2. CHANGE EXPLORATION

As a data model to represent changes, DBChEx uses the

notion of change-cubes [2], which are characterized as fol-

lows:

Definition 1. A change c is a quadruple of the form

a set of changes a change-cube C = {c

the changes, combinations of (t, id, p) are unique.

A change ht, id, p, vi describes that at a certain point in

tions of properties are modeled through setting its value to

the Null-value (⊥).

This format is flexible enough to handle changes in both

data and schema, and general enough to integrate data from

various different formats or domains. Thus, the DBChEx

tool serves the exploration of changes from various sources of

data, individually and jointly to reveal correlations between

those changes. During the demonstration of DBChEx we

show how Wikipedia infobox data and IMDB movie data

changes over time.

change exploration workflow using DBChEx.

3. ARCHITECTURE

Figure 2 shows the general components of DBChEx. The

user interacts with the system through a web client. This

web client communicates with a node.js backend via http or

WebSockets. The backend, in turn, uses a database via a

SQL interface – in our current implementation MonetDB [8]

– and a data clustering service that is currently implemented

in Apache Spark [15].

3.1 Web client

of change-cubes. This set can be sorted by multiple statis-

tics, effectively turning that set into an ordered list. On the

right side of the interface, two different tabs display previ-

ously visited operator sequences as well as bookmarks. The

user-interface is designed to keep the user engaged and re-

duce the amount of required text input. Through clicking

on elements that appear interesting, the user can modify the

current operator sequence.

primary navigation through the changes: they represent

While browsing through the space of possible sequences,

the user might fear to lose track. Since this can cause coun-

terproductive hesitation, we introduce two features that are

inspired by web browsers: a history of previously executed

operator sequences and the possibility to bookmark cer-

tain operator sequences that the user considers interesting

enough to save for future investigation. The tool also offers

the ability to explore datasets collaboratively: the URL in

the browser reflects the current state of the exploration, so

that the user can share the current view by simply copying

for each of the cubes, i.e., a normalized measure for their

degree of variation. These statistics and visualizations can

serve as a recommendation and guidance to the user for fur-

ther analysis. The user can click on any of the elements to

apply a filter that removes unwanted entities or properties

from the change-cubes.

At each point in the exploration it is possible to switch to

visualizations that are in sync with the current exploration

progress and guide further exploration steps. A helpful visu-

alization for volatility is a heatmap that allows users to see

the fields in the heat map activates a corresponding slice that

focuses only on the changes that are responsible for the color

of that field. At the moment, the visualizations in DBChEx

are still rudimentary, but further visualizations are easy to

add. Especially visualizations that rely on animations for

the time domain similar to Gapminder

or map-based visu-

alizations [11] could be very helpful.

While focusing on certain topics can already be helpful,

sometimes it is difficult to spot interesting subgroups through

Figure 3: The DBChEx web user interface, here with one line/change-cube per settlement

Figure 4: A heatmap for changes on selected Wikipedia settlement entities and selected infobox properties, color-coded relative

to the absolute number of changes.

tering is an additional feature of the change-cubes that can

serve as a filter or grouping criterion. Additionally, the clus-

perfect at first and in the course of exploration, the user will

notice opportunities for improvement. Due to the closed-

loop approach, our tool offers the possibility to implement

these directly and to create a view on the change-cube with

just a few clicks, which implements these insights. For ex-

ample, the user can detect a renaming of a property, link

these two properties, and profit from a longer history avail-

able for this new merged property. Users employ the already

mentioned query operators to select the parts of the change-

cube to be changed. The modification operators differ from

end needs to translate the operator sequences to SQL queries

queries that return the removed and added changes. By

simply linking the original change table by EXCEPT or UNION

with the two subqueries, we obtain the modified cube.

This modified cube may contain changes that are either

(i) inconsistent or (ii) redundant. Two changes are incon-

sistent, if they both set the value for the same property of

the same entity at the same time and do not agree on that

value. So they violate the uniqueness-constraint mentioned

in Section 2. For now, we rely again on the database to de-

tinct infobox templates that are merged or attributes that

are renamed. For IMDB, the user can also observe schema-

changes, but much less frequently. IMDB is an example that

has attributes of highly different volatility. For instance, the

ity in contrast to its primaryTitle. In the future we plan

to make DBChEx an open-source tool and also provide our

datasets in an accessible way.

of Chicago again, but unlike Alice he focuses on changes

that contain Chicago in the value dimension. Here he finds

a lot of changes on the same day that update the sub-

division name3 of various locations in Chicago. At this

point the following feature of DBChEx can help Bob: Once

the user has found an interesting change, the tool provides

a dataset-specific link back to the original source of the

change. For Wikipedia, the tool provides context informa-

tion (user, comment) and a link to the diff-page of the re-

vision, while for other datasets the tool could for example

Figure 5 gives a short overview of a small exploration sce-

nario on IMDB, for which we have gathered 47 daily snap-

shots. Assume that Alice first performs a split by property,

which results in one change-cube per property as shown in

Figure 5a. For each of the change-cubes, the tool displays a

number of statistics, for example the distribution of changes

over time. A large spike of changes for the properties episo-

deNumber and seasonNumber on 2018-02-02 catches Alice’s

eye. For further inspection, she prunes all other change-

cubes and keeps only those two change-cubes for both prop-

Time Data Changes (original) Changes (transformed)

2018-02-05 t001|"nm001,nm002" 2018-02-05,t001,principalCast,"nm001,nm002"

2018-02-06 t001|"nm001,nm003" 2018-02-06,t001,principalCast,"nm001,nm003" 2018-02-05,t001|1,nconst,"nm003"

2018-02-10

their seasonNumber to 1. She looks up some of those enti-

ties and realizes that all of them are from the same series

“One Piece”. On that day more than 800 episodes of that

series got merged into one season, which leads her to the

conclusion that there must be a (semi-)automatic way that

allows users to perform such bulk changes. Her assump-

tion was substantiated when she inspected the other spike

in Figure 5a for changes of the property parentTconst, which

identifies the parent TV series. On 2018-02-03 this property

was changed for 206 episodes from tt0338640 to tt0209736.

approach to have a longer history of data available across

this schema change. principalCast was previously a comma-

separated list of nconst references. By concatenating tconst

and the position of individual nconst elements as order-

ing, new entity IDs are created that correspond to the new

schema. These new entities have exactly one property nconst,

while the other properties (category, job, characters) are all

Null. This transformation may turn a change before 2018-

02-10 into several changes affecting several entities, but some

of the changes caused by the schema change may also dis-

results created by the Bellman tool [6], Dasu et al. have ex-

plored how data and schema changes in a database can be

observed through a limited set of metadata [5]. That work

has focussed on the case of only limited access to the data-

base. In contrast, we assume full access to the database and

its changes, and are thus able to focus on more fine-grained

change exploration. Another related field is the visual ana-

lytics of data [7]. There are a large variety of visualizations

for time-oriented data [1], some of which are also imple-

mented by commercial tools, such as Tableau

. However,

tion through the large space of possible explorations. Given

meaningful transformations to time series, time series explo-

ration techniques and tools [16] can also be used to visualize

and interpret change behavior.

6. CONCLUSIONS

With DBChEx we present an interactive exploration tool

that enables the user to explore changes of data and schema

in an innovative way: it operates on the newly defined change-

cubes through a set of exploration primitives. A unique

feature is that our framework treats changes as first-class

dependent data. That is, instead of exploring changing data,

our tool supports the user in exploring data changes. We

embrace the fact that these data changes also need some

modelling through an iterative, closed-loop approach, which

allows users to improve modeling decisions in the course of

their exploration.

Our current approach scales as far as the database can

answer the queries in reasonable time. However, for larger

or streaming datasets there are possible improvements for

[2] T. Bleifuß, L. Bornemann, T. Johnson, D. V.

[3] L. Bornemann, T. Bleifuß, D. Kalashnikov,

[4] U. Cetintemel, M. Cherniack, J. DeBrabant, Y. Diao,

[6] T. Dasu, T. Johnson, S. Muthukrishnan, and

[7] M. F. De Oliveira and H. Levkowitz. From visual data

[8] S. Idreos, F. Groffen, N. Nes, S. Manegold, K. S.

[10] N. Kamat, P. Jayachandran, K. Tunga, and A. Nandi.

[11] D. Mashima, S. Kobourov, and Y. Hu. Visualizing

[12] S. Sarawagi, R. Agrawal, and N. Megiddo.