gramps-project.org

Writing code to find out something about your family tree in Gramps isn’t particularly hard, but it is tedious. And I find the more tedious code is to write, the more errors I make.

Let’s take a look at some random example of what it takes to do something in Gramps. Consider the problem of finding out how many people were born in March. Basically, we need to go through the database, get the reference (if one) to the birth event, look up the birth event, if found, get the date from it, get the month and compare it. That might look like this:

count = 0
for p in db.iter_people():
    birth_ref = p.get_birth_ref()
    if birth_ref:
        event = db.get_event_from_handle(birth_ref.ref)
        if event:
            date = event.get_date_object()
            if date.get_month() == 3:
                count += 1

This example is not unusual; you’d have to do something like this whatever it is that you are doing. And this is a simple example. It can get much more complicated. But even in this “simple” example, you have to know a lot about the gen.lib objects that make up Gramps (things like Person, Event, and Date) but you also need to know the interface to the database (things like iter_people, get_event_from_handle) and how things are linked together (handles and refs).

I’ve been working on Gramps-Connect, a web version of Gramps, and it starts from a completely different premise. As it is based on a query language (and built from Django), the above question can be done in one line:

Person.objects.filter(birth__month1=3).count()

This form takes a bit of getting used to, as it has some different syntax from regular Python. You need to read the double-underscore as a dot. You still have to know the names of the tables and fields, but everything else is just standard Django. There are some nice things here: there is no reference to the database, and there is no explicit mention of handles to manually lookup related data.

I wondered if we can take some of the ideas from Django and apply them to Gramps to end up with a better interface. I’m not that interested in writing a query engine for Gramps (Gramps is implemented in a schema-less graph-like datastore, not anything like a relational database), but I would like to simplify the manner in which developers interact with Gramps.

Here is how you would solve the above using this experimental Gramps addon:

from libaccess import *
init(dbstate.db)
len([p for p in Person.all() if p.birth.date.month == 3])

That’s it. Now there’s quite a bit going on here, but I think the most interesting is that this works even if a person doesn’t have a birth event, or if the birth event happens to be missing (eg, the database is in an inconsistent state). By allowing this, it makes chained accessors (item.item.item) possible to use to get to the endpoint to make the comparison (month == 3). Also, one doesn’t need to know anything about the database or gen.lib objects other than the field names we’re interested in.

The Gramps development team will be looking at alternative interfaces for Gramps 3.3 for a variety of reasons. Perhaps you have some ideas.

You can play (and make changes to) this experimental interface by running Gramps trunk, and using libaccess from gramps-addons. Or wait a couple of months for Gramps 3.2!

-Doug

One gift I give to my family each year at this time is a calendar made with Gramps’ Calendar Report. You can print a subset of your family, print their birthdays, anniversaries, and even show holidays from your country, in your locale’s language. Here is a sample page from that report:

I usually put an old picture on the cover of the calendar, but this year I thought I’d do something different. I had been thinking about something similar to the Surname Cloud Gramplet:

That is a nice representation of the data, and also useful: you can double-click any of the surnames and you’ll get a list of all of the people with that surname. But I was looking for something more artistic. Then I remembered Wordle. Wordle let’s you make images like:

I noticed in the Wordle “advanced” options that one can post a list of words, and their weightings. It wouldn’t take much to make the Surname Cloud Gramplet output the right information, so I created the Wordle Gramplet. It outputs the data, you paste into Wordle, and voila:

If you’d like to make a Gramps Wordle yourself:

1. Download the zip from WordleGramplet.zip
2. For Gramps 3.1, take just the file WordleGramplet.py from the zip file and put it in your .gramps plugins directory.
3. For Gramps 3.2 (trunk), put the unzipped WordleGramplet directory in your .gramps plugins directory.
4. Start Gramps, go to the Gramplet View, right-click in an open area, select “Add”, and select the Wordle Gramplet
5. You can change some of the defaults by detaching the Gramplet, and selecting the Options area:
   
6. You can change the number of different font sizes to use, select a filter to use, and click Save.
7. Go to Wordle.net/advanced and paste the output of the gramplet (just the “words: weights”) into the top box
8. To add the year, put “2010: weight” in the box too, where “weight” is the number representing the size of the year. In the example above, I made it twice the weight of the most common word (eg, “2010: 10″).
9. You will need to de-select “Remove numbers” under “Language” in the Wordle editor. Change the font, layout, colors, etc.
10. I used the “open in window” option and made the window big, then took a screen capture.

That’s it! If you have questions, please leave a comment below. If you make a Gramps-Wordle, leave a link to it. Happy New Year!

-Doug

My recent work on the gramps database objects has led me to reflect upon their structure from an abstract data type point of view. I’m hoping that this blog entry will start a discussion that will ultimately lead to the development of a true ADT-based structure for gramps databases with multiple implementations.

Current structure using BSDDB

The current implementation of gramps uses BSDDB as the storage engine.  This has served us well and provides an efficient back end with good Python wrappers to access it.  (For those wanting more detail, see the official documentat Read the rest of this entry »

If you were looking for a heart-warming story of coping with loss, I’m afraid you’ll have to look further. I’m not talking about that kind of closure but rather this one.

When I was supposed to be enjoying good weather and good beer, I found myself thinking about functional programming instead. Now, that’s a big topic that I’ll let you explore on your own if you like, but one aspect of it that I thought I could apply to my work was closures. So, what’s a closure? An example would help:

def f(x):
    def g(y):
          return x + 1
    return g

add2 = f(2)

Now, whenever I call add2 in my program, I can add two to something else, like this:

print “two plus two equals”, add2(2)

Let’s walk through what happens when I call function “f”. You’ll see that the first thing f does is define function g. However, g uses (in its return statement) the value ‘x’ that was passed to f. Essentially, that value of x is bound to the function g that is defined. After the definition of g, the value of g (that is, the reference to the function) is passed back to the caller of f. At that point, I could delete f if I wanted to, since what I will really use is the result of f — a custom function.

OK, this is a toy example. What about something we can actually use? Well, in my latest commits to read.py, I’ve done just that. I added several iterator functions that are built using closures. Open it up and look around line 770. There you should see the definition of a closure “_f,” that defines a function “g” that is returned to the caller. In this case, “g” is an generator function (because of the “yield” statement). When “_f” is called in lines 782-789, it is given an argument which is another function in the same module — one that returns a cursor over the corresponding object type (people, families, events, …, etc.). However, those functions take an argument, as all normal methods do, of the object instance (usually called ’self’). So, the function “g” that “_f” defines uses that argument to pass to the cursor function.

Starting at line 791, you should see another closure defined and used. I used the same name since the name of the closure function is not important, (nor is the name of the function defined inside it). In fact, I could (and maybe should) delete “_f” when I’m done with it, just to be tidy.

So what’s the advantage of this approach? For one thing, it is succinct, and allows for a compact expression that you can see at a glance. In these cases, I’ve replaced 8 methods (plus a possible helper method) with one closure definition and a bunch of function calls. The common code is self-contained and leaks no attributes to the namespace around it (a good practice of functional programming).

I bet you’re wondering what the catch is. Well, there are a couple:

1. You can’t override the closure function or anything inside it. Since this happens at the definition of the class, nothing overridden in a derived class is available for use when the closure is called. That means that I cannot override “get_person_cursor” in a derived class and expect the override to be used when my closure is called.

2. Since you are creating a custom function with each call to the closure, you generate somewhat more object code. In contrast, with the approach used in other methods in this module, I would define separate methods for each object type (like get_person_cursor) and call a helper function from within it (__get_cursor in that case). If the closed function is short (like these ones are), you probably come out about even; if the closed function is long and involved, that would mean several almost-identical functions in the object code and you might wind up with a larger file after compilation.

Where to next? For the moment, I’d just like to leave it out there and see what happens. Look for breakages, etc. Down the road, this is just one more tool to help us build good code. Within the domain of functional programming, there are lots more.

At last I’m ready to release the fruits of my labor. Today I will be making two large commits to gramps trunk:

1. Changes to the basic database objects and other modules to activate the new code.
2. Changes to the proxy database objects to (hopefully) reduce their processing time.

The first group “flips the switch,” so to speak, on the new modules read.py and write.py. These contain the methods formerly spread between base.py and dbdir.py. The modified base.py is now strictly abstract (or aspires to be!) As their names imply, read.py contains methods necessary to read the database; write.py contains methods used to add, update and delete database records. I’ve been working with these for more than a month and have tested them on databases large and small, running through all typical functions from creation to reporting, import to export, and utility functions like check and repair, rebuild secondary indices and rebuild reference maps. Now is the time to turn it all loose and find and fix the breakages I missed!

The second group of changes concerns the proxy databases — work I began then suspended until the larger work neared completion. I hope that you see performance improvements from these changes. I know that I do. One interesting thing I found is that, when considering a record for inclusion, each proxy object would call the get_”object”_from_handle methods and check the results. This often meant double calls, since proxy a might be running through the objects in its database, which can cause it to call proxy b first. But it happens that if proxy b is already responding to an iterator call, it has already filtered the results it is passing to proxy a, so it is redundant for proxy a to call proxy b again for each record. I circumvented this by implementing a dynamic get_unfiltered_”object” method, that drills down to the underlying “real” database. You can see how it is used in the new predicate functions (e.g. “include_person”). The new methods are built on-the-fly using a __getattr__ hook.

The committed revisions for these two updates are 13139 and 13140.

As mentioned in the previous article, I’ve been working on restructuring the database objects to remove obsolete code and refactor the objects by functional area. One key area that I mentioned before has to do with managing the undo/redo capability.

In the main gramps window under “Edit”, there are three items that may or may not be grayed out: Undo, Redo and Undo History. These rely on methods in the database objects that manage a separate database where transactions are kept in case the user makes a mistake and wishes to undo (or redo) some change. These methods work closely with (but are separate from) the transactions discussed in part I. When you change something, the process from a high level is:

1. Start a new transaction
2. Write the specifics of each record change (old and new data) to the undo database
3. Commit the transaction, which causes:
   1. Records to be added, deleted or updated
   2. A reference to the transaction added to a special transaction list

The transaction list is what drives the undo/redo menu. The index of the current entry is kept in a special variable, “undoindex.” When a user wishes to undo or redo a transaction, the program takes the transaction pointed to by the undoindex (or the undoindex + 1 for redo) and reverses it — changing an add to a delete, a delete to an add, and replacing new data with old data for updates (the reverse for a redo operation).

Today, the undo database is a BSDDB recno database managed in part by the transaction processor and in part by methods in the main database class. I’ve been able to consolidate this work in a new undoredo module, which also contains a level of abstraction so that we can use any kind of storage for the undo database, as long as it supports methods like append and get. This means that it is trivial (and my test code does this) to use a simple Python list for the same thing. So, why use a database at all? It’s a matter of size and scalability.

Currently, there is a hard-coded limit of 1000 entries in the undo transaction list. However, a single transaction can comprise many records in the undo database, with people and families and events and sources and references between them all requiring updating, so 1000 entries can easily mean several thousand records in the undo database. Considering that a single record is usually between 250 and 500 bytes (though it can be much larger), the amount of storage to hold everything in memory can be a little much. (I suppose this is a matter for discussion, since RAM is generally cheap these days.)

The new undoredo module is about ready to go, but a problem remains. In fact, it is a problem today. If we reach our limit of 1000 entries, the program starts overwriting the undo data, which not only means that you can’t reliably undo transactions anymore, but also that you can’t use the “Abandon Changes and Quit” option from the main menu. I’m not aware of any complaints, but that may be just because 1000 is a lot of updates — probably more than the average user will make during a single session.

The question is, “Should we worry about this?” We can ignore the problem — after all, this is the way things work today — or we can try to design a better way. Some options I can think of:

1. Do away with the 1000 transaction limit. As long as the updates are kept on disk, the transactions referred to by the list are usually under 50 bytes. Do the math and you can see that this does not grow too quickly.

2. Keep some limit but enhance the undo/redo functionality to work on a ring — possibly even keeping the transaction list as a linked list where the tail points back to the head (thus making a ring) with an attribute that identifies the current entry.

3. Do nothing since apparently no one has hit this problem so far.

4. Something else?

Here’s where I would like your input. Please do some thinking and try to work out some answers to the above problem. Let’s try to solve this together!

The classes and methods that manage the gramps database are at the heart of every operation in gramps — from importing, entering and editing data to producing reports and exporting data to other formats. I recently became intensely interested in the operation of this part of the codebase, and began a project to understand it and streamline it where possible. This article is the first in a series I plan to write about that work.

The gramps database is made up of several files typically stored under the .gramps./grampsdb/ directory, where is a name that is generated when a database is created. If you look at the contents of that directory, you will see files whose names betray their contents: person.db, family.db, event.db etc. along with various cross-reference files. Storing the data in several files increases the robustness of the system (since a problem in one file does not normally affect the others).

Today, most of the work of managing this structure is done by two modules: base.py and dbdir.py, both in the src/gen/db directory of the source tree. These are rich and complicated modules containing several classes and many, many methods. As I began to study these modules, I saw that they were deeply interconnected (largely due to the history of development) and that they contained important subfunctions alongside the basic methods to retrieve, add, modify and delete data. I then began to look for ways to consolidate similar functionality and factor the code by major function group.

Note: None of these changes have been committed to SVN yet. I want to build and run many unit tests as well as integration tests before I do that, though I might commit some of the independent (and currently unused) new modules for safe keeping. Meanwhile I just rsync them between two different machines.

Transactions:

The first major subfunction I worked on was transaction processing. When gramps is ready to write changes to the database, it does not do so directly. A simple change made via one of the editors may actually affect several of the files in the database structure. To protect those files, the changes are grouped into a transaction. Once all the changes are gathered into the transaction, they are then committed together. Also, the original change is saved for possible undoing later on.

The code for saving and eventually writing the data to the database is currently split between a Transaction class and methods in dbdir.py and base.py. I realized that there these methods could be brought together in one class that could then be placed in its own module for safekeeping. Thus was the GrampsDbTxn class born.

The GrampsDbTxn class handles all the functions of storing transactions and writing them to the database. It also contains a level of abstraction in that it does not matter what the underlying database access method is, so long as it supports put() and delete() methods as well as (possibly null-operation) methods for starting a database-access-method transaction and committing it later. Today, gramps uses BSDDB as our DBMS and it contains a (lower-level) transaction function similar to the one GrampsDbTxn provides for a gramps database.

To begin a gramps transaction, one calls the transaction_begin method of the database, usually like this:

transaction = self.db.transaction_begin()

Then, as changes are made, calls are made to add those changes to the currently-active transaction, usually like this:

transaction.add(…)

However, due to the interconnectedness of the gramps database files, you will see modules calling commit_person() or commit_family() or remove_person() etc., which modules eventually call transaction.add().

Later, when all the changes that are to be made are done, the transaction.commit() method is called, which does several things:

1. Begins a DBMS-level transaction
2. Adds any new records
3. Updates any existing records
4. Calls any signal handlers to let them know about the changes
5. Deletes any records that are so marked
7. Updates the cross-reference files
8. Commits the DBMS-level transaction

Note: It is important that step 4 precede step 5 since to give the signal handler the opportunity to inspect the to-be-deleted data before it is gone.

Oh, I almost forgot to mention that, when transaction.add() is called, the information regarding the pending operation is also written to the undo database (a separate file in the gramps subdirectory) so that the transaction can be undone (and later redone!) if necessary. (I plan to write more about the undo/redo operations in a later installment.)

A BSDDB wrapper:

As mentioned above, BSDDB contains a transaction mechanism of begin…commit. It is actually a natural mechanism for the Python “with” statement which uses context-managers. Unfortunately, these are not directly supported by BSDDB, so I built a little wrapper class for the purpose. Today, you would need to write:

from bsddb import db
env = db.DBEnv()
…
env.txn_begin()
…
db.put()
db.delete()
…
env.txn_commit()

With the new wrapper class, you can write:

with BSSDBtxn(..) as txn:
    txn.put()
    txn.delete()

and the commit method is called automatically when you fall off the end of the “with” statement (unless you call txn.abort() in the middle). The result is more compact and robust code. I plan to exploit this whereever possible.

While I was at it, I also added context management to the GrampsDbTxn class, so it is possible to write:

with GrampsDbTxn(…) as txn:
    self.db.commit_person()
    …etc..

and the GrampsDbTxn commit method is automatically called at the end.

Well, that’s surely enough for now! Look for Part II, which will examine the undo database

According the subversion log it has been two years since I last committed anything. Whilst I have been lurking on the mailing list for the last two years I have not had much time for doing much else. I have been fielding support queries on the MacPorts port for gramps but that is about it.

Recently I have been helping Emrys Williams to build the new native OS X port and this has fired me up to get a bit more active.

I hope to make some real progress this time on the research planning side of things. A long time ago I helped to implement support for Repositories, but this was meant to be just the start of a set of features to support people in making the most of the short opportunities we get to do real research. I want to develop some tools that will make GRAMPS your research friend.

Lets hope that I can make some progress this time and it is not another two years before real life gets out of the way.

Richard

PS.

I was at EuroPython this year and at the start of Bruce Eckel’s keynote he said how he had arrived at the conference after a bad international flight, tense and in a bad mood. But when he walked in to the conference he sensed the atmosphere and thought Ah Python people and immediately relaxed.

Well after two years away and involvement in various other projects I have the same reaction with I get back into GRAMPS. Ah GRAMPS people

It is nice to be back, at least for a while.

Before leaving on holiday, I committed yesterday evening the present state of the reworked event tabs for version 3.2

There is still time to tweak this or change things. I like the state it is in though. To recap, what is it about? The idea is to show in the list of events also the family events. This makes the person editor more useful as it shows all events a person is involved in, allows to see possible errors more easily, and it is an often requested feature.

The way things are done is by changing the event list in a tree list, with nodes for the personal events and for every family. See the screenshot:

Person event list

Let me tackle the main comment people will have: is it not possible to just show a flat list with a column or color indicating what are the personal and family events? Yes, that would be possible, but it is not done for the following reasons:

• The person editor is not a report. The Timeline quickview is a report that can do ordering. It could be made a gramplet making it perhaps more usefull. The most important use case for the person editor is to enter new information. Therefore, information overload must be avoided.
• The event tab must be understood by looking at it, so as not to scare new users.
• The event tab must educate users also in good practices. It is important to understand the difference between personal events, and family events. This distinction comes from the GEDCOM standard. In GRAMPS events are stand alone objects that can be shared, with a unique event reference for every person or family it is coupled too. This is a great feature, but can lead to uncertainty in how to use the events. The new event tab should make the use case obvious: a family event is an event the parents of a family share (and perhaps influences the children), whereas all other events should be personal events shared between all participants.

So what is possible on this new event tab?

1. Everything that was possible in the old event list
2. See which families the person is a parent in. Specifically, one can now already in the person editor see how many times a person married, when this happened, …, without needing to open the relationship view. This implicates that a family which has no events is also shown in the event tab.
3. Reorder these families using the up/down buttons on the group line. This up to now was only possible in the relationship view but not many users know this is even possible. The tooltip of the reorder button now clearly states it too.
4. Go with one click on the edit button to the families a person is part of. That is, one cannot change the family events in the person editor, if one tries that, the family editor opens instead. Now many people are annoyed they have to move from a person editor to a view to open the family or the editor of the spouse. This can now be avoided.
5. Drag a family event to the personal events so as to make a copy of it as a personal event. There should be no reason to do this as the person is already part of the family event, but yes, it is possible now.

One thing people will note is that family events of families a person is child in are not shown. My interpretation of family events is that they first and foremost involve the parents of a family, not the children, though obviously something like a divorce might have a big impact on the children. I’ll see how users react to determine to include family events as a child always, never, or via a preference option. My fear is adding parent family events will start to make the event list very crowded, and be confusing to newer users and to what a family event really is about.

Well, having changed person events, also the family event tab has undergone a similar change:

Family event list

Again, here only the events of the parents are shown after the family events, no children events. Again, I’m not sure how relevant children events are for a family in se. A nice gramplet/quickview seems like a better idea as determining what is relevant can be very difficult. Eg, death of a child is important to a family if it happens while the parents are still alive, but not afterwards as the family in essence does not exist anymore. So what to show? Let’s see how this presentation is recieved and go from there.

The family event tab does not add real extra features as the person event tab does. One can edit the parents from the event tab now, but one could already to that in the family editor. One sees all events, but in present editor father and mother birth and death where already shown too.

Some things to note. Sorting in the columns works, but only sorts inside the groups. The database order is what is shown on start up, and as soon as one starts to drag the view will also change and revert to this original order. After all, drop changes how things are stored in the family tree. One obtains the original order also by clicking on the first column, so clicking there does not sort on description, but reverts to database order. In the present version there was no way to return to the original database order after doing a sort.

As a post scriptum, a screenshot of the final version of the names tab, for those not running the development version.

Person Name list

Note that there are 3 new columns. One now sees the group as value of the name, if the name has sources attached to it, and a preview of the first note if that is present with the name.

Comments?

Benny