119
edits
Changes
no edit summary
{{out man tip|The advice on this page was for older versions of dateGramps so may not work for you. Please update as needed.}}{{stub}}
Comparison of performance on large datasets between different Gramps versions
If you want to work with a large database, read [[Tips for large databases]].
==General setup==
Comparison should be with equal hardware, and on the same datasets to be fair. Optimal representation may be chosen, so for Gramps, tests are done in the native database format, called GRAMPS GRDB format or GRAMPS XML format.
Should somebody want to publish results of commercial software under windows, this is allowed, but should be fair: same hardware and dataset, so test on a dual-boot machine, and use the internal format of the program.
=== Genealogical datasets ===
{{man warn|Warning|Private datasets will not be shared under any reason. <br><br>Free datasets are given under the following copyright: use for testing of genealogical programs only, no publication, no sharing. They have been created with free information on the net of which where the users posting author explicitly state it can stated their dataset may be used re-distributed freely.<br><br> Should However, should you however feel certain data is misplaced, or that the original posting author does did not have the right to distribute the data, please contact us to remove any information as necessary.}}
'''FAQ'''
* ''My computer hangs on upon open, eating memory?'' These are LARGE datasets, so do NOT open them directly. For Gramps open them as follows: create a new Family Tree. Open it and go to the import menu and import the dataset.
* ''What is tar.bz?'' This is a compression format. You must uncompress the file before importing it
* ''Can you provide the GEDCOM?'' No. Offering GEDCOM has the danger of attracting sample would tend to much attract excessive traffic to this sitenot related to Gramps. If you need must have GEDCOM, you should could install Gramps, import the dataset, and then choose "Export to GEDCOM".
* ''What is in these files?'' See summary at the bottom of this page.
{| {{prettytable}}
|-
!Test Code !nameFile Name
!Download size
!People
!Size(MB)!CopyrightLicense
|-
|<!-- Code -->[[#Summary_of_database_test_d01|d01]]|<!-- File Name -->Doug's test GEDCOM| <!-- Download size --> - | <!-- People --> 100993|<!-- Size (MB) -->32MB|<!-- License -->Private
|-
|<!-- Code -->[[#Summary_of_database_test_d02|d02]]|[http://www.gramps<!-- File Name --project.org/files/stresstestdata><strike>testdb80000</testdb80000.gramps testdb80000]strike>| <!-- Download size --> 11.2MB| <!-- People --> 82688 |<!-- Size (MB) -->70MB|<!-- License -->Testing only, no sharing, no publication <br>{{man menu|*** NOTE: THIS FILE IS MISSING. <br>IF ANYONE HAS A COPY, PLEASE CONTACT nickwebmaster@gramps-project.org ***}}
|-
|<!-- Code -->[[#Summary_of_database_test_d03|d03]]|<!-- File Name -->[http://www.gramps-project.org/files/stresstestdata/testdb120000.gramps.tar.gz testdb120000]| <!-- Download size --> 14.8MB | <!-- People --> 124032|<!-- Size (MB) -->88 MB|<!-- License -->Testing only, no sharing, no publication
|-
|<!-- Code -->[[#Summary_of_database_test_d03|d03_alternate]]|<!-- File Name -->[http://www.gramps-project.org/files/stresstestdata/test_2011-09-07.gramps.tar.bz2 test_2011-09-07.gramps]| <!-- Download size --> 11.9MB | <!-- People --> 124032|<!-- Size (MB) -->88.4MB|<!-- License -->Testing only, no sharing, no publication (d03 for Gramps 3.3.x)
|-
|<!-- Code -->[[#Summary_of_database_test_d04|d04]]|<!-- File Name -->Jean-Raymond's test GEDCOM [http://forum.geneanet.org/index.php?topic=389170.0 french forum]| <!-- Download size --> -| <!-- People --> 52699| <!-- Size (MB) -->13.6MB|<!-- License -->Private
|-
|<!-- Code -->[[#Summary_of_database_test_d05|d05]]|<!-- File Name -->[http://www.gramps-project.org/files/stresstestdata/places.gramps places.gramps]| <!-- Download size --> 2.5MB | <!-- People --> 65598 place objects|<!-- Size (MB) -->15.3MB|<!-- License -->Testing only, no sharing, no publication
|-
|<!-- Code -->[[#Summary_of_database_test_d05|d06 ]] (same as d05, but gramps42 format)|<!-- File Name -->[[Media:Places-2.gramps.zip]]| <!-- Download size --> 2.8MB | <!-- People --> 65598 place objects (expanded)|<!-- Size (MB) -->22MB|<!-- License -->Testing only, no sharing, no publication|-|<!-- Test Code -->|<!-- File Name -->|<!-- Download size -->|<!-- People -->|<!-- Size (MB) -->|<!-- License -->
|}
{| {{prettytable}}
|-
!Hardware Code
!Processor
!clock
{| {{prettytable}}
|-
!Test Code !! test Test Description
|-
|T01 || Time to import GEDCOM/GRAMPS in empty native file format (GRDB)
|-
|T01_a || Time to import GEDCOM/GRAMPS XML in empty native file format (BSDDB)
|-
|T02 || Size native file format (GRDB)
{| {{prettytable}}
|-
!Comp Hardware Code !! Gramps !! data !! T01 !! T02
|-
|H03 ||bgcolor="#ffa0a0"| 2.2.4 notrans || d01 (xml)||bgcolor="#ffa0a0"| 2h || 542.6MB (v11)
{| {{prettytable}}
|-
!Comp Hardware Code !! data !! Gramps !! T03 !! T04 !! T05 !! T06 !! T07 !! T08 !! result
|-
|H02 || d01 ||bgcolor="#ffa0a0"| 2.2.4 || T03 = 4m17s || T04 = ? || T05 = ?/? || T06 = ? || T07 = ? || T08 = ||bgcolor="#ffa0a0"|
Number of repositories: 5
Number of notes: 1509
== User Stories ==
Running the tests can be slow, so here some user testimonies about Gramps Performance
=== Robert 2012-10, version 3.3.1 ===
I work with a database of 141,000+names currently without difficulty
(Gramps 3.3.1-1 on Fedora 16).
Initial start is fairly slow though.
First time to load each view is slow, but subsequent visits to views is
almost immediate.
Initial view load times:
* people 11 to 12 secs
* relationship abt 7 secs
* family 3 to 4 secs
* events 7 to 8 secs
* places 3 to 4 secs
* notes 11 to 12 secs
* ancestry view abt 1 sec or less
* Media abt 2 secs (although I only have about 1000 media in database)
* Repositories almost immediate
* sources about 1 sec - (time selecting a source varies according to number references for that source - my worst case is a civil registry which has about twice as many references as people in my database).
=== JohnBoyTheGreat 2019-12, version 5.1.1 ===
Import tested with the [https://mhss.sk.ca/FH/GRANDMA.shtml GRANDMA Mennonite database] of 1.4 million people. by user on reddit!! https://www.reddit.com/r/gramps/comments/dzevcl/database_size_limit_for_gramps/fb6hdbj/
''
FOLLOWUP...
It's been a few weeks since I asked whether anyone had attempted to use GRAMPS with the GRANDMA Mennonite database of 1.4 million people.
Based upon the suggestion above, I tried to load the Catalog of Life database...it took several days and it seemed to be working, but it eventually locked up GRAMPS. However, it seemed to be working. But, since I was only loading the Catalog of Life to test it, I decided not to waste time trying again.
In the meantime, I had ordered the GRANDMA database, but the organization selling it somehow gave me a bad download code and I couldn't reach them for several days (holiday and weekend). It was more than a week later before I was able to try loading the GRANDMA database into GRAMPS.
The result was SUCCESS!
It took about three days to load the GRANDMA database into GRAMPS, after giving GRAMPS a realtime priority in Windows. Setting GRAMPS to realtime sped up the loading considerably. My computer is rather fast compared to many, so anyone who wants to do the same thing should consider that it could take a week or longer to load up a huge database like GRANDMA.
At this point I have the GRANDMA database loaded into GRAMPS and running okay. It's really slow to switch to various functions, but it works okay once you get to each part of the GRAMPS program. One difficulty I ran into was scrolling through the 1.4 million records. It moves through the records quickly, but there are so many that you can't just use your cursor to pull to the surname you want to explore. Instead, I have to move the slider to a surname as close as possible, then scroll repeatedly until I find it. That can be 10-20 spins of the mouse wheel, so the process can be exhausting when you are looking for many different names.
My next step is going to be to copy the individual profiles which I need to a second family tree database that will be much, much smaller.
CONCLUSION: GRAMPS can handle 1.4 million records in a database. It's slow and takes several days to load, but it works.
''
== Possible Future Optimizations ==
** [http://www.tamurajones.net/TwoHugeGEDCOMFiles.xhtml Two Huge GEDCOM Files]
** [http://www.tamurajones.net/GedFan0.4.0.0.xhtml GedFan] - creates GEDCOM files, so-called fan files, which are used to test genealogy applications, and thus determine the capacity of those application, expressed as a fan value.
==See also==
* [[GEPS 016: Enhancing Gramps Processing Speed]]
[[Category:Developers/General]]
[[Category:Documentation|Performance]]
