Difference between revisions of "GEPS 045: Place Model Enhancements"

From Gramps
Jump to: navigation, search
m (Addons)
 
(46 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
Rethinking Gramps Place data model
 
Rethinking Gramps Place data model
  
One of the items on the Gramps 5.1 Roadmap is to look into better GEDCOM support for places.  We could continue to force fit GEDCOM import data to Gramps, or look into expanding Gramps data model to make for a better fit.
+
One of the items on the Gramps [[5.2 Roadmap]] is to look into better GEDCOM support for places.  We could continue to force fit GEDCOM import data to Gramps, or look into expanding Gramps data model to make for a better fit.
 
== Place Enhancements ==
 
== Place Enhancements ==
 
=== Place.Type: ===
 
=== Place.Type: ===
Line 7: Line 7:
 
Gov (http://gov.genealogy.net/search/index) and GEDCOM L group support multiple types, each with a subordinate date range, GEDCOM L group support a citation for each type as well
 
Gov (http://gov.genealogy.net/search/index) and GEDCOM L group support multiple types, each with a subordinate date range, GEDCOM L group support a citation for each type as well
 
==== Recommendation ====
 
==== Recommendation ====
Make place types a list.  Each element should have a subordinate date range and may have a citation (another decision).
+
Make place types a list.  Each element should have a subordinate date range and will have a citation list. (done)
 +
 
 +
We should also provide a routine similar to the current place displayer to return a place type for a particular date. (done, gen.utils.location.__get_type)
  
We should also provide a routine similar to the current place displayer to return a place type for a particular date.
 
 
=== Place Types: ===
 
=== Place Types: ===
 
Changes to place types, do not actually modify the data model, but dealing with them better will certainly impact a lot of code.
 
Changes to place types, do not actually modify the data model, but dealing with them better will certainly impact a lot of code.
Line 22: Line 23:
 
#: An additional level might be the country specific types.
 
#: An additional level might be the country specific types.
 
#) Provide a conversion of the Gov type to the closest Gramps PlaceType, possibly adding a few more Gramps PlaceTypes.  If we did that, I think it would be reasonable to store the original Gov type in a place fact.
 
#) Provide a conversion of the Gov type to the closest Gramps PlaceType, possibly adding a few more Gramps PlaceTypes.  If we did that, I think it would be reasonable to store the original Gov type in a place fact.
 +
#) Keep our standard Place Types, allow GOV types to be added as custom types as needed.
 +
#: Add a concept of Place type groups, which make it possible for multiple types to be identified as Country (for example).  The Groups are G_COUNTRY, G_REGION, G_PLACE, G_UNPOP, G_BUILDING.
 +
#: The GUI type selector is enhanced to support additional levels corresponding to the groups (only when needed).
 +
#: The Edit/Preferences/Place Types (new panel) is added to allow editing of the place type names, addition and removal of place types, and assignment to groups.  It is also possible to add and remove groups.
 +
 +
 +
The third option is accepted for development. (done)
 +
 
=== Place.Name: ===
 
=== Place.Name: ===
 
Gramps supports multiple names, each with a language and date range.
 
Gramps supports multiple names, each with a language and date range.
Line 41: Line 50:
 
The GeoNames data and Place Cleanup Gramplet currently store colloquial place names ('Big Apple' for New York City USA) as an alternate name with no language code.
 
The GeoNames data and Place Cleanup Gramplet currently store colloquial place names ('Big Apple' for New York City USA) as an alternate name with no language code.
 
==== Recommendation ====
 
==== Recommendation ====
#) Make colloquial name part of a more general place fact list.  Each would have a date, and possibly a citation (another decision) associated with it.
+
#) Our primary name and alternate names become part of a more general place name list.  Each has a date, language, and a citation list associated with it.
#) Add an abbreviation to each place name.  There would be potentially different abbreviations for each language and date.  In Reports and graphs, when abbreviations were specified to be used, the language/date specific abbreviation would be used instead of place displayer.
+
#) Add an abbreviation list to each place name.  Each abbreviation has a type and value.  In Reports and graphs, when abbreviations were specified to be used, the language/date specific name abbreviation would be used instead by the place displayer.
#) For place name abbreviations, when editing the name, if an abbreviation is present somewhere in the name list, default the value to that abbreviation.
+
#) For place name abbreviations, the place display format editor allows selection of the abbreviation by type of first available in the list for a particular name. If no abbreviation is present, use a place displayer routine uses the full name.
#) For reports/graphs when using the abbreviation, if no abbreviation is present, use a place displayer type of routine that uses the standard name for date/lang for each level until an abbreviation is encountered.
+
 
 +
Done
 +
 
 
=== Postal Code: ===
 
=== Postal Code: ===
 
Gramps currently offers a single field 'Code' to store postal codes.  It may have also been used for other purposes in the past.
 
Gramps currently offers a single field 'Code' to store postal codes.  It may have also been used for other purposes in the past.
Line 53: Line 64:
 
==== Recommendation ====
 
==== Recommendation ====
 
One of the following:
 
One of the following:
#) Make the postal code(s) part of a more general place fact list.  Each code would have a date, and possibly a citation (another decision) associated with it.
+
#) Make the postal code(s) part of a more general place attribute list.  Each code (attribute) would have a citation list associated with it.  Since an attribute doesn't have a date, the date would be appended as text to the attribute value (Ex: "12345; Date: JAN 12 2019"). (Done)
#) Make postal codes a listEach element should have a subordinate date range and may have a citation (another decision).
+
 
 
=== MAIDENHEAD_LOCATOR ===
 
=== MAIDENHEAD_LOCATOR ===
 
GEDCOM L has added this to the Place import/export.
 
GEDCOM L has added this to the Place import/export.
Line 60: Line 71:
 
The Maidenhead Locator divides the world into small areas (micro-fields) marked by 8 letters and numbers.  It was originally conceived for use in ham radio, but it is also used as a geographic locator in some other applications.  https://en.wikipedia.org/wiki/Maidenhead_Locator_System
 
The Maidenhead Locator divides the world into small areas (micro-fields) marked by 8 letters and numbers.  It was originally conceived for use in ham radio, but it is also used as a geographic locator in some other applications.  https://en.wikipedia.org/wiki/Maidenhead_Locator_System
 
==== Recommendation ====
 
==== Recommendation ====
Store the Maidenhead locator in a more general place fact list.  Date and Citation parts of the fact would not be used.
+
Store the Maidenhead locator in the place attribute list.  Date and Citation parts would not be used. (Done)
 +
 
 
=== ADMINISTRATIVE_IDENTIFIER ===
 
=== ADMINISTRATIVE_IDENTIFIER ===
 
GEDCOM L has added this to the Place import/export.
 
GEDCOM L has added this to the Place import/export.
Line 66: Line 78:
 
The official or public identifier for a location object, such as community code, ISO 3166 for countries and states.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
 
The official or public identifier for a location object, such as community code, ISO 3166 for countries and states.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
 
==== Recommendation ====
 
==== Recommendation ====
Store the ADMINISTRATIVE IDENTIFIER in a more general place fact list along with the Date and Citation. Merge the TYPE_OF_ADMINISTRATIVE_IDENTIFIER into the fact type.
+
Store the ADMINISTRATIVE IDENTIFIER in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype, the TYPE_OF_ADMINISTRATIVE_IDENTIFIER would be appended as text to the attribute value (Ex: "BlahBlah; Type: whatever"). (Done)
 +
 
 
=== DEMOGRAPHIC DATA ===
 
=== DEMOGRAPHIC DATA ===
 
GEDCOM L has added this to the Place import/export.
 
GEDCOM L has added this to the Place import/export.
 
Demographics about people to the place object, such as number of households, number of people, occupations, and so on.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
 
Demographics about people to the place object, such as number of households, number of people, occupations, and so on.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
 
==== Recommendation ====
 
==== Recommendation ====
Store the DEMOGRAPICAL_DATA in a more general place fact list along with the Date and Citation. Merge the TYPE_OF_DEMOGRAPICAL_DATA into the fact type.
+
Store the DEMOGRAPICAL_DATA in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype or date, the TYPE_OF_DEMOGRAPICAL_DATA and/or date would be appended as text to the attribute value (Ex: "43210; Type: Population; Date: 2017"). (Done)
 +
 
 
=== EVENT_DETAIL ===
 
=== EVENT_DETAIL ===
 
GEDCOM L has added this to the Place import/export.
 
GEDCOM L has added this to the Place import/export.
Line 78: Line 92:
 
==== Recommendation ====
 
==== Recommendation ====
 
Add an Event reference list to the Place.  The list could contain one of the following:
 
Add an Event reference list to the Place.  The list could contain one of the following:
#) EventRef;  The Event ref role would be set to 'Place'
+
#) EventRef;  The Event ref role would be set to 'Place' (Done)
#) Direct pointer to Event object.  (My preference)
+
#) Direct pointer to Event object.  (Rejected)
 +
 
 
== Enclosed by ==
 
== Enclosed by ==
 
The following items would be part of the Gramps place reference, which currently only includes a date and the reference to another place.
 
The following items would be part of the Gramps place reference, which currently only includes a date and the reference to another place.
Line 89: Line 104:
 
GEDCOM L adds a citation to the Enclosed by
 
GEDCOM L adds a citation to the Enclosed by
 
==== Recommendation ====
 
==== Recommendation ====
No changeThe Hierarchical relationship can be inferred from the type of the enclosing place, so it appears to me to be redundantI also think that a citation here adds little value.
+
* The Place Type Enclosure (PlaceRef) is extended with a PlaceHierTypeIt can be one of ADMIN, RELI, GEOG, CULT or JUDI (or other custom types)The PlaceRef also has an added citation list.
 +
 
 +
== Place Attributes ==
 +
To deal with several new facts associated with places, a Place Attributes list is added.  Each place attribute includes the standard type, value, notes and citations.
 +
 
 +
Facts that also include a date or subtype are encoded as text in the Attribute.value.  Several new attribute types are added to support places AIDN (Administrative Identifier), DMGD (Demographic data), MAIDEN (Maidenhead Locator), and POSTAL (Postal code).
 +
 
 +
The Attribute Type selector in the GUI has a second level menu called place with these types in it.
  
This presumes that Place Types are extended to include additional types in multiple categories.
 
== Place facts ==
 
To deal with several new facts associated with places, I propose a generalized Place Facts list.  Each place fact would include the type, value, date, and possibly citation.
 
 
== Citations ==
 
== Citations ==
GEDCOM L has added citations to most of its additional data. To me this seems excessive.  I think we need to decide if we support citations individually, or just citations on the Place object.  This would affect the proposed Place Fact structure, as well as Place Name, Type, Postal codes, and Place Enclosed by.
+
GEDCOM L has added citations to most of its additional data.
 +
 
 +
The Place object has citation lists attached to Places, Place Attributes, Place Names, Place Types, and Place Enclosed by.
 +
 
 
== ToDo ==
 
== ToDo ==
 
=== Top priority ===
 
=== Top priority ===
 
If we do these first and carefully, we should be able to run Gramps with new trees.
 
If we do these first and carefully, we should be able to run Gramps with new trees.
 +
 
gen.lib.place
 
gen.lib.place
* update object, get_schema, serialize/unserialize etc.
+
* ✔ update object, get_schema, serialize/unserialize etc.
* Add add_eventref, get/set/merge_eventreflist
+
* ✔ Add add_eventref, get/set/merge_eventreflist
* Add add/get/set/merge_place_facts
+
* ✔ Add add/get/set/merge place attributes
* add_place_type, get/set/merge_place_types. Consider keeping the most modern place type (based on date) with admin hierarchy at top of list (makes views.placemodel a bit faster to pick type to view and allows for get_place_type to return top value for short term compatibility).
+
* ✔ get/set/merge_place_types (get/set effects only the top of the list of types). New add_place_type, Supports sortin the most modern place type (based on date) with admin hierarchy at top of list (makes views.placemodel a bit faster to pick type to view and allows for get_place_type to return top value for compatibility).
* Remove get/set title (or covert to use place.displayer)
+
* <s>Remove get/set title</s>
* Get/set_name – Do we want to have this separate, or as just top of a list of names?
+
* &#x2714; Get/set_name – Place.name removed, one unified list of names, so these methods only effect the top of a list of names.  Also remove get/set alternative names, and added add_name (which can sort the most recent name to top of list).
* Add get/set/merge_abbreviation
+
* &#x2714; Add get/set/merge_abbreviation
* Remove or modify get/set_code (if modify, set/return top postal code fact)
+
* &#x2714; Remove get/set_code and Place.code (on import/upgrade this is moved to Place Attribute)
* Remove get/set/merge_alternative_locations (Note: need to move to notes for upgrade)
+
* <s>Remove get/set/merge_alternative_locations</s>
* fix up other methods as needed.
+
* &#x2714; fix up other methods as needed.
* Do we add citationrefs to place names, types, placerefs?
+
* &#x2714; add citationrefs to place names, types, placerefs
gen.lib.placefact – new.  Similar to gen.lib.attribute but with date.
+
<s>gen.lib.placefact – new.  Similar to gen.lib.attribute but with date.</s>
gen.lib.placefacttype – new. Similar to lib.eventtype, but for place facts.
+
 
gen.lib.placetype – update for more place types and hierarchy
+
<s>gen.lib.placefacttype – new. Similar to lib.eventtype, but for place facts.</s>
gui.views.placemodel – update for changed serialization.  Do we still want to display 'Code'?  Propose not (if not, also change plugins.lib.libplaceview for missing column).
+
 
 +
&#x2714; gen.lib.placetype – update for place groups and support three classifications of types; standard Gramps types, numbered custom types (supports GOV types), and full custom types.  Also update place type combo menu to allow much larger type list in two level menu.
 +
 
 +
&#x2714; gui.views.placemodel – update for changed serialization.  Remove 'Code' also change plugins.lib.libplaceview for missing column.
 +
 
 +
&#x2714; gen.lib.placeabbrev, gen.lib.placeabbrevtype, gen.lib.placehiertype
 +
 
 
=== 2nd priority ===
 
=== 2nd priority ===
 
==== Editors ====
 
==== Editors ====
 
I'm assuming that the place editor is not significantly changed in style from the current one.
 
I'm assuming that the place editor is not significantly changed in style from the current one.
* Add place type editor icon to place type? Instead of combo.
+
* &#x2714; Add place type editor icon to place type. Place Type combo remains but only affects top of list.
* Place type editor gui.editors.editplacetype? Combo for type and Add date and citationref
+
* &#x2714; Remove 'Code' from Place editor
* Add place Types tab gui.editors.displaytabs.placetypeembeddedlist and placetypemodel
+
* &#x2714; Add place Types tab gui.editors.displaytabs.placetypeembeddedlist and placetypemodel
* Remove Title edit capability
+
* <s>Remove Title edit capability</s>
* Remove Code (data is moved to place facts tab).
+
* &#x2714; Update Place Names tab for unified list of names, abbreviations
* Add Place Facts tab gui.editors.displaytabs.placefactembeddedlist and placefactsmodel
+
* &#x2714; Add Place Attributes tab gui.editors.displaytabs.placeatttrembeddedlist
* Add place Events tab - gui.editors.displaytabs.placeeventembeddedlist and placeeventmodel
+
* &#x2714; Add place Events tab - gui.editors.displaytabs.placeeventembeddedlist
* Duplicate place editor work for gui.editors.editplaceref
+
* &#x2714; Duplicate place editor work for gui.editors.editplaceref
* gui.editors.editevent – fix DeleteEventQuery to scan for places containing events
+
* &#x2714; New Place type editor gui.editors.editplacetype. Combo for type and Add date and a citationref tab
 +
* &#x2714; gui.editors.editevent – fix DeleteEventQuery to scan for places containing events
 +
* &#x2714; Update Place Names Editor for unified list of names.
 +
* &#x2714; Add abbreviations and citations tabs to place name editor gui.editors.displaytabs.placeabbrevembedlist and placeabbrevmodel
 +
* &#x2714; New Place Abbreviations Editor gui.editors.editplaceabbrev
 +
* &#x2714; Add citations tab to gui.editors.editplaceref
 +
 
 
==== Views ====
 
==== Views ====
* Add Place Facts and Events Gramplets to Place views plugins.lib.libplaceview
+
* &#x2714; Add Place Attributes and Events Gramplets to Place views plugins.lib.libplaceview
* Modify Events Gramplet plugins.gramplets.events to deal with place events (remove age column).
+
* &#x2714; Modify Events Gramplet plugins.gramplets.events to deal with place events (remove age column).
* Add Place Facts Gramplet plugins.gramplets.placefacts similar to Attributes Gramplet but with dates
+
* &#x2714; update Attributes Gramplet plugins.gramplets.attributes for places
 +
* &#x2714; plugins.lib.maps.placeselection update to use place groups instead of types
 +
* &#x2714; plugins.lib.maps.geography update for place changes
 +
* &#x2714; plugins.view.eventview fix DeleteEventQuery to support place events
 +
* &#x2714; plugins.view.geoplaces modify marker color to use much larger place types
 +
* &#x2714; plugins.view.placetreeview fix add place to add heierarchy type
 +
 
 
==== Import/export/upgrade ====
 
==== Import/export/upgrade ====
* Update dtd
+
* &#x2714; Update dtd
* Update XML export
+
* &#x2714; Update XML export
* Update XML import
+
* &#x2714; Update XML import
* Modify bsddb update routines plugins.db.bsddb.upgrade
+
* &#x2714; Modify bsddb update routines plugins.db.bsddb.upgrade
* Create dbapi update routine plugins.db.pbapi.upgrade, and hooks into db loaders
+
* &#x2714; Create dbapi update routine plugins.db.pbapi.upgrade, and hooks into db loaders
 
DB upgrade or XML import should:
 
DB upgrade or XML import should:
* put legacy place title into a Note, if it differs from the place displayer.
+
* &#x2714; Move Code field into place attribute
* Move Code field into place fact
+
* &#x2714; add new citations lists
* Move alternate Location into Note or place Fact
+
* &#x2714; change place type to type list with date and citations
 +
* &#x2714; change place name and place alternative names to place name list with new abbreviations
 +
* &#x2714; change PlaceRef to add citations and hierarchy
 +
* &#x2714; add placetype custom information to metadata
 +
 
 
==== Merge ====
 
==== Merge ====
* Fix gen.merge.mergeeventquery to check for place events
+
* &#x2714; Fix gen.merge.mergeeventquery to check for place events
 +
* &#x2714; Remove Code from gui.merge.mergeplace and its glade file
 +
 
 +
==== Place Displayer ====
 +
* &#x2714; Update place displayer gen.display.place to support rules for place types, groups, and abbreviations etc.
 +
* &#x2714; Update gen.utils.location to support date for multiple place types, hierarchy, and to return abbreviations
 +
* &#x2714; rewrite gui.editors.editplaceformat and its glade file
 +
 
 +
These now support:
 +
* multiple place formats (like before)
 +
* each format supports a heirarchy type and language, and allows display order reversal
 +
* each format can have multiple rules, which can be added, removed and reordered (order is executed top to bottom)
 +
* each rule can apply to all places, or just one place (useful for country specific rules)
 +
* each rule can apply to a place Group, Type, or the street/number
 +
* For Group rules, you can select the Group from the standard and custom groups
 +
* for Type rules, you can select the Type from the standard and custom types
 +
* for the Group or Type rules, you can choose to display All, largest, or smallest of matching types, or hide them.
 +
* for the Group or Type rules, you can choose to show an abbreviation from the abbreviation types, or the first available abbreviation type, or None (which shows the associated name, instead of the abbreviation).
 +
* for street/number rules you can choose to display street/number, number/street, or Hidden
 +
* The Place Formats are stored in the db NOT in a system wide file, so only applies to the current tree.  This is due to potential references to specific places or customized types or groups which may not be present in other trees.
 +
 
 +
==== Edit/Preferences/Place Types ====
 +
* &#x2714; Updated gui.configure with new Place Types panel
 +
 
 +
This now supports:
 +
* Shows the place type list on left side, along with used indication, and a comment
 +
* Shows place groups for selected Place Type on right side
 +
* An Entry at top right allows the Place Type name to be changed
 +
* The Groups are shown as check boxes, which can be changed
 +
* There is a search box at bottom of dialog
 +
* You can add new types via the Add button at bottom
 +
* You can remove types that are not used and are not standard
 +
* You can hide types (from type selection combobox menus) with the Hide button.
 +
* You can add new Groups via a right-click over one of the current Group checkbox items
 +
* You can remove Groups via a right-click over one of the current Group checkbox items
 +
* The customized place type/group information is stored in the current db and so applies to only a specific tree.
 +
* The customized place type/group information is stored in the XML for backup.
 +
 
 
=== 3rd priority ===
 
=== 3rd priority ===
 
==== Import/Export ====
 
==== Import/Export ====
* Update Gedcom export from GEPS 043 work to support place model https://github.com/gramps-project/gramps/pull/469
+
* &#x2714; Update Gedcom import/export from GEPS 043 work to support place model https://github.com/gramps-project/gramps/pull/469
* Update Gedcom import from GEPS 043 work to support place model
+
* &#x2714; Update importer for place changes; csv
* Update importer for place changes; csv
+
* &#x2714; Update importer for place changes;  geneweb
* Update importer for place changes;  geneweb
+
* &#x2714; Update importer for place changes; progen
* Update importer for place changes; progen
+
* (N/A) Update importer for place changes;vcard
* Update importer for place changes;vcard
+
* &#x2714; Update exporter for place changes; csv
* Update exporter for place changes; csv
+
* (N/A) Update exporter for place changes;ftree
* Update exporter for place changes;ftree
+
* (N/A) Update exporter for place changes;geneweb
* Update exporter for place changes;geneweb
+
* (N/A) Update exporter for place changes;vcalendar
* Update exporter for place changes;vcalendar
+
* (N/A) Update exporter for place changes;vcard
* Update exporter for place changes;vcard
+
 
 
==== Reports ====
 
==== Reports ====
* Place Format Editor?
+
* &#x2714; Plugins.webreport.narrativeweb, etc.  Add Place Formats option, Update to use place groups, show place attributes.
* Narweb, no obvious changes.
+
* &#x2714; Plugins.graph.gvfamilylines – update to use place Displayer/Formats.
* Plugins.graph.gvfamilylines – Uses_get_main_location, may want to update to use place abbreviation.
+
* &#x2714; Plugins.graph.gvrelgraph – Uses_place displayer, update to use place Displayer/Formats.
* Plugins.graph.gvrelgraph – Uses_place displayer, may want to update to use place abbreviation.
+
* &#x2714; Plugins.libllibstubkeyword – updated to support place groups for Country, State (largest REGION), County (smallest Region), City (largest Place).  Also fixed up postal to find in place attributes. Title now uses standard Place Displayer with preferred format, Place sub-codes 1-5 can use the first five Place displayer Formats.
* Plugins.libllibstubkeyword – May want to rethink this, at a minimum include the place abbreviation as a choice. Uses get_main_location…
+
* &#x2714; Plugins.drawreport.statisticschart – Uses_place displayer, update to use place Displayer/Formats.
* Plugins.lib.maps.geography; add_place_from_kml needs minor update
+
* &#x2714; Plugins.textreport.ancestorreport – Uses_place displayer, update to use place Displayer/Formats.
* Plugins.lib.maps.placeselection may need to do something with get_location, depends on specific place types.
+
* &#x2714; Plugins.textreport.placereport – added more names and types to the report, fixed to avoid problems with Place Events.
 
* Plugins.mapservices.eniroswedenmap  uses some specific place types…
 
* Plugins.mapservices.eniroswedenmap  uses some specific place types…
* Plugins.textreport.placereport – may want to fill out data with new stuff
+
 
 
==== Tools ====
 
==== Tools ====
 
* Generate Testcases  may want to add some additional place items
 
* Generate Testcases  may want to add some additional place items
 
* Check & Repair; check_events needs to look at places for references.
 
* Check & Repair; check_events needs to look at places for references.
 
==== Filters ====
 
==== Filters ====
* Gen.filters.rules.has_data place type scan should look at whole list
+
* &#x2714; gen.proxy.filter - support place attributes and event refs for sanitizing notes
* New filter rules for place facts, place events
+
* &#x2714; gen.proxy.private - support place names, types and their citation lists as well as the new eventrefs and attributes.
* Place Sidebar filter; remove code, maybe add fact filter
+
* &#x2714; gen.proxy.referencedbyselection - support place names, types and their citation lists as well as the new eventrefs and attributes.
 +
* &#x2714; gen.proxy.proxybase - added new get_placehier_types, get_placeabbr_types, get_place_attribute_types methods.
 +
* &#x2714; gen.filters.rules.place.has_data - remove "code" place type scan should look at whole list
 +
* &#x2714; New filter rules for place attributes, place events (gen.filters.rules.place._hasattribute, _hasevent)
 +
* &#x2714; gui.filters.sidebar._placessidebarfilter - Place Sidebar filter; remove code
 +
 
 
==== Addons ====
 
==== Addons ====
* CheckPlaceTitles – don't need this anymore?
+
* [[Addon:CheckPlaceTitles|CheckPlaceTitles]] – don't need this anymore?
 +
* &#x2714; D3Charts/DescendantIndentedTree - has get_title for a place
 
* DataEntryGramplet – has set_title for a place
 
* DataEntryGramplet – has set_title for a place
* DescendantBookReport – has get_title for a place
+
* &#x2714; DescendantBookReport – has get_title for a place
* DetailedDescendantBookReport– has get_title for a place
+
* &#x2714; DetailedDescendantBookReport– has get_title for a place
* DescendantsLines\substkw – same fixes as for Plugins.libllibstubkeyword, or just use the plugins.lib version.
+
* &#x2714; DescendantsLines\substkw – stop override for PlaceFormat from Plugins.libllibstubkeyword, so just use the plugins.lib version.
* DynamicWeb – uses place.get_title place get_main_location, get_alternate_locations, get_code
+
* &#x2714; DynamicWeb – uses place.get_title place get_main_location, get_alternate_locations, get_code (done, works, but may want to enhance with more display of place multiple types, attributes, events...).
* ExtractCity – uses set_code
+
* &#x2714; ExtractCity – uses set_code
* GetGov – modify to store data in correct parts of place, adapt to updated Place Types with hierarchy and date.
+
* &#x2714; GetGov – modify to store data in correct parts of place, adapt to updated Place Types with hierarchy and date.  
* GraphView – may want to add support for place abbreviation
+
* &#x2714; GraphView – add support for place selectable formats
* ImportMerge – make sure algorithm doesn't break on Event/Place/Event loop
+
* &#x2714; ImportMerge – make sure algorithm doesn't break on Event/Place/Event loop
* PlaceCleanup – Need to store data in correct parts of place, change Code handling, change abbreviation handling, store dated facts, multiple place types etc.
+
* &#x2714; lxml - update to support new schema
* PlaceCompletion – don't support this anymore?
+
* &#x2714; PlaceCleanup – Need to store data in correct parts of place, change Code handling, change abbreviation handling, store attributess, multiple place types etc.
* Sqlite.ExportSql – needs update if we want to support
+
* &#x2714; [[Addon:Place_completion_tool|PlaceCompletion]] – don't support this anymore?
* Sqlite.ImportSql – needs update if we want to support
+
* &#x2714; Sqlite.ExportSql – update for new place data (Not compatible with previous sql files)
 +
* &#x2714; Sqlite.ImportSql – update for new place data (Not compatible with previous sql files)
 +
 
 
==== Documentation ====
 
==== Documentation ====
* Update Data model diagram
+
* Update [[Gramps_Data_Model|Data model diagram]]
 
* Update wiki (entering data, place name editor, …)
 
* Update wiki (entering data, place name editor, …)
 +
 
==== Test code ====
 
==== Test code ====
 
* Db tests
 
* Db tests
* Import/export tests
+
* &#x2714; Import/export/report tests
 
* Filter tests
 
* Filter tests
 +
 
== Comments ==
 
== Comments ==
 
== See also ==
 
== See also ==
 
*[[GEPS 006: Better Place handling]]
 
*[[GEPS 006: Better Place handling]]
 
*[[GEPS 043: Improving GEDCOM support for Places]]
 
*[[GEPS 043: Improving GEDCOM support for Places]]
 +
*[[GEPS 045: Place Model Enhancements - Place Changes Screenshots]]
  
 
[[Category:GEPS|P]]
 
[[Category:GEPS|P]]

Latest revision as of 13:32, 2 February 2022

Rethinking Gramps Place data model

One of the items on the Gramps 5.2 Roadmap is to look into better GEDCOM support for places. We could continue to force fit GEDCOM import data to Gramps, or look into expanding Gramps data model to make for a better fit.

Place Enhancements

Place.Type:

Gramps only supports a single place type. Gov (http://gov.genealogy.net/search/index) and GEDCOM L group support multiple types, each with a subordinate date range, GEDCOM L group support a citation for each type as well

Recommendation

Make place types a list. Each element should have a subordinate date range and will have a citation list. (done)

We should also provide a routine similar to the current place displayer to return a place type for a particular date. (done, gen.utils.location.__get_type)

Place Types:

Changes to place types, do not actually modify the data model, but dealing with them better will certainly impact a lot of code.

GeoNames currently doesn’t do much to provide place types. It currently uses PCLx, for Country, PPLx for 'Populated Place', and ADM1-ADMn for different level of administrative place. These do not map well to Gramps PlaceTypes. The Place Cleanup Gramplet tries to do it anyway, but allows user modification.

Gov currently has over 260 place types in several broad categories. The categories are Administrative, Civil, Religious (church), Judicial, populated place, transportation, and a few miscellaneous others. They are still adding more. The GetGov Gramplet currently stores the returned German place type as a Custom Gramps place type, which is of course not translatable. For example, Deutsches Reich, or USA gets a PlaceType of 'Staat', rather than country. While the system has a lot of types it is very German centric, many types have only German names in their definition documents although some of the more commonly used types have at least English translations and some have several other translations.

Recommendation

One of;

  1. ) Adopt the Gov type system. We would have to do a lot of translation work on many types, or indicate that the type is local to certain countries (so could use the local words). Our GUI would probably have to use at least a two level hierarchy in the PlaceType selector, possibly three levels, to help users find appropriate types for manually entered places or filtering.
    One of the levels (top level?) would be the category type.
    An additional level might be the country specific types.
  2. ) Provide a conversion of the Gov type to the closest Gramps PlaceType, possibly adding a few more Gramps PlaceTypes. If we did that, I think it would be reasonable to store the original Gov type in a place fact.
  3. ) Keep our standard Place Types, allow GOV types to be added as custom types as needed.
    Add a concept of Place type groups, which make it possible for multiple types to be identified as Country (for example). The Groups are G_COUNTRY, G_REGION, G_PLACE, G_UNPOP, G_BUILDING.
    The GUI type selector is enhanced to support additional levels corresponding to the groups (only when needed).
    The Edit/Preferences/Place Types (new panel) is added to allow editing of the place type names, addition and removal of place types, and assignment to groups. It is also possible to add and remove groups.


The third option is accepted for development. (done)

Place.Name:

Gramps supports multiple names, each with a language and date range.

GEDCOM supports two additional parts of a name; neither of which seem to be used much and are not currently supported by Gramps.

FONE (Place phonetic variation) and its subordinate TYPE. This seems to be used for far eastern places for example if hiragana was used to provide a reading of a name written in kanji, then the <PHONETIC_TYPE> value would indicate kana. The TYPE is defined as [<user defined> | hangul | kana]

ROMN (PLACE_ROMANIZED_VARIATION:= {Size=1:120}) and its subordinate TYPE. The romanized variation of the place name is written in the same form prescribed for the place name used in the superior <PLACE_NAME> context. The method used to romanize the name is indicated by the line_value of the subordinate <ROMANIZED_TYPE>, for example if romaji was used to provide a reading of a place name written in kanji, then the <ROMANIZED_TYPE> subordinate to the ROMN tag would indicate ‘romaji’. The TYPE is defined as [<user defined> | pinyin | romaji | wadegiles]

GEDCOM L group adds in _NAMC (place name addition) for each name. Their documentation currently says "Addition to PLACE_NAME, can be appended to PLACE_NAME in non-delimited reports to make the label unique". This is for suffixes like "am Main" in "Frankfurt am Main".

GEDCOM L group adds in ABBR with a subordinate TYPE. ABBREVIATION_OF_NAME Abbreviation for the place name, whereby the type of abbreviation can be further explained with the optional TYPE.

GEDCOM L group adds in a citation list for each name.

The GeoNames data and Place Cleanup Gramplet currently store place name abbreviations as an alternate name with the 'abbr' language code.

The GeoNames data and Place Cleanup Gramplet currently store colloquial place names ('Big Apple' for New York City USA) as an alternate name with no language code.

Recommendation

  1. ) Our primary name and alternate names become part of a more general place name list. Each has a date, language, and a citation list associated with it.
  2. ) Add an abbreviation list to each place name. Each abbreviation has a type and value. In Reports and graphs, when abbreviations were specified to be used, the language/date specific name abbreviation would be used instead by the place displayer.
  3. ) For place name abbreviations, the place display format editor allows selection of the abbreviation by type of first available in the list for a particular name. If no abbreviation is present, use a place displayer routine uses the full name.

Done

Postal Code:

Gramps currently offers a single field 'Code' to store postal codes. It may have also been used for other purposes in the past.

GeoNames, GOV, GEDCOM L all support multiple postal codes. GEDCOM L adds in a subordinate DATE range and citation for each code.

The Place Cleanup Gramplet currently stores multiple postal codes in the code field separated by commas.

Recommendation

One of the following:

  1. ) Make the postal code(s) part of a more general place attribute list. Each code (attribute) would have a citation list associated with it. Since an attribute doesn't have a date, the date would be appended as text to the attribute value (Ex: "12345; Date: JAN 12 2019"). (Done)

MAIDENHEAD_LOCATOR

GEDCOM L has added this to the Place import/export.

The Maidenhead Locator divides the world into small areas (micro-fields) marked by 8 letters and numbers. It was originally conceived for use in ham radio, but it is also used as a geographic locator in some other applications. https://en.wikipedia.org/wiki/Maidenhead_Locator_System

Recommendation

Store the Maidenhead locator in the place attribute list. Date and Citation parts would not be used. (Done)

ADMINISTRATIVE_IDENTIFIER

GEDCOM L has added this to the Place import/export.

The official or public identifier for a location object, such as community code, ISO 3166 for countries and states. GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.

Recommendation

Store the ADMINISTRATIVE IDENTIFIER in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype, the TYPE_OF_ADMINISTRATIVE_IDENTIFIER would be appended as text to the attribute value (Ex: "BlahBlah; Type: whatever"). (Done)

DEMOGRAPHIC DATA

GEDCOM L has added this to the Place import/export. Demographics about people to the place object, such as number of households, number of people, occupations, and so on. GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.

Recommendation

Store the DEMOGRAPICAL_DATA in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype or date, the TYPE_OF_DEMOGRAPICAL_DATA and/or date would be appended as text to the attribute value (Ex: "43210; Type: Population; Date: 2017"). (Done)

EVENT_DETAIL

GEDCOM L has added this to the Place import/export.

GEDCOM L has added an event reference to the place.

Recommendation

Add an Event reference list to the Place. The list could contain one of the following:

  1. ) EventRef; The Event ref role would be set to 'Place' (Done)
  2. ) Direct pointer to Event object. (Rejected)

Enclosed by

The following items would be part of the Gramps place reference, which currently only includes a date and the reference to another place.

HIERARCHICAL_RELATIONSHIP

GEDCOM L has added this to the Place import/export.

Values are [POLI | RELI | GEOG | CULT] to differentiate political (administrative), ecclesiastical, geographical or cultural attributions. GEDCOM L includes this to provide some indication of the type of enclosure. For example a Church might be enclosed by a city and enclosed by a parish, with different relationships.

Citation

GEDCOM L adds a citation to the Enclosed by

Recommendation

  • The Place Type Enclosure (PlaceRef) is extended with a PlaceHierType. It can be one of ADMIN, RELI, GEOG, CULT or JUDI (or other custom types). The PlaceRef also has an added citation list.

Place Attributes

To deal with several new facts associated with places, a Place Attributes list is added. Each place attribute includes the standard type, value, notes and citations.

Facts that also include a date or subtype are encoded as text in the Attribute.value. Several new attribute types are added to support places AIDN (Administrative Identifier), DMGD (Demographic data), MAIDEN (Maidenhead Locator), and POSTAL (Postal code).

The Attribute Type selector in the GUI has a second level menu called place with these types in it.

Citations

GEDCOM L has added citations to most of its additional data.

The Place object has citation lists attached to Places, Place Attributes, Place Names, Place Types, and Place Enclosed by.

ToDo

Top priority

If we do these first and carefully, we should be able to run Gramps with new trees.

gen.lib.place

  • ✔ update object, get_schema, serialize/unserialize etc.
  • ✔ Add add_eventref, get/set/merge_eventreflist
  • ✔ Add add/get/set/merge place attributes
  • ✔ get/set/merge_place_types (get/set effects only the top of the list of types). New add_place_type, Supports sortin the most modern place type (based on date) with admin hierarchy at top of list (makes views.placemodel a bit faster to pick type to view and allows for get_place_type to return top value for compatibility).
  • Remove get/set title
  • ✔ Get/set_name – Place.name removed, one unified list of names, so these methods only effect the top of a list of names. Also remove get/set alternative names, and added add_name (which can sort the most recent name to top of list).
  • ✔ Add get/set/merge_abbreviation
  • ✔ Remove get/set_code and Place.code (on import/upgrade this is moved to Place Attribute)
  • Remove get/set/merge_alternative_locations
  • ✔ fix up other methods as needed.
  • ✔ add citationrefs to place names, types, placerefs

gen.lib.placefact – new. Similar to gen.lib.attribute but with date.

gen.lib.placefacttype – new. Similar to lib.eventtype, but for place facts.

✔ gen.lib.placetype – update for place groups and support three classifications of types; standard Gramps types, numbered custom types (supports GOV types), and full custom types. Also update place type combo menu to allow much larger type list in two level menu.

✔ gui.views.placemodel – update for changed serialization. Remove 'Code' also change plugins.lib.libplaceview for missing column.

✔ gen.lib.placeabbrev, gen.lib.placeabbrevtype, gen.lib.placehiertype

2nd priority

Editors

I'm assuming that the place editor is not significantly changed in style from the current one.

  • ✔ Add place type editor icon to place type. Place Type combo remains but only affects top of list.
  • ✔ Remove 'Code' from Place editor
  • ✔ Add place Types tab gui.editors.displaytabs.placetypeembeddedlist and placetypemodel
  • Remove Title edit capability
  • ✔ Update Place Names tab for unified list of names, abbreviations
  • ✔ Add Place Attributes tab gui.editors.displaytabs.placeatttrembeddedlist
  • ✔ Add place Events tab - gui.editors.displaytabs.placeeventembeddedlist
  • ✔ Duplicate place editor work for gui.editors.editplaceref
  • ✔ New Place type editor gui.editors.editplacetype. Combo for type and Add date and a citationref tab
  • ✔ gui.editors.editevent – fix DeleteEventQuery to scan for places containing events
  • ✔ Update Place Names Editor for unified list of names.
  • ✔ Add abbreviations and citations tabs to place name editor gui.editors.displaytabs.placeabbrevembedlist and placeabbrevmodel
  • ✔ New Place Abbreviations Editor gui.editors.editplaceabbrev
  • ✔ Add citations tab to gui.editors.editplaceref

Views

  • ✔ Add Place Attributes and Events Gramplets to Place views plugins.lib.libplaceview
  • ✔ Modify Events Gramplet plugins.gramplets.events to deal with place events (remove age column).
  • ✔ update Attributes Gramplet plugins.gramplets.attributes for places
  • ✔ plugins.lib.maps.placeselection update to use place groups instead of types
  • ✔ plugins.lib.maps.geography update for place changes
  • ✔ plugins.view.eventview fix DeleteEventQuery to support place events
  • ✔ plugins.view.geoplaces modify marker color to use much larger place types
  • ✔ plugins.view.placetreeview fix add place to add heierarchy type

Import/export/upgrade

  • ✔ Update dtd
  • ✔ Update XML export
  • ✔ Update XML import
  • ✔ Modify bsddb update routines plugins.db.bsddb.upgrade
  • ✔ Create dbapi update routine plugins.db.pbapi.upgrade, and hooks into db loaders

DB upgrade or XML import should:

  • ✔ Move Code field into place attribute
  • ✔ add new citations lists
  • ✔ change place type to type list with date and citations
  • ✔ change place name and place alternative names to place name list with new abbreviations
  • ✔ change PlaceRef to add citations and hierarchy
  • ✔ add placetype custom information to metadata

Merge

  • ✔ Fix gen.merge.mergeeventquery to check for place events
  • ✔ Remove Code from gui.merge.mergeplace and its glade file

Place Displayer

  • ✔ Update place displayer gen.display.place to support rules for place types, groups, and abbreviations etc.
  • ✔ Update gen.utils.location to support date for multiple place types, hierarchy, and to return abbreviations
  • ✔ rewrite gui.editors.editplaceformat and its glade file

These now support:

  • multiple place formats (like before)
  • each format supports a heirarchy type and language, and allows display order reversal
  • each format can have multiple rules, which can be added, removed and reordered (order is executed top to bottom)
  • each rule can apply to all places, or just one place (useful for country specific rules)
  • each rule can apply to a place Group, Type, or the street/number
  • For Group rules, you can select the Group from the standard and custom groups
  • for Type rules, you can select the Type from the standard and custom types
  • for the Group or Type rules, you can choose to display All, largest, or smallest of matching types, or hide them.
  • for the Group or Type rules, you can choose to show an abbreviation from the abbreviation types, or the first available abbreviation type, or None (which shows the associated name, instead of the abbreviation).
  • for street/number rules you can choose to display street/number, number/street, or Hidden
  • The Place Formats are stored in the db NOT in a system wide file, so only applies to the current tree. This is due to potential references to specific places or customized types or groups which may not be present in other trees.

Edit/Preferences/Place Types

  • ✔ Updated gui.configure with new Place Types panel

This now supports:

  • Shows the place type list on left side, along with used indication, and a comment
  • Shows place groups for selected Place Type on right side
  • An Entry at top right allows the Place Type name to be changed
  • The Groups are shown as check boxes, which can be changed
  • There is a search box at bottom of dialog
  • You can add new types via the Add button at bottom
  • You can remove types that are not used and are not standard
  • You can hide types (from type selection combobox menus) with the Hide button.
  • You can add new Groups via a right-click over one of the current Group checkbox items
  • You can remove Groups via a right-click over one of the current Group checkbox items
  • The customized place type/group information is stored in the current db and so applies to only a specific tree.
  • The customized place type/group information is stored in the XML for backup.

3rd priority

Import/Export

  • ✔ Update Gedcom import/export from GEPS 043 work to support place model https://github.com/gramps-project/gramps/pull/469
  • ✔ Update importer for place changes; csv
  • ✔ Update importer for place changes; geneweb
  • ✔ Update importer for place changes; progen
  • (N/A) Update importer for place changes;vcard
  • ✔ Update exporter for place changes; csv
  • (N/A) Update exporter for place changes;ftree
  • (N/A) Update exporter for place changes;geneweb
  • (N/A) Update exporter for place changes;vcalendar
  • (N/A) Update exporter for place changes;vcard

Reports

  • ✔ Plugins.webreport.narrativeweb, etc. Add Place Formats option, Update to use place groups, show place attributes.
  • ✔ Plugins.graph.gvfamilylines – update to use place Displayer/Formats.
  • ✔ Plugins.graph.gvrelgraph – Uses_place displayer, update to use place Displayer/Formats.
  • ✔ Plugins.libllibstubkeyword – updated to support place groups for Country, State (largest REGION), County (smallest Region), City (largest Place). Also fixed up postal to find in place attributes. Title now uses standard Place Displayer with preferred format, Place sub-codes 1-5 can use the first five Place displayer Formats.
  • ✔ Plugins.drawreport.statisticschart – Uses_place displayer, update to use place Displayer/Formats.
  • ✔ Plugins.textreport.ancestorreport – Uses_place displayer, update to use place Displayer/Formats.
  • ✔ Plugins.textreport.placereport – added more names and types to the report, fixed to avoid problems with Place Events.
  • Plugins.mapservices.eniroswedenmap uses some specific place types…

Tools

  • Generate Testcases may want to add some additional place items
  • Check & Repair; check_events needs to look at places for references.

Filters

  • ✔ gen.proxy.filter - support place attributes and event refs for sanitizing notes
  • ✔ gen.proxy.private - support place names, types and their citation lists as well as the new eventrefs and attributes.
  • ✔ gen.proxy.referencedbyselection - support place names, types and their citation lists as well as the new eventrefs and attributes.
  • ✔ gen.proxy.proxybase - added new get_placehier_types, get_placeabbr_types, get_place_attribute_types methods.
  • ✔ gen.filters.rules.place.has_data - remove "code" place type scan should look at whole list
  • ✔ New filter rules for place attributes, place events (gen.filters.rules.place._hasattribute, _hasevent)
  • ✔ gui.filters.sidebar._placessidebarfilter - Place Sidebar filter; remove code

Addons

  • CheckPlaceTitles – don't need this anymore?
  • ✔ D3Charts/DescendantIndentedTree - has get_title for a place
  • DataEntryGramplet – has set_title for a place
  • ✔ DescendantBookReport – has get_title for a place
  • ✔ DetailedDescendantBookReport– has get_title for a place
  • ✔ DescendantsLines\substkw – stop override for PlaceFormat from Plugins.libllibstubkeyword, so just use the plugins.lib version.
  • ✔ DynamicWeb – uses place.get_title place get_main_location, get_alternate_locations, get_code (done, works, but may want to enhance with more display of place multiple types, attributes, events...).
  • ✔ ExtractCity – uses set_code
  • ✔ GetGov – modify to store data in correct parts of place, adapt to updated Place Types with hierarchy and date.
  • ✔ GraphView – add support for place selectable formats
  • ✔ ImportMerge – make sure algorithm doesn't break on Event/Place/Event loop
  • ✔ lxml - update to support new schema
  • ✔ PlaceCleanup – Need to store data in correct parts of place, change Code handling, change abbreviation handling, store attributess, multiple place types etc.
  • PlaceCompletion – don't support this anymore?
  • ✔ Sqlite.ExportSql – update for new place data (Not compatible with previous sql files)
  • ✔ Sqlite.ImportSql – update for new place data (Not compatible with previous sql files)

Documentation

Test code

  • Db tests
  • ✔ Import/export/report tests
  • Filter tests

Comments

See also