Results tagged “Perl”

Backlog of Achievements

Having been busy, but distracted, I've neglected to update with some backend work.  It follows in summary:

  • Refactored away the URL table.  Previously there had been a URL table which stored the urls as LONGTEXT objects and let the MANUSCRIPT and SAINT_SOURCE tables reference it by ID.  This was theoretically elegant, but kind of useless.  I replaced the URL_ID with a proper URL in both of those tables, mapped the data in and dropped the URL table.  This was done entirely in SQL, without a single line of application code needing editing, which perhaps made me more proud that it should
  • Initial work on a date-graph page is complete.  This gives a histogram of the dates of the DB contents, only for manuscripts with entries though.  It is not complete, but working and useful (see below)
  • Added the ability to add native css to the colors file.  This is outside the css_raw column in the COLORS table, that gets inserted into the named class.  Now there's a stand-alone CSS file that gets appended.  This was needed to implement different colors on the first letter of an entry, since those need pseudo-classes
  • Added a little secret tool to visualize ALL the manuscripts in the DB.  It's just a toy, but also only took about 45 minutes of coding

CoKLDB Date Distribution.png

Saint Detail Pages

One of the last uncoded basic functions is now in place, Saint detail pages.  At the bottom of the Saint popup, where there are links to the matching MS, there's now a link to full details.  This page includes all the data in the Saint table, as well as all of the matching entries, for comparing spellings, etc.  Eventually there will be checkboxes as another way into the "compare" mode, but not yet.  There is, however, a basic perm-link setup for this.  Similar in syntax to that for the saints /p/v100/saint/NNNN will take to to the saint with that ID, eg. /p/v100/saint/2850 will give you ALL the possible ways that "All Saints" can be written.

Future plans also include some graphing powers, but like the MS pages, those are part of future work

Tech Debt - Manuscripts class

In many way the BOHdb::Manuscripts class is the messiest of the underlying data manipulation classes. As odd as it seems, it was mostly written before the work coalesced in the BOHdb::Manuscript class that under-girds it, and there are some colossal inefficiencies in the code, particularly in loading the manuscript metadata. One of the oddities, how one controls which manuscripts are loaded into the object, has been ironed out at least. There are now a couple of “restriction” functions, one to set/add them and another to clear them. This means that the load_all function, which previously loaded manuscripts based on the passed restrictions, and then forgot them, ONLY loads manuscripts; it uses restrictions stored in the object previously. This does make the code a bit more verbose:

 $mss->load_all({'calendar_entries' =>1 });

is now

 $mss->set_restriction({'calendar_entries' =>1 });
 $mss->load_all();

But it should make it more usable.

A second piece of tech debt was also paid off, there are now basic unit tests for the Manuscripts class, there had previously been none

Complete List of Saints

One of the final new features for MVP has been completed, the list of all the saints, sorted by date, with their IDs included, in a table.  This is the UI for manually linking calendar entries, transcribed from a manuscript, to the saints in the DB tables.  Previously this work had been done by referencing the DB table directly through SQL tools, now that underlying access is uneeded. 

This work included using the Data Tables library for layout and sorting.  As a side effect the main manuscript table was migrated to use this library as well

Short names

A lot of moving parts moved to add one new feature, there's now a "Browse Saint by Name".  This required adding a short_name column to the saints table (easy), and populating it with the minimal name of the saint, so removing all the titles (Bishop, Virgin, Martyr, etc) and all the localizing info (of Benevento, eg).  That required some regex parsing of the saints and then a bunch of manual fixup, since the regex can't tell that the 'Virgin' of "of the Blessed Virgin Mary" is important to keep, even though in "Agnes, Virgin" it is not.  Once that was completed, the new field was added to the Saints class and a new get_by_name function added to Saints, allowing one to retrieve a hash with the key as the short_name, rather than date.  Along with that the get_by_name can restrict by letter of the alphabet, if you only want saints from a-c or somesuch.

All of that was rolled up into the new browse, which uses a series of vertical tabs to segment the alphabet into manigable lists of saints.  The final display mode is mostly similar to "saints by date", with the same issues around gridlines to work out, but it's all live and working. 

Tasks remaining are a better popup when you click a saint, probably including all name variants from the MS, and better display UI/UX

DateConverter class

Today's addition is a fast DateConverter.  the Date class can be used for conversion, usually by creating an object and then getting values, eg.

my $date = BOHdb::Date->new({ 'name' => 'id', 'values' => 283});
my $month = $date->get_month_num();
my $day = $date->get_standard_day();

However if that's the only reason to create the Date object, it's sorta inefficient to hit the DB in order to convert a date. For simple conversion between day_id (1-365) and month/day numbers the DateConverter class is MUCH faster.  It's had the three relevant columns of the DB turned into big perl arrays and coded into the class directly, so a conversion is simply an array lookup.  There are also the 2 functions in there to generate the static arrays, in case the backed data ever changes, the class is easy to update.  The user only uses the 2 get functions, get_date_by_id and get_date_by_month_day. Both return a little hash with all the relevant data

{
   'month' => 'October',
   'day' => '10',
   'id' => '283',
   'month_num' => '10'
},

A quick timing test:

ok 1 - get_date_by_id returns a hashref in 0.000009 seconds
...
ok 5 - BOHdb::Date created in 0.027421 seconds

Calendars and More Calendars

  • Unit tests are written for calendar class, 12 tests so far
  • Initial code up of the compare_calendar functions is complete.  There's nothing to DO with the comparison results yet, but the comparisons seem to work
  • Add a language field to the calendar DB
  • Cleaned up the display of the calendar color sublist

Bulk Loader, rev. 1

Something that has been on the "should do" list for a while and I finally did is a bulk loader for calendar entries.  This is a command-line app that takes a csv file and inserts it into the DB of entries.  It requires that the columns match the expectation, but there's an excel template in Git that makes that easy, and it requires that the calendar object is already there, as it takes calendar ID as it's argument.  Running it looks like this:

[aaronm@Shamash bohdb]$ bin/loader.pl --csv ~/Dropbox/manuscripts/Books\ of\ Hours/Add\ MS\ 35313.csv --calendar=46 
366 rows loaded from /Users/aaronm/Dropbox/manuscripts/Books of Hours/Add MS 35313.csv at bin/loader.pl line 47.
365 rows prepared to insert at bin/loader.pl line 83.
[aaronm@Shamash bohdb]$

This is rough code, we're doing direct SQL transactions rather than using the relevant Calendar object, but it makes the insert faster and lets the transaction work.  At some point there will be a better loader, probably with a web interface, and that will do things "right".  BL MS Add 35313 is the first calendar loaded with the new code.

Saints Crossreferencing

The first of the clever functions has been implemented in a basic way, saints are cross-referenced.  In a calendar listing, any saint with a proper identity is slightly bolder-face and a hyperlink.  Clicking will bring up a dialog with the cannonical name, date and links to see all MS either containing the saint, or missing the saint, see screenshot below. 

At the moment this only works for the primary saint, not the secondary one.  Breaking the link into 2 parts is a harder problem and will be tackled later-on.  The lookup works for both however.

Screen Shot 2016-02-21 at 4.49.37 PM.png