-
Notifications
You must be signed in to change notification settings - Fork 2
/
DEPLOYNOTES
1071 lines (757 loc) · 40.1 KB
/
DEPLOYNOTES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.. _DEPLOYNOTES:
DEPLOYNOTES
***********
Instructions for installation & upgrade notes.
Installation
============
Software Dependencies
---------------------
We recommend the use of `pip <http://pip.openplans.org/>`_ and `virtualenv
<http://virtualenv.openplans.org/>`_ for environment and dependency
management in this and other Python projects. If you don't have them
installed we recommend ``sudo easy_install pip`` and then ``sudo pip install
virtualenv``.
Configure the environment
^^^^^^^^^^^^^^^^^^^^^^^^^
When first installing this project, you'll need to create a virtual environment
for it. The environment is just a directory. You can store it anywhere you like;
in this documentation it’ll live right next to the source. For instance, if the
source is in /home/thekeep/src, consider creating an environment in
/home/thekeep/env. To create such an environment, su into apache's user
and::
virtualenv --no-site-packages /home/thekeep/env
This creates a new virtual environment in that directory. Source the activation
file to invoke the virtual environment (requires that you use the bash shell)::
. /home/thekeep/env/bin/activate
Once the environment has been activated inside a shell, Python programs
spawned from that shell will read their environment only from this
directory, not from the system-wide site packages. Installations will
correspondingly be installed into this environment.
.. Note::
Installation instructions and upgrade notes below assume that
you are already in an activated shell.
Install python dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Keep depends on several python libraries. The installation is mostly
automated, and will print status messages as packages are installed. If there
are any errors, pip should announce them very loudly.
To install python dependencies, cd into the repository checkout and::
pip install -r pip-dependencies
If you are a developer or are installing to a continuous ingration server
where you plan to run unit tests, code coverage reports, or build sphinx
documentation, you probably will also want to::
pip install -r build-dependencies
pip install lxml --upgrade --force-reinstall --no-binary :all:
After this step, your virtual environment should contain all of the
dependencies for The Keep.
Alternatively, you can use the fabric deploy/build script to install
dependencies for you::
pip install fabric
fab install_deps
Known Issues
""""""""""""
* As of 01/2011, there is a problem installing **python-magic** with pip; this
is apparently caused by either an incorrect dependency on a ctypes PyPi package
or a problem with the PyPi ctypes package that causes it to be uninstallable.
As a work-around, python-magic can be manually installed without dependencies::
pip install --no-deps python-magic
You may need to comment out python-magic in the pip-dependencies file to allow
the other packages to install correctly.
* As of 03/2011, installing **eullocal** from git via pip does not install
the eullocal template themes correctly. The easiest way to fix this is to
manually create a symbolic link from the root of your virtualenv to the
eullocal theme directory::
cd /home/thekeep/env
ln -s src/eullocal/themes/
Install/Configure System Dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Fedora
""""""
The Keep expects to connect to a Fedora-Commons repository version
3.4.x. Fedora should be configured as follows:
* LDAP login filter
* To support large file ingest, the storage length for uploaded file
should be increased from the default value of 5 minutes, e.g.:
.. code-block:: xml
<param name="uploadStorageMinutes" value="15">
* XACML policy enforcement enabled with a combining algorithm of
Permit-Overrides. You should have the following in the fedora.fcfg:
.. code-block:: xml
<param name="XACML-COMBINING-ALGORITHM" value="com.sun.xacml.combine.OrderedPermitOverridesPolicyAlg"/>
<param name="ENFORCE-MODE" value="enforce-policies"/>
Fedora xacml repository policies should be installed from
https://svn.library.emory.edu/svn/fedora/repository-policies/ (or from
the most recently tagged version of the repository-policies, as
identified in version-specific upgrade notes). This checkout should
include a policy file named
**permit-keep-content-if-keep-admin-or-server.xml** in the directory
of Keep-specific policies. The default configuration should be
suitable for a production environment, but if you are deploying the
policies to another environment, you should update the policy to
include the IP address of the server where the web application will
run.
The Keep relies on EULindexer for searching and indexing. See the
`EULindexer Deploy Notes
<https://github.com/emory-libraries/eulindexer/blob/master/DEPLOYNOTES.rst>`_
for additional Fedora configuration requirements for that program.
Celery/RabbitMQ
"""""""""""""""
We use celery for asynchronous tasks (currently only WAV to MP3 audio
encoding), which requires a task broker. We recommend RabbitMQ. For
more information on configuring this, please see
http://celeryq.org/docs/getting-started/broker-installation.html
Solr/EULindexer
"""""""""""""""
The Keep uses Solr for searching Fedora content. The Solr schema
included with the source code at ``solr/schema.xml`` should be used as
the Solr schema configuration. Solr should be configured to
autocommit index changes within a fairly brief time period (perhaps
1-3 seconds); edit the Solr configuration file ``solrconfig.xml`` and
set a value for ``maxTime`` in the ``autoCommit`` section.
The url for accessing the configured Solr instance should be added to
Keep ``localsettings.py`` as **SOLR_SERVER_URL**.
Repository content created and updated by the Keep should be indexed
using **EULindexer**. To add the Keep to an installed and configured
instance of EULindexer, add the Keep indexdata url to the
eulindexer ``localsettings.py``, e.g.::
INDEXER_SITE_URLS = {
'keep': 'http://keep.library.emory.edu:/indexdata/',
}
To populate the index initially, or to reindex all content, run the
``reindex`` script that is available in EULindexer::
python manage.py reindex -s keep
Misc
""""
The WAV to MP3 audio conversion task uses FFMPEG to do the actual
encoding. On Ubuntu systems, this can be installed using ``sudo
apt-get install ffmpeg``.
.. Note::
You may have to add the repo ppa:jon-severinsson/ffmpe to install ffmpeg
Install the Application
-----------------------
Apache
^^^^^^
After installing dependencies, copy and edit the wsgi and apache
configuration files in src/apache inside the source code checkout. Both will
probably require some tweaking for paths and such.
The application provides an RSS podcast feed intended for iTunes on
the MARBL reading room kiosk. The URLs for the podcast feeds and MP3
files should be IP restricted at the Apache level to allow only those
machines that are permitted access to the content. The url patterns
that should be restricted are::
/audio/feeds/[0-9]+/
/audio/*/access.mp3 * is the fedora object pid
Configuration
^^^^^^^^^^^^^
Configure application settings by copying localsettings.py.sample to
localsettings.py and editing for local database, LDAP, fedora, PID manager,
eXist-DB and key configuration.
eXist-DB
""""""""
The Keep requires access to an eXist Database in order to
auto-generate Fedora collection objects based on EAD Finding Aid
documents. All EXISTDB_* settings in the localsettings example file
should be configured for access to an eXist 1.4.x instance with
Finding Aids content. It is recommended to configure eXist access
using guest access or a non-privileged account.
PID Manager
"""""""""""
Commenting out or removing any of the PIDMAN_* configuration options will
raise an error if the ``DEV_ENV`` setting is ``False`` (as is normal for a
production install).
The configured PIDMAN_USER must have permission inside the pid
manager to add pids and targets.
----
After configuring all settings, initialize the db with all needed
tables and initial data using::
python manage.py syncdb
Load Fedora fixtures and control objects to the configured repository
using::
python manage.py syncrepo
Ingest clean-up cron job
^^^^^^^^^^^^^^^^^^^^^^^^
Configure the ingest cleanup script to run as a cron job. This is a custom
manage.py script that cleans up the ingest staging temp directory so that
any uploaded files that are not successfully ingested into Fedora will
be cleaned up. The script cleans up files older than the age specified in
localsettings.py. When run with verbosity 0, only errors will be reported::
python manage.py ingest_cleanup -v 0
.. NOTE::
This script will delete all files older than a specified age
in the configured directory, so it should **not** be run as a
privileged user.
Celery Daemon
^^^^^^^^^^^^^
The celery worker needs to be running for asynchronous tasks. To run through
django do::
python manage.py celeryd -Q keep
Note that we are using a different default queue so that keep-specific tasks
are not sent to a non-keep celery daemon.
See http://ask.github.com/celery/cookbook/daemonizing.html for instructions
on configuring celery to run as a daemon.
.. _mimetype-config:
Custom MIME Types
^^^^^^^^^^^^^^^^^
The AFF and AD1 disk image formats do not (yet) have official MIME types.
Also the 64 bit WAV format W64 AKA RIFF Wave has been added.
We have defined local mimetypes and you will need to update local magic files
so these formats can be recognized. The necessary definitions are included
with the source code in a file called ``magic_extras.txt``. On staging
or production servers, a newly compiled magic file should be generated to
replace the existing file (e.g., ``/usr/share/file/magic.mgc``). For
development systems, as a simpler solution, developers can instead add
the contents of ``magic_extras.txt`` to $HOME/.magic or /etc/magic.
Notes for Developers
--------------------
The following instructions can be ignored when deploying to a staging
or production environment, but may be helpful to a developer working
on the project or running automated tests.
Session configuration
^^^^^^^^^^^^^^^^^^^^^
As of `Release 1.0 - Min Items, part IV`_ the project default Session configuration is set to mark
cookies as secure. To enable login over HTTP (e.g. when developing with Django's runserver),
you will need to override this in your ``localsettings.py``. See the commented out example in
``localsettings.py.sample``.
Fedora XACML policy
^^^^^^^^^^^^^^^^^^^
The version of the Fedora XACML policy checked into subversion is
configured for the production environment. When this policy is
deployed to a development environment, you will want to edit the
deployed version to include the IP addresses of any servers that will
need to access Keep content for development or testing.
Test Setup
^^^^^^^^^^
Certain unit tests pass test user credentials to Fedora, in order to test and
simulate accessing Fedora as the logged in user. For tests to work properly,
the unit test user should be defined (with appropriate permissions)
as a fedora account. See :mod:`keep.audio.tests` for account details.
Upgrade Notes
=============
Release 2.7
-----------
* New local settings must be configured to support for Rushdie pid
and reuse and cleanup: **PIDMAN_RUSHDIE_DOMAIN** and
**PIDMAN_RUSHDIE_UNUSED_URI**
Release 2.6.5
-------------
* The Keep has been updated for eXist 2.2 compatilibity; the **EXIST_**
configurations in ``localsettings.py`` should be updated to match
settings for the current findingaids database.
* The MARBL collection label and MODS title should be updated to reflect
the full title, in order to match the subarea in the findingaids:
"Stuart A. Rose Manuscript, Archives, and Rare Book Library"
* In the localsettings.py this version starts to require the
`FEDORA_MANAGEMENT_USER` and `FEDORA_MANAGEMENT_PASSWORD` in order to
run management commands that query objects in Fedora. Please make sure
that these two local variables are defined in the `localsettings.py`.
Please also make sure that all migrations are run during the deployment
process.
* Please also make sure that all migrations are run during the deployment
process.
Release 2.6.2 - MacMillan Law Library
-------------------------------------
* Run syncrepo to load the new top-level collection object to Fedora::
$ python manage.py migrate
Release 2.5
-----------
* Run database migrations to add the new download disk image permission
and add it to Born Digital Curator group.
$ python manage.py migrate
* This release adds new fields to the Solr schema; Solr should be updated
with the project schema.xml, and all content will be need to reindexed.
* The AFF Disk Image migration requires that the FTKimager command-line
tools be installed on the server where celery tasks are run:
http://accessdata.com/product-download under "Command Line Versions",
or use one of these links:
* `Fedora/Red Hat 32bit <thttp://accessdata.com/product-download/digital-forensics/fedora-and-red-hat-version-32bit-3.1.1>`_
* `Fedora/Red Hat 64bit <http://accessdata.com/product-download/digital-forensics/fedora-and-red-hat-version-x64-3.1.1>`_
* `Debian/Ubuntu 32bit <http://accessdata.com/product-download/digital-forensics/debian-and-ubuntu-32bit-3.1.1>`_
* `Debian/Ubuntu 64bit <http://accessdata.com/product-download/digital-forensics/debian-and-ubuntu-x64-3.1.1>`_
* `Mac OS 10.5/10.6 <http://accessdata.com/product-download/digital-forensics/mac-os-10.5-and-10.6x-version-3.1.1>`_
* `Windows 32bit <http://accessdata.com/product-download/digital-forensics/windows-32bit-3.1.1>`_
* This release includes a new celery task for AFF migration, so the celery
daemon should be restarted after deploy.
* Run the script to migrate AD1 disk images, specifying the directory
where born digital curators have provided the migration content::
$ python manage.py migrate_ad1_diskimages <directory>
* Run the script to migrate AFF disk images; note that AFF and generated E01
temporary files will be placed in the **INGEST_STAGING_TEMP_DIR**
configured in `localsettings.py`::
$ python manage.py migrate_aff_diskimages
Release 2.4
-----------
* The configuration for LDAP has changed; update ``localsettings.py``
based on the example LDAP configuration in ``localsettings.py.dist``.
* Upgrade to django 1.8 and django migrations. Installations with an existing
database will need to fake initial migration for existing tables::
$ python manage.py migrate accounts
$ python manage.py migrate --fake-initial
$ python manage.py migrate
* For new installations, the default Group definitions will no longer
be automatically created. To load them manually, run::
$ python manage.py loaddata initial_groups
* Configure the site not to use external Javascript and to suppress
Google Analytics code for requests from configured Researcher IP
addresses by setting **EULTHEME_NO_EXTERNAL_JS** and
**RESEARCHER_NO_ANALYTICS** both to True in ``localsettings.py``.
* This update includes a new mimetype for the E01 Expert Witness Format;
server magic files should be updated based on ``magic_extras.txt``.
Release 2.3 - Fedora 3.8 migration support
------------------------------------------
* Update the database for configurable downtime::
$ python manage.py syncdb
Release 2.2.1 - Health Sciences Library
---------------------------------------
* Run syncrepo to load the new top-level collection object to Fedora::
$ python manage.py migrate
Release 2.2 - Ye ol DM Video
----------------------------
* add old_dm and accessions to DB config
* set OLD_DM_MEDIA_ROOT = 'http://webprd001.library.emory.edu/harvest/preservation/' and make sure fedora server can access it via the web
* set MIGRATION_VIDEO_ROOT to the mnt point where OLD_DM_MEDIA_ROOT points
* If PID_ALIASES exists in localsettings remove it.
Release 2.1.2 - ETD Library
---------------------------
* run command to create ETD library
$ manage.py syncrepo
Release 2.1 - Video Front End
-----------------------------
* run command to create Video permission::
$ manage.py syncdb
* Replace current solr schema with one from the project
Release 2.0 - Video Ingest
--------------------------
* run command to create Video content model::
$ manage.py syncrepo
* run commands to create Video perms::
$ manage.py syncdb
$ manage.py migrate video
* setup ftp user and directory on staging server to have separate video and diskimage directories and appropriate
users and restrictions.
* Update permit-keep-content-if-keep-admin-or-server.xml with Video-1.0 section and reload policies
Release 1.10
------------
* Add and configure **KEEP_MANUAL_URL** in ``localsettings.py``
* Compile magic file with info from magic_extras.txt
* install ``ffmpeg`` or equivalent system package
Release 1.9
-----------
* Now requires that **mediainfo** be installed on the server for use with
:mod:`pymediainfo`, in order to support ingesting 32bit WAV files.
* Run ``python manage.py migrate`` to update user profiles in the database
for latest version of :mod:`eullocal`.
* Now requires lame version 3.99.5 or later, in order to convert 32bit float
WAV files.
Release 1.8
------------
* Run ``python manage.py syncdb`` to ensure that newly defined, more granular
permissions and pre-defined groups are loaded to the database. Note that
site admins may need to defined and configure group membership before
users will be able to use the site.
* The Keep now requires Solr 4. A new Solr core should be created, the
**SOLR_SERVER_URL** configuration in ``localsettings.py`` should be updated,
and all site content should be reindexed using eulindexer.
* Be sure to restart the eulindexer service so that it picks up the new Solr
configuration and server connection information.
* The iTunes podcast feed functionality has been removed. This requires the
following changes:
* The cron job to run the new_feed_notice manage.py script should be
removed or disabled (the script no longer exists)
* Configurations in localsettings.py for **MAX_ITEMS_PER_PODCAST_FEED**
and **FEED_ADMINS** are no longer used and should be removed.
* All apache configurations restricting access to audio feeds and audio
download URLS by IP address and/or LDAP username should be *removed*;
access restriction is now managed within the django app.
Release 1.7.1
-------------
* To support ingesting disk image files around 80-100 GB, you should ensure
that the WSGI inactivity timeout configured in Apache is set fairly high,
probably at least 1 hour (3600).
Release 1.7
-----------
* Set up a shared network space where designated Keep users can upload
large disk image content for ingest. This directory should be mounted
on the Django app server and made read/write accessible by the Keep
application user, and should be mounted on the Fedora server in a path
where security policies allow file ingest. Configure the appropriate
paths in **LARGE_FILE_STAGING_DIR** and
**LARGE_FILE_STAGING_FEDORA_DIR** in ``localsettings.py``.
* The Fedora XACML policy ``deny-unallowed-file-resolution.xml`` must
be modified to allow File URI ingest from the directory that is set up
as the large-file staging directory on the fedora server by logged
in users.
Release 1.6
-----------
* To support AFF and AD1 disk image files, custom mimetypes are required.
See :ref:`mimetype-config` for setup instructions.
* Run ``python manage.py syncdb`` to update the database schema for
new models.
* Restart the eulindexer service indexing the deployed site so it will
start picking up and indexing any new Disk Image content.
* Update XACML policies in Fedora with the latest version to allow
access to disk image content.
Release 1.5.2
-------------
* Django settings have been updated to follow Django 1.5 conventions.
In particular, check ``localsettings.py.dist`` for changes to
**CACHES** and **LOGGING**.
* Make sure to restart the celery daemon so that it will pick up the
new batch status update task.
Release 1.5
-----------
* This update includes Solr index changes; update Solr with the latest
``schema.xml`` and reindex site content.
* The allowed difference between original audio and generated access copies
must now be configured via **AUDIO_ALLOWED_DURATION_DISCREPANCY** in
``localsettings.py``.
* Update Apache configurations to reference the WSGI file in the new location
(``keep/wsgi.py``), following Django 1.4 conventions.
* Update site and database to work with celery 3.0.
* Update the database for the newest version of :mod:`celery`::
python manage.py migrate djcelery
* Celery broker should now be configured using **BROKER_URL** instead of
individual **BROKER_** settings; see ``localsettings.py.dist`` for
an example.
* The celery worker should now be started via::
python manage.py celery worker -Q keep
Be sure to update any cron jobs or init scripts that use the old
``celeryd`` syntax.
* If you are not using the WSGI script included with the source code, add the
following to your wsgi script::
import djcelery
djcelery.setup_loader()
* Allowed duration discrepancy when generating access copies of audio files
is now configurable in ``localsettings.py`` using **AUDIO_ALLOWED_DURATION_DISCREPANCY**.
* Previously ingested and migrated Arrangement objects were associated with
the Rushdie collection incorrectly (using the isMemberOf relation instead of
isMemberOfCollection). This can be corrected in the Django console as follows::
python manage.py shell
.. code-block:: python
from keep.common.fedora import Repository
from keep.arrangement.models import ArrangementObject
repo = Repository(username='fedoraAdmin', password='fedoraAdmin')
objs = repo.get_objects_with_cmodel(ArrangementObject.ARRANGEMENT_CONTENT_MODEL, type=ArrangementObject)
for obj in objs:
if obj._deprecated_collection is not None:
if obj._deprecated_collection.exists and obj._deprecated_collection.has_requisite_content_models:
# populate collection if not set
if not obj.collection:
obj.collection = obj._deprecated_collection
# remove deprecated collection
del obj._deprecated_collection
# save changes
if obj.rels_ext.isModified():
obj.save('converting incorrect collection membership from isMemberOf to isMemberOfCollection')
.. Note::
In a development or staging environment where a collection is incorrectly referenced but does not
exist, you may want to check the deprecated collection by pid. In particular, it is important
not to remove all ``isMemberOf`` relations, because Rushdie Mailbox objects are associated with
their corresponding Email account objects via ``isMemberOf``.
* This release includes several one-time scripts to import content and verdict
for Rushdie's 5300c computer. These should be run by a developer or someone familiar with
both the site and the content as follows, in this order:
1. Import series and verdict information for non-email content based on a CSV file, which will be
provided by born-digital archivist or project manager::
python manage.py import_verdict --csv=/path/to/csvfile
Note that this step may identify duplicate records, which will require manual clean up; any
clean up should be completed **before** running the next script (import_content).
2. Import file content for non-email objects. This will require the 5300c disk
image be mounted on the local file system, using a command something like this::
sudo mount 5300c.img -o loop -t hfs /mnt/rushdie_5300c
Then run the script to import file content from the disk image, providing it the pid
for the 5300c processing batch (aka "simple collection")::
python manage.py import_content /mnt/rushdie_5300c emory:bm4pr
3. Import email messages from the 5300c disk image; pid should be the processing batch (aka "simple collection")
id; file path should be the location of Eudora email content::
python manage.py ingest_5300c_email emory:bm4pr /path/to/Eudora\ Folder/ --user=<user> --password=
4. Import series and verdict information for email content based on a CSV file::
python manage.py import_verdicts --email --csv=/path/to/5300c_email_verdicts.csv
The output from each of these scripts should be collected and sent to project managers and/or
born-digital archivists for review.
* This release includes a new manage.py script for sending notification when the number of available
iTunes podcast changes. Update the database with the new table that will be used to keep track of
the number of feeds::
python manage.py syncdb
Update ``localsettings.py`` with configurations for sending emails, optionally including `SERVER_EMAIL`_,
`EMAIL_SUBJECT_PREFIX`_ (see ``localsettings.py.dist``), and set any necessary SMTP configuration (see
`Django email settings`_). Add the email addresses for the users who should receive this
notification to the new **FEED_ADMINS** setting.
.. _SERVER_EMAIL: https://docs.djangoproject.com/en/1.4/ref/settings/#server-email
.. _EMAIL_SUBJECT_PREFIX: https://docs.djangoproject.com/en/1.4/ref/settings/#email-subject-prefix
.. _Django email settings: https://docs.djangoproject.com/en/1.4/ref/settings/#email-host
Run the script once to populate the current count in the database::
python manage.py new_feed_notice
Configure this script to run as a nightly cron job.
Release 1.4 - search and ingest enhancements
--------------------------------------------
* The Keep has been updated to make use of Django's
:mod:`~django.contrib.staticfiles` application for managing static
files (i.e., website images, CSS, and Javascript). The fabric
deploy script will run the ``collectstatic`` manage command, which
will collect all static files into the configured **STATIC_ROOT**
path in ``settings.py`` (currently a ``static`` directory inside the
deploy). Existing apache configurations should be updated; instead
of multiple aliases for sitemedia and genlib_media directories, a
single alias should be set up for the ``/static`` URL, pointing at
the configured static directory. (See latest
``apache/the_keep.conf`` for an example.)
* This update includes Solr index changes; update Solr with the latest
``schema.xml`` and reindex site content.
* This update requires modifications to Fedora XACML policies; update
Fedora with the latest polices and and reload them.
* Run syncdb to load the Born Digital group using the following command::
python manage.py syncdb
* When the code is deployed to staging the ENABLE_BETA_WARNING
variable should be set in localsettings.py::
ENABLE_BETA_WARNING = True
Release 1.3 - Search-based Report generation
--------------------------------------------
* Fabric deploy script has been updated to follow latest fabfile best
practices & local conventions. Run ``fab -l`` and ``fab -d deploy``
for usage details.
* The Keep has been updated to work with Solr 3.3. Use the included
solr configuration configurations or the Solr tarball created by the
fabric deploy script to create a new Solr core and then use
eulindexer to index all Keep content.
* Now that the ``old_dm`` migration has been completed, migration code
has been removed. Database configurations for **old_dm** and
**accessions** databases should be removed from ``localsettings.py``.
* We are now using :mod:`south` to manage database migrations for
emory_ldap (part of :mod:`eullocal`) and :mod:`djcelery`. To update
an existing database, you will need a few extra steps. First, run
``syncdb`` to create the database tables for :mod:`south`. Then, to
update emory_ldap, you will need to fake the initial migration, run
migrations 0002 and 0003 normally, and then fake step 0004::
python manage.py migrate --fake emory_ldap 0001
python manage.py migrate emory_ldap 0003
python manage.py migrate --fake emory_ldap 0004
For :mod:`djcelery`, all djcelery_ and celery_ tables should be
dropped from the database and recreated via migrate::
python manage.py migrate djcelery
Release 1.2 - Verdict App
-------------------------
* Configure the LOGGING section in the localsettings.py file on the remote server
based on the example in the localsettings.py.dist file.
Log rotation need to be set for this file as well.
* Devloper should run the fabric deploy script:
- On the local workstation, copy settings.json.dist to settings.json and configure
- run::
fab load_config:settings.json deploy
* TheKeep XACML policies should be updated to the appropriate branch or tag:
* Staging: https://svn.library.emory.edu/svn/fedora/repository-policies/branches/release_1.9
* Production: https://svn.library.emory.edu/svn/fedora/repository-policies/tags/release_1.9
* reload policies on fedora
* The solr schema should be updated:
- Copy <project base>/solr/schema.xml to the appropriate solr core
- Solr should be restarted
* The following manage commands should be run::
python ./manage.py syncdb
python ./manage.py syncrepo
* The paths in the http config file should be verified since some have changed or may be different from enviroment to enviroment..
* At this point systems should move the link to point to the current version.
* Contact developer for these tasks:
- Assign appropriate permissions for each user in the database
- Load P5300c objects to fedora using the command::
python ./manage.py load_arrangement <CSV file> <master collection pid> <new simple collection name> -u <user> --password
* Reindex the site:
- Make sure that eulindexer has an entry for "keep" in localsettings.py > INDEXER_SITE_URLS .
- Staging:: https://testkeep.library.emory.edu/indexdata/
- Production:: https://keep.library.emory.edu/indexdata/
- EULIndexer should be restarted
.. Note::
When running this command you must be in the eulindexer virtual
environment.
- Run the command::
python ./manage.py reindex -s keep
- After the above commad finishes make sure the indxer is runnng so new items can be indexed
Release 1.1 - Metadata Migration
--------------------------------
* The kiosk RSS feed now links to files in the old Digital Masters datastore
once their metadata records are in The Keep. This requires setting an
**OLD_DM_MEDIA_ROOT** in localsettings. It should point to the root URL for
preservation files. Our correct production setting is the one listed in
``localsettings.py.sample``.
* This release relies on new Fedora content model objects. Execute the
following command to load them::
python manage.py syncrepo
* The Keep now uses Solr for searching and eulindexer for indexing
items. Follow the directions in `Solr/EULindexer`_ to set up a new
Solr index and add the Keep to EULindexer. Before indexing existing
content, run a one-time clean-up script to update existing objects
in Fedora::
python manage.py cleanup_metadata
After this script completes, configure EULindexer and run its reindex
script as described in `Solr/EULindexer`_ to populate the Solr index for
existing repository content. Make sure that the indexer script is running
before migrating new content, so that newly ingested items will
automatically be indexed.
* After existing content has been reindexed and the website has been
verified to work correctly, prepare to migrate legacy metadata. First
dry-run the migration script::
python manage.py migrate_metadata -n -v2 -c migrate.csv > migrate.out
Developers and project stakeholders will review migration output
(``migrate.csv`` and ``migrate.out``). This review process will probably
result in a request for a ``load_ead`` command as performed in Release 0.7
below. Once the review and any additional configuration are complete, run
the final metadata migration::
python manage.py migrate_metadata -c migrate.csv > migrate.out
The application does not use the CSV or output files. After the migration
is complete, these files may be archived and/or deleted.
Release 1.0.3
-------------
* This release depends on a newer version of python-eulcore. Activate
the virtualenv and update::
pip install -r pip-dependencies
* Previously, the code did not add a name to PID records when
generating ARKs for new objects. To update the PID records with
labels for objects that have already been created, the following
code can be used in the django console:
.. code-block:: python
from keep.common.fedora import Repository
from django.conf import settings
from pidservices.djangowrapper.shortcuts import DjangoPidmanRestClient
repo = Repository()
pidman = DjangoPidmanRestClient()
objs = repo.find_objects(owner=settings.FEDORA_OBJECT_OWNERID)
for obj in objs:
pidman.update_ark(obj.noid, name=obj.label)
Release 1.0.2
-------------
The Fedora XACML policy has been corrected for the production
environment. Please update the Fedora policies to this tag:
https://svn.library.emory.edu/svn/fedora/repository-policies/tags/release_1.6.1
See `Fedora`_ configuration notes for additional information on
possible additional steps for customizing this policy (for
non-production environment).
Release 1.0 - Min Items, part IV
--------------------------------
* We are using a newer version of python-eulcore. To update::
. /home/keep/env/bin/activate
pip install -r pip-dependencies
* Django Session default configuration has been modified in ``settings.py``. By default,
the application will now store session data in the django cache (already in use by the application and configured via
``CACHE_BACKEND``), mark cookies as secure (HTTPS transfer only), expire after one week, and expire when the browser
is closed. Any of these configurations may be overridden in ``localsettings.py``. For more details on configuration
options, see `Django Sessions documentation <http://docs.djangoproject.com/en/1.2/topics/http/sessions/>`_ .
.. Note::
Developers are affected by this configuration change and should consult `Notes for Developers`_.
Release 0.9 - Min Items, part III
---------------------------------
* The application now includes an asynchronous task for generating MP3 files,
which requires LAME, celery configuration, and a celery broker. See
`Celery/RabbitMQ`_ and `Misc`_ in `Install/Configure System
Dependencies`_. Celery and broker configurations are now required in
localsettings.py (see localsettings.py.sample; search for BROKER).
Running tasks requires that a celery worker process be run; see the
`Celery Daemon`_ section in `Install the Application`_.
New database tables in the relational database are required for the
celery functionality. To initialize them, run::
python manage.py syncdb
* The application now provides an RSS podcast feed intended for iTunes on the
MARBL reading room kiosk. Due to limitations in the amount of data that iTunes
can harvest in a single feed, the podcast feed generation logic expects a setting
for MAX_ITEMS_PER_PODCAST_FEED. See localsettings.py.sample for example
configuration and recommended value.
When deployed in production, Apache should be configured to restrict access to
the podcast feed and MP3 URLs. See the `Apache`_ section of `Install the
Application`_ for details.
* Dependencies have been addded for celery tasks and MP3 error checking.
To install the new dependencies::
. /home/keep/env/bin/activate
pip install -r pip-dependencies
* The application now uses an eulcore task result module to report on celery tasks,
which requires additional tables in the relational database. To initialize them,
sync the db::
python manage.py syncdb
* New datastreams have been added to the Fedora Audio object. Any audio objects
loaded in the staging environment should be removed when this update is deployed.
This can be done in the django console as follows. Start the console::
python manage.py shell
And then run:
.. code-block:: python
from keep.common.fedora import Repository
from keep.audio.models import AudioObject
repo = Repository()
repo.get_objects_with_cmodel(AudioObject.AUDIO_CONTENT_MODEL)
objs = repo.get_objects_with_cmodel(AudioObject.AUDIO_CONTENT_MODEL)
for o in objs:
repo.purge_object(o.pid)
* As part of branding this project with its new name, "The Keep", the pseudo-owner
used for repository-level permissions has been changed. The Fedora repository
policies should be updated to
https://svn.library.emory.edu/svn/fedora/repository-policies/branches/release_1.3
Any existing objects will need to be either removed and re-generated or updated
with the new owner ID. Updating existing objects can be done in the django
console. Start the console::
python manage.py shell
Then run the following:
.. code-block:: python
from keep.common.fedora import Repository
from django.conf import settings
repo = Repository()
objs = repo.find_objects(owner='euterpe-project')
for o in objs:
o.owner = settings.FEDORA_OBJECT_OWNERID
o.save()
* As part of branding this project as "The Keep", the main source directory
is now ``src/keep`` instead of ``src/digitalmasters`` .
Release 0.8 - Min Items, part II
--------------------------------
* The app now depends on the standard django ``sites`` module. This module
stores some data in the relational database. To initialize it, sync the db::
python manage.py syncdb
After restarting the web app, configure the default site by replacing the
``example.com`` domain with the domain of the live webapp.
* Django caching for lists based on Fedora objects requires defining a
CACHE_BACKEND in localsettings.py (see localsettings.py.sample)