Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Adding indexer that serializes RDF to disk.
  • Loading branch information
escowles committed Oct 14, 2015
1 parent 2053e07 commit 8ca0720
Show file tree
Hide file tree
Showing 16 changed files with 1,054 additions and 0 deletions.
99 changes: 99 additions & 0 deletions fcrepo-indexing-disk/README.md
@@ -0,0 +1,99 @@
#Fedora Indexing Service (Disk)

This application serializes RDF to disk for [Fedora4](http://fcrepo.org) objects that are
updated.

Additional background information on this service is available on the Fedora Wiki on the
[Integration Services page](https://wiki.duraspace.org/display/FEDORA4x/Integration+Services).

##Building

To build this project use

MAVEN_OPTS="-Xmx1024m" mvn install

##Running from the command line

To run the project you can execute the following Maven goal

mvn camel:run

##Deploying in OSGi

This project can be deployed in an OSGi container. For example using
[Apache ServiceMix](http://servicemix.apache.org/) or
[Apache Karaf](http://karaf.apache.org), you can run the following
command from its shell:

feature:repo-add mvn:org.fcrepo.camel/fcrepo-camel-toolbox/LATEST/xml/features
feature:install fcrepo-indexing-disk

##Deploying in Tomcat/Jetty

If you intend to deploy this application in a web container such as Tomcat or Jetty,
please refer to the documentation in the
[fcrepo-camel-webapp](https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/tree/master/fcrepo-camel-webapp)
project.

##Configuration

The application can be configured by creating a file in
`$KARAF_HOME/etc/org.fcrepo.camel.indexing.disk.cfg`. The following
values are available for configuration:

In the event of failure, the maximum number of times a redelivery will be attempted.

error.maxRedeliveries=10

If the fedora repository requires authentication, the following values
can be set:

fcrepo.authUsername=<username>
fcrepo.authPassword=<password>
fcrepo.authHost=<host realm>

The baseUrl for the fedora repository.

fcrepo.baseUrl=localhost:8080/fcrepo/rest

If you would like to index only those objects with a type `indexing:Indexable`,
set this property to `true`

indexing.predicate=false

The JMS connection URI, used for connecting to a local or remote ActiveMQ broker.

jms.brokerUrl=tcp://localhost:61616

The camel URI for the incoming message stream.

input.stream=activemq:topic:fedora

The camel URI for handling reindexing events.

disk.reindex.stream=activemq:queue:disk.reindex

The base directory where RDF should be written.

disk.baseDir=/path/to/directory

The filename extension to use for RDF files.

disk.extension=ttl

The RDF serialization to use (one of JSON-LD, N3, N-TRIPLE, RDF/XML, RDF/XML-ABBREV, TURTLE".

disk.serialization=TURTLE

The location of the internal Audit trail, if using the `fcrepo-audit` extension module.
Nodes at this location will not be indexed.

audit.container=/audit

By editing this file, any currently running routes will be immediately redeployed
with the new values.

For more help see the Apache Camel documentation

http://camel.apache.org/

265 changes: 265 additions & 0 deletions fcrepo-indexing-disk/pom.xml
@@ -0,0 +1,265 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<artifactId>fcrepo-camel-toolbox</artifactId>
<groupId>org.fcrepo.camel</groupId>
<version>4.3.1-SNAPSHOT</version>
</parent>

<artifactId>fcrepo-indexing-disk</artifactId>
<packaging>bundle</packaging>

<name>Fedora Indexing Service with a Disk backend</name>
<description>Camel-based indexing service for serializing object RDF to disk</description>

<properties>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-blueprint</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http4</artifactId>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
</dependency>
<dependency>
<groupId>org.fcrepo.camel</groupId>
<artifactId>fcrepo-camel</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>

<!-- logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>

<!-- Testing & Camel Plugin -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-test-blueprint</artifactId>
</dependency>

<dependency>
<groupId>com.jayway.awaitility</groupId>
<artifactId>awaitility</artifactId>
</dependency>

<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
</dependency>

<!--
<dependency>
<groupId>org.apache.jena</groupId>
<artifactId>jena-fuseki</artifactId>
</dependency>
-->

<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<defaultGoal>install</defaultGoal>

<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
</plugin>

<!-- to generate the MANIFEST-FILE of the bundle -->
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
</plugin>

<!-- to run the example using mvn camel:run -->
<plugin>
<groupId>org.apache.camel</groupId>
<artifactId>camel-maven-plugin</artifactId>
</plugin>

<!-- enable checkstyle plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
</plugin>

<!-- reserve network ports for integration testing -->
<!-- add configuration file to artifact set for OSGi deployment -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<configuration>
<portNames>
<portName>fcrepo.dynamic.test.port</portName>
<portName>fcrepo.dynamic.jms.port</portName>
<portName>fcrepo.dynamic.stomp.port</portName>
<portName>jetty.dynamic.stop.port</portName>
</portNames>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>src/main/cfg/org.fcrepo.camel.indexing.disk.cfg</file>
<type>cfg</type>
<classifier>configuration</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>

<!-- integration testing -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<argLine>${jacoco.agent.it.arg}</argLine>
<systemPropertyVariables>
<fcrepo.dynamic.jms.port>${fcrepo.dynamic.jms.port}</fcrepo.dynamic.jms.port>
<fcrepo.dynamic.stomp.port>${fcrepo.dynamic.stomp.port}</fcrepo.dynamic.stomp.port>
<fcrepo.dynamic.test.port>${fcrepo.dynamic.test.port}</fcrepo.dynamic.test.port>
<jetty.dynamic.test.port>${jetty.dynamic.test.port}</jetty.dynamic.test.port>
</systemPropertyVariables>
</configuration>
</plugin>


<!-- Launch jetty for integration testing with fedora -->
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<configuration>
<systemProperties>
<force>true</force>
<systemProperty>
<name>fcrepo.home</name>
<value>${project.build.directory}/fcrepo-data</value>
</systemProperty>
<systemProperty>
<name>fcrepo.dynamic.jms.port</name>
<value>${fcrepo.dynamic.jms.port}</value>
</systemProperty>
<systemProperty>
<name>fcrepo.dynamic.stomp.port</name>
<value>${fcrepo.dynamic.stomp.port}</value>
</systemProperty>
</systemProperties>
<scanIntervalSeconds>10</scanIntervalSeconds>
<stopKey>STOP</stopKey>
<stopPort>${jetty.dynamic.stop.port}</stopPort>
<daemon>true</daemon>
</configuration>
<executions>
<execution>
<id>start-jetty</id>
<phase>pre-integration-test</phase>
<goals>
<goal>stop</goal>
<goal>start</goal>
</goals>
<configuration>
<httpConnector>
<port>${fcrepo.dynamic.test.port}</port>
</httpConnector>
<scanIntervalSeconds>0</scanIntervalSeconds>
<daemon>true</daemon>
<contextHandlers>
<contextHandler implementation="org.eclipse.jetty.webapp.WebAppContext">
<war>${project.build.directory}/fcrepo.war</war>
<contextPath>/fcrepo</contextPath>
</contextHandler>
</contextHandlers>
</configuration>
</execution>

<execution>
<id>stop-jetty</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>

<!--Bring in fcrepo webapp for integration testing-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy</id>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.fcrepo</groupId>
<artifactId>fcrepo-webapp</artifactId>
<version>${project.version}</version>
<type>war</type>
<outputDirectory>${project.build.directory}</outputDirectory>
<destFileName>fcrepo.war</destFileName>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>

</plugins>
</build>

</project>

0 comments on commit 8ca0720

Please sign in to comment.