MarkLogic Production Deployments
How to make them
clean/repeatable/testable
February 15, 2011
Copyright © 2011 Flatirons Solutions Corporation
1
Who Am I?
 Brad Rix
[email protected]
Technical Lead for Content Technology Practice
 Flatirons Solutions Corporation
Flatirons Solutions is a Colorado based system integrator and
long-time MarkLogic partner specializing in XML publishing,
dynamic content delivery, and digital asset management for
both commercial and government clients.
Copyright © 2011 Flatirons Solutions Corporation
2
Why Do I Care About This Stuff?
 Problem:
 You’ve written some cool XQuery
 It does great things that will help your business/customer
 But oops: the more you write code, the harder it gets:
 New developers break code they don’t understand
 Deployments to new environment don’t work and the issues are hard to
resolve
 More code being written causes more instabilities which causes longer
debug/deploy cycles
 Which results in you spending less time writing cool XQuery and more
time fixing the problems described above
 Answer:
 Invest some time and engineering to create a
build/test/deploy/continuous-integration infrastructure which will
greatly improve the ratio of (time doing fun stuff/time doing un-fun
stuff)
Copyright © 2011 Flatirons Solutions Corporation
3
Agenda
 General Goals
 Automatic Deployments
 Automatic Configuration
 Automated Unit Testing
 Hudson Automated Builds
 Lessons Learned
Copyright © 2011 Flatirons Solutions Corporation
4
General Deployment Goals
 Repeatable Procedures
 Ability to automate deployment of XQuery modules
 Ability to automate building of environments to replicate
exact configuration among dev/test/production
environments
 Ability to run unit tests against the code & environment
 Automate running of unit tests
Copyright © 2011 Flatirons Solutions Corporation
5
Automated Deployments
 XQuery modules stored in database
 Used ant task to deploy modules
 Use Docs HTTP server or create a single HTTP server that has
a single module stored that manages deployment
 Ant task zips up files and POST them to the HTTP server on
the Docs database where the simple XQuery module is
deployed.
 Setting collections/permissions/etc on the files as they are
ingested.
Copyright © 2011 Flatirons Solutions Corporation
6
Automated Deployments (ant task)
 First zip modules into a single zip file
<target name="package“>
<mkdir dir="${dist.home}"/>
<zip destfile="${dist.home}/${ml. name}.zip">
<fileset dir="${build.home}/src/${ml. name}"/>
</zip>
</target>
Copyright © 2011 Flatirons Solutions Corporation
7
Automated Deployment (ant task)
 Properties set in build.properties configurable per environment
 Post zip file to preloaded endpoint in Docs database
<target name=“deploy">
<exec executable="wget" dir="${dist.home}" failonerror="true"
vmlauncher="true">
<arg value="--user=${ml.ingest.user}"/>
<arg value="--password=${ml.ingest.password}"/>
<arg value="--header=Content-Type:application/zip"/>
<arg value="--post-file=${ml.name}.zip"/>
<arg
value="${ml.db.docs.host}/admin/loadModulesDB.xqy?modulesdb=
${ml.modules.db}&amp;serverroot=/"/>
</exec>
</target>
Copyright © 2011 Flatirons Solutions Corporation
8
Automated Deployment (module)
Modules parse zip file to get file (snippet):
for $zip-input in xdmp:zip-manifest($xquery-moduleszip)//*:part
return
Get Module:
xdmp:zip-get($xquery-modules-zip, $filename,
<options xmlns="xdmp:zip-get">
</options>)
Then load module into database:
xdmp:document-insert($uri, $doc, $permissions)
Copyright © 2011 Flatirons Solutions Corporation
9
Automated Configuration
 Goal is to be able to configure all database configurations
including Application servers/application settings/database
settings/range indexes/security roles/etc.
 Create a configuration file that contains all of the required
information.
 Allow the configuration to change and able to re-run on new
or existing environment
 Run the configuration via ant task or xquery module
Copyright © 2011 Flatirons Solutions Corporation
10
Automated Database Configuration
 Configuration file per environment
 Partial Configuration of database
<database name=“MyDBName”>
<word-positions>true</word-positions>
<element-value-positions>true</element-value-positions>
<element-range-indexes>
<index>
<scalar-type>string</scalar-type>
<namespace-uri>http://mhhe.com/meta/resolved</namespace-uri>
<local-name>type</local-name>
<collation>http://marklogic.com/collation/</collation>
<range-value-position>false</range-value-position>
</index>
…..
Copyright © 2011 Flatirons Solutions Corporation
11
Automated Configuration (ant task)
 Ant task to call a module to configure the server.
 Post the file to an endpoint or have the module load the
configuration from a file.
<target name="setupdatabase">
<runxquery
module="admin/configureEnv.xqy?source=filesystem&amp;u
ri=${ml.home.url}/Docs/config/environmentconfig.xml&amp;env=${ml.environment}&amp;restart=true"
host="${ml.db.docs.host}" user="${ml.user}"
password="${ml.password}"/>
</target>
Copyright © 2011 Flatirons Solutions Corporation
12
Configuration Module
 Configuration module will read the configuration file
 Read the XML options and run the appropriate admin
services API functions to configure the server/database.
 Optionally restart the server depending on parameters that
require restart to change
Copyright © 2011 Flatirons Solutions Corporation
13
Sample Function to Create Indexes
let $database-name := $database-config/@name
let $dbid := xdmp:database($database-name)
(: remove old ones first :)
let $cluster-config := admin:database-delete-range-element-index($cluster-config,$dbid,
admin:database-get-range-element-indexes ($cluster-config, $dbid))
let $retv :=
for $index in $database-config/element-range-indexes/index
let $rangespec := admin:database-range-element-index($index/scalar-type, $index/namespace-uri,
$index/local-name, $index/collation, $index/range-value-position )
return
try {
xdmp:set($cluster-config, admin:database-add-range-element-index($cluster-config, $dbid,
$rangespec) )
} catch ($err) {
if ($err/error:code eq "ADMIN-DUPLICATEITEM") then
()
else
error(xs:QName("ERROR"), xs:string($err/error:message))
}
return $cluster-config
Copyright © 2011 Flatirons Solutions Corporation
14
XQuery Unit Testing
 Know that all existing modules work as expected
 Changes in modules still work for previous expected
behavior
 At Flatirons we developed a java XQueryUnit that utilizes
XML Unit for differencing XML files
 Framework uses ant to run all unit tests
 Incorporated into entire build process so that the deploy
target can run the tests to ensure that everything works
properly in this environment
Copyright © 2011 Flatirons Solutions Corporation
15
Unit Tests (configuration file)
 Call a module and expect known return
 If XML does not match return, then test will fail
<xqu:test xqu:name="HelloWorldInline">
<xqu:xqy_test>
<xqu:xqy_test_file xqu:uri="modules/util/testHelloWorld.xqy" />
</xqu:xqy_test>
<xqu:expected_result>
<xqu:xml_result>
<xqu:xml_result_src>
<hello>
world
</hello>
</xqu:xml_result_src>
</xqu:xml_result>
</xqu:expected_result>
</xqu:test>
Copyright © 2011 Flatirons Solutions Corporation
16
Unit Tests – Testing Restful Endpoint
 To test an http restful call into MarkLogic, write an XQuery
wrapper that performs the xdmp:http-get (or post).
 Return the data back from the module and compare that to
expected output in the configuration file.
Copyright © 2011 Flatirons Solutions Corporation
17
Run Unit Tests
 Run ant test
test:
[junit] Testsuite: com.mhhe.xmlunit.UnitTestSuite
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 3.078
sec
[junit]
[junit] Testcase: HelloWorldInline took 0.453 sec
[junit] Testcase: HelloWorldFile took 0.016 sec
[junit] Testcase: TestCreateProject took 2.109 sec
[junit] Testcase: TestFavorites took 0.172 sec
[junit] Testcase: TestGetAsset took 0.219 sec
Copyright © 2011 Flatirons Solutions Corporation
18
Unit Tests (Failed Test)
[junit]
[junit] Testcase: HelloWorldInline took 0.313 sec
[junit] FAILED
[junit] Unexpected error received during test
'HelloWorldInline': junit.framework.AssertionFailedError:
org.custommo
nkey.xmlunit.Diff
[junit] [different] Expected text value 'world' but was '
[junit]
failed_test
[junit]
' - comparing <hello ...>world</hello> at
/hello[1]/text()[1] to <hello ...>
[junit]
failed_test
[junit]
</hello> at /hello[1]/text()[1]
Copyright © 2011 Flatirons Solutions Corporation
19
Automated Build Testing (Hudson)
 Continuous build testing with Hudson (http://hudson-ci.org/)
 Hudson is a self contained web application (just install and run).
Includes Jetty web server as part of deployment.
 Run builds on a schedule (hourly) as each developer changes code.
 Build will run full configuration/ code deploy/ unit tests at each
interval
 If any failure then e-mail user who checked in code (along with
buildmeister)
 Quick demo of Hudson now….
Copyright © 2011 Flatirons Solutions Corporation
20
Lessons Learned
 Tightly type the input and output of all modules
 This will detect when function is called with empty or wrong
parameters quickly.
 Disable Function Mapping (will attempt to map calls to other
functions)
 declare option xdmp:mapping "false";
 Unit Test as much as possible
Copyright © 2011 Flatirons Solutions Corporation
21