{"id":100,"date":"2017-05-20T14:32:56","date_gmt":"2017-05-20T12:32:56","guid":{"rendered":"http:\/\/sahits.ch\/blog\/openpatrician\/?p=100"},"modified":"2017-05-20T14:32:56","modified_gmt":"2017-05-20T12:32:56","slug":"embedded-databases-for-statistic-storage","status":"publish","type":"post","link":"http:\/\/sahits.ch\/blog\/openpatrician\/2017\/05\/20\/embedded-databases-for-statistic-storage\/","title":{"rendered":"Embedded databases for statistic storage"},"content":{"rendered":"<p>Currently OpenPatrician only allows a view of the current state of affairs for example for fleet size, reputation or available wares. Even the personal balance records only the current and the past week. For statistical purposes these data should be recorded for the whole game time. This calls for a database. This post details the findings of the evaluation of these:<\/p>\n<ul>\n<li>MongoDB<\/li>\n<li>Neo4j<\/li>\n<li>H2<\/li>\n<\/ul>\n<p><!--more--><\/p>\n<p>The whole evaluation was done based upon Spring Boot, which makes it easy to use certain databases.<\/p>\n<h1>Criteria<\/h1>\n<ul>\n<li>As the database is part of the application, it has to be integrated or in other words embedded<\/li>\n<li>Ease of use to insert and query the data mainly based on date ranges<\/li>\n<li>As the statistics have to be part of the savegame the content of the database needs to be savable and loadable<\/li>\n<li>Performance for adding records, selecting records, saving and loading<\/li>\n<\/ul>\n<h1>Common parts<\/h1>\n<p>Some of the code is used in all the examples:<\/p>\n<p>The data model for the Fleet statistic:<\/p>\n<pre class=\"brush:java\">package ch.sahits.game.test.database;\r\n\r\nimport lombok.Getter;\r\nimport org.springframework.data.annotation.Id;\r\n\r\nimport java.time.LocalDate;\r\nimport java.time.format.DateTimeFormatter;\r\n\r\n@EqualsAndHashCode\r\npublic class ShipRecord {\r\n    @Id\r\n    private String id;\r\n    @Getter\r\n    private String playerUuid;\r\n    @Getter\r\n    private LocalDate date;\r\n    @Getter\r\n    private int fleetSize;\r\n\r\n    public ShipRecord(String playerUuid, LocalDate date, int fleetSize) {\r\n        this.playerUuid = playerUuid;\r\n        this.date = date;\r\n        this.fleetSize = fleetSize;\r\n    }\r\n\r\n    @Override\r\n    public String toString() {\r\n        return \"ShipRecord{\" +\r\n                \"id='\" + id + '\\'' +\r\n                \", playerUuid='\" + playerUuid + '\\'' +\r\n                \", date=\" + date.format(DateTimeFormatter.BASIC_ISO_DATE) +\r\n                \", fleetSize=\" + fleetSize +\r\n                '}';\r\n    }\r\n}<\/pre>\n<p class=\"brush:java\">As all evaluations are done in the same maven module here is the pom.xml the specialties in it will be pointed out in the respective sections:<\/p>\n<h1>MongoDB<\/h1>\n<h2>Setup<\/h2>\n<p>The setup is easy if you know how to tweak it.\u00a0 To enable MongoDB with Spring Boot the starter module is required:<\/p>\n<pre class=\"brush:xml\">&lt;dependency&gt;\r\n    &lt;groupId&gt;org.springframework.boot&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;spring-boot-starter-data-mongodb&lt;\/artifactId&gt;\r\n    &lt;version&gt;${spring.boot.version}&lt;\/version&gt;\r\n&lt;\/dependency&gt;<\/pre>\n<p>To enable the embedded MongoDB this dependency is needed as well:<\/p>\n<pre class=\"brush:xml\">&lt;dependency&gt;\r\n    &lt;groupId&gt;de.flapdoodle.embed&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;de.flapdoodle.embed.mongo&lt;\/artifactId&gt;\r\n    &lt;version&gt;2.0.0&lt;\/version&gt;\r\n&lt;\/dependency&gt;<\/pre>\n<p>On the first start up this will automatically download and install the MongoDB so that it can be used in embedded mode. So there is no separate installation necessary.<\/p>\n<h2>Querying<\/h2>\n<p>With MongoDB the quering can be done with a Repository. This is an typed interface (based along the same premise as JpaRepository) which generates the queries automatically based on the method name. The neat thing here is that with IntelliJ (2017.1.3 Ultimate) you get suggestions how to construct the method name based on the domain model:<\/p>\n<pre class=\"brush:java\">package ch.sahits.game.test.database;\r\n\r\nimport org.springframework.data.mongodb.repository.MongoRepository;\r\n\r\nimport java.time.LocalDate;\r\nimport java.util.List;\r\n\r\npublic interface FleetRepository extends MongoRepository&lt;ShipRecord, String&gt; {\r\n    List&lt;ShipRecord&gt; findAllByPlayerUuid(String playerUuid);\r\n\r\n    \/**\r\n     * Find all recods of a player between two dates (exclusive)\r\n     *\/\r\n    List&lt;ShipRecord&gt; findAllByPlayerUuidAndDateBetweenOrderByDate(String playerUuid, LocalDate startDate, LocalDate endDate);\r\n}\r\n<\/pre>\n<p>For more complex queries there is also the possibility with MongoTemplate which was not further investigated.<\/p>\n<h2>Serializability<\/h2>\n<p>By default the embedded database is an in memory database. To configure the storage location these two properties have to be set:<\/p>\n<pre>spring.mongodb.embedded.storage.database-dir=${user.home}\/.OpenPatrician\/mongodb\r\nspring.mongodb.embedded.storage.oplog-size=0\r\n<\/pre>\n<p>The database then can be dumped and restored with these two additional beans:<\/p>\n<pre class=\"brush:java\">    private void dumpMongoDB() {\r\n        \/\/ ensure directories are ok\r\n        File tempDirectory = new File(mongodbDumpLocation);\r\n        tempDirectory.delete();\r\n        tempDirectory.mkdir();\r\n        try {\r\n            mongoDBDumper.start();\r\n        } catch (IOException e) {\r\n            e.printStackTrace();\r\n        }\r\n    }\r\n\r\n    private void restoreMongoDB() {\r\n        try {\r\n            mongoDBRestorer.start();\r\n        } catch (IOException e) {\r\n            e.printStackTrace();\r\n        }\r\n    }\r\n\r\n    @Bean\r\n    public MongoDumpExecutable mongoDBDumper(IMongodConfig mongodConfig) throws IOException {\r\n        int port = mongodConfig.net().getPort();\r\n        IMongoDumpConfig mongoDumpConfig = new MongoDumpConfigBuilder()\r\n                .version(Version.Main.PRODUCTION)\r\n                .net(new Net(port, Network.localhostIsIPv6()))\r\n                .out(mongodbDumpLocation)\r\n                .build();\r\n\r\n        return MongoDumpStarter.getDefaultInstance().prepare(mongoDumpConfig);\r\n    }\r\n\r\n    @Bean\r\n    public MongoRestoreExecutable mongoDbrestorer(IMongodConfig mongodConfig) throws IOException {\r\n        int port = mongodConfig.net().getPort();\r\n        IMongoRestoreConfig mongoRestoreConfig = new MongoRestoreConfigBuilder()\r\n                .version(Version.Main.PRODUCTION)\r\n                .net(new Net(port, Network.localhostIsIPv6()))\r\n                .dropCollection(true)\r\n                .dir(mongodbDumpLocation)\r\n                .build();\r\n\r\n        return MongoRestoreStarter.getDefaultInstance().prepare(mongoRestoreConfig);\r\n    }<\/pre>\n<h2>Perofmance<\/h2>\n<table>\n<tbody>\n<tr>\n<th>Operation<\/th>\n<th>Total duration<\/th>\n<th>Average duration<\/th>\n<\/tr>\n<tr>\n<td>Insert 100&#8217;000 records<\/td>\n<td>\u00a019.3s<\/td>\n<td>\u00a00.19ms<\/td>\n<\/tr>\n<tr>\n<td>Lookup records within a 2 year range with a total of 100&#8217;000 records<\/td>\n<td>\u00a01352.15s<\/td>\n<td>\u00a0135.21ms<\/td>\n<\/tr>\n<tr>\n<td>Dump 10&#8217;000 records 1&#8217;000 times<\/td>\n<td>\u00a01063.87s<\/td>\n<td>1063.87ms<\/td>\n<\/tr>\n<tr>\n<td>Restore the dumped records<\/td>\n<td>\u00a03067.20s<\/td>\n<td>3067.20ms<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The long times for queries might be due to the fact that every query is logged. When the application crashes or is terminated the MongoDB has to be removed manually. This is probably something that could be worked around with using finally blocks to close the process cleanly.<\/p>\n<h2>Further readings<\/h2>\n<ul>\n<li>Spring Boot with MongoDB: <a href=\"https:\/\/spring.io\/guides\/gs\/accessing-data-mongodb\/\">https:\/\/spring.io\/guides\/gs\/accessing-data-mongodb\/<\/a><\/li>\n<li>Embedded MongoDB: <a href=\"https:\/\/github.com\/flapdoodle-oss\/de.flapdoodle.embed.mongo\">https:\/\/github.com\/flapdoodle-oss\/de.flapdoodle.embed.mongo<\/a><\/li>\n<li>Embedded MongoDB Restore and Dump Test: <a href=\"https:\/\/github.com\/flapdoodle-oss\/de.flapdoodle.embed.mongo\/blob\/master\/src\/test\/java\/de\/flapdoodle\/embed\/mongo\/MongoDumpExecutableTest.java\">https:\/\/github.com\/flapdoodle-oss\/de.flapdoodle.embed.mongo\/blob\/master\/src\/test\/java\/de\/flapdoodle\/embed\/mongo\/MongoDumpExecutableTest.java<\/a><\/li>\n<\/ul>\n<h1>Neo4j<\/h1>\n<h2>Setup<\/h2>\n<p>From the Spring side this dependency is needed:<\/p>\n<pre class=\"brush:xml\">&lt;dependency&gt;\r\n    &lt;groupId&gt;org.springframework.boot&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;spring-boot-starter-data-neo4j&lt;\/artifactId&gt;\r\n    &lt;version&gt;${spring.boot.version}&lt;\/version&gt;\r\n&lt;\/dependency&gt;<\/pre>\n<p>To enable Neo4j two additional dependencies are needed:<\/p>\n<pre class=\"brush:xml\">&lt;dependency&gt;\r\n    &lt;groupId&gt;org.neo4j&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;neo4j-kernel&lt;\/artifactId&gt;\r\n    &lt;version&gt;3.2.0&lt;\/version&gt;\r\n&lt;\/dependency&gt;\r\n&lt;dependency&gt;\r\n    &lt;groupId&gt;org.neo4j&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;neo4j-lucene-index&lt;\/artifactId&gt;\r\n    &lt;version&gt;3.2.0&lt;\/version&gt;\r\n&lt;\/dependency&gt;\r\n&lt;dependency&gt;\r\n    &lt;groupId&gt;org.neo4j&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;neo4j-ogm-embedded-driver&lt;\/artifactId&gt;\r\n    &lt;version&gt;2.1.2&lt;\/version&gt;\r\n&lt;\/dependency&gt;<\/pre>\n<p>In addition to that we need the annotation @EnableNeo4jRepositories on the application configuration. The model class also needs to be annotated with @NodeEntity and the entities id is annotated with @GraphId instead of @Id and the id is of type Long. Neo4j also requires a default constructor.<\/p>\n<p>When using the file based approach on Windows there are some issues with the URL, which forced me to provide my own Configuration bean:<\/p>\n<pre class=\"brush:java\">@Bean\r\npublic Configuration neo4jConfiguration(Neo4jProperties properties) {\r\n    Configuration configuration = new Configuration();\r\n    DriverConfiguration driverConfiguration = configuration.driverConfiguration();\r\n    File f = new File(System.getProperty(\"user.home\")+\"\/.OpenPatrician\/neo4j.db\");\r\n    driverConfiguration.setDriverClassName(\"org.neo4j.ogm.drivers.embedded.driver.EmbeddedDriver\");\r\n    driverConfiguration.setURI(\"file:\/\/\/Users\/ahotz\/.OpenPatrician\/neo4j.db\");\r\n    if(properties.getUsername() != null &amp;&amp; properties.getPassword() != null) {\r\n        driverConfiguration.setCredentials(properties.getUsername(), properties.getPassword());\r\n    }\r\n    if(properties.getCompiler() != null) {\r\n        configuration.compilerConfiguration().setCompilerClassName(properties.getCompiler());\r\n    }\r\n\r\n    return configuration;\r\n}<\/pre>\n<h2>Querying<\/h2>\n<p>The queries can be done through a Repository in the same way as the MongoDB, only that the interface extends GraphRepository:<\/p>\n<pre class=\"brush:java\">public interface FleetNeo4jRepository extends GraphRepository&lt;ShipRecord&gt;<\/pre>\n<p><strong>At this point Neo4j was no further investigated due to various unresolved dependencies with non available interface implementations and no documentation at all.<\/strong><\/p>\n<h2>Further readings<\/h2>\n<ul>\n<li>Spring Boot with Neo4j: <a href=\"https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-nosql.html\">https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-nosql.html<\/a><\/li>\n<li>Neo4j: <a href=\"http:\/\/neo4j.com\/docs\/ogm-manual\/current\/reference\/#reference:getting-started\">http:\/\/neo4j.com\/docs\/ogm-manual\/current\/reference\/#reference:getting-started<\/a><\/li>\n<\/ul>\n<h1>H2<\/h1>\n<h2>Setup<\/h2>\n<p>There are some dependencies that are needed:<\/p>\n<pre class=\"brush:xml\">&lt;dependency&gt;\r\n    &lt;groupId&gt;org.springframework.boot&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;spring-boot-starter-data-jpa&lt;\/artifactId&gt;\r\n    &lt;version&gt;${spring.boot.version}&lt;\/version&gt;\r\n&lt;\/dependency&gt;\r\n&lt;dependency&gt;\r\n    &lt;groupId&gt;com.h2database&lt;\/groupId&gt;\r\n    &lt;artifactId&gt;h2&lt;\/artifactId&gt;\r\n&lt;\/dependency&gt;<\/pre>\n<p>The entity is annotated with the javax.persistance annotations for @Entity and the for the ID with @Id.<\/p>\n<h2>Querying<\/h2>\n<p>The repository works along the same lines as for MangoDB. The interface extends the CrudRepository.<\/p>\n<h2>Serializability<\/h2>\n<p>H2 provides tools to create a backup and restore it. However their premise is that when restoring the database is empty. Therefore creating a backup and immediately restore it again will fail. For that reason the script that is created in the backup needs to be tweaked to drop all objects to start with:<\/p>\n<pre class=\"brush:java\">\/\/ Create a dump and add a statement to drop everything to make the restore work.\r\ntry {\r\n    String backupFile = \"h2.backup.zip\";\r\n    String tempOutputFilenName = \"out.zip\";\r\n    Script.main(\"-url\", \"jdbc:h2:file:~\/.OpenPatrician\/h2.db;DB_CLOSE_ON_EXIT=FALSE\",  \"-user\", \"sa\", \"-script\", tempOutputFilenName, \"-options\", \"compression\", \"zip\");\r\n    File f = new File(tempOutputFilenName);\r\n    ZipFile zipFile = new ZipFile(tempOutputFilenName);\r\n    final ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(backupFile));\r\n    for(Enumeration e = zipFile.entries(); e.hasMoreElements(); ) {\r\n        ZipEntry entryIn = (ZipEntry) e.nextElement();\r\n        zos.putNextEntry(new ZipEntry(entryIn.getName()));\r\n\r\n        InputStream is = zipFile.getInputStream(entryIn);\r\n        byte[] firstBytes = \"DROP ALL OBJECTS\".getBytes();\r\n        zos.write(firstBytes);\r\n        byte[] buf = new byte[1024];\r\n        int len;\r\n        while ((len = (is.read(buf))) &gt; 0) {\r\n            zos.write(buf, 0, (len &lt; buf.length) ? len : buf.length);\r\n        }\r\n        zos.closeEntry();\r\n    }\r\n    zos.close();\r\n    f.delete();\r\n} catch (SQLException |IOException e) {\r\n    e.printStackTrace();\r\n}\r\n\/\/ Restore\r\ntry {\r\n    RunScript.main(\"-url\", \"jdbc:h2:file:~\/.OpenPatrician\/h2.db;DB_CLOSE_ON_EXIT=FALSE\",  \"-user\", \"sa\", \"-script\", \"h2.backup.zip\", \"-options\", \"compression\", \"zip\");\r\n} catch (SQLException e) {\r\n    e.printStackTrace();\r\n}<\/pre>\n<h2>Performance<\/h2>\n<table>\n<tbody>\n<tr>\n<th>Operation<\/th>\n<th>Total duration<\/th>\n<th>Average duration<\/th>\n<\/tr>\n<tr>\n<td>Insert 100&#8217;000 records<\/td>\n<td>\u00a027.12s<\/td>\n<td>\u00a00.27ms<\/td>\n<\/tr>\n<tr>\n<td>Lookup records within a 2 year range with a total of 100&#8217;000 records<\/td>\n<td>\u00a05443.15s<\/td>\n<td>\u00a0544.31ms<\/td>\n<\/tr>\n<tr>\n<td>Dump 10&#8217;000 records 1&#8217;000 times<\/td>\n<td>\u00a0757.06s<\/td>\n<td>\u00a0757.06ms<\/td>\n<\/tr>\n<tr>\n<td>Restore the dumped records<\/td>\n<td>\u00a0295.44s<\/td>\n<td>\u00a0295.44ms<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Further readings<\/h2>\n<ul>\n<li>Spring boot SQL databases: <a href=\"https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-sql.html\">https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-sql.html<\/a><\/li>\n<li>H2 documentation backup and restore: <a href=\"http:\/\/www.h2database.com\/html\/tutorial.html#upgrade_backup_restore\">http:\/\/www.h2database.com\/html\/tutorial.html#upgrade_backup_restore<\/a><\/li>\n<li>StackOverflow Backup&amp;Restore: <a href=\"http:\/\/stackoverflow.com\/questions\/44078484\/stopping-and-restarting-a-file-based-h2-database-from-a-spring-application\">http:\/\/stackoverflow.com\/questions\/44078484\/stopping-and-restarting-a-file-based-h2-database-from-a-spring-application<\/a><\/li>\n<\/ul>\n<h1>Comparison<\/h1>\n<p>The setup for H2 is easier than for for MongoDB, however it is manageable for both. The solutions for serializing the database need some work for both MongoDB and H2. Both provide tools to do the job, however for H2 the generated output cannot be used as is and needs to be tweaked.<\/p>\n<table>\n<tbody>\n<tr>\n<th>What<\/th>\n<th>MongoDB<\/th>\n<th>Neo4J<\/th>\n<th>H2<\/th>\n<\/tr>\n<tr>\n<td>Setup<\/td>\n<td>\u00a0+<\/td>\n<td>-\u221e<\/td>\n<td>\u00a0++<\/td>\n<\/tr>\n<tr>\n<td>Serializablity<\/td>\n<td>\u00a0+<\/td>\n<td><\/td>\n<td>\u00a0&#8211;<\/td>\n<\/tr>\n<tr>\n<td>Insert 100&#8217;000 records AVG<\/td>\n<td>\u00a00.19ms<\/td>\n<td><\/td>\n<td>\u00a00.27ms<\/td>\n<\/tr>\n<tr>\n<td>Lookup records within a 2 year range with a total of 100&#8217;000 records AVG<\/td>\n<td>\u00a0135.21ms<\/td>\n<td><\/td>\n<td>\u00a0544.31ms<\/td>\n<\/tr>\n<tr>\n<td>Dump 100&#8217;000 records 100&#8217;000 times AVG<\/td>\n<td>1063.87ms<\/td>\n<td><\/td>\n<td>\u00a0757.06ms<\/td>\n<\/tr>\n<tr>\n<td>Restore the dumped records AVG<\/td>\n<td>3067.20ms<\/td>\n<td><\/td>\n<td>\u00a0295.44ms<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h1>Further reading<\/h1>\n<ul>\n<li>Starting point for various No-SQL setups with Spring Boot: <a href=\"https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-nosql.html\">https:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/boot-features-nosql.html<\/a><\/li>\n<li>Default Spring Boot application properties: <a href=\"http:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/common-application-properties.html\">http:\/\/docs.spring.io\/spring-boot\/docs\/current\/reference\/html\/common-application-properties.html<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Currently OpenPatrician only allows a view of the current state of affairs for example for fleet size, reputation or available wares. Even the personal balance records only the current and the past week. For statistical purposes these data should be recorded for the whole game time. This calls for a database. This post details the [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[18,19],"class_list":["post-100","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-database","tag-evaluation"],"_links":{"self":[{"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/posts\/100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/comments?post=100"}],"version-history":[{"count":15,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/posts\/100\/revisions"}],"predecessor-version":[{"id":116,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/posts\/100\/revisions\/116"}],"wp:attachment":[{"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/media?parent=100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/categories?post=100"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/sahits.ch\/blog\/openpatrician\/wp-json\/wp\/v2\/tags?post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}