feed.xml

<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> <channel><title>OnDataEngineering</title><description>A collaborative site for independent, critical and technical thinking on the use cases, architectural patterns and technologies relating to the transformation and preparation of data for exploitation.</description><link>https://ondataengineering.github.io/</link><atom:link href="https://ondataengineering.github.io/feed.xml" rel="self" type="application/rss+xml" /> <item><title>The Mid Week News 18/09/2019</title><link>https://ondataengineering.github.io/blog/2019/09/18/the-mid-week-news/</link><pubDate>Wed, 18 Sep 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Right - news time again.&lt;/p&gt; &lt;p&gt;Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-calcite/&quot;&gt;Apache Calcite&lt;/a&gt; 1.21 is out - you might never have heard of it, but it’s probably being used by many of the data tools you use on a daily basis for query parsing and optimization - &lt;a href=&quot;https://calcite.apache.org/news/2019/09/11/release-1.21.0/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cloudera have announced the release of Cloudera Streams Management - bundling their Kafka management console and replication tool - &lt;a href=&quot;https://blog.cloudera.com/announcing-the-general-availability-of-cloudera-streams-management/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Interested in Apache Samza, hear about LinkedIn’s journey with it - &lt;a href=&quot;https://www.infoq.com/news/2019/09/linkedin-apache-samza/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the ever reliable The Morning Papers - Procella, YouTube’s unified OLTP/OLAP (HTAP) database - &lt;a href=&quot;https://blog.acolyer.org/2019/09/11/procella/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDBCloud&lt;/a&gt; (their time series database as a service on AWS, Azure and GCP) has hit 2.0 and has now gone serverless - &lt;a href=&quot;https://www.influxdata.com/blog/influxdb-cloud-2-0-launches-as-a-serverless-platform-for-time-series-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/google-cloud-dataproc/&quot;&gt;Google Cloud DataProc&lt;/a&gt; now supports (in alpha) running your Spark jobs on Kubernetes (GKE) rather than YARN VMs - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/alpha-access-to-cloud-dataproc-jobs-on-gke&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More on on &lt;a href=&quot;/technologies/google-cloud-dataproc/&quot;&gt;Google Cloud DataProc&lt;/a&gt; Spark on GKE from ZDNet - &lt;a href=&quot;https://www.zdnet.com/article/google-announces-alpha-of-cloud-dataproc-for-kubernetes/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; is now available on &lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google Cloud Platform&lt;/a&gt; if you’re looking for a cloud agnostic Hadoop as a service offering (that runs on Google Cloud) - &lt;a href=&quot;https://www.qubole.com/blog/announcing-general-availability-of-qubole-on-google-cloud/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Interested in replicating data between &lt;a href=&quot;/technologies/apache-kafka&quot;&gt;Kafka&lt;/a&gt; clusters - Cloudera have a post on on MirrorMaker 2 which is based on &lt;a href=&quot;/technologies/apache-kafka/kafka-connect/&quot;&gt;Kafka Connect&lt;/a&gt; - &lt;a href=&quot;https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/09/18/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 11/09/2019</title><link>https://ondataengineering.github.io/blog/2019/09/11/the-mid-week-news/</link><pubDate>Wed, 11 Sep 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Apologies - we’ve been off on holiday again, hence the radio silence. But we’re back, and with a big old news bump.&lt;/p&gt; &lt;p&gt;Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; release 5.26 is out, with even better Spark performance&lt;/li&gt; &lt;li&gt;And Amazon have also announced &lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; 6.0, with support for Hadoop 3.1 and running Spark jobs in Docker containers&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-orc/&quot;&gt;Apache ORC&lt;/a&gt; 1.6 is out if you’re looking for columnar data storage on HDFS&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; 6.0 is finally out if you’re looking for mature shared nothing MPP database&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-carbondata/&quot;&gt;Apache CarbonData&lt;/a&gt; 1.6 is out if you’re looking for indexed storage of data on HDFS with supports for batch inserts and updates&lt;/li&gt; &lt;li&gt;Version 0.5 of the &lt;a href=&quot;/technologies/apache-nifi/registry/&quot;&gt;NiFi Registry&lt;/a&gt; is out if you’re looking to configuration manage your NiFi flows&lt;/li&gt; &lt;li&gt;Version 0.4 of &lt;a href=&quot;/technologies/apache-myriad/&quot;&gt;Apache Myriad&lt;/a&gt; is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/zenko/cloudserver/&quot;&gt;Zenko CloudServer&lt;/a&gt; has just released version 8.2&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Are you running an &lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; version prior to 5.0 - if so there’s an XML bomb attack - &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-12401&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;ApacheIoTDB - the Apache open source time series database focusing on IoT use cases has it’s first official release @ 0.8 - &lt;a href=&quot;https://iotdb.apache.org/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the ever excellent The Morning Paper, a review of a paper that used “the TPC-H benchmark to assess Redshift, Redshift Spectrum, Athena, Presto, Hive, and Vertica to find out what works best and the trade-offs involved” - &lt;a href=&quot;https://blog.acolyer.org/2019/08/30/choosing-a-cloud-dbms/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; is now available on &lt;a href=&quot;/tech-vendors/microsoft-azure&quot;&gt;Azure&lt;/a&gt; - &lt;a href=&quot;https://www.elastic.co/blog/elasticsearch-service-on-elastic-cloud-now-available-on-microsoft-azure&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/confluent-open-source&quot;&gt;Confluent Schema Registry&lt;/a&gt; is now available as a cloud service in Confluent Cloud - &lt;a href=&quot;https://www.confluent.io/blog/confluent-cloud-schema-registry-generally-available&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Looking for an open source &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;object store&lt;/a&gt; - Datanami have the latest on MinIO - &lt;a href=&quot;https://www.datanami.com/2019/08/26/minio-enjoying-role-in-emerging-cloud-architecture/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt;’s Q2 results are better than expected - &lt;a href=&quot;https://www.datanami.com/2019/09/06/cloudera-rebounds-in-q2-beats-expectations/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;StreamSets have announced StreamsetsTranformer - a graphical tool for creating &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt; pipelines that’s part of their DataOps Platform - &lt;a href=&quot;https://streamsets.com/blog/streamsets-transformer-is-here/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Using &lt;a href=&quot;/technologies/google-cloud-storage/&quot;&gt;Google Cloud Storage&lt;/a&gt; with &lt;a href=&quot;/technologies/apache-hadoop&quot;&gt;Hadoop&lt;/a&gt; - Google have a new version of their Cloud Storage Connector for Hadoop out with a bunch of performance improvements and locking for directory modifications - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/new-release-of-cloud-storage-connector-for-hadoop-improving-performance-throughput-and-more&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;ApacheDolphinScheduler has just been accepted into the Apache Incubator - originally called Easy Scheduler, donated by Analysys, it’s a tool for distributed ETL scheduling - &lt;a href=&quot;https://cwiki.apache.org/confluence/display/INCUBATOR/DolphinSchedulerProposal&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/09/11/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 14/08/2019</title><link>https://ondataengineering.github.io/blog/2019/08/14/the-mid-week-news/</link><pubDate>Wed, 14 Aug 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s time for the mid week news again. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Looking for a Kafka alternative, &lt;a href=&quot;/technologies/pravega/&quot;&gt;Pravega&lt;/a&gt; has just had a 0.5 release&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-ranger/&quot;&gt;Apache Ranger&lt;/a&gt; 2.0 is out - be interesting to see what the convergence roadmap with ApacheSentry is&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hue/&quot;&gt;Hue&lt;/a&gt; 4.5 is out, with a bunch of improved SQL integrations and Kubernetes/Docker support&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Datanami, (shock/horror) why Data Catalogs are important, and what some of the options are - &lt;a href=&quot;https://www.datanami.com/2019/08/07/data-catalogs-seen-as-difference-makers-in-big-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;If you’re learning &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt;, Confluent now have an entire site of tutorials for you - &lt;a href=&quot;https://www.confluent.io/blog/announcing-apache-kafka-tutorials&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From &lt;a href=&quot;/tech-vendors/amazon-web-services/&quot;&gt;Amazon Web Services&lt;/a&gt;, deploying data lakes using AWSLakeFormation - &lt;a href=&quot;https://aws.amazon.com/blogs/big-data/building-securing-and-managing-data-lakes-with-aws-lake-formation/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via InfoQ, details on how Badoo handle 20 billion events per day, staring LiveStreamingDaemon (an open source replacement for Facebook’s Scribe), ORC, HDFS, Exasol, Spark and CubeDB (an in memory multi-key counter store) - &lt;a href=&quot;https://www.infoq.com/news/2019/08/badoo-20-billion-events-per-day&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Couple from Datanami - firstly their latest thoughts on the future of &lt;a href=&quot;/technologies/apache-hadoop&quot;&gt;Hadoop&lt;/a&gt; - &lt;a href=&quot;https://www.datanami.com/2019/08/12/re-imagining-big-data-in-a-post-hadoop-world/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And also from Datanami, an update on Lucidworks who provide &lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; based products, and who’ve just secured a new funding round - &lt;a href=&quot;https://www.datanami.com/2019/08/12/investors-search-for-lucidworks/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More information on AWSLakeFormation, this time from InfoQ - &lt;a href=&quot;https://www.infoq.com/news/2019/08/aws-lake-formation-ga/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/08/14/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 07/08/2019</title><link>https://ondataengineering.github.io/blog/2019/08/07/the-midweek-news/</link><pubDate>Wed, 07 Aug 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Another big dump of new releases and product news this week. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; and &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch-Hadoop&lt;/a&gt; 7.3 are now out&lt;/li&gt; &lt;li&gt;Alpha 2 of &lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; on Kubernetes is also out - &lt;a href=&quot;https://www.elastic.co/blog/announcing-elastic-cloud-on-kubernetes-eck-0-9-0-alpha-2&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-cdh&quot;&gt;Cloudera Enterprise&lt;/a&gt; 6.3 is out, with upgrades to Kafka, HBase, Impala and Kudu&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-altus/director/&quot;&gt;Cloudera Altus Director&lt;/a&gt; is also up to 6.3&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/confluent-enterprise/&quot;&gt;Confluent Enterprise&lt;/a&gt; and &lt;a href=&quot;/technologies/confluent-open-source/&quot;&gt;Confluent Open Source&lt;/a&gt; are up to 5.3, with support for Kubernetes and Role Based Access Control&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; 2.14 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-accumulo/&quot;&gt;Apache Accumulo&lt;/a&gt; 2.0 is out, with a new fluent API, Bulk Import API, custom table stats, prioritisable scan executors, and of course, official Docker support&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; 5.25 is out, with a couple of Spark performance improvements&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;StreamSets Data Collector&lt;/a&gt; 3.10 is out, with support for ingesting data directly from NiFi&lt;/li&gt; &lt;li&gt;Delta Lake, the open sourced &lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt;, is up to 0.3 - &lt;a href=&quot;https://databricks.com/blog/2019/08/02/announcing-delta-lake-0-3-0-release.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From ZDNet, DGraph - an open source graph database written in Go - has just received a funding round - &lt;a href=&quot;https://www.zdnet.com/article/you-can-go-your-own-graph-database-way-dgraph-secures-115m-to-pursue-its-opinionated-path/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;If you’re interested in Brooklin, the open source tool from LinkedIn for moving streaming data around, InfoQ have a presentation for you - &lt;a href=&quot;https://www.infoq.com/presentations/linkedin-streams-brooklin/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapRs&lt;/a&gt; long running will they/won’t they saga has come to an end with their purchase by HPE - Datanami ask why - &lt;a href=&quot;https://www.datanami.com/2019/08/05/what-hpe-sees-in-mapr-technologies/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera, best practice for deploying &lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Apache Hadoop&lt;/a&gt;, including your Linux configuration - &lt;a href=&quot;https://blog.cloudera.com/how-to-deploy-apache-hadoop-clusters-like-a-boss/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;An intro to Prefect, an evolution of &lt;a href=&quot;/technologies/apache-airflow/&quot;&gt;Apache Airflow&lt;/a&gt; to support modern data applications - open source with commercial backing - &lt;a href=&quot;https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Looks like &lt;a href=&quot;/tech-vendors/microsoft-azure/&quot;&gt;Microsoft&lt;/a&gt; has purchased BlueTalon, which could mean better fine grained access control over your data ecosystem in Azure - &lt;a href=&quot;https://blogs.microsoft.com/blog/2019/07/29/microsoft-acquires-bluetalon-simplifying-data-privacy-and-governance-across-modern-data-estates/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon have announced PartiQL, an evolution of SQL and SQL++ for querying relational, NoSQL and other structured data - &lt;a href=&quot;https://aws.amazon.com/blogs/opensource/announcing-partiql-one-query-language-for-all-your-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/08/07/the-midweek-news/</guid> </item> <item><title>The Mid Week News 31/07/2019</title><link>https://ondataengineering.github.io/blog/2019/07/31/the-mid-week-news/</link><pubDate>Wed, 31 Jul 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Apologies - forgot to say I was going on holiday! Apologies for the lack of updates, and the bigger than usual update this week. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; is up to release 5.12&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-knox/&quot;&gt;Apache Knox&lt;/a&gt; 1.3 is out, with support for &lt;a href=&quot;/technologies/hue&quot;&gt;Hue&lt;/a&gt;, &lt;a href=&quot;/technologies/cloudera-manager/&quot;&gt;Cloudera Manager&lt;/a&gt; and &lt;a href=&quot;/technologies/apache-livy/&quot;&gt;Apache Livy&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; 8.2 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-data-science-workbench/&quot;&gt;Cloudera Data Science Workbench&lt;/a&gt; 1.6 is out, with support for multiple editors including Jupyter, RStudio and PyCharm&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; Enterprise 2.3 is out, with support for role based access control&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Cloudera, details on YuniKorn, their new universal resource scheduler that can underpin both Kubernetes and Hadoop YARN resource scheduling - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/07/yunikorn-a-universal-resource-scheduler/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, thoughts on &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;Object Stores&lt;/a&gt; and &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems&quot;&gt;ScaleOutFilesystems&lt;/a&gt; - &lt;a href=&quot;https://www.datanami.com/2019/07/17/object-and-scale-out-file-systems-fill-hadoop-storage-void/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s a bunch of vulnerabilities announced for &lt;a href=&quot;/technologies/apache-storm/&quot;&gt;Apache Storm&lt;/a&gt; - CVE-2018-11779; CVE-2018-1320 and CVE-2019-0202&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/microsoft-azure-data-lake-store&quot;&gt;Azure Data Lake Store&lt;/a&gt; now supports Azure Blob Storage API as well as an HDFS compatible API - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/silo-busting-2-0-multi-protocol-access-for-azure-data-lake-storage/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, Ascend has launched with an Autonomous Dataflow Service - looks interesting, if anyone has any thoughts please share - &lt;a href=&quot;https://www.datanami.com/2019/07/19/ascend-launches-from-stealth-with-19m/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Redis now has a module that turns it into a &lt;a href=&quot;/tech-categories/time-series-databases/&quot;&gt;Time Series Databases&lt;/a&gt; - &lt;a href=&quot;https://redislabs.com/blog/redistimeseries-ga-making-4th-dimension-truly-immersive/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-cdh&quot;&gt;CDH&lt;/a&gt; now supports and will include &lt;a href=&quot;/technologies/apache-phoenix/&quot;&gt;Apache Phoenix&lt;/a&gt; for SQL over HBase - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/07/apache-phoenix-for-cdh/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Google Big Query now has a new user interface, GIS functions, persistent UDFs and more - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/new-persistent-user-defined-functions-increased-concurrency-limits-gis-and-encryption-functions-and-more&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the ODBS Industry Watch blog, an interview on LeanXcale, a hybrid Transactional Analytical (HTAP) database - I’m still not sure what to make of these, thoughts welcomed - &lt;a href=&quot;http://www.odbms.org/blog/2019/07/on-leanxcale-database-interview-with-patrick-valduriez-and-ricardo-jimenez-peris/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/07/31/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 17/07/2019</title><link>https://ondataengineering.github.io/blog/2019/07/17/the-mid-week-news/</link><pubDate>Wed, 17 Jul 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;News news news again. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kudu/&quot;&gt;Apache Kudu&lt;/a&gt; 1.10 is out, with table backup/restore, metadata sync with Hive Metastore, and native fine-grained authentication via ApacheSentry&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-dataplane/data-lifecycle-manager/&quot;&gt;Data Lifecycle Manager&lt;/a&gt; (part of the DataPlane Platform) is up to 1.5 if you’re looking for a tool to replicate HDFS and Hive data between clusters&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-dataplane/data-analytics-studio/&quot;&gt;Data Analytics Studio&lt;/a&gt; (part of the DataPlane Platform) is up to 1.3 is you’re looking for a tool to run and diagnose performance issues with Hive queries&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; have announced the licensing model for the new company - TLDR, they’re sticking with Apache and AGPL licences, sticking with the Apache foundation, and the ex-Cloudera commercial components will all be open sourced - &lt;a href=&quot;http://vision.cloudera.com/our-commitment-to-open-source-software/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the Starburst blog, part sales pitch, but a good case for separation of storage and compute and keeping your architecture open - &lt;a href=&quot;https://www.starburstdata.com/technical-blog/the-power-of-optionality-in-big-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;LinkedIn have open sourced Brooklin, their tool for replicating streaming data between streaming data stores and/or databases - we’ve added this to our list of &lt;a href=&quot;/tech-categories/data-ingestion/&quot;&gt;Data Ingestion&lt;/a&gt; technologies - &lt;a href=&quot;https://engineering.linkedin.com/blog/2019/brooklin-open-source&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Databricks Runtime 5.5 is out - &lt;a href=&quot;https://databricks.com/blog/2019/07/16/announcing-databricks-runtime-5-5-and-runtime-5-5-for-machine-learning.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;This looks really interesting - from Datanami, an intro to Dagster, an open source tool for creating data applications using a functional paradigm, with support for a range of languages and integrations out of the box - &lt;a href=&quot;https://www.datanami.com/2019/07/16/dagster-emerges-to-simplify-data-app-development/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/07/17/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 10/07/2019</title><link>https://ondataengineering.github.io/blog/2019/07/10/the-mid-week-news/</link><pubDate>Wed, 10 Jul 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s time for our weekly dose of the news. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-arrow/&quot;&gt;Apache Arrow&lt;/a&gt; 0.14 is out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From the ever excellent The Morning Paper - a SIGMOD’19 paper from &lt;a href=&quot;/technologies/apache-beam&quot;&gt;Beam&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-calcite&quot;&gt;Calcite&lt;/a&gt; and &lt;a href=&quot;/technologies/apache-flink&quot;&gt;Flink&lt;/a&gt; experts on unifying SQL over tables and streams - &lt;a href=&quot;https://blog.acolyer.org/2019/07/03/one-sql-to-rule-them-all/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via The Register, it looks like Netezza is finally dead - discontinued by IBM - &lt;a href=&quot;https://www.theregister.co.uk/2019/07/03/rip_netezza_ibms_fpgapowered_data_warehousing_dream/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Register this time - it looks like &lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapR&lt;/a&gt; has missed it’s deadline for sale - &lt;a href=&quot;https://www.theregister.co.uk/2019/07/04/mapr_misses_deadline_for_sale/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From InfoQ, details on LinkedIn’s Brooklin that they use for moving streaming data around - apparently being open sourced soon - &lt;a href=&quot;https://www.infoq.com/news/2019/07/brooklin-data-streaming-service&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Looking for an &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt; alternative - Apache Pulsar has hit 2.4 - &lt;a href=&quot;https://github.com/apache/pulsar/releases/tag/v2.4.0&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/07/10/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 03/07/2019</title><link>https://ondataengineering.github.io/blog/2019/07/03/the-mid-week-news/</link><pubDate>Wed, 03 Jul 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;News time again! Quieter this week. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/alluxio/&quot;&gt;Alluxio&lt;/a&gt; 2.0 is out if you’re looking for a distributed analytics storage layer / cache&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-druid&quot;&gt;Apache Druid&lt;/a&gt; 0.15 is out if you’re looking for real time OLAP / star schema queries on real time data&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/scality-ring/&quot;&gt;Scality RING&lt;/a&gt; 8 is out - object storage with support for hybrid cloud&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Datanami - entity resolution and Senzing. If you haven’t heard of either it’s well worth a read - &lt;a href=&quot;https://www.datanami.com/2019/06/27/inside-jeff-jonas-big-plan-to-democratize-entity-resolution/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDnet - an update on data catalogs, and specifically Waterline Data - &lt;a href=&quot;https://www.zdnet.com/article/multi-cloud-data-catalogs-the-easy-way-using-metadata-and-machine-learning-by-waterline-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Matt Turck has his 2019 Data &amp;amp; AI thoughts out, along with his technology landscape picture that’s always worth a look - &lt;a href=&quot;https://mattturck.com/data2019/&quot;&gt;part 1&lt;/a&gt;; &lt;a href=&quot;https://mattturck.com/2019trends/&quot;&gt;part 2&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Interested in a deep dive on the &lt;a href=&quot;/technologies/apache-kylin/&quot;&gt;Apache Kylin&lt;/a&gt; OLAP engine - their blog has you covered - &lt;a href=&quot;http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/07/03/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 26/06/2019</title><link>https://ondataengineering.github.io/blog/2019/06/26/the-mid-week-news/</link><pubDate>Wed, 26 Jun 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s time for our weekly news summary again, and it’s a but of a bumper week. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; 5.20 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt; 2.3 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-calcite/&quot;&gt;Apache Calcite&lt;/a&gt; 1.20 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; 7.2 is out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;More from Datanami on &lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapR&lt;/a&gt;, who are apparently close to finding a buyer for the company - &lt;a href=&quot;https://www.datanami.com/2019/06/18/mapr-says-its-close-to-deal-to-sell-company/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hue&quot;&gt;Hue&lt;/a&gt; now supports &lt;a href=&quot;/technologies/apache-atlas/&quot;&gt;Apache Atlas&lt;/a&gt; as a metadata catalog backend - potentially worth a look if you’re an HDP user - &lt;a href=&quot;http://gethue.com/realtime-catalog-search-with-hue-and-apache-atlas/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, Data.world (an enterprise knowledge graph vendor) has brought Capsenta, who have a relational database to knowledge graph virtualization bridge - &lt;a href=&quot;https://www.zdnet.com/article/data-world-joins-forces-with-capsenta-to-bring-knowledge-graph-based-data-management-and-consumer-grade-ui-to-the-enterprise/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;DatabricksRuntime 5.4 is out, including DataBricksConnect, auto optimisation of table layouts and support for the AWS Glue metastore - &lt;a href=&quot;https://databricks.com/blog/2019/06/20/announcing-databricks-runtime-5-4.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;AWSGlue now supports workflows - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2019/06/aws-glue-now-provides-workflows-to-orchestrate-etl-workloads/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Qubole think that using &lt;a href=&quot;/technologies/amazon-s3/&quot;&gt;Amazon S3&lt;/a&gt; Select (where filters on S3 queries are run server side) can speed up &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt; processing by 2.9x - [link](https://www.qubole.com/blog/amazon-s3-select-integration/&lt;/li&gt; &lt;li&gt;Bloor have a Market Update on hybrid real-time operational/transactional and analytic processing - [link](https://www.bloorresearch.com/research/hybrid-real-time-data-processing/&lt;/li&gt; &lt;li&gt;SwiftStack have announced a Data Analytics storage solution build on SwiftStackStorage, SwiftStack1Space for multi-cloud federation and &lt;a href=&quot;/technologies/alluxio/&quot;&gt;Alluxio&lt;/a&gt; for in memory caching and federation over HDFS - [link](https://www.swiftstack.com/blog/2019/06/24/multi-cloud-data-lake-%e2%80%93-evolving-from-descriptive-to-predictive-to-cognitive-analytics/&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; have just run a preview of CDP, which they’re calling an Enterprise Data Cloud - [link](http://vision.cloudera.com/cloudera-provides-first-look-at-cloudera-data-platform-the-industrys-first-enterprise-data-cloud/&lt;/li&gt; &lt;li&gt;Datanami have their latest views on the future of &lt;a href=&quot;/technologies/apache-hadoop&quot;&gt;Hadoop&lt;/a&gt; - [link](https://www.datanami.com/2019/06/24/hitting-the-reset-button-on-hadoop/&lt;/li&gt; &lt;li&gt;Oracle have rationalised their analytics products under a single brand - ZDNet has the scoop - [link](https://www.zdnet.com/article/oracle-analytics-honing-18-products-down-to-a-single-brand/&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/06/26/the-mid-week-news/</guid> </item> <item><title>DataKitchen DataOps Platform</title><link>https://ondataengineering.github.io/technologies/datakitchen-devops-platform/</link><pubDate>Tue, 25 Jun 2019 08:00:00 +0000</pubDate> <description> &lt;p&gt;Platform to enable aoption of DataOps practices for data engineering, science, and analytic teams. These DataOps practices combine ideas in Agile development, DevOps, statistical process control, data science model deployment, and test data automation through a series of steps in a collaborative workflow. Within DataKitchen&apos;s product, users start with &quot;Kitchens.&quot; Each Kitchen represents a place to do work: production environments, development sandboxes, etc. Kitchens are a collection of the data, data stores, tools (ETL, data science, visualization), code or configuration used by those tools, git branch, and the necessary servers and software. These collections can be created, merged, or shut down. A Kitchen can also be a current environment already available in the organization. Kitchens can be individual or shared across groups. When working in Kitchen, team members create and run &quot;Recipes.&quot; Each Recipe is a directed graph of steps. A Recipe represents the workflow pipeline used to deliver analytics: acquire data, transform data, call a machine learning model, and visualize data. A Recipe utilized the tools that DataKitchen&apos;s customers already own. As Recipes are running, tests are embedded in the Recipe to detect values, ranges, distribution, frequency, implied and enforced integrity, and other business based checks on data or processing. &quot;Order&quot; metadata is created that is no just about lineage and descriptors of the data and jobs, but also includes statistics such as wall-clock time, processing requirements, test data outputs, and more. Alerts are delivered if selected tests fail. If not found, data and process errors reduce the business users trust. DataKitchen&apos;s customers see a meaningful reduction in the number of data errors, incorrect results, and late deliveries. Another DataKitchen goal is to reduce the time it takes to move changes from development into production. When business customers request new changes, DataKitchen can continually and automatically deploy those changes. A challenge is to make sure that those changes do not cause regressions, functional errors, or performance problems in production. The embedded Recipe tests serve a dual role in resolving this problem. In production environments, those tests provide surveillance and alerts, but in development, those tests make sure that any change in code or configuration does not cause a problem further down the pipeline. DataKitchen allows users on different teams and location to collaborate. One method is to use &quot;Ingredients&quot; to create sharable, reusable, component services. Multiple Recipes can call Ingredients, and they have a standard, function-like API. The platform was designed and implemented for secure multi-tenant multi-cloud, and multi-environment deployments. Finally, interfacing with Data Kitchen is supported by a user interface, command line or APIs. This is important because Data Kitchen can accept data and metadata from other processes you may already have in place. The DataKitchen DevOps Platform is a commercial product, available as a managed service with optional on-prem agent installation, and was first released in 2014.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;June 2019&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;product-links&quot;&gt;Product Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://www.datakitchen.io/platform.html&quot;&gt;https://www.datakitchen.io/platform.html&lt;/a&gt; - homepage&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://datakitchen.readme.io/docs&quot;&gt;https://datakitchen.readme.io/docs&lt;/a&gt; - documentation&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;dataops-links&quot;&gt;DataOps Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://www.dataopsmanifesto.org/&quot;&gt;https://www.dataopsmanifesto.org/&lt;/a&gt; - DataOps Manifesto&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://www.datakitchen.io/dataops-cookbook-main.html&quot;&gt;https://www.datakitchen.io/dataops-cookbook-main.html&lt;/a&gt; - DataOps Cookbook&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://paper.li/datakitchen_io/1525109892#/&quot;&gt;https://paper.li/datakitchen_io/1525109892#/&lt;/a&gt; - DataOps News&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://medium.com/data-ops&quot;&gt;https://medium.com/data-ops&lt;/a&gt; - DataOps Blog&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://www.youtube.com/playlist?list=PLVbsAdgZXvtyy6HVKCP0HChjCcq2oW3eK&quot;&gt;https://www.youtube.com/playlist?list=PLVbsAdgZXvtyy6HVKCP0HChjCcq2oW3eK&lt;/a&gt; - DataOps Videos&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://www.slideshare.net/datakitchen99&quot;&gt;https://www.slideshare.net/datakitchen99&lt;/a&gt; - DataOps SlideShare&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://www.datakitchen.io/blog&quot;&gt;https://www.datakitchen.io/blog&lt;/a&gt; - DataKitchen blog&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/datakitchen-devops-platform/</guid> </item> <item><title>The Mid Week News 19/06/2019</title><link>https://ondataengineering.github.io/blog/2019/06/19/the-mid-week-news/</link><pubDate>Wed, 19 Jun 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Once again with the weekly news - will it ever stop! Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-atlas/&quot;&gt;Apache Atlas&lt;/a&gt; 1.2 is out with a bunch of minor updates to the 1.x line. 2.0 came out last month if you’re looking for the latest version&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; 5.24 is out, with new versions of Flink, Presto, and Hue and Spark performance improvements&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-bigtop/&quot;&gt;Apache Bigtop&lt;/a&gt; 1.4 is out, with an Hadoop bump to 2.8.5 and a new integration test framework&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Datanami - Attunity and WANdiscoLiveMigrator, and how they can help you migrate your data to the cloud - &lt;a href=&quot;https://www.datanami.com/2019/06/12/great-cloud-migration-opens-data-opportunities/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;GridGain, commercial vendors for &lt;a href=&quot;/technologies/apache-ignite/&quot;&gt;Apache Ignite&lt;/a&gt;, now includes an Hadoop Data Lake Accelerator in their commercial products, giving in memory bi-directional caching of your Hadoop data - &lt;a href=&quot;https://www.gridgain.com/resources/blog/gridgain-data-lake-accelerator-released-today&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Datanami have some thoughts on Qubole Quantum, their new serverless query engine based on &lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt; - &lt;a href=&quot;https://www.datanami.com/2019/06/13/serverless-sql-engine-targets-cloud-analytics/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Starburst have announced that &lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt; how supports &lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt;, the newly open sourced data storage for Spark and Hadoop that supports ACID transactions - &lt;a href=&quot;https://www.starburstdata.com/technical-blog/starburst-presto-databricks-delta-lake-support/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapR&lt;/a&gt; have an update (that doesn’t say anything particularly new) on their financial situation - &lt;a href=&quot;https://mapr.com/blog/mapr-update-june-13/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, how Teradata has moved from hardware to software, and from perpetual licences to subscription licences - &lt;a href=&quot;https://www.datanami.com/2019/06/14/teradata-turns-40-takes-off-gloves-readies-for-a-fight/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Solutions Review, a summary of Gartner’s recent cool vendors for Data Management (DataDDO, DataKitchen, Panoply) - &lt;a href=&quot;https://solutionsreview.com/data-management/gartner-names-3-cool-vendors-in-data-management-for-2019/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Databricks have announced Databricks Connect, a library that allows seamless execution of &lt;a href=&quot;/technologies/apache-spark&quot;&gt;Spark&lt;/a&gt; code into notebooks, IDEs and custom apps, with execution happening on a remote cluster - &lt;a href=&quot;https://databricks.com/blog/2019/06/14/databricks-connect-bringing-the-capabilities-of-hosted-apache-spark-to-applications-and-microservices.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet, &lt;a href=&quot;/technologies/microsoft-azure-data-lake-store&quot;&gt;Azure Data Lake Store&lt;/a&gt; and &lt;a href=&quot;/technologies/microsoft-azure-blob-storage&quot;&gt;Azure Blob Storage&lt;/a&gt; are now both supported by Okera if you’re looking for more granular data acccess controls - &lt;a href=&quot;https://www.zdnet.com/article/azure-data-lake-storage-gets-okera-security-and-governance-platform-support/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/06/19/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 12/06/2019</title><link>https://ondataengineering.github.io/blog/2019/06/12/the-mid-weeks-news/</link><pubDate>Wed, 12 Jun 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Time for the weekly news again. Remember, you can get daily news updates from our twitter feed (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;StreamSets Data Collector&lt;/a&gt; 3.9 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; 2.13 is out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Datanami - looks like both Tom Reilly and Mike Olson will be retiring from &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; this summer - &lt;a href=&quot;https://www.datanami.com/2019/06/06/cloudera-ceo-reilly-to-retire-after-poor-1q-results/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Qubole have a new serverless &lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt; offering out called Quantum - pay per query and let them manage the Presto cluster - &lt;a href=&quot;https://www.qubole.com/blog/technical-overview-quantum-serverless-engine/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Interested in Hadoop &lt;a href=&quot;/technologies/apache-hadoop/ozone/&quot;&gt;Ozone&lt;/a&gt; - the project has a new blog post on on securing it - &lt;a href=&quot;https://blogs.apache.org/ozonesecurity/entry/security-in-apache-hadoop-ozone&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Cloudera have a bunch of performance metrics out relating to Erasure Coding in &lt;a href=&quot;/technologies/apache-hadoop/hdfs/&quot;&gt;HDFS&lt;/a&gt; - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/06/hdfs-erasure-coding-in-production/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, an excellent article on the current state of the &lt;a href=&quot;/technologies/apache-hadoop&quot;&gt;Hadoop&lt;/a&gt; market, and in particular &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; and &lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapR&lt;/a&gt; - &lt;a href=&quot;https://www.datanami.com/2019/06/10/hadoop-struggles-and-bi-deals-whats-going-on/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Bloor have a new paper out on TestDataManagement, looking at the different methodologies - &lt;a href=&quot;https://www.bloorresearch.com/research/test-data-management-2/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/06/12/the-mid-weeks-news/</guid> </item> <item><title>The Mid Week News 05/06/2019</title><link>https://ondataengineering.github.io/blog/2019/06/05/the-mid-week-news/</link><pubDate>Wed, 05 Jun 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;If you’ve been following us on twitter (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;) then you’ll have seen all this news already.&lt;/p&gt; &lt;p&gt;If not - here’s your new weekly news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; R56 is out for AWS, Azure and Oracle&lt;/li&gt; &lt;li&gt;Hortonworks (Cloudera) &lt;a href=&quot;/technologies/hortonworks-dataplane/data-steward-studio/&quot;&gt;Data Steward Studio&lt;/a&gt; 1.5 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-storm/&quot;&gt;Apache Storm&lt;/a&gt; has a big 2.0 release out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From the code972 blog, thoughts on the AWS Open Distro for Elastic Search from someone who knows what they’re talking about. See also his posts on why you shouldn’t use AWS Elastic Service, and running &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; on Kubernetes - &lt;a href=&quot;https://code972.com/blog/2019/03/116-dont-confuse-awss-open-distro-for-elasticsearch-with-altruism&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon Managed Streaming for Apache Kafka is now generally available - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2019/05/amazon_managed_streaming_for_apache_kafka_amazon_msk_is_now_generally_available/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.infoq.com/news/2019/06/aws-managed-kafka-ga/&quot;&gt;InfoQ&lt;/a&gt;&lt;/li&gt; &lt;li&gt;I always through their tech looks pretty interesting, but it look’s like &lt;a href=&quot;/tech-vendors/mapr/&quot;&gt;MapR&lt;/a&gt; are having financial troubles, although MapR have published a response - &lt;a href=&quot;https://www.datanami.com/2019/05/30/after-funding-falls-through-mapr-seeks-a-buyer-to-avoid-shut-down/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://mapr.com/blog/an-update-from-mapr/&quot;&gt;MapR&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Are you into or looking at &lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; - they’ve got a bunch of new Katas out if you’re looking to learn it - &lt;a href=&quot;https://beam.apache.org/blog/2019/05/30/beam-kata-release.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Qubole - scaling &lt;a href=&quot;/technologies/apache-hive/&quot;&gt;Apache Hive&lt;/a&gt; through the use of a dedicate cluster for running HiveServer2 - &lt;a href=&quot;https://www.qubole.com/blog/increase-scalability-of-hiveserver2/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - the latest on Snowflake, the cloud native analytical database - &lt;a href=&quot;https://www.datanami.com/2019/05/29/snowflake-rides-cloud-wave-to-great-heights/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Qubole, and normally I’d not link to something like this, but this feels a reasonable stab at the difference between a Data Lake and a Data Warehouse - &lt;a href=&quot;https://www.qubole.com/blog/data-lakes-vs-data-warehouses/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - how IBM is Turning DB2 into an AI database - &lt;a href=&quot;https://www.datanami.com/2019/06/03/how-ibm-is-turning-db2-into-an-ai-database/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera, part 2 of their blog on the introduction of attribute based access control to &lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/06/cdh6-2-cloudera-search-attribute-based-access-control-part-2/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Elastic - how to secure your &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; cluster using all the security features now available for free - &lt;a href=&quot;https://www.elastic.co/blog/tips-to-secure-elasticsearch-clusters-for-free-with-encryption-users-and-more&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Snowflake on &lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google Cloud Platform&lt;/a&gt; has been announced - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/announcing-snowflake-on-google-cloud-platform&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Solutions Review, cloud data integrator Matillion has secured a bunch of VC funding. Anyone any experience with this, and happy to write us a tech summary? &lt;a href=&quot;https://solutionsreview.com/data-integration/matillion-nabs-series-c-funding-for-cloud-data-warehouse-integration/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/06/05/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 29/05/2019</title><link>https://ondataengineering.github.io/blog/2019/05/29/the-mid-week-news/</link><pubDate>Wed, 29 May 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s that time of the week again, so let’s look at the news.&lt;/p&gt; &lt;p&gt;Next week however, we’re going to try something slightly difference. We’ll do the news daily (ish) over our twitter account (&lt;a href=&quot;https://twitter.com/OnDataEng&quot;&gt;@OnDataEng&lt;/a&gt;) for those that like Twitter, with this weekly post then summarising it for those that like RSS/newsletters, with the format of this weekly post likely to change over time. &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-nifi/registry/&quot;&gt;NiFi Registry&lt;/a&gt; has hit 0.4&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cloudera have a post on Conjuctive access control groups in &lt;a href=&quot;/technologies/apache-solr&quot;&gt;Solr&lt;/a&gt; document level access control - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/05/cdh6-2-cloudera-search-attribute-based-access-control-part-1/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Firstly from Confluent - &lt;a href=&quot;/tech-categories/schema-registries/&quot;&gt;Schema Registries&lt;/a&gt; and why you need them, and an intro to &lt;a href=&quot;/technologies/confluent-open-source&quot;&gt;Confluent Schema Registry&lt;/a&gt; - &lt;a href=&quot;https://www.confluent.io/blog/schemas-contracts-compatibility&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And as a follow up from Confluent - 17 ways to mess up your &lt;a href=&quot;/technologies/confluent-open-source&quot;&gt;Confluent Schema Registry&lt;/a&gt; - &lt;a href=&quot;https://www.confluent.io/blog/17-ways-to-mess-up-self-managed-schema-registry&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Morning Paper - improving memory compression by compressing objects not cache lines - &lt;a href=&quot;https://blog.acolyer.org/2019/05/24/zippads/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via Solutions review, Syncsort DMX Change Data Capture is now Syncsort Connect CDC - &lt;a href=&quot;https://blog.syncsort.com/2019/05/big-data/connect-cdc-real-time-streaming-data/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.syncsort.com/en/products/Connect-CDC&quot;&gt;home page&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Autoscale is now in preview with &lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;Azure HDInsight&lt;/a&gt; - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/drive-higher-utilization-of-azure-hdinsight-clusters-with-autoscale/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/05/29/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 22/05/2019</title><link>https://ondataengineering.github.io/blog/2019/05/22/the-mid-week-news/</link><pubDate>Wed, 22 May 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Once again, it’s time for the mid week news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-atlas/&quot;&gt;Apache Atlas&lt;/a&gt; has hit 2.0&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-avro/&quot;&gt;Apache Avro&lt;/a&gt; has hit 1.9&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; has hit 8.1&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.19&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; and &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch Hadoop&lt;/a&gt; has been bumped to 6.8 and 7.1&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; have more details on their 6.0 release&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Let’s call this out explicitly, but the 6.8 and 7.1 Elastic releases are solely to make security (TLS and RBAC, which were open sourced not so long ago) free, presumably in response to the new Elastic Open Distro - &lt;a href=&quot;https://www.elastic.co/blog/security-for-elasticsearch-is-now-free&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Also from Elastic, they’ve announced Elastic Cloud on Kubernetes (ECK) - &lt;a href=&quot;https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And from DataStax, they’ve announced Constellation, their &lt;a href=&quot;/technologies/apache-cassandra&quot;&gt;Cassandra&lt;/a&gt; as a service offering - &lt;a href=&quot;https://www.datastax.com/2019/05/datastax-announces-constellation-a-cloud-native-data-platform&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/datastax-constellation-apache-cassandra-as-a-service-announced/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet, the future of SAP Hana - &lt;a href=&quot;https://www.zdnet.com/article/the-future-of-sap-hana/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Latest updates from Microsoft on Cosmos DB - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/a-cosmonaut-s-guide-to-the-latest-azure-cosmos-db-announcements/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And from Datanami - how ScyllaDB (an Apache Cassandra-compatible database) handles both OLAP and OLTP - &lt;a href=&quot;https://www.datanami.com/2019/05/14/scylladb-gives-cohabitation-of-olap-and-oltp-a-shot/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/05/22/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 15/05/2019</title><link>https://ondataengineering.github.io/blog/2019/05/15/the-mid-week-news/</link><pubDate>Wed, 15 May 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It feel’s like deja-vu, but it’s time for the news again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-oozie/&quot;&gt;Apache Oozie&lt;/a&gt; is up to 0.4-alpha&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; is up to R56 for AWS&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From the &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; blog - using temporal tables in their Streaming SQL and Table API - &lt;a href=&quot;https://flink.apache.org/2019/05/14/temporal-tables.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Elastic - field aliases in &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; - &lt;a href=&quot;https://www.elastic.co/blog/introducing-field-aliases-in-elasticsearch&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Also from Elastic this week - common performance issues with &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; and how to fix them - &lt;a href=&quot;https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera - information on MirrorMaker 2, replicating data between &lt;a href=&quot;/technologies/apache-kafka&quot;&gt;Kafka&lt;/a&gt;ka instances based on Kafka Connect - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/05/kafka-replication-the-case-for-mirrormaker-2/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And also from Cloudera, a couple of &lt;a href=&quot;/technologies/apache-hadoop&quot;&gt;Hadoop&lt;/a&gt; / &lt;a href=&quot;/technologies/apache-hive&quot;&gt;Hive&lt;/a&gt; posts on managing small files and partitions in Hive - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/05/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/&quot;&gt;small files&lt;/a&gt;; &lt;a href=&quot;https://blog.cloudera.com/blog/2019/05/partition-management-in-hadoop/&quot;&gt;partition management&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Microsoft - introducing Azure SQL Database Edge - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-sql-database-edge-enabling-intelligent-data-at-the-edge/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.infoq.com/news/2019/05/Azure-SQL-Database-Edge&quot;&gt;InfoQ&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And more from Microsoft - new performance and capability fixes for Azure SQL Data Warehouse - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-sql-data-warehouse-releases-new-capabilities-for-performance-and-security/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;ZDNet and Datanami have updates on Confluent Cloud - &lt;a href=&quot;https://www.zdnet.com/article/confluent-makes-apache-kafka-cloud-native/&quot;&gt;ZDNet&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/05/13/kafka-in-the-cloud-who-needs-clusters-anyway/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From LinkedIn, an introduction to Ambry - their “distributed, highly available and horizontally scalable immutable object store optimized to store and serve media” - &lt;a href=&quot;https://engineering.linkedin.com/blog/2019/05/introducing-data-compaction-in-ambry&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Confluent, a couple of posts on managing multi-clusters with Confluent Control Centre and consuming data from &lt;a href=&quot;/technologies/apache-kafka&quot;&gt;Kafka&lt;/a&gt; - &lt;a href=&quot;https://www.confluent.io/blog/dawn-of-kafka-devops-managing-multi-cluster-kafka-connect-and-ksql-with-confluent-control-center&quot;&gt;managing multi-clusters&lt;/a&gt;; &lt;a href=&quot;https://www.confluent.io/blog/apache-kafka-data-access-semantics-consumers-and-membership&quot;&gt;data access&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And lastly, from Influx Data - getting started with &lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; 2.0 and announcing their InfluxDB Cloud 2.0 Beta Free Tier - &lt;a href=&quot;https://www.influxdata.com/blog/getting-started-with-influxdb-2-0-scraping-metrics-running-telegraf-querying-data-and-writing-data/&quot;&gt;getting started&lt;/a&gt;; &lt;a href=&quot;https://www.influxdata.com/blog/announcing-influxdb-cloud-2-0-beta-free-tier/&quot;&gt;influxdb cloud free tier&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/05/15/the-mid-week-news/</guid> </item> <item><title>Google Cloud Composer</title><link>https://ondataengineering.github.io/technologies/google-cloud-composer/</link><pubDate>Tue, 14 May 2019 00:00:00 +0000</pubDate> <description> &lt;p&gt;Managed workflow orchestration service built on Apache Airflow that&apos;s designed for running data integration tasks on a repeated schedule. Implemented on a micro-service architecture, the Airflow database and web server are implemented on App Engine and access protected using Identify-Aware Proxy (an enterprise security model that enables employees to work from untrusted networks without the use of a VPN), while the scheduler, executor and worker nodes are implemented on Kubernetes Engine. Integrated with Cloud Storage for staging DAGs, plugins, data dependencies and Stackdriver for real-time logging and monitoring of Airflow service and workflow logs. Manageable via a web (Cloud Platform Console and Airflow web interfaces), command line interface (Cloud SDK) or an RPC and REST API. Allows custom Airflow plugins and Python dependencies from the Python Package Index to be installed. Priced an an hourly rate (charged per minute) based on the size of a Cloud Composer environment, which is in addition to any Kubernetes Engine, Compute Engine or Persistent Disk and network egress charges.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Other Names&lt;/td&gt;&lt;td&gt;Cloud Composer&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;May 2019 - v1.3&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2&gt;Related Technologies&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Packages&lt;/td&gt;&lt;td&gt;&lt;a href=&quot;/technologies/apache-airflow/&quot;&gt;Apache Airflow&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/composer/&quot;&gt;https://cloud.google.com/composer/&lt;/a&gt; - homepage&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/composer/docs/concepts/versioning/composer-versions&quot;&gt;https://cloud.google.com/composer/docs/concepts/versioning/composer-versions&lt;/a&gt; - bundle services version list&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/composer/docs/release-notes&quot;&gt;https://cloud.google.com/composer/docs/release-notes&lt;/a&gt; - release notes&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/composer/docs/&quot;&gt;https://cloud.google.com/composer/docs/&lt;/a&gt; - documentation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://github.com/GoogleCloudPlatform/python-docs-samples/&quot;&gt;https://github.com/GoogleCloudPlatform/python-docs-samples/&lt;/a&gt; - sample Composer workflows&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;p&gt;See &lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google Cloud Platform&lt;/a&gt; updates&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/google-cloud-composer/</guid> </item> <item><title>Apache Airflow</title><link>https://ondataengineering.github.io/technologies/apache-airflow/</link><pubDate>Tue, 14 May 2019 00:00:00 +0000</pubDate> <description> &lt;p&gt;A workflow management system designed for orchestrating repeated data integration tasks on a schedule, with workflows configured in Python as a Directed Acyclic Graph (DAG) of tasks. A scheduler is responsible for identifying tasks to be run, with an executor responsible for determining where tasks should run (with support for local execution or remote execution using Celery, Dask, Mesos and Kubernetes, with the ability to define custom executors). Supports periodic execution of workflows (based on a schedule interval), sensor operators (that wait until some condition is true, e.g. a file exists), automatic retry of failed tasks, catchup of historic task executions, task templating, triggers and complex dependancies, shared connection configuration, configurable job parallelism, variables that can be configured through the UI, re-usable sub DAGs and experimental support for data lineage with integration to Apache Atlas. Packaged with a wide variety of prebuilt &apos;operators&apos; for data integration; databases (MySQL, PostgreSQL, Oracle), Hadoop (Hive, Pig, Sqoop) and cloud services (Amazon Web Services, Google Cloud Platform and Microsoft Azure services), with the ability to write your own. Comes with a command line and web interface to manage and monitor workflows and perform administrative actions on the environment and an experimental REST API. Persists workflow management state and operational metadata in either a MySQL or PostgreSQL relational database and queryable using SQL via the web interface to create simple charts. Includes a security model with support for a range of authentication methods including LDAP, Kerberos (limited), OAuth and Google Authentication. Originally developed at Airbnb and donated to the Apache Foundation&apos;s incubator program in June 2015. Under active development with a wide range of contributors. Commercial support is available from a variety of vendors who distribute it as a standalone managed service (Astronomer and Google), to run on Kubernetes (Astronomer), or part of wider managed data service offering (Qubole).&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Other Names&lt;/td&gt;&lt;td&gt;Airflow&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Vendors&lt;/td&gt;&lt;td&gt;&lt;a href=&quot;/tech-vendors/apache/&quot;&gt;The Apache Software Foundation&lt;/a&gt;, &lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google Cloud Platform&lt;/a&gt;, Astronomer&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial Open Source&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;May 2019 - 1.10&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2&gt;Related Technologies&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Is packaged by&lt;/td&gt;&lt;td&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt;, &lt;a href=&quot;/technologies/google-cloud-composer/&quot;&gt;Google Cloud Composer&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;release-history&quot;&gt;Release History&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt; &lt;td&gt;version&lt;/td&gt; &lt;td&gt;release date&lt;/td&gt; &lt;td&gt;release links&lt;/td&gt; &lt;td&gt;release comment&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;1.7&lt;/td&gt; &lt;td&gt;2016-03-28&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-March28,2016&quot;&gt;announcement&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;1.8&lt;/td&gt; &lt;td&gt;2017-03-19&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-March19,2017&quot;&gt;announcement&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;1.9&lt;/td&gt; &lt;td&gt;2018-01-02&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Jan2,2018&quot;&gt;announcement&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;1.10&lt;/td&gt; &lt;td&gt;2018-08-20&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Aug20,2018&quot;&gt;announcement&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://airflow.apache.org&quot;&gt;https://airflow.apache.org&lt;/a&gt; - documentation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://github.com/jghoman/awesome-apache-airflow&quot;&gt;https://github.com/jghoman/awesome-apache-airflow&lt;/a&gt; - a curated list of resources about Apache Airflow (incubating)&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements&quot;&gt;https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements&lt;/a&gt; - announcements&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://github.com/apache/incubator-airflow/releases&quot;&gt;https://github.com/apache/incubator-airflow/releases&lt;/a&gt; - details of releases&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/apache-airflow/</guid> </item> <item><title>The Mid Week News 08/05/2019</title><link>https://ondataengineering.github.io/blog/2019/05/08/the-mid-week-news/</link><pubDate>Wed, 08 May 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s news time again, and we’ve used a real date this time… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-drill/&quot;&gt;Apache Drill&lt;/a&gt; as hit 1.16&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; is up to 5.23&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-s3/&quot;&gt;Amazon S3&lt;/a&gt; now supports &lt;a href=&quot;https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/&quot;&gt;Batch Operations&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;ZDNet have all the data updates from &lt;a href=&quot;/tech-vendors/microsoft-azure/&quot;&gt;Microsoft&lt;/a&gt; Build - &lt;a href=&quot;https://www.zdnet.com/article/microsoft-builds-its-data-story-in-the-cloud-and-at-the-edge/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Microsoft, details on the latest updates to Azure Data Factory and Azure SQL Data Warehouse - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/analytics-in-azure-remains-unmatched-with-new-innovations/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Google, how Google Cloud DataPrep (based on Trifacta) helps you manage Data Quality - &lt;a href=&quot;https://cloud.google.com/blog/products/gcp/improving-data-quality-for-machine-learning-and-analytics-with-cloud-dataprep&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Bloor have all the latest on Teradata Vantage, and their split of storage and computer - &lt;a href=&quot;https://www.bloorresearch.com/2019/05/teradata-vantage/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - some people have got very large cloud bills - &lt;a href=&quot;https://www.datanami.com/2019/05/02/cloud-analytics-proving-costly-for-some/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google&lt;/a&gt; have some best practices for data governance in the cloud - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/principles-and-best-practices-for-data-governance-in-the-cloud&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Microsoft have announced high performance C# and F# &lt;a href=&quot;/technologies/apache-spark&quot;&gt;Spark&lt;/a&gt; libraries - &lt;a href=&quot;https://www.infoq.com/news/2019/04/microsoft-net-apache-spark&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;eBay have announced Beam, an open source distributed RDF store - &lt;a href=&quot;https://github.com/eBay/beam&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Confluent have a post on optimising Kafka Streams applications - &lt;a href=&quot;https://www.confluent.io/blog/optimizing-kafka-streams-applications&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/05/08/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 31/04/2019</title><link>https://ondataengineering.github.io/blog/2019/05/01/the-mid-week-news/</link><pubDate>Wed, 01 May 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;One big announcement this week, but otherwise it’s another very quiet one… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; has hit 2.21&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;The big news this week is that Databricks have open sourced &lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt;, their transactional data lake storage layer that runs over HDFS/S3 - &lt;a href=&quot;https://databricks.com/blog/2019/04/24/open-sourcing-delta-lake.html&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://delta.io&quot;&gt;home page&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/04/24/databricks-donates-delta-code-to-open-source&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/a-standard-for-storing-big-data-apache-spark-creators-release-open-source-delta-lake/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Datanami have an interesting article on Data Pipeline Automation, with a range of tools referenced that we should probably look at at some point - &lt;a href=&quot;https://www.datanami.com/2019/04/24/data-pipeline-automation-the-next-step-forward-in-dataops&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/05/01/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 24/04/2019</title><link>https://ondataengineering.github.io/blog/2019/04/24/the-mid-week-news/</link><pubDate>Wed, 24 Apr 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s a light week for the news, but let’s have a quick look anyway… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;None this week&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;ODBMS Industry Watch has an interview with Merv Adrian of Gartner on his views of the current database market, Hadoop and the Cloud - &lt;a href=&quot;http://www.odbms.org/blog/2019/04/on-the-database-market-interview-with-merv-adrian/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;An nice deeply technical blog post from the &lt;a href=&quot;/technologies/pravega/&quot;&gt;Pravega&lt;/a&gt; team on how it provides performant support for events of any size - &lt;a href=&quot;http://blog.pravega.io/2019/04/22/events-big-or-small-bring-them-on/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera - fine grained access control on Impala over Kudu - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/04/fine-grained-authorization-with-apache-kudu-and-impala/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the Knoldus blog - are Knowledge Graphs (see our &lt;a href=&quot;/tech-categories/graph-databases/&quot;&gt;Graph Databases&lt;/a&gt; tech category) the future of Data Lakes - &lt;a href=&quot;https://blog.knoldus.com/are-knowledge-graphs-the-future-of-data-lakes/&quot;&gt;links&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Microsoft have open sourced Data Accelerator for &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt;, a tool for simplifying creating streaming pipelines - &lt;a href=&quot;https://cloudblogs.microsoft.com/opensource/2019/04/16/microsoft-open-sources-data-accelerator-for-apache-spark/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Google have a bunch of updates they’ve announced for &lt;a href=&quot;/technologies/google-cloud-dataproc/&quot;&gt;Google Cloud DataProc&lt;/a&gt; - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/new-open-source-tools-in-cloud-dataproc-process-data-at-cloud-scale&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Elastic, what’s new in Lucene 8 - &lt;a href=&quot;https://www.elastic.co/blog/whats-new-in-lucene-8&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From DataBricks, using Spark with &lt;a href=&quot;/technologies/amazon-s3&quot;&gt;S3&lt;/a&gt; Select - &lt;a href=&quot;https://databricks.com/blog/2019/04/17/running-peta-scale-spark-jobs-on-object-storage-using-s3-select.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/04/24/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 17/04/2019</title><link>https://ondataengineering.github.io/blog/2019/04/17/the-mid-week-news/</link><pubDate>Wed, 17 Apr 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Tick tock, tick tock - it’s time for the news again, and there’s a big bunch of announcements this week… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; and &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch-Hadoop&lt;/a&gt; have hit the big 7.0&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; has hit 2.2&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;Azure HDInsight&lt;/a&gt; has hit 4.0 (Hadoop 3.x)&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; not supports GCP&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; has hit 5.22&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; has hit 1.8&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-myriad/&quot;&gt;Apache Myriad&lt;/a&gt; has hit 0.3&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kylin/&quot;&gt;Apache Kylin&lt;/a&gt; has a 3.0-alpha out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; has hit 2.0 alpha 8&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cloudera Flow Management and Cloudera Edge Management (part of the new Cloudera DataFlow, nee &lt;a href=&quot;/technologies/hortonworks-data-flow&quot;&gt;HDF&lt;/a&gt;) are now generally available - &lt;a href=&quot;http://vision.cloudera.com/announcing-the-general-availability-of-cloudera-flow-management-and-cloudera-edge-management/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Also from Cloudera, an update on Workload XM, and it’s beautiful visualisations of Spark jobs - &lt;a href=&quot;https://blog.cloudera.com/blog/2019/04/demystifying-spark-jobs-to-optimize-for-cost-and-performance/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s a bunch of stuff out of the recent Google NEXT conference (thanks to longtime friend of the site Jeff for the heads-up) - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/google-cloud-smart-analytics-accelerates-your-business-transformation&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/04/10/google-cloud-unveils-slew-of-new-data-management-and-analytics-services/&quot;&gt;Datanami view&lt;/a&gt; &lt;ul&gt; &lt;li&gt;Cloud Data Fusion (beta) - a managed version of Cask Data Application Platform (CDAP) - an open source framework for building data analytic applications, which Google acquired back in May 2018&lt;/li&gt; &lt;li&gt;Big Query BI Engine (beta) - in memory analysis service that accelerates query response times from BigQuery for Looker, Google Data Studio, and Google Sheets&lt;/li&gt; &lt;li&gt;Connected Sheets (beta) - ability to access BigQuery through Google Sheets&lt;/li&gt; &lt;li&gt;Data Catalog (beta) - managed metadata management service&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;ZDNet have a couple of good summaries of the open source partnership announcements at Google NEXT, including their thoughts on what this means for open source - &lt;a href=&quot;https://www.zdnet.com/article/google-cloud-next-19-postmortem-thomas-kurian-wants-the-right-people/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/google-cloud-gives-open-source-data-vendors-a-break-will-that-save-open-source/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Datanami have a story on a new high performance massively parallel in memory graph database from Cray alumni called Trovares xGT - &lt;a href=&quot;https://www.datanami.com/2019/04/15/startup-trovares-brings-hpc-to-graph-analytics/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/04/17/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 10/04/2019</title><link>https://ondataengineering.github.io/blog/2019/04/10/the-mid-week-news/</link><pubDate>Wed, 10 Apr 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Where does time go? It’s time for the news again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-arrow/&quot;&gt;Apache Arrow&lt;/a&gt; 0.13 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; 5.18 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-altus/director/&quot;&gt;Cloudera Altus Director&lt;/a&gt; 6.2 is out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Google have decided to partner with a bunch of Open Source ISVs such as Elastic on Confluent rather than create their own offerings based on the open source products - &lt;a href=&quot;https://www.datanami.com/2019/04/09/google-extends-olive-branch-to-open-source-tech/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.theregister.co.uk/2019/04/09/google_cloud_keynote/&quot;&gt;The Register&lt;/a&gt;; &lt;a href=&quot;https://www.elastic.co/blog/elastic-and-google-team-up-to-bring-a-more-native-elasticsearch-service-experience-on-google-cloud&quot;&gt;Elastic&lt;/a&gt;; &lt;a href=&quot;https://www.confluent.io/blog/announcing-confluent-cloud-for-apache-kafka-native-service-on-google-cloud-platform&quot;&gt;Confluent&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt; has gone GA - &lt;a href=&quot;https://databricks.com/blog/2019/04/04/announcing-databricks-runtime-5-3.html&quot;&gt;blog&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/04/08/how-databricks-keeps-data-quality-high-with-delta/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - Apache Spark is Great, But Not Perfect - &lt;a href=&quot;https://www.datanami.com/2019/04/03/apache-spark-is-great-but-its-not-perfect/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet - where is Confluent Going? - &lt;a href=&quot;https://www.zdnet.com/article/where-is-confluent-going/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Microsoft are crowing about their performance, price and security for their Azure analytics services again - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/want-to-evaluate-your-cloud-analytics-provider-here-are-the-three-questions-to-ask/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;The Register has an update on Teradata’s move to the cloud - &lt;a href=&quot;https://www.theregister.co.uk/2019/04/08/teradata_vantage_as_a_service/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/04/10/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 03/04/2019</title><link>https://ondataengineering.github.io/blog/2019/04/03/the-mid-week-news/</link><pubDate>Wed, 03 Apr 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Some bigish releases this week. Let’s delve into the news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-cdh/&quot;&gt;CDH&lt;/a&gt; 6.2 is out, including new releases of &lt;a href=&quot;/technologies/cloudera-manager/&quot;&gt;Cloudera Manager&lt;/a&gt; and &lt;a href=&quot;/technologies/cloudera-navigator/&quot;&gt;Cloudera Navigator&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-impala/&quot;&gt;Apache Impala&lt;/a&gt; 3.2 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt; 2.2 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hue/&quot;&gt;Hue&lt;/a&gt; 4.4 is out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; 7.0 RC1 is out&lt;/li&gt; &lt;li&gt;Confluent &lt;a href=&quot;/technologies/confluent-open-source/&quot;&gt;Open Source&lt;/a&gt; and &lt;a href=&quot;/technologies/confluent-enterprise&quot;&gt;Enterprise&lt;/a&gt; have hit 5.2&lt;/li&gt; &lt;li&gt;Hortonworks DataPlane &lt;a href=&quot;/technologies/hortonworks-dataplane/data-lifecycle-manager/&quot;&gt;Data Lifecycle Manager&lt;/a&gt; and &lt;a href=&quot;/technologies/hortonworks-dataplane/data-steward-studio/&quot;&gt;Data Steward Studio&lt;/a&gt; are both up to 1.4&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cloudera have announced Cloudera DataFlow (their new name for &lt;a href=&quot;/technologies/hortonworks-data-flow&quot;&gt;HDF&lt;/a&gt;), with a couple of new products - &lt;a href=&quot;http://vision.cloudera.com/introducing-cloudera-edge-management-and-cloudera-flow-management/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;MapR have announced key new features in their next &lt;a href=&quot;/technologies/mapr-converged-data-platform/&quot;&gt;MapR Converged Data Platform&lt;/a&gt; release, including support for Spark and Drill on Kubernetes, and support for their filesystem as a Kubernetes persistent volume - &lt;a href=&quot;https://mapr.com/products/whats-new/compute-storage/&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://mapr.com/blog/deploying-native-spark-and-drill-applications-in-kubernetes-just-got-easier/&quot;&gt;blog post&lt;/a&gt;; &lt;a href=&quot;https://www.theregister.co.uk/2019/04/02/mapr_kubernetes_launch/&quot;&gt;The Register&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/mapr-brings-tight-integration-with-kubernetes/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Cloudera’s annual report is out, and The Register have their summary - &lt;a href=&quot;https://www.theregister.co.uk/2019/04/01/cloudera_annual_report/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Datanami have an interview with Doug Cutting - &lt;a href=&quot;https://www.datanami.com/2019/04/01/heres-what-doug-cutting-says-is-hadoops-biggest-contribution&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Morning Paper - Calvin - fast distributed transactions for partitioned database systems - &lt;a href=&quot;https://blog.acolyer.org/2019/03/29/calvin-fast-distributed-transactions-for-partitioned-database-systems/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/microsoft-azure-blob-storage&quot;&gt;Azure Blob Storage&lt;/a&gt; lifecycle management is generally available - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-blob-storage-lifecycle-management-now-generally-available/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Also on &lt;a href=&quot;/technologies/microsoft-azure-blob-storage&quot;&gt;Azure Blob Storage&lt;/a&gt;, high performance block blobs are now aailable across all storage tiers - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/high-throughput-with-azure-blob-storage/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon have announced a new Glacier Deep Storage storage class for &lt;a href=&quot;/technologies/amazon-s3/&quot;&gt;Amazon S3&lt;/a&gt; - &lt;a href=&quot;https://aws.amazon.com/blogs/aws/new-amazon-s3-storage-class-glacier-deep-archive/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0212&quot;&gt;CVE-2019-0212&lt;/a&gt; - &lt;a href=&quot;/technologies/apache-hbase/&quot;&gt;Apache HBase&lt;/a&gt; 2.x incorrectly applied Kerberos authorization to users of the HBase REST server&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/04/03/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 27/03/2019</title><link>https://ondataengineering.github.io/blog/2019/03/27/the-mid-week-news/</link><pubDate>Wed, 27 Mar 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Wow, yet another week has passed. Let’s peek at the news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-calcite/&quot;&gt;Apache Calcite&lt;/a&gt; is up to 1.19&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-nifi/minifi/&quot;&gt;Apache NiFi MiNiFi C++&lt;/a&gt; is up to 0.6&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; and &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch-Hadoop&lt;/a&gt; have hit 6.7&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;There’s (always) more Cloudera (Hortonworks) information, this time on their new data platform from the recent DataWorks Summit - &lt;a href=&quot;https://www.theregister.co.uk/2019/03/20/cloudera_data_platform_details/&quot;&gt;The Register summary&lt;/a&gt;; &lt;a href=&quot;https://www.theregister.co.uk/2019/03/21/cloudera_mick_hollison_interview/&quot;&gt;The Register interview&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/the-new-cloudera-hortonworks-hadoop-100-open-source-50-boring/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;On the continuing story of open source licences, the latest take from ZDNet - &lt;a href=&quot;https://www.zdnet.com/article/open-source-growing-pains-is-open-core-the-answer/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Confluence, doing watermarks and triggers in &lt;a href=&quot;/technologies/apache-kafka/kafka-streams/&quot;&gt;Kafka Streams&lt;/a&gt; which doesn’t believe in them - &lt;a href=&quot;https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And again from Confluence, distributed tracing of &lt;a href=&quot;/technologies/apache-kafka&quot;&gt;Kafka&lt;/a&gt; based solutions using Zipkin - &lt;a href=&quot;https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Azure Premium &lt;a href=&quot;/technologies/microsoft-azure-blob-storage&quot;&gt;Blob Storage&lt;/a&gt; is now generally available - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-premium-block-blob-storage-is-now-generally-available/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Short thoughts from Datanmi on &lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; 6 - &lt;a href=&quot;https://www.datanami.com/2019/03/20/pivotal-extends-greenplum-postgresql-on-cloud-foundry/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/03/27/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 20/03/2019</title><link>https://ondataengineering.github.io/blog/2019/03/20/the-mid-week-news/</link><pubDate>Wed, 20 Mar 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Ok, let’s scour the interwebs for the latest news: &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kudu&quot;&gt;Apache Kudu&lt;/a&gt; is up to 1.9&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; has hit the big 8.0&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has a 6.x beta out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-data-flow/&quot;&gt;Hortonworks DataFlow&lt;/a&gt; is up to 3.4&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; is now up to R55 for Azure and Oracle clouds&lt;/li&gt; &lt;li&gt;Hortonworks &lt;a href=&quot;/technologies/schema-registry/&quot;&gt;Schema Registry&lt;/a&gt; is up to 0.7&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;StreamSets Data Collector&lt;/a&gt; is up to 3.8&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Pivotal, a reminder that &lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; is stil out there and awesome - &lt;a href=&quot;https://content.pivotal.io/blog/pivotal-greenplum-postgres&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Register, a report on Cloudera cloud strategy - &lt;a href=&quot;https://www.theregister.co.uk/2019/03/14/cloudera_enterprise_data_cloud/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Azure Databricks now supports &lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Delta&lt;/a&gt;, GitHub integration and deployment in Azure virtal networks - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-databricks-vnet-injection-devops-version-control-and-delta-availability/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And on &lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt;, there’s a new blog post on performing upserts - &lt;a href=&quot;https://databricks.com/blog/2019/03/18/efficient-upserts-into-data-lakes-with-databricks-delta.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;You can now migrate Oracle ETL jobs to AWS Glue - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2019/03/aws-schema-conversion-tool-adds-support-for-migrating-oracle-etl/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/03/20/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 13/03/2019</title><link>https://ondataengineering.github.io/blog/2019/03/13/the-mid-week-news/</link><pubDate>Wed, 13 Mar 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Roll up for this weeks news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; has hit 2.11&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; is up to 5.21&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;This is the big news of the week - AWS have forked &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; (although they claim it’s not a fork) as the Open Distro for Elasticsearch, and have added in a bunch of altering, SQL, security, audit and fine-grained access control features - &lt;a href=&quot;https://opendistro.github.io/for-elasticsearch/&quot;&gt;homepage&lt;/a&gt; &lt;ul&gt; &lt;li&gt;From AWS, their rationale in a couple of blog posts - &lt;a href=&quot;https://aws.amazon.com/blogs/aws/new-open-distro-for-elasticsearch/&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/blogs/opensource/keeping-open-source-open-open-distro-for-elasticsearch/&quot;&gt;rationale&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Response from Elastic - &lt;a href=&quot;https://www.elastic.co/blog/on-open-distros-open-source-and-building-a-company&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Analysis from a bunch of sites - &lt;a href=&quot;https://www.theregister.co.uk/2019/03/12/aws_elasticsearch_distro/&quot;&gt;The Register&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/03/12/search-war-unfolding-for-control-of-elasticsearch/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.influxdata.com/blog/aws-intends-for-their-new-project-to-be-an-elasticsearch-fork/&quot;&gt;Influx&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And we’ve posted these previous, but some background on the recent noise about cloud providers killing open source - &lt;a href=&quot;https://www.infoq.com/articles/will-cloud-computing-kill-open-source&quot;&gt;InfoQ&lt;/a&gt;, &lt;a href=&quot;https://www.datanami.com/2018/12/24/cloud-backlash-grows-as-open-source-gets-less-open/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.influxdata.com/blog/copyleft-and-community-licenses-are-not-without-merit-but-they-are-a-dead-end/&quot;&gt;Influx&lt;/a&gt;; &lt;a href=&quot;https://www.confluent.io/blog/license-changes-confluent-platform&quot;&gt;Confluent&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;From ZDNet, and excellent article on standardisation coming to &lt;a href=&quot;/tech-categories/graph-databases/&quot;&gt;Graph&lt;/a&gt; and &lt;a href=&quot;/tech-categories/rdf-databases/&quot;&gt;RDF&lt;/a&gt; - &lt;a href=&quot;https://www.zdnet.com/article/graph-data-standardization-its-just-a-graph-making-gravitational-waves-in-the-real-world/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Starburst (the commercial product based on &lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt;) has a new major release - &lt;a href=&quot;https://www.starburstdata.com/technical-blog/announcing-starburst-enterprise-302e-with-mission-control/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via DZone, a pretty good article (because it doesn’t talk about features) on choosing between &lt;a href=&quot;/technologies/apache-solr&quot;&gt;Solr&lt;/a&gt; and &lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; - &lt;a href=&quot;https://opensourceconnections.com/blog/2019/02/28/stop-worrying-solr-elasticsearch/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Datanami has a couple of blogs on Spark and why it’s been successful and is growing - &lt;a href=&quot;https://www.datanami.com/2019/03/11/what-makes-apache-spark-sizzle-experts-sound-off/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/03/08/a-decade-later-apache-spark-still-going-strong/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Solr has a new vulnerabliity - &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0192&quot;&gt;CVE-2019-0192&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Azure Premium &lt;a href=&quot;/technologies/microsoft-azure-blob-storage&quot;&gt;Blob Storage&lt;/a&gt; is in public preview - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/azure-premium-blob-storage-public-preview/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Couple of updates from the Hue blog - &lt;a href=&quot;http://gethue.com/self-service-impala-sql-query-troubleshooting/&quot;&gt;Impala query explain plans and troubleshooting&lt;/a&gt; and &lt;a href=&quot;http://gethue.com/hue-in-docker/&quot;&gt;Hue in Docker&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And finally, from the Flink blog, how to integrate &lt;a href=&quot;/technologies/apache-flink&quot;&gt;Flink&lt;/a&gt; and Prometheus - &lt;a href=&quot;https://flink.apache.org/features/2019/03/11/prometheus-monitoring.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/03/13/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 06/03/2019</title><link>https://ondataengineering.github.io/blog/2019/03/06/the-mid-week-news/</link><pubDate>Wed, 06 Mar 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Ok, after the bumper two week edition last week, it’s slim pickings this week… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;None&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Datanami, can &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;Object Stores&lt;/a&gt; now compete with &lt;a href=&quot;/technologies/apache-hadoop/hdfs/&quot;&gt;HDFS&lt;/a&gt; for analytic workloads - &lt;a href=&quot;https://www.lightbend.com/blog/a-glimpse-at-the-future-of-apache-spark-30-with-deep-learning-and-kubernetes&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Following on from recent &lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; support for cloud object storage, they have an article on deploying Greenplum in the cloud - &lt;a href=&quot;https://greenplum.org/deploying-in-the-cloud/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon have a custom &lt;a href=&quot;/technologies/apache-spark&quot;&gt;Spark&lt;/a&gt; output committer that’s optimised for working with &lt;a href=&quot;/technologies/amazon-s3/&quot;&gt;Amazon S3&lt;/a&gt; storage - &lt;a href=&quot;https://aws.amazon.com/blogs/big-data/improve-apache-spark-write-performance-on-apache-parquet-formats-with-the-emrfs-s3-optimized-committer/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;If you’re an Ambari user (for example you’re running HDP), here’s how to use &lt;a href=&quot;/technologies/hue/&quot;&gt;Hue&lt;/a&gt; with your cluster - &lt;a href=&quot;http://gethue.com/configure-ambari-hdp-with-hue/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - is stronger data protections coming to the US following GDPR in the EU - &lt;a href=&quot;https://www.datanami.com/2019/02/28/surviving-the-coming-data-governance-wave&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via DZone, a video presentatoin of thoughts of what might be in &lt;a href=&quot;/technologies/apache-spark&quot;&gt;Spark&lt;/a&gt; 3.0 - &lt;a href=&quot;https://www.lightbend.com/blog/a-glimpse-at-the-future-of-apache-spark-30-with-deep-learning-and-kubernetes&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And finally, from the Apache Incubator, there’s a new proposal for DataSketches, a project from Yahoo for providing highly efficient approximations with mathematical guarentees to expensive complex computations on streaming data - &lt;a href=&quot;https://wiki.apache.org/incubator/DataSketchesProposal&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/03/06/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 27/02/2019</title><link>https://ondataengineering.github.io/blog/2019/02/27/the-mid-week-news/</link><pubDate>Wed, 27 Feb 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Right, we’ve back from our short break, so let’s catch up on the news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-nifi&quot;&gt;NiFi&lt;/a&gt; has hit 1.9&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt; has new features&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; 7.0 beta 1 is now out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; is up to 5.16&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; 2.0 alpha 4 is out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Azure have new announcements relating to their Data Services, including Azure Data Explorer (now added to our &lt;a href=&quot;/tech-vendors/microsoft-azure&quot;&gt;Azure&lt;/a&gt; vendor page) - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/individually-great-collectively-unmatched-announcing-updates-to-3-great-azure-data-services/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Forrester have a new Cloud Hadoop/Spark Platforms Wave out (now included in our &lt;a href=&quot;/tech-categories/hadoop-distributions/&quot;&gt;Hadoop Distributions&lt;/a&gt; page), and Cloudera are crowing about their position - &lt;a href=&quot;https://www.forrester.com/report/The+Forrester+Wave+Cloud+HadoopSpark+Platforms+Q1+2019/-/E-RES142663&quot;&gt;Forrester&lt;/a&gt;; &lt;a href=&quot;http://vision.cloudera.com/clouderas-and-hortonworks-data-platform-in-the-cloud-named-among-leaders-in-new-forrester-wave/&quot;&gt;Cloudera&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; blog - Blink, a fork from Alibaba that improves batch processing performance that’s being folded back in - &lt;a href=&quot;https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More from the &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; blog - best practices in monitoring - &lt;a href=&quot;https://flink.apache.org/news/2019/02/25/monitoring-best-practices.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s some stuff doing the rounds on the success of &lt;a href=&quot;/technologies/apache-arrow/&quot;&gt;Apache Arrow&lt;/a&gt; - &lt;a href=&quot;https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces46&quot;&gt;Apache&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/apache-arrow-the-little-data-accelerator-that-could/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;DB-Engines (the great database ranking site), has updated the way it handles multi-model databases - &lt;a href=&quot;https://db-engines.com/en/blog_post/80&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From InfoQ, a presentation from WePay on their use of Debezium to stream MySQL database changes into Google BigQuery - &lt;a href=&quot;https://www.infoq.com/presentations/wepay-database-streaming&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon have been doing some work so that Spark better handles node loss - &lt;a href=&quot;https://aws.amazon.com/blogs/big-data/spark-enhancements-for-elasticity-and-resiliency-on-amazon-emr/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;LinkedIn have a writeup from their community event on the future of Hadoop - &lt;a href=&quot;https://engineering.linkedin.com/blog/2019/02/the-present-and-future-of-apache-hadoop--a-community-meetup-at-l&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Qlik have aquired Attunity - &lt;a href=&quot;https://www.zdnet.com/article/qlik-to-acquire-attunity-for-560m/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Google have acquired Alooma - &lt;a href=&quot;https://cloud.google.com/blog/topics/inside-google-cloud/google-announces-intent-to-acquire-alooma-to-simplify-cloud-migration&quot;&gt;Google&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/02/21/google-doubles-down-on-cloud-data-migration/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.theregister.co.uk/2019/02/20/google_buys_alooma/&quot;&gt;The Register&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Uber have open sourced AresDB, a “GPU-Powered Open Source, Real-time Analytics Engine”, now included in out &lt;a href=&quot;/tech-categories/analytical-databases/&quot;&gt;Analytical Databases&lt;/a&gt; list - &lt;a href=&quot;https://eng.uber.com/aresdb/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Solr has a CVE advisory out - &lt;a href=&quot;http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2017-3164&quot;&gt;CVE-2017-3164&lt;/a&gt; - SSRF issue in Apache Solr&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/02/27/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 13/02/2019</title><link>https://ondataengineering.github.io/blog/2019/02/13/the-mid-week-news/</link><pubDate>Wed, 13 Feb 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;I’m off next week, so no news then, and when I’m back we’ll see if we can kick start some more content… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; has hit 2.10&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; has hit 7.7&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudbreak/&quot;&gt;Cloudbreak&lt;/a&gt; is up to 2.9&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; is up to R55 on AWS&lt;/li&gt; &lt;li&gt;Hortonworks &lt;a href=&quot;/technologies/hortonworks-dataplane/streams-messaging-manager/&quot;&gt;Streams Messaging Manager&lt;/a&gt; is up to 1.2&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; has it’s 2.0 alpha release out&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/pravega/&quot;&gt;Pravega&lt;/a&gt; is up to 0.4&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;LinkedIn have move on Cruise Control, their open source product for managing &lt;a href=&quot;/technologies/apache-kafka&quot;&gt;Kafka&lt;/a&gt; clusters - &lt;a href=&quot;https://engineering.linkedin.com/blog/2019/02/introducing-kafka-cruise-control-frontend&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Something interested from Forrester on Data Governance, and where this might be going - &lt;a href=&quot;https://go.forrester.com/blogs/data_governance_takes_a_turn_toward_ambient/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;We should have a look at this at some point - Datanami have an article on Lentiq EdgeLake, a fully federated lake product/architecture - &lt;a href=&quot;https://www.datanami.com/2019/02/07/lentiq-launches-edgelake-with-some-fanfare/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Qubole have an update on Rubix, their open source distributed object store cache, on using it with Spark - &lt;a href=&quot;https://www.qubole.com/blog/increase-apache-spark-performance-with-rubix-distributed-cache/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From the ever excellent Juku.IT - how to manage your data in a multi-cloud strategy - &lt;a href=&quot;https://www.juku.it/isnt-it-time-to-rethink-your-cloud-strategy/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;At the risk of propagating Cloudera adverts - their view on why Gartner have positioned them furthest for “Completeness of Vision” in the recent Magic Quadrant for Data Management Solutions for Analytics - &lt;a href=&quot;http://vision.cloudera.com/three-takeaways-from-gartners-2019-magic-quadrant-for-data-management-solutions-for-analytics/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/02/13/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 06/02/2019</title><link>https://ondataengineering.github.io/blog/2019/02/06/the-mid-week-news/</link><pubDate>Wed, 06 Feb 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Another week rolls round, so let’s do the news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-data-science-workbench/&quot;&gt;Cloudera Data Science Workbench&lt;/a&gt; is up to 1.5&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; has hit 6.5, along with &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch Hadoop&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; is up to 2.1&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-accumulo/&quot;&gt;Apache Accumulo&lt;/a&gt; has is second 2.0 alpha release out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;As well as the previous announcements on the single Hadoop product line, Cloudera have now announced Cloudera DataFlow (CDF) - the future of HDF in the new Cloudera - &lt;a href=&quot;http://vision.cloudera.com/cloudera-dataflow/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt; now has an open source foundation behind it, the Presto Software Foundation - &lt;a href=&quot;https://www.starburstdata.com/technical-blog/the-presto-software-foundation/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-airflow&quot;&gt;Apache Airflow&lt;/a&gt; has now graduated from the Apache Incubator - &lt;a href=&quot;https://wiki.apache.org/incubator/January2019&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/databricks-delta/&quot;&gt;Databricks Delta&lt;/a&gt; now supports querying data as it was any any point in history - &lt;a href=&quot;https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Confluent have (another) post on why event driven architectures are the future - &lt;a href=&quot;https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Google have announced their own Kubernetes integration for &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt; for Google Cloud Platform - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/data-analytics-meet-containers-kubernetes-operator-for-apache-spark-now-in-beta&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Solutions Review have their analysis of the latest Gartner Magic Quadrant for Data Management Solutions for Analytics - &lt;a href=&quot;https://solutionsreview.com/data-management/whats-changed-2019-gartner-magic-quadrant-for-data-management-solutions-for-analytics/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Starburst, their thoughts on the separation of compute and storage for analytics - &lt;a href=&quot;https://www.starburstdata.com/technical-blog/art-of-abstraction/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Register, Databricks has another £250m in funding, including from Microsoft - &lt;a href=&quot;https://www.theregister.co.uk/2019/02/05/databricks_series_e_250m/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Bloor have updated their Graph Database Market Update - &lt;a href=&quot;https://www.bloorresearch.com/technology/graph-databases/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;GridGain have announced that they now have a support offering for &lt;a href=&quot;/technologies/apache-ignite/&quot;&gt;Apache Ignite&lt;/a&gt; alongside their GridGain offerings - &lt;a href=&quot;https://www.gridgain.com/resources/blog/gridgain-introduces-first-support-offering-apacher-ignitetm-users&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/02/06/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 30/01/2019</title><link>https://ondataengineering.github.io/blog/2019/01/30/the-mid-week-news/</link><pubDate>Wed, 30 Jan 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;News time again, and it’s a quiet week… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Apache Hadoop&lt;/a&gt; has hit 3.2&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; has a 2.0 alpha release out&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From the ever excellent The Morning Papers - offloading MPP query processing to the network switch - &lt;a href=&quot;https://blog.acolyer.org/2019/01/28/the-case-for-network-accelerated-query-processing/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Confluent has raised another $125m - &lt;a href=&quot;https://www.confluent.io/blog/confluent-raises-a-125m-series-d-funding-round&quot;&gt;blog&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And on the subject on Confluent, ZDNet have a good piece on them - &lt;a href=&quot;https://www.zdnet.com/article/confluent-shows-open-source-paradigm-shifts-cloud-and-commercial-success-can-all-co-exist/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Register, the number of malicious attacks on Hadoop clusters are on the rise - &lt;a href=&quot;https://www.theregister.co.uk/2019/01/24/hadoop_malware_attack/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And from Bloor, their latest paper on comparative costs of different Data Integration platforms based on customer surveys (requires payment for commercial use) - &lt;a href=&quot;https://www.bloorresearch.com/research/comparative-costs-and-uses-for-data-integration-platforms-4th-edition/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/01/30/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 23/01/2019</title><link>https://ondataengineering.github.io/blog/2019/01/23/the-mid-week-news/</link><pubDate>Wed, 23 Jan 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Right, time for the news again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-arrow/&quot;&gt;Apache Arrow&lt;/a&gt; has hit 0.12&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.16&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hudi/&quot;&gt;Hudi&lt;/a&gt;, which we looked at recently, and was only submitted to the Apache Incubator two weeks ago has now entered incubation - &lt;a href=&quot;http://incubator.apache.org/projects/hudi.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;I’d not considered this before, but Alluxio have a post on using &lt;a href=&quot;/technologies/alluxio/&quot;&gt;Alluxio/Tachyon&lt;/a&gt; (but would equally apply to other in memory &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems&quot;&gt;Hadoop Compatible Filesystems&lt;/a&gt;) to reduce the performance impact of using object stores or remote (cloud) storage - &lt;a href=&quot;https://www.alluxio.com/blog/deploying-big-data-workloads-on-object-storage-without-performance-penalty&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From from Datanami on 2019 Big Data trends - &lt;a href=&quot;https://www.datanami.com/2019/01/21/10-big-data-trends-to-watch-in-2019/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Bloor have their latest &lt;a href=&quot;/tech-categories/graph-databases/&quot;&gt;Graph&lt;/a&gt; and &lt;a href=&quot;/tech-categories/rdf-databases/&quot;&gt;RDF&lt;/a&gt; database market update out -&lt;a href=&quot;https://www.bloorresearch.com/research/graph-database-market-update-2019/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet, looks like MariaDB now have a unified product supporting both AX analytics and TX transactional nodes under a common database engine - &lt;a href=&quot;https://www.zdnet.com/article/mariadb-unifies-its-platform/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Register, an update on Teradata, their strategy and new CEO - &lt;a href=&quot;https://www.theregister.co.uk/2019/01/16/teradata_new_ceo/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From The Morning Papers - SageDB, a database “where learned models pervade every aspect of a database system” - &lt;a href=&quot;https://blog.acolyer.org/2019/01/16/sagedb-a-learned-database-system/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Confluent - testing &lt;a href=&quot;/technologies/apache-kafka/kafka-streams/&quot;&gt;Kafka Streams&lt;/a&gt; - &lt;a href=&quot;https://www.confluent.io/blog/stream-processing-part-2-testing-your-streaming-application&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/01/23/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 16/01/2019</title><link>https://ondataengineering.github.io/blog/2019/01/16/the-mid-week-news/</link><pubDate>Wed, 16 Jan 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;We’re a week into our new year, and it’s time for the news again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-flume/&quot;&gt;Apache Flume&lt;/a&gt; is up to 1.9 (over a year since it’s last release)&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kylin/&quot;&gt;Apache Kylin&lt;/a&gt; is up to 2.6&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;StreamSets Data Collector&lt;/a&gt; is up to 3.7&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Datanami have the lowdown on the new &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; roadmap &lt;a href=&quot;https://www.datanami.com/2019/01/10/cloudera-unveils-cdp-talks-up-enterprise-data-cloud/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/tech-vendors/amazon-web-services/&quot;&gt;Amazon Web Services&lt;/a&gt; has a new NoSQL Document DB service compatible with MongoDB - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-documentdb-with-mongodb-compatibility-generally-available/&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/blogs/aws/new-amazon-documentdb-with-mongodb-compatibility-fast-scalable-and-highly-available/&quot;&gt;blog&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Parquet performance with Hive has got a boost from &lt;a href=&quot;/technologies/cloudera-cdh&quot;&gt;CDH&lt;/a&gt; 6.0 with support for vectorisation - &lt;a href=&quot;https://blog.cloudera.com/blog/2018/12/faster-swarms-of-data-accelerating-hive-queries-with-parquet-vectorization/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Hortonworks are continuing to push federation capabilities in &lt;a href=&quot;/technologies/apache-hive&quot;&gt;Hive&lt;/a&gt; - &lt;a href=&quot;https://hortonworks.com/blog/query-federation-with-hive/&quot;&gt;JDBC federation with push down&lt;/a&gt;; &lt;a href=&quot;https://hortonworks.com/blog/introducing-hive-kafka-sql/&quot;&gt;Kafka federation&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And Hortonworks are also talking about their better &lt;a href=&quot;/technologies/apache-hive&quot;&gt;Hive&lt;/a&gt; and &lt;a href=&quot;/technologies/apache-spark&quot;&gt;Spark&lt;/a&gt; integration via their Apache Hive Warehouse Connector - &lt;a href=&quot;https://hortonworks.com/blog/hive-warehouse-connector-use-cases/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/01/16/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 09/01/2019</title><link>https://ondataengineering.github.io/blog/2019/01/09/the-mid-week-news/</link><pubDate>Wed, 09 Jan 2019 07:30:00 +0000</pubDate> <description> &lt;p&gt;Hello - and welcome back. I hope you’ve had a good break. Let’s see what’s happened whilst we’ve been gone… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cloudera have released v6.1 of &lt;a href=&quot;/technologies/cloudera-cdh/&quot;&gt;CDH&lt;/a&gt;, &lt;a href=&quot;/technologies/cloudera-manager/&quot;&gt;Cloudera Manager&lt;/a&gt;, &lt;a href=&quot;/technologies/cloudera-navigator/&quot;&gt;Cloudera Navigator&lt;/a&gt; and &lt;a href=&quot;/technologies/cloudera-altus/director/&quot;&gt;Cloudera Altus Director&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-calcite/&quot;&gt;Apache Calcite&lt;/a&gt; has hit 1.18&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-drill/&quot;&gt;Apache Drill&lt;/a&gt; has hit 1.15&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-knox/&quot;&gt;Apache Knox&lt;/a&gt; has hit 1.2&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; is up to 5.20&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-dataplane/data-analytics-studio/&quot;&gt;Hortonworks Data Analytics Studio&lt;/a&gt; has hit 1.2&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/opentsdb/&quot;&gt;OpenTSDB&lt;/a&gt; has hit 2.4&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; has a 7.0-alpha2 release out if you’re brave/interested&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-datafu/&quot;&gt;Apache DataFu&lt;/a&gt;] is up to 1.5&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Alibaba has acquired Data Artisans, the backers of &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; - &lt;a href=&quot;https://www.datanami.com/2019/01/08/alibaba-acquires-apache-flink-backer-data-artisans&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;The &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt; + &lt;a href=&quot;/tech-vendors/hortonworks/&quot;&gt;Hortonworks&lt;/a&gt; merger is now complete - &lt;a href=&quot;http://vision.cloudera.com/the-new-cloudera/&quot;&gt;Cloudera&lt;/a&gt;; &lt;a href=&quot;https://hortonworks.com/blog/welcome-brand-new-cloudera/&quot;&gt;Hortonworks&lt;/a&gt;; &lt;a href=&quot;https://www.theregister.co.uk/2019/01/07/cloudera_hortonworks_merger_completed/&quot;&gt;The Register&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/cloudera-and-hortonworks-merger-closes-quo-vadis-big-data/&quot;&gt;ZDNet&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/01/03/merger-with-hortonworks-complete-cloudera-looks-to-future/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;It’s a new year, so everyone has their review of 2018 and look forward to 2019 articles out - &lt;a href=&quot;https://www.zdnet.com/article/data-crystal-balls-looking-glasses-and-boiling-frogs-reviewing-2018-predicting-2019/&quot;&gt;ZDNet&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/big-data-2019-cloud-redefines-the-database-and-machine-learning-runs-it/&quot;&gt;ZDNet again&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/predictions-for-2019-in-data-analytics-and-ai/&quot;&gt;And again ZDNet&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2019/01/02/industry-speaks-big-data-prognostications-for-2019/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hudi/&quot;&gt;Hudi&lt;/a&gt;, which we looked at recently, has just been submitted to the Apache Incubator - &lt;a href=&quot;https://wiki.apache.org/incubator/HudiProposal&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Cloudera has a new edition of &lt;a href=&quot;/technologies/cloudera-data-science-workbench/&quot;&gt;Cloudera Data Science Workbench&lt;/a&gt; coming that runs over Kubernetes (instead of Hadoop) as is targetted at Machine Learning in the cloud called Cloudera Machine Learning - &lt;a href=&quot;https://www.cloudera.com/about/news-and-blogs/press-releases/2018-12-05-cloudera-announces-preview-of-cloud-native-machine-learning-platform-to-accelerate-the-industrialization-of-ai.html&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/cloudera-machine-learning-release-takes-cloud-native-path/&quot;&gt;ZDNet view&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s more on the move to more restrictive open source licences to combat the exploitation of open source technologies by cloud providers - &lt;a href=&quot;https://www.datanami.com/2018/12/24/cloud-backlash-grows-as-open-source-gets-less-open/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://www.influxdata.com/blog/copyleft-and-community-licenses-are-not-without-merit-but-they-are-a-dead-end/&quot;&gt;Influx&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Via DBEngines - a proposal for GQL, a new standard graph query langage - &lt;a href=&quot;https://db-engines.com/en/blog_post/78&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-airflow&quot;&gt;Apache Airflow&lt;/a&gt; is now a top level project - &lt;a href=&quot;https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces44&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Apache Gearpump (a streaming service based on a micro-service Actor model) which was incubating has now been retired - &lt;a href=&quot;http://incubator.apache.org/projects/gearpump&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2019/01/09/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 19/12/2018</title><link>https://ondataengineering.github.io/blog/2018/12/19/the-mid-week-news/</link><pubDate>Wed, 19 Dec 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;Right - the last news before Christmas. We’ll be back in the new year - hope you all have a lovely holiday (whatever you’re doing)… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-data-platform/&quot;&gt;Hortonworks Data Platform&lt;/a&gt; has hit 3.1&lt;/li&gt; &lt;li&gt;Hortonworks &lt;a href=&quot;/technologies/hortonworks-dataplane/data-lifecycle-manager/&quot;&gt;Data Lifecycle Manager&lt;/a&gt; and &lt;a href=&quot;/technologies/hortonworks-dataplane/data-steward-studio/&quot;&gt;Data Steward Studio&lt;/a&gt; have both hit 1.3&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; has hit 2.9&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-solr/&quot;&gt;Apache Solr&lt;/a&gt; has hit 7.6&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elastic-cloud/&quot;&gt;Elastic Cloud&lt;/a&gt; has hit 2.0&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.15&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;It’s in the HDP 3.1 and HDF 3.3 release links, but we’ll call this out separately. As part of these releases, you can now run Hive and Druid queries over Kafka queues - &lt;a href=&quot;https://hortonworks.com/blog/democratizing-analytics-within-kafka-three-new-access-patterns/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And again from Hortonworks - when to use HBase vs Hive vs Druid - &lt;a href=&quot;https://hortonworks.com/blog/big-data-processing-engines-which-one-do-i-use-part-1/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And one more from Hortonworks - &lt;a href=&quot;/technologies/apache-hive&quot;&gt;Hive&lt;/a&gt; (LLAP + Tez) is now twice as fast as part of HDP 3.0 due to support for materialized views, SQL constraints, query result caching and vectorization - &lt;a href=&quot;https://hortonworks.com/blog/2x-faster-bi-interactive-queries-with-hdp-3-0/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Under the covers with &lt;a href=&quot;/technologies/mapr-converged-data-platform/&quot;&gt;MapR Converged Data Platform&lt;/a&gt;apR - &lt;a href=&quot;https://mapr.com/blog/mapr-data-platform-under-the-hood/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet - the rise of Kubernetes for Big Data, and thoughts on Ozone and the Hortonworks Open Data Initiative - &lt;a href=&quot;https://www.zdnet.com/article/the-rise-of-kubernetes-epitomizes-the-move-from-big-data-to-flexible-data/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera, tracking and assesing &lt;a href=&quot;/technologies/apache-impala/&quot;&gt;Apache Impala&lt;/a&gt; performance - &lt;a href=&quot;https://blog.cloudera.com/blog/2018/12/assessment-of-apache-impala-performance-using-cloudera-manager-metrics-part-1-of-3/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;GraphIt - a new DSL for specifying graph algorithms that delivers huge performance increases - &lt;a href=&quot;https://www.datanami.com/2018/12/10/graphit-promises-big-speedup-in-graph-processing/&quot;&gt;Datanami&lt;/a&gt;; &lt;a href=&quot;https://blog.acolyer.org/2018/12/12/graphit-a-high-performance-graph-dsl/&quot;&gt;The Morning Paper&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Juku.IT - thoughts on Amazon Output - their new hybrid cloud - &lt;a href=&quot;https://www.juku.it/the-cloud-is-more-hybrid-now/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;This is worth reading from InfoQ - will Cloud Computing Kill Open Source - does the use of open source software by the big cloud providers without contributing back destroy the commercial models for investing in open source - &lt;a href=&quot;https://www.infoq.com/articles/will-cloud-computing-kill-open-source&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And on the same note, Confluence have updated the licence for their open source &lt;a href=&quot;/technologies/confluent-open-source/&quot;&gt;Confluent Open Source&lt;/a&gt; components to prevent provisioning of them as a service - &lt;a href=&quot;https://www.confluent.io/blog/license-changes-confluent-platform&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From ZDNet - IBM have a new multi-cloud strategy - &lt;a href=&quot;https://www.zdnet.com/article/ibm-bets-on-a-multi-cloud-future/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Last one from Hortonworks today - mounting storage inside YARN containers using CNCF Container Storage Interface (csi) drivers - &lt;a href=&quot;https://hortonworks.com/blog/open-hybrid-architecture-running-stateful-containers-on-yarn/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-arrow&quot;&gt;Arrow&lt;/a&gt; has a new expression compiler and execution kernel - &lt;a href=&quot;https://arrow.apache.org/blog/2018/12/05/gandiva-donation/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;YugaByte DB (a multi-model db), and running Presto queries over it - &lt;a href=&quot;https://blog.yugabyte.com/presto-on-yugabyte-db-interactive-olap-sql-queries-made-easy-facebook/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/12/19/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 12/12/2018</title><link>https://ondataengineering.github.io/blog/2018/12/12/the-mid-week-news/</link><pubDate>Wed, 12 Dec 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;Apologies for the radio silence - I’ve been ill. But to make up for it, it’s a monster bumper news this week… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-bigtop/&quot;&gt;Apache BigTop&lt;/a&gt; has hit 1.3&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; has hit 1.7&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-gobblin/&quot;&gt;Apache Gobblin&lt;/a&gt; has hit 0.14&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-ignite/&quot;&gt;Apache Ignite&lt;/a&gt; has hit 2.7&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-impala/&quot;&gt;Apache Impala&lt;/a&gt; is up to 3.1&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt; has hit 2.1&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-parquet/&quot;&gt;Apache Parquet&lt;/a&gt; has hit a 1.11 release of it’s Map Reduce implementation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/cloudera-cdh/&quot;&gt;CDH&lt;/a&gt; and &lt;a href=&quot;/technologies/cloudera-manager/&quot;&gt;Cloudera Manager&lt;/a&gt; is up to 5.16; &lt;a href=&quot;/technologies/cloudera-navigator/&quot;&gt;Cloudera Navigator&lt;/a&gt; to 2.15&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; has hit 6.5, along with &lt;a href=&quot;/technologies/elasticsearch-hadoop/&quot;&gt;Elasticsearch Hadoop&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.14&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-data-flow/&quot;&gt;Hortonworks Data Flow&lt;/a&gt; is up to 3.3&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole&lt;/a&gt; has hit R54&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;Streamsets Data Collector&lt;/a&gt; has hit 3.6&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-hadoop/ozone/&quot;&gt;Apache Hadoop Ozone&lt;/a&gt; is up to Ozone 0.3 alpha&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Some Apache Incubator updates &lt;ul&gt; &lt;li&gt;Quickstep (a high performance database engine) has been retired from the incubator due to lack of activity&lt;/li&gt; &lt;li&gt;Iceberg (the file based table store) from Netflix and IoTDB (a time series db) have both been accepted into the Incubator&lt;/li&gt; &lt;li&gt;Griffen (the Data Quality Service platform built on Apache Hadoop and Apache Spark) has graduated&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;Hortonworks have a blog post on Ambari 2.7 and why it’s great (months after it’s release) - &lt;a href=&quot;https://hortonworks.com/blog/whats-great-apache-ambari-2-7/&quot;&gt;https://hortonworks.com/blog/whats-great-apache-ambari-2-7/&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/elasticsearch/&quot;&gt;Elasticsearch&lt;/a&gt; now supports Kubernetes deployments, with Helm charts available from Elastic - &lt;a href=&quot;https://www.elastic.co/blog/alpha-helm-charts-for-elasticsearch-kibana-and-cncf-membership&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Hortonworks have more on their cloud journey for &lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Hadoop&lt;/a&gt; and &lt;a href=&quot;/technologies/hortonworks-data-platform/&quot;&gt;HDP&lt;/a&gt; - &lt;a href=&quot;https://hortonworks.com/blog/open-hybrid-architecture-why-participating-in-the-cloud-native-computing-foundation/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Microsoft have more features announced for &lt;a href=&quot;/technologies/microsoft-azure-data-lake-store/&quot;&gt;Azure Data Lake Store&lt;/a&gt; Gen2 - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/azure-data-lake-storage-gen2-preview-more-features-more-performance-better-availability/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Also from Microsoft, a comprehensive guide for the Information they’ve published on &lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;HDInsight&lt;/a&gt; - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/get-up-to-speed-with-azure-hdinsight-the-comprehensive-guide/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon have a monster pile of AWS announcements: &lt;ul&gt; &lt;li&gt;For &lt;a href=&quot;/technologies/amazon-s3/&quot;&gt;Amazon S3&lt;/a&gt; we have &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/s3-glacier-api-simplification/&quot;&gt;full Glacier integration&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/s3-object-lock/&quot;&gt;object lock&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/s3-glacier-deep-archive/&quot;&gt;Glacier Deep Archive&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-operations/&quot;&gt;batch operations&lt;/a&gt;; &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-s3-block-public-access/&quot;&gt;blocking public access&lt;/a&gt; and &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/s3-intelligent-tiering/&quot;&gt;Intelligence Tiering&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon Textract is a new OCR service - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-textract-now-in-preview-easily-extract-text-and-data-from-virtually-any-document/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon Lake Formation is a new service for setting up an S3 data lake, populating it with data from a range of source systems, and then securing the data - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/announcing-aws-lake-formation/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon Timestream is a new Time Series database - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/announcing-amazon-timestream/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;You have now pause and resume EC2 instances backed by EBS - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/amazon-ec2-now-lets-you-pause-and-resume-your-workloads/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s a new Kafka managed service (Managed Streaming for Kafka - MSK) - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-managed-streaming-for-kafka-in-public-preview/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And you can now get Lustre filesystems as a service on AWS - &lt;a href=&quot;https://aws.amazon.com/about-aws/whats-new/2018/11/amazon-fsx-lustre/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;And from Azure this week, you can now get MariaDB as a manged service - &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/announcing-the-general-availability-of-azure-database-for-mariadb/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;RedHat have acquired NooBaa - an object storage solution - &lt;a href=&quot;https://www.juku.it/red-hat-acquires-noobaa-beefing-up-its-storage-portfolio/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More from Hortonworks on Ozone - &lt;a href=&quot;https://hortonworks.com/blog/open-hybrid-architecture-o3-the-new-rocket-ship/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - an article on Pachydern, an interesting alternative to Hadoop - &lt;a href=&quot;https://www.datanami.com/2018/11/20/inside-pachyderm-a-containerized-alternative-to-hadoop/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Samza 1.0 is out - &lt;a href=&quot;https://engineering.linkedin.com/blog/2018/11/samza-1-0--stream-processing-at-massive-scale&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.zdnet.com/article/real-time-data-processing-just-got-more-options-linkedin-releases-apache-samza-1-0-streaming/&quot;&gt;ZDNet&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - looks like Cloudera have a new ML platform coming - &lt;a href=&quot;https://www.datanami.com/2018/12/05/cloudera-gives-a-peek-at-future-ml-platform/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/12/12/the-mid-week-news/</guid> </item> <item><title>Hudi</title><link>https://ondataengineering.github.io/technologies/hudi/</link><pubDate>Thu, 15 Nov 2018 00:00:00 +0000</pubDate> <description> &lt;p&gt;Spark library for managing tabular structured data on Hadoop that supports atomic transactions, near real time ingestion and quering, incremental reading of data for further processing and upserts, updates &amp; deletes. Data is stored in HDFS, with a folder for each table partition, and with data files chunked by Hadoop block size (with each chunk allocated a unique fileid). Supports two storage mechanisms - Copy on Write (maintains a data as a Parquet file for each chunk that&apos;s re-written for updates and deletes) and Merge on Read (also maintains data as a Parquet file, however new data for a chunk is written to an Avro delta file, with an async background compaction process to merge all new delta files into the Parquet file for a chunk). Data is queryable via Hive, Presto and SparkSQL via a custom InputReaders through three views - Read Optimised (only queries Parquet files), Real Time (queries both Parquet and Avro delta files, merging in the deltas at query time) and Incremental (only reads Avro files to provide new data since a given commit). Supports strongly consistent atomic transactional commits (with a commit log (the timeline) used to prevent data from being queried until it is commmitted, and with support for automatically rolling back failed commits and the ability to manually rollback specific commits) and read isolation (all data filenames include the commit id meaning data files are never modified once committed, with a cleanup process to remove old redudant files). Compactions are non blocking, lock free and asynchronous, with pluggable strategies for prioritising compactions. All records must have a unique key, with a key lookup (either via bloom filter of external HBase table) used to identify updates and identify which chunk that update should be applied to. Also pluggable to support alternative storage formats to Parquet and Avro if required. Spark APIs includes support for incremental reads, bulk inserts, upserts and Spark SQL, and includes integration with Hive and Presto (including a Hive Metadata sync tool that incrementally pushes table and partition metadata to the Hive metastore for Hive and Presto), a CLI, the ability to generate Graphite metrics and a number of utilities (including the ability to stream data from Kafka and Sqoop into Hudi). Created at Uber where it&apos;s used in production, and open sourced in December 2016. Name stands for Hadoop Upserts anD Incrementals.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Other Names&lt;/td&gt;&lt;td&gt;Hoodi&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Open Source&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;November 2018&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://uber.github.io/hudi/index.html&quot;&gt;https://uber.github.io/hudi/index.html&lt;/a&gt; - homepage and documentation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://github.com/uber/hudi&quot;&gt;https://github.com/uber/hudi&lt;/a&gt; - source code&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/70937&quot;&gt;https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/70937&lt;/a&gt;; &lt;a href=&quot;https://conferences.oreilly.com/strata/strata-ca-2017/public/schedule/detail/56511&quot;&gt;https://conferences.oreilly.com/strata/strata-ca-2017/public/schedule/detail/56511&lt;/a&gt; - intro presentations&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://eng.uber.com/hoodie/&quot;&gt;https://eng.uber.com/hoodie/&lt;/a&gt;; &lt;a href=&quot;https://eng.uber.com/uber-big-data-platform/&quot;&gt;https://eng.uber.com/uber-big-data-platform/&lt;/a&gt; - blog posts&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://eng.uber.com/tag/hoodie/&quot;&gt;https://eng.uber.com/tag/hoodie/&lt;/a&gt; - blog posts&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/hudi/</guid> </item> <item><title>The Mid Week News 14/11/2018</title><link>https://ondataengineering.github.io/blog/2018/11/14/the-mid-week-news/</link><pubDate>Wed, 14 Nov 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s Wednesday again, which means it’s time for the news again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.13&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt; has hit 2.4&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/influxdb/&quot;&gt;InfluxDB&lt;/a&gt; has hit 1.7&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt; has hit 5.18&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;MapR have announced their Clarity program aimed as disaffected Cloudera and Hortonworks clusters - &lt;a href=&quot;https://mapr.com/blog/the-mapr-clarity-program-is-your-clear-path-to-ai-hybrid-and-multi-cloud-containers-and-operational-analytics/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2018/11/07/mapr-targets-cloudera-hortonworks-customers-with-clarity-release/&quot;&gt;Datanami view&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Uber have a great article on the evolution of their 100+ Petabyte Big Data Platform - &lt;a href=&quot;https://eng.uber.com/uber-big-data-platform/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.infoq.com/news/2018/11/uber-big-data-evolution&quot;&gt;InfoQ view&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Airbnb have a post on how they use &lt;a href=&quot;/technologies/apache-druid/&quot;&gt;Apache Druid&lt;/a&gt; - &lt;a href=&quot;https://medium.com/airbnb-engineering/druid-airbnb-data-platform-601c312f2a4c&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Netflix have donated Iceberg - their on disk structured table store to the Apache foundation - &lt;a href=&quot;https://wiki.apache.org/incubator/IcebergProposal&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And on the subject of &lt;a href=&quot;/technologies/apache-druid/&quot;&gt;Druid&lt;/a&gt; - how to configure it to use Minio (and more generally any object store) for deep storage - &lt;a href=&quot;https://cleanprogrammer.net/how-to-configure-druid-to-use-minio-as-deep-storage/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;MapR have started a series on scale out multi-purpose data platforms (any why theirs is the best) - &lt;a href=&quot;https://mapr.com/blog/in-search-of-a-data-platform/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;RedHat have announced AMQ Streams - a Kafka distribution that runs on OpenShift - &lt;a href=&quot;https://www.redhat.com/en/blog/announcing-red-hat-amq-streams-apache-kafka-red-hat-openshift&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2018/11/12/red-hat-adds-kafka-streaming-to-openshift/&quot;&gt;Datanmi view&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt; now supports 200K partitions per cluster - &lt;a href=&quot;https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;First of a few from Datanami - a view on Cloud Warehouses - &lt;a href=&quot;https://www.datanami.com/2018/11/08/whats-driving-the-cloud-data-warehouse-explosion/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And also from Datanami - Talend have brought Stitch - &lt;a href=&quot;https://www.datanami.com/2018/11/07/21615/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Azure Event Hubs now has a Kafka compatible end point API - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/announcing-the-general-availability-of-azure-event-hubs-for-apache-kafka/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Azure SQL Data Warehouse has a bunch of new features out - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/azure-sql-data-warehouse-taking-scalability-security-and-manageability-to-new-heights/&quot;&gt;link&lt;/a&gt;; &lt;a href=&quot;https://azure.microsoft.com/en-gb/blog/row-level-security-is-now-supported-for-azure-sql-data-warehouse/&quot;&gt;row level security blog&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And we close with a couple of &lt;a href=&quot;/technologies/apache-hive/&quot;&gt;Apache Hive&lt;/a&gt; security announcements - &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-11777&quot;&gt;CVE-2018-11777&lt;/a&gt;: Blocking local resource access in HiveServer2 and &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-1314&quot;&gt;CVE-2018-1314&lt;/a&gt;: Hive explain query not being authorized&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/11/14/the-mid-week-news/</guid> </item> <item><title>Databricks Delta</title><link>https://ondataengineering.github.io/technologies/databricks-delta/</link><pubDate>Thu, 08 Nov 2018 00:00:00 +0000</pubDate> <description> &lt;p&gt;Storage layer for tabular structured data within the Databricks Unified Analytics Platform that supports ACID transactions and data skipping. Data is persisted to Amazon S3 or Azure Blob Storage as Parquet files with metadata stored in a Hive Metastore, and includes full integration with Spark Structured Streaming and Spark SQL. Supports batch appends, overwrites, updates, upserts and deletes and streaming appends or overwrites, with new data written as new delta files (with changes collapsed during reads) supported by a transaction log. Allows multiple writers able to simultaneously modify a dataset, and ensures readers are always presented with a consistent view through the use of snapshots. Includes support for a number of SQL management extensions, including viewing the transaction history (describe history), accessing previous versions of datafiles (by timestamp or version), collapsing delta files to improve performance (optimize) and removing old files left around to support snapshooted reads (vacuum). Supports performant reads through standard Hive partitioning (including support for partition pruning) and data skipping (reducing data read based on recorded min/max values for data files which can be enhanced by z ordering data). Also supports views over tables and backward compatible schema changes, including support for auto addition of new fields based on input data. Currently in preview, having been first announced in October 2018.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;April 2019&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;updates&quot;&gt;Updates&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;2019-02-27 - &lt;a href=&quot;https://databricks.com/blog/2019/02/19/new-databricks-delta-features-simplify-data-pipelines.html&quot;&gt;blog post&lt;/a&gt;&lt;/li&gt; &lt;li&gt;2019-04-09 - &lt;a href=&quot;https://databricks.com/blog/2019/04/04/announcing-databricks-runtime-5-3.html&quot;&gt;GA in Databricks Runtime 5.3&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://databricks.com/product/databricks-delta&quot;&gt;https://databricks.com/product/databricks-delta&lt;/a&gt; - homepage&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://docs.databricks.com/delta/index.html&quot;&gt;https://docs.databricks.com/delta/index.html&lt;/a&gt; - documentation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://databricks.com/blog/2017/10/25/databricks-delta-a-unified-management-system-for-real-time-big-data.html&quot;&gt;https://databricks.com/blog/2017/10/25/databricks-delta-a-unified-management-system-for-real-time-big-data.html&lt;/a&gt; - initial blog post&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://databricks.com/blog/2018/07/31/processing-petabytes-of-data-in-seconds-with-databricks-delta.html&quot;&gt;https://databricks.com/blog/2018/07/31/processing-petabytes-of-data-in-seconds-with-databricks-delta.html&lt;/a&gt; - data skipping and z-ordering blog post&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://databricks.com/blog/2018/07/19/simplify-streaming-stock-data-analysis-using-databricks-delta.html&quot;&gt;https://databricks.com/blog/2018/07/19/simplify-streaming-stock-data-analysis-using-databricks-delta.html&lt;/a&gt; - streaming analysis blog post&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://databricks.com/blog/category/company/product&quot;&gt;https://databricks.com/blog/category/company/product&lt;/a&gt; - Databricks product blog&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/databricks-delta/</guid> </item> <item><title>The Mid Week News 07/11/2018</title><link>https://ondataengineering.github.io/blog/2018/11/07/the-mid-week-news/</link><pubDate>Wed, 07 Nov 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;No technology updates this week, but a bunch of news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;None this week&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;From Hortonworks - &lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Hadoop&lt;/a&gt; is thriving! &lt;a href=&quot;https://hortonworks.com/blog/apache-hadoop-thriving/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And again from Hortonworks - more on their Open Hybrid Artitecture - &lt;a href=&quot;https://hortonworks.com/blog/open-hybrid-architecture-bringing-cloud-native-to-on-premises/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Cloudera, how to protect yourself from the recent Hadoop malware attacks (and how trivially easy this is with Altus obviously) - &lt;a href=&quot;http://blog.cloudera.com/blog/2018/11/protecting-hadoop-clusters-from-malware-attacks/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Google, an intro to &lt;a href=&quot;/technologies/google-cloud-dataproc/&quot;&gt;Cloud DataProc&lt;/a&gt; - their Hadoop as a service offering - &lt;a href=&quot;https://cloud.google.com/blog/products/data-analytics/run-apache-spark-and-apache-hadoop-workloads-with-flexibility-and-predictability-with-cloud-dataproc&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And again from Google - a comparison of &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;object stores&lt;/a&gt; and &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems/&quot;&gt;HDFS&lt;/a&gt; - &lt;a href=&quot;https://cloud.google.com/blog/products/storage-data-transfer/hdfs-vs-cloud-storage-pros-cons-and-migration-tips&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami - RockSet, a new SQL cloud service with automatic ingest and indexing of data, including streaming data, has emerged from stelth - &lt;a href=&quot;https://www.datanami.com/2018/11/01/rockset-sql-cloud-service-emerges-from-stealth/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And again from Datanami, there’s a new major version of Dremio out, and I’m hoping to do a tech summary on this shortly - &lt;a href=&quot;https://www.datanami.com/2018/10/30/dremio-fleshes-out-data-platform/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;I’m not sure where I came across this, but I’ve added LeoFS to our &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;object stores&lt;/a&gt; page - &lt;a href=&quot;https://leo-project.net/leofs/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Amazon Redshift is now 3.5x faster! &lt;a href=&quot;https://aws.amazon.com/blogs/big-data/performance-matters-amazon-redshift-is-now-up-to-3-5x-faster-for-real-world-workloads/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Databricks, you can now do pivots in &lt;a href=&quot;/technologies/apache-spark/spark-sql/&quot;&gt;Spark SQL&lt;/a&gt; - &lt;a href=&quot;https://databricks.com/blog/2018/11/01/sql-pivot-converting-rows-to-columns.html&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;DZone have a RefCard on temporal support in Oracle, SQL Server and MariaDB - &lt;a href=&quot;https://dzone.com/refcardz/temporal-data-processing&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/11/07/the-mid-week-news/</guid> </item> <item><title>Thoughts On Schema Registries</title><link>https://ondataengineering.github.io/blog/2018/11/02/thoughts-on-schema-registries/</link><pubDate>Fri, 02 Nov 2018 07:45:00 +0000</pubDate> <description> &lt;p&gt;So I want to spend the next couple of weeks thinking about how you store structured data in unstructured data stores (specifically &lt;a href=&quot;/tech-categories/streaming-data-stores/&quot;&gt;Streaming Data Stores&lt;/a&gt;, &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems/&quot;&gt;Hadoop Compatible File Systems&lt;/a&gt; and &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;Object Stores&lt;/a&gt;).&lt;/p&gt; &lt;p&gt;This week, we’re looking at &lt;a href=&quot;/tech-categories/streaming-data-stores/&quot;&gt;Streaming Data Stores&lt;/a&gt; &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;We need to look at this because messages in streaming data stores are just collections of bytes, with all serialisation and de-serialisation is done in the client, effectively meaning we’re doing schema on read/write. So the client needs to know what schema to use to serialise or de-serialise the data.&lt;/p&gt; &lt;p&gt;With batch data on disk you could use self describing data - i.e. you embed the schema with the data using something like &lt;a href=&quot;/technologies/apache-avro/&quot;&gt;Avro&lt;/a&gt;, however that doesn’t work for streaming data as it’s just not practical to embed the schema into every message, so your schema has to live somewhere else.&lt;/p&gt; &lt;p&gt;You could embed the schema into you code, and with a bit of care you can have a shared schema that multiple jobs use. However this doesn’t allow anything outside of your code base to re-use this schema - if you’re using third party tools that don’t integrate with your configuration management tool (e.g. NiFi or StreamSets), then you have no way of sharing that schema.&lt;/p&gt; &lt;p&gt;So the final option is to use a dedicated schema registry - a tool that will manage your schemas for you, and serve them up over an API based on the topic name that can be used by clients to fetch the schema when they want to read and write data. You will however need to make sure that the client or tool you’re using supports the schema registry. As an added bonus, they support schema compatibility checks, in that if you say you want your schemas to be forward or backward compatible (or both), it will validate that any changes to schemas conform to this.&lt;/p&gt; &lt;p&gt;And for your schema registry you basically have two options at the moment.&lt;/p&gt; &lt;p&gt;The first is the Confluent Schema Registry, part of the &lt;a href=&quot;/technologies/confluent-open-source/&quot;&gt;Confluent Open Source&lt;/a&gt; bundle of &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Apache Kafka&lt;/a&gt;. It supports Avro schemas, and is integrated into Kafka APIs, Kafka Connect, Kafka Streams, NiFi and StreamSets.&lt;/p&gt; &lt;p&gt;The second is a more recent addition, with Hortonworks’ open source &lt;a href=&quot;/technologies/schema-registry/&quot;&gt;Schema Registry&lt;/a&gt; tool. It does broadly the same thing, supporting Avro schema, with integration into the Kafka APIs plus NiFi and their Streaming Analytics Manager. And it’s designed to be more general purpose, with support for serving templates, machine learning models or business rules mooted for the future.&lt;/p&gt; &lt;p&gt;Next week we’ll take about schema management for data in &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems/&quot;&gt;Hadoop Compatible File Systems&lt;/a&gt; and &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;Object Stores&lt;/a&gt;) and the &lt;a href=&quot;/technologies/apache-hive/hive-metastore/&quot;&gt;Hive Metastore&lt;/a&gt;, which fulfils a similar function, but with some extra complications.&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/11/02/thoughts-on-schema-registries/</guid> </item> <item><title>Schema Registries</title><link>https://ondataengineering.github.io/tech-categories/schema-registries/</link><pubDate>Fri, 02 Nov 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;Our list of and information on schema registries, including the Hive Metastore, the Confluent and Hortonworks Schema Registries, and alternatives to these.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Category Definition&lt;/h2&gt; &lt;!-- Tech Vendor metadata --&gt; &lt;p&gt;Tools that support the definition, management and serving of &lt;a href=&quot;/tech-categories/data-storage-formats/&quot;&gt;Data Storage Format&lt;/a&gt; schemas for use in the serialisation and de-serialisation of data, primarily with &lt;a href=&quot;/tech-categories/streaming-data-stores/&quot;&gt;Streaming Data Stores&lt;/a&gt;. Will support an API (and often a web user interface) for managing and retrieving schemas, and will often support schema evolution checks (ensuring changes are forward or backward compatible or both), and an SDK that integrates with clients to allow structured data to be read and written directly. May also support serving of the libraries required to perform serialisation/de-serialisation, and high availability configurations. &lt;!--more--&gt;&lt;/p&gt; &lt;h2 id=&quot;open-source-streaming-data-schema-registries&quot;&gt;Open Source Streaming Data Schema Registries&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt; &lt;td&gt;&lt;a href=&quot;/technologies/confluent-open-source/&quot;&gt;Confluent Schema Registry&lt;/a&gt;&lt;/td&gt; &lt;td&gt;Central definition of schemas for reading and writing from/to Kafka topics, with support for a range of technologies (including the Kafka APIs, Kafka Connect, Kafka Streams, NiFi and StreamSets)&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;&lt;a href=&quot;/technologies/schema-registry/&quot;&gt;Hortonworks Schema Registry&lt;/a&gt;&lt;/td&gt; &lt;td&gt;Central definition of Avro schemas for use in NiFi, Kafka Producers/Consumers and Streaming Analytics Manager&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;Avro Schema Registry&lt;/td&gt; &lt;td&gt;Compatible with Confluent’s Schema Registry API, but re-implemented in Ruby backed by Postgres - &lt;a href=&quot;https://github.com/salsify/avro-schema-registry&quot;&gt;https://github.com/salsify/avro-schema-registry&lt;/a&gt;&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;Landoop Schema Registry UI&lt;/td&gt; &lt;td&gt;Web based user interface for the Confluent Schema Registry - &lt;a href=&quot;https://github.com/Landoop/schema-registry-ui&quot;&gt;https://github.com/Landoop/schema-registry-ui&lt;/a&gt;&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;streaming-data-schema-registry-alternatives&quot;&gt;Streaming Data Schema Registry Alternatives&lt;/h2&gt; &lt;p&gt;Schemas can of course be managed and maintained if your configuration management tool, however these will not be available outside of your code base (e.g. to third party tools such as NiFi or StreamSets).&lt;/p&gt; &lt;p&gt;Cloudera have a three part blog post on how to roll your own schema management tool for &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Kafka&lt;/a&gt; using a Kafka topic to store your schemas - &lt;a href=&quot;http://blog.cloudera.com/blog/2018/07/robust-message-serialization-in-apache-kafka-using-apache-avro-part-1/&quot;&gt;part1&lt;/a&gt;; &lt;a href=&quot;http://blog.cloudera.com/blog/2018/07/robust-message-serialization-in-apache-kafka-using-apache-avro-part-2/&quot;&gt;part2&lt;/a&gt;; &lt;a href=&quot;http://blog.cloudera.com/blog/2018/08/robust-message-serialization-in-apache-kafka-using-apache-avro-part-3/&quot;&gt;part3&lt;/a&gt;&lt;/p&gt; &lt;h2 id=&quot;hive-metastore&quot;&gt;Hive Metastore&lt;/h2&gt; &lt;p&gt;The &lt;a href=&quot;/technologies/apache-hive/hive-metastore/&quot;&gt;Hive Metastore&lt;/a&gt; fulfils a similar function for data stored in &lt;a href=&quot;/tech-categories/hadoop-compatible-filesystems/&quot;&gt;Hadoop Compatible File Systems&lt;/a&gt; and &lt;a href=&quot;/tech-categories/object-stores/&quot;&gt;Object Stores&lt;/a&gt;, however serves a wider range of table metadata (including how it’s structured on disk), and doesn’t include some features like schema lifecycle management.&lt;/p&gt; &lt;p&gt;&lt;a href=&quot;https://www.slideshare.net/Hadoop_Summit/sharing-metadata-across-the-data-lake-and-streams-103204119&quot;&gt;This presentation&lt;/a&gt; from Hortonworks describes their view of the future of the Hive Metastore, including it’s separation from Hive and integration with the schema registry.&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/tech-categories/schema-registries/</guid> </item> <item><title>The Mid Week News 31/10/2018</title><link>https://ondataengineering.github.io/blog/2018/10/31/the-mid-week-news/</link><pubDate>Wed, 31 Oct 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;ok - let’s do this week’s news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-nifi/&quot;&gt;Apache Nifi&lt;/a&gt; has hit 1.8&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-beam/&quot;&gt;Apache Beam&lt;/a&gt; has hit 2.8&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-kudu/&quot;&gt;Apache Kudu&lt;/a&gt; is up to 1.8&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Following on from our thinking on Kubernetes, Confluent have a post on running &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Kafka&lt;/a&gt; on Kubernetes - &lt;a href=&quot;https://www.confluent.io/blog/apache-kafka-kubernetes-could-you-should-you&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From InfoQ, an interview on &lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;HDSight&lt;/a&gt; - &lt;a href=&quot;https://www.infoq.com/news/2018/10/HDInsight-Chatterjee&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Dremio 3.0 is out and ZDNet have a write up - &lt;a href=&quot;https://www.zdnet.com/article/dremio-3-0-adds-catalog-containers-enterprise-features/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, it looks like there’s a bot out targeting unsecured &lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Hadoop&lt;/a&gt; clusters - &lt;a href=&quot;https://www.datanami.com/2018/10/29/latest-bot-targets-hadoop-clusters/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Still more on the Cloudera Hortonworks merger, this time Datanami looking at the product roadmap - &lt;a href=&quot;https://www.datanami.com/2018/10/24/new-cloudera-plots-a-course-toward-a-unified-future/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Pinot, the Open source realtime distributed OLAP datastore from LinkedIn, has been accepted into the Apache Incubator - &lt;a href=&quot;http://incubator.apache.org/projects/pinot.html&quot;&gt;http://incubator.apache.org/projects/pinot.html&lt;/a&gt;&lt;/li&gt; &lt;li&gt;IoTDB (a new massive scale IoT time series DB) and Sharding Sphere (middleware for distributed databases) have both been submitted to the Apache Incubator - &lt;a href=&quot;https://wiki.apache.org/incubator/IoTDBProposal&quot;&gt;IoTDB&lt;/a&gt;; &lt;a href=&quot;https://wiki.apache.org/incubator/ShardingSphereProposal&quot;&gt;ShardingSphere&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-impala/&quot;&gt;Apache Impala&lt;/a&gt; has two security announcements (fixed in 3.0.1) - &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-11785&quot;&gt;CVE-2018-11785&lt;/a&gt; and &lt;a href=&quot;https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-11792&quot;&gt;CVE-2018-11792&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More from Data Artisans on stateful processing with &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; - &lt;a href=&quot;https://data-artisans.com/blog/stateful-stream-processing-apache-flink-state-backends&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/10/31/the-mid-week-news/</guid> </item> <item><title>The Mid Week News 24/10/2018</title><link>https://ondataengineering.github.io/blog/2018/10/24/the-mid-week-news/</link><pubDate>Wed, 24 Oct 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s Wednesday, which means it’s time to catch up on the news… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-carbondata/&quot;&gt;Apache CarbonData&lt;/a&gt; is up to 1.5&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/greenplum/&quot;&gt;Greenplum&lt;/a&gt; has hit 5.12&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hortonworks-data-platform-search/&quot;&gt;HDP Search&lt;/a&gt; has a 4.0 release&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/hue/&quot;&gt;Hue&lt;/a&gt; has hit 4.3&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;There’s more views on the Cloudera - Hortonworks merger, this time with a more negative view, including a post from MapR brilliantly titled “Two Wrongs Don’t Make A Right” - &lt;a href=&quot;https://mapr.com/blog/two-wrongs-dont-make-a-right/&quot;&gt;MapR&lt;/a&gt;; &lt;a href=&quot;https://www.bloorresearch.com/2018/10/cloudera-and-hortonworks-what-do-i-think/&quot;&gt;Bloor Research&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2018/10/18/is-hadoop-officially-dead/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Teradata have been talking about their new strategy and there’s commentry - &lt;a href=&quot;https://www.zdnet.com/article/is-teradatas-march-to-cloud-and-commodity-its-destiny/&quot;&gt;ZDNet&lt;/a&gt;; &lt;a href=&quot;https://www.datanami.com/2018/10/17/inside-teradatas-audacious-plan-to-consolidate-analytics/&quot;&gt;Datanami&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And a view on SAP, this time from The Register - &lt;a href=&quot;https://www.theregister.co.uk/2018/10/18/sap_q3_reults/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;The Register have a summary of the latest Gartner Object and Distributed File Storage Magic Quadrant - &lt;a href=&quot;https://www.theregister.co.uk/2018/10/23/eight_suppliers_move_in_gartners_latest_object_mq/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;This was interesting - from Cloudera a post on the work they do to manage and rationalise third party library dependancies in Hadoop - &lt;a href=&quot;http://blog.cloudera.com/blog/2018/10/third-party-libraries-in-c6/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;More InfoQ, then best practices for deploying &lt;a href=&quot;/technologies/apache-kafka/&quot;&gt;Kafka&lt;/a&gt; - &lt;a href=&quot;https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;There’s a big post on the internals of &lt;a href=&quot;/technologies/pravega&quot;&gt;Pravega&lt;/a&gt;, the Kafka challenger - &lt;a href=&quot;http://blog.pravega.io/2018/10/17/pravega-internals/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;And &lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;HDInsight&lt;/a&gt; now supports caching of object storage using the RubiX framework open sourced by Qubole - &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/apache-spark-speedup-with-hdinsight-io-cache/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/10/24/the-mid-week-news/</guid> </item> <item><title>The Future of Hadoop...</title><link>https://ondataengineering.github.io/blog/2018/10/19/the-future-of-hadoop/</link><pubDate>Fri, 19 Oct 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;Ok - this is our last post on Hadoop for the time being, so let’s try and predict the future… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;We started talking about the future of Hadoop when we looked at Hortonworks’ Open Hybrid Architecture Initiative &lt;a href=&quot;/blog/2018/09/21/thoughts-on-hortonworks-open-hybrid-architecture-initiative/&quot;&gt;a few weeks ago&lt;/a&gt;. In that we talked about the current positioning of Hadoop as a multi-purpose storage and compute platform for analytics, the move towards the separation of compute and storage, and the containerisation of Hadoop and the workloads that run on it.&lt;/p&gt; &lt;p&gt;I’m going to try to not repeat any of that, but today I do want to imagine what Hadoop might look like in 3 to 5 years time. And we’ll base this on the assumption that separation of storage and compute is going to happen, i.e. services will not assume or need these to be co-located.&lt;/p&gt; &lt;p&gt;So, in not particular order, some predictions:&lt;/p&gt; &lt;p&gt;HDFS (or more specifically the HCFS specification) will become the standard for accessing volumes of data for analytics. It’s almost there already, but there are a few kinks to be worked out with accessing data in object store technologies, and maybe even remote HDFS clusters.&lt;/p&gt; &lt;p&gt;HDFS will evolve into a multi-purpose clustered storage platform. This is obviously the direction being driven by Ozone, giving HDFS a generic underlying block storage layer on which HDFS, object store and other access routes (streaming data, NoSQL etc.) can be layered. This is a pretty competitive market however, not only are all the big storage vendors playing in this space, but MapR have this tech already - the &lt;a href=&quot;/technologies/mapr-fs/&quot;&gt;MapR-FS&lt;/a&gt; supports HDFS, NFS, S3 (via an embedded Minio instance), Kafka (via &lt;a href=&quot;/technologies/mapr-es/&quot;&gt;MapR-ES&lt;/a&gt;) and NoSQL document and wide column (via &lt;a href=&quot;/technologies/mapr-db/&quot;&gt;MapR-DB&lt;/a&gt;) APIs - and there’s a bunch of specialist storage and database technology vendors in this space.&lt;/p&gt; &lt;p&gt;YARN as a standard for resource management for compute jobs will go, replaced by Kubernetes. This is starting to happen already - Spark and Flink already support Kubernetes, with jobs executed through the deployment of a set of transient dockerised processes. Kubernetes will need to beef up it’s schedulers a bit, but it will be the primary standard that cluster processing frameworks use to orchestrate their jobs. And note that this isn’t running Hadoop on Kubernetes (that some people like BlueData and Robin) can deliver today - this is about running individual transient compute jobs on Kubernetes rather than a persistent Hadoop cluster (that then does it’s own local resource management).&lt;/p&gt; &lt;p&gt;YARN as a standard for resource management of long lived processes will die, probably even before it gets going. There’s been efforts to make long live services on Hadoop (HBase, Kafka etc.) integrate with YARN for a while (through Hoya, Koya to &lt;a href=&quot;/technologies/apache-slider/&quot;&gt;Apache Slider&lt;/a&gt;), and the latest version of YARN has support for long lived processes built in. But why integrate all the persistent services that run on Hadoop (Solr, Impala, Kudu, HBase etc.) with YARN when you can just deploy them using containers and let a container management layer handle the resource management. And note that for most of these long running services they’re pretty much good to run on docker containers today, as long as those that require HDFS are happy running over external object storage.&lt;/p&gt; &lt;p&gt;YARN as a cluster resource manager will (probably) go. The reason I say probably is that you’ll still be able to buy a physical Hadoop platform, that will support multi-purpose storage (with an HCFS interface) and compute management (with a Kubernetes interface). However, will this be an evolution of YARN under the covers, or as I suspect, will the brave decision be made to ditch it and just integrate a Kubernetes distribution.&lt;/p&gt; &lt;p&gt;And that leaves (for me) the most important and valuable part of Hadoop - the shared services - metadata, security, audit, data management. They’re still immature, but my hope is that these become a set of standard integration points that a wide range of analytical tools use to enable centralised management and governance of your analytics ecosystem. And (finally) there will be some standardisation of the tooling in this space - whether it’s Sentry or Ranger, Altas or Navigator, or DataPlane will remain to be seen.&lt;/p&gt; &lt;p&gt;So what does that mean Hadoop will be in five years time? It’ll be a collection of analytical platforms and tools, bound together by some common standards for storage, compute, security, management and governance, that can be individually deployed in the public cloud or on premesis, with the option of using local cloud tech or an out of the box physical cluster on premises.&lt;/p&gt; &lt;p&gt;And an offering of a wide range of analytical tools that integrate together, where management, maintenance and governance costs are minimised, and that can be deploy wherever and however you want sounds pretty good to me.&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/10/19/the-future-of-hadoop/</guid> </item> <item><title>Thoughts On Hadoop Service Providers</title><link>https://ondataengineering.github.io/blog/2018/10/17/thoughts-on-hadoop-service-providers/</link><pubDate>Wed, 17 Oct 2018 07:45:00 +0000</pubDate> <description> &lt;p&gt;So last week we looked at a bunch of cloud Hadoop service providers. Time to recap and expouse some thoughts… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Deploying Hadoop is hard and complex, especially if you’re looking to make it secure, stable and implement a bunch of best practice stuff. And wouldn’t it be nice if we could automate the process and make it repeatable, allowing us to create clusters when needed and tear them down when we’re done?&lt;/p&gt; &lt;p&gt;&lt;a href=&quot;/technologies/amazon-emr/&quot;&gt;Amazon EMR&lt;/a&gt;, &lt;a href=&quot;/technologies/azure-hdinsight/&quot;&gt;Azure HDInsight&lt;/a&gt; and &lt;a href=&quot;/technologies/google-cloud-dataproc/&quot;&gt;Google Cloud Dataproc&lt;/a&gt; are the big cloud providers offerings in this space, and other cloud providers have similar offerings. Fundamentally they allow you to programatically specify and create an Hadoop cluster with one or more services pre-installed. They all support a bunch of standard stuff - some sort of selection of the Hadoop services you want pre-installed, a bunch of automatic configuration of Hadoop, streamlined usage of cloud storage (with encryption), custom bootstrap actions etc., and are all priced as a premium on the costs for the raw storage and compute that you consume.&lt;/p&gt; &lt;p&gt;But using any of these are going to tie you into those cloud platforms and their distributions of Hadoop. EMR and Cloud Dataproc both run their own distributions (although with some exploitation of &lt;a href=&quot;/technologies/apache-bigtop/&quot;&gt;Apache BigTop&lt;/a&gt;) - Google’s is more limited (Spark, MapReduce, Pig &amp;amp; Hive), EMR much broader (adding a range of other techs, including Flink, Presto, TensorFlow, Hue and Zeppelin). HDInsight is based on the Hortonworks Data Platform, and is the broadest of the lot adding in Kafka, Storm, Hive LLAP and (with an enterprise security add on) Ranger security. And although you’re tied into the cloud vendor, this is not neccesarily a bad thing - you get full integration with their security, audit and management tools, and if you’re going all in with a cloud vendor this can make a lot of sense.&lt;/p&gt; &lt;p&gt;The alternative is you align yourself to an “independant” Hadoop vendor - you’re still aligning your self to a distribution, but you now have the freedom to deploy it wherevers most appropriate - on premises, or on whichever cloud vendor works for you. Your options here were (primarily) &lt;a href=&quot;/tech-vendors/cloudera/&quot;&gt;Cloudera&lt;/a&gt;, &lt;a href=&quot;/tech-vendors/hortonworks/&quot;&gt;Hortonworks&lt;/a&gt; and &lt;a href=&quot;/tech-vendors/mapr&quot;&gt;MapR&lt;/a&gt;, however Cloudera nand Hortonworks have just announced they’re planning to merge. And all these options have tooling to programatically deploy Hadoop on cloud infrastructure, giving a similar experience to the cloud vendor offerings. For Cloudera it’s &lt;a href=&quot;/technologies/cloudera-altus/director/&quot;&gt;Director&lt;/a&gt;, Hortonworks have &lt;a href=&quot;/technologies/cloudbreak/&quot;&gt;Cloudbreak&lt;/a&gt; and MapR the MapR Orbit Cloud Suite.&lt;/p&gt; &lt;p&gt;But all these options are targeted at Hadoop administrators - a lot is automated, but you still need to have a pretty deep understanding of Hadoop, you’re still responsible for managing (starting, stopping and scaling) your cluster (although EMR, Dataproc and Cloudbreak have some support for auto scaling), and you’ll need to be comfortable by customising your cluster through bootstrap scripts.&lt;/p&gt; &lt;p&gt;&lt;a href=&quot;/technologies/qubole-data-service/&quot;&gt;Qubole Data Service&lt;/a&gt; feels like it’s targeting a slightly different market - in that it tries to automate as much of the cluster management as possible. You still need to spec your cluster and select the cloud infrastructure you want to run it on, but it will then manage it for you - automatically starting, stopping and scaling it based on the current workload to make sure youre cloud infrastructure costs are being minimised. And it also works hard to provide a much richer user interface, allowing analysts to manage their data in the cloud (including ingesting it and push it back to a wide range of cloud databases) and giving them rich query/job editors and Zeppelin notebooks. It feels more like the vision of an Haodop managed service.&lt;/p&gt; &lt;p&gt;Which leaves us with &lt;a href=&quot;/technologies/cloudera-altus/&quot;&gt;Cloudera Altus&lt;/a&gt;. This feels like Cloudera’s attempt to at a slightly higher level service. Unlike Qubole however, it has a range of offerings targeted at slightly different user communities, but it feels like it’s trying to differentiate itself from the traditional cloud Hadoop offerings.&lt;/p&gt; &lt;p&gt;As always - see our &lt;a href=&quot;/tech-categories/hadoop-distributions/&quot;&gt;Hadoop Distributions&lt;/a&gt; page for our full list of on premises and cloud based Hadoop distributions.&lt;/p&gt; &lt;p&gt;Right - that’s it for today. One more post this week on Hadoop and then we’ll move on to something new.&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/10/17/thoughts-on-hadoop-service-providers/</guid> </item> <item><title>The Mid Week News 17/10/2018</title><link>https://ondataengineering.github.io/blog/2018/10/17/the-mid-week-news/</link><pubDate>Wed, 17 Oct 2018 07:30:00 +0000</pubDate> <description> &lt;p&gt;It’s news time again… &lt;!--more--&gt;&lt;/p&gt; &lt;p&gt;Technology updates (details are on the relevant technology pages):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/technologies/apache-accumulo/&quot;&gt;Apache Accumulo&lt;/a&gt; has a 2.0 alpha release out&lt;/li&gt; &lt;li&gt;Hortonworks DataPlane &lt;a href=&quot;/technologies/hortonworks-dataplane/data-lifecycle-manager/&quot;&gt;Data Lifecycle Manager&lt;/a&gt; has hit 1.2&lt;/li&gt; &lt;li&gt;Hortonworks DataPlane &lt;a href=&quot;/technologies/hortonworks-dataplane/streams-messaging-manager/&quot;&gt;Streams Messaging Manager&lt;/a&gt; has hit 1.1&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Other technology news:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Hortonworks have a couple of posts on &lt;a href=&quot;/technologies/apache-hadoop/ozone/&quot;&gt;Ozone&lt;/a&gt; - &lt;a href=&quot;https://hortonworks.com/blog/apache-hadoop-ozone-object-store-overview/&quot;&gt;overview&lt;/a&gt;; &lt;a href=&quot;https://hortonworks.com/blog/apache-hadoop-ozone-object-store-architecture/&quot;&gt;architecture&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Confluent have another post on how event driven architectures are the future, but there’s some interesting stuff there on on-demand data stores that could be interesting from an analytics point of view - &lt;a href=&quot;https://www.confluent.io/blog/event-driven-2-0&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Data Artisans - checkpointing and Kafka offsets with &lt;a href=&quot;/technologies/apache-flink/&quot;&gt;Apache Flink&lt;/a&gt; - &lt;a href=&quot;https://data-artisans.com/blog/how-apache-flink-manages-kafka-consumer-offsets&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;A nice intro on ingestion of data into Hadoop using &lt;a href=&quot;/technologies/streamsets-data-collector/&quot;&gt;StreamSets&lt;/a&gt; - &lt;a href=&quot;https://streamsets.com/blog/modernizing-hadoop-ingest-beyond-flume-and-sqoop/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Looks like Snowflake has just secured a bunch more funding - &lt;a href=&quot;https://www.theregister.co.uk/2018/10/11/tricorn_snowflake_pockets_450m_in_another_massive_funding_round/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/technologies/google-cloud-storage/&quot;&gt;Google Cloud Storage&lt;/a&gt; is seeing some small changes - &lt;a href=&quot;https://cloud.google.com/blog/products/storage-data-transfer/store-it-analyze-it-back-it-up-cloud-storage-updates-bring-new-replication-options&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;li&gt;From Datanami, an update on Teradata strategy - &lt;a href=&quot;https://www.datanami.com/2018/10/09/new-teradata-focuses-on-answers-not-analytics/&quot;&gt;link&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/blog/2018/10/17/the-mid-week-news/</guid> </item> <item><title>Qubole Data Service</title><link>https://ondataengineering.github.io/technologies/qubole-data-service/</link><pubDate>Fri, 12 Oct 2018 00:00:00 +0000</pubDate> <description> &lt;p&gt;Hadoop as a managed service over AWS, Azure, Google Gloud Platform and Oracle Cloud. Supports Airflow, Hadoop, Presto and Spark cluster types, automatic management (starting, stopping and scaling) of clusters based on workload, automatic shared Hive metastores within accounts, role based access control (to accounts, clusters and UI/API functionality, with Hive authorisation to manage access to data), connectivity to external databases (Data Stores), labelling of clusters and routing of commands by label (allowing graceful cluster upgrades), custom node bootstrap commands, encryption, auditing, data caching (on AWS only via open source Rubix project), ODBC/JDBC drives. Has a rich web based user interface that supports exploration of data (in Hadoop, object stores and connected external databases), a command composer with auto completion (supporting Hive, Presto, Pig, Shell, Spark and Worklow commands) with auto completion and command history, parameterisable command templates, data management (import, export and upload), a visual query builder (Smart Query), Zeppelin based notebooks (including publication of public read only notebook views), command schedulers, cluster management and a range of usage and cluster metrics and graphs. Also supports a REST API. Priced per hour based on the cloud infrastructure being used, which is in addition to any cloud vendor costs. Launched in 2013.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Other Names&lt;/td&gt;&lt;td&gt;QDS&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;May 2019 - R56 (AWS/Azure/Oracle)&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2&gt;Related Technologies&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Packages&lt;/td&gt;&lt;td&gt;&lt;a href=&quot;/technologies/apache-airflow/&quot;&gt;Apache Airflow&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Apache Hadoop&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-hive/&quot;&gt;Apache Hive&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-pig/&quot;&gt;Apache Pig&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-sqoop/&quot;&gt;Apache Sqoop&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-tez/&quot;&gt;Apache Tez&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-zeppelin/&quot;&gt;Apache Zeppelin&lt;/a&gt;, &lt;a href=&quot;/technologies/presto/&quot;&gt;Presto&lt;/a&gt;, TensorFlow&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;release-history&quot;&gt;Release History&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt; &lt;td&gt;version&lt;/td&gt; &lt;td&gt;release date&lt;/td&gt; &lt;td&gt;release links&lt;/td&gt; &lt;td&gt;release comment&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R54&lt;/td&gt; &lt;td&gt;2018-11-29&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://www.qubole.com/blog/release-54/&quot;&gt;blog post&lt;/a&gt;; &lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotesR54/index.html&quot;&gt;AWS&lt;/a&gt;; &lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-AzureR54/index.html&quot;&gt;Azure&lt;/a&gt;; &lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-OracleR54/index.html&quot;&gt;Oracle&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R55 (AWS)&lt;/td&gt; &lt;td&gt;2019-02-06&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotesR55/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R55 (Azure)&lt;/td&gt; &lt;td&gt;2019-03-15&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-AzureR55/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R55 (Oracle)&lt;/td&gt; &lt;td&gt;2019-03-15&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-OracleR55/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;GCP&lt;/td&gt; &lt;td&gt;2019-04-10&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://www.qubole.com/blog/qubole-google-deliver-unified-user-experience/&quot;&gt;announcement&lt;/a&gt;; &lt;a href=&quot;https://www.qubole.com/blog/technical-overview-of-qubole-on-gcp/&quot;&gt;tech overview&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R56 (AWS)&lt;/td&gt; &lt;td&gt;2019-05-08&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotesR56/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R56 (Oracle)&lt;/td&gt; &lt;td&gt;2019-05-29&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-OracleR56/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td&gt;R56 (Azure)&lt;/td&gt; &lt;td&gt;2019-05-30&lt;/td&gt; &lt;td&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/releasenotes-AzureR56/index.html&quot;&gt;release notes&lt;/a&gt;&lt;/td&gt; &lt;td&gt; &lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://www.qubole.com/&quot;&gt;https://www.qubole.com/&lt;/a&gt; - homepage&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/&quot;&gt;https://docs.qubole.com/en/latest/&lt;/a&gt; - docs&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/admin-guide/osversionsupport.html&quot;&gt;https://docs.qubole.com/en/latest/admin-guide/osversionsupport.html&lt;/a&gt; - supported component versions&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://rubix.readthedocs.io/en/latest/&quot;&gt;https://rubix.readthedocs.io/en/latest/&lt;/a&gt; - Rubix&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://www.qubole.com/blog/&quot;&gt;https://www.qubole.com/blog/&lt;/a&gt; - Qubole blog&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://docs.qubole.com/en/latest/release-notes/index.html&quot;&gt;https://docs.qubole.com/en/latest/release-notes/index.html&lt;/a&gt; - release notes&lt;/li&gt; &lt;/ul&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/qubole-data-service/</guid> </item> <item><title>Google Cloud DataProc</title><link>https://ondataengineering.github.io/technologies/google-cloud-dataproc/</link><pubDate>Thu, 11 Oct 2018 00:00:00 +0000</pubDate> <description> &lt;p&gt;Service for dynamically provisioning Hadoop clusters on Google Compute Engine based on a single standard set of Hadoop services. Supports selection of virtual machines (including custom machine types and machines with GPUs), usage of custom VM images, a claimed cluster startup time of less than 90 seconds, local storage and HDFS filesystem, programmatic execution of jobs, workflows (parameterisable operations that create clusters, run jobs and then delete the cluster), manual and automatic scaling, initialisation actions (to install extra services or run scripts, with a set of open source actions available), optional components (automatic addition of extra services), automatic deletion of clusters (based on time, usage or idleness), integration with Stackdriver Logging and Monitoring and encryption of data in HDFS and Cloud Storage. Manageable via the Google Cloud Console Web UI and SDK plus an RPC and REST API. Priced an an hourly rate (charged per second) based on the specification of the VMs being used, which is in addition to any Compute Engine or Persistent Disk charges.&lt;/p&gt; &lt;!-- Tech Category metadata --&gt; &lt;h2&gt;Technology Information&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Other Names&lt;/td&gt;&lt;td&gt;Google DataProc, DataProc&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Type&lt;/td&gt;&lt;td&gt;Commercial&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Last Updated&lt;/td&gt;&lt;td&gt;October 2018 - v1.3&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2&gt;Related Technologies&lt;/h2&gt; &lt;table&gt; &lt;tbody&gt; &lt;tr&gt;&lt;td&gt;Packages&lt;/td&gt;&lt;td&gt;&lt;a href=&quot;/technologies/apache-hadoop/&quot;&gt;Apache Hadoop&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-hive/&quot;&gt;Apache Hive&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-pig/&quot;&gt;Apache Pig&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-spark/&quot;&gt;Apache Spark&lt;/a&gt;, &lt;a href=&quot;/technologies/apache-tez/&quot;&gt;Apache Tez&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;h2 id=&quot;links&quot;&gt;Links&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/dataproc/&quot;&gt;https://cloud.google.com/dataproc/&lt;/a&gt; - homepage&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions&quot;&gt;https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions&lt;/a&gt; - bundle services version list&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/dataproc/docs/release-notes&quot;&gt;https://cloud.google.com/dataproc/docs/release-notes&lt;/a&gt; - release notes&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://cloud.google.com/dataproc/docs/&quot;&gt;https://cloud.google.com/dataproc/docs/&lt;/a&gt; - documentation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://github.com/GoogleCloudPlatform/dataproc-initialization-actions&quot;&gt;https://github.com/GoogleCloudPlatform/dataproc-initialization-actions&lt;/a&gt; - open source initialization actions&lt;/li&gt; &lt;/ul&gt; &lt;h2 id=&quot;news&quot;&gt;News&lt;/h2&gt; &lt;p&gt;See &lt;a href=&quot;/tech-vendors/google-cloud-platform/&quot;&gt;Google Cloud Platform&lt;/a&gt; updates&lt;/p&gt; </description> <guid isPermaLink="true">https://ondataengineering.github.io/technologies/google-cloud-dataproc/</guid> </item> </channel> </rss>