GoldenGate 12.3.0.1.2 was released

GoldenGate 12.3.0.1.2 (or December release) was uploaded to edelivery.oracle.com. If you use GoldenGate Microservice Architecture then it is essential to download new version because it has some new features and bugs fixed.

I couldn’t find instructions how to upgrade GG 12.3 Initial Release to GG 12.3.0.1.2 December Release. So I chose the following method:

1) Install GG 12.3.0.1.2 to new directory (for example to directory /u01/app/oracle/product/gg/12.3_ma/db122_december  — this is GG for DB 12c)

2) Move existing directory (I moved /u01/app/oracle/product/gg/12.3_ma/db122 —> /u01/app/oracle/product/gg/12.3_ma/db122_initial)

3) Create link  for old directory which will point to last installed version (link /u01/app/oracle/product/gg/12.3_ma/db122 ——> /u01/app/oracle/product/gg/12.3_ma/db122_december)

Restart everything. You should see new version in Service Manager

image

GoldenGate 12.3: Microservice Architecture (MA)

The Microservices Architecture (MA) for Oracle GoldenGate is a new REST API Microservices-based architecture that allows you to install, configure, monitor, and manage Oracle GoldenGate services using a web-based UI.

Really there are two versions of GoldenGate now: classic and microservice. Classic architecture has standard extract, replicat, pump and receiver. It is managed by classic ggsci. Microservice Architecture (MA) has different types of processes and managed using Admin Client or using web UI. See architecture of GoldenGate MA below

imageOracle GoldenGate MA is designed with the industry-standard HTTP communication protocol and JSON data interchange format.

Classic architecture was managed using ggsci console and had weak authentication and authorization tools. Oracle GoldenGate MA has ability to verify identity using basic authentication and using SSL client certificates.

GoldenGate MA processes

Oracle GoldenGate MA  uses different types of processes to perform same tasks as GoldenGate Classic. Let’s talk a little bit about new processes:

Service Manager. Something like (and replacement of) Manager process.  This is watchdog for other processes.

Administration Server. Something like ggsci console. Operates as central control entity. You use it to create and manage other processes. The key feature of Administration Server is REST API which can be accesses from any HTTP or HTTPS client.

You can add, delete or alter GoldenGate processes, edit configuration files, add users and assign roles using Administration Server.

Receiver Server. Something like collector. It can receive trail files from remote server. However it replaces multiple collectors because it is multithreaded. Receiver was designed to be protocol agnostic – so it supports HTTPS, HTTP, UDT (reliable UDP) and classic GoldenGate TCP transports. By default it uses HTTPS protocol.

Distribution Server. Something like pump.  But again this multithreaded process which can handle multiple trail at the same time. So it will replace multiple pumps. And again it supports multiple protocols: WebSockets for HTTPS-based streaming, which relies on SSL security, UDT, SOCKS5, HTTP. It also support Passive mode to initiate connection from remote side.

Performance Metrics Server. This is process which collects and saves information from other processes (extracts, replicats, etc). All GoldenGate processes push information to Performance Metrics Server. Now this is the only processes which writes data to GoldenGate datastore (Berkley DB). You can use Performance Metrics Server to query various metrics, view logs, process statuses, monitor system utilization, etc.

Admin Client. It is a command line utility (similar to ggsci) used to create, configure and manage processes. Admin Client uses REST API to accomplish its tasks.


Continue reading ‘GoldenGate 12.3: Microservice Architecture (MA)’ »

GoldenGate 12.3: announcement, new features and installation

Oracle has release new version of GoldenGate 12.3 in 18 August. This is very long awaited version – it postponed 2 or 3 times because of some very important new features. See some useful links for GoldenGate 12.3:

Continue reading ‘GoldenGate 12.3: announcement, new features and installation’ »

GoldenGate Studio: Quick Start

imageIntroduction

You work with GoldenGate then you know that there was tool GoldenGate Director for configuration, deployment, monitoring and management. Really these responsibilities are distributed among development and support. So there should be two tools: for development and for support. Oracle is following this ideology and has created two tools:

 

  • GoldenGate Monitor (or GG plugin for OEM if you are OEM fan) — for management and monitoring
  • GoldenGate Studio — for design and deployment

What do we need from development tools? I have the following list in my head:

  • WYSIWYG/drag’n’drop interface
  • easy moving configuration through the path Dev/QA/Prod
  • versioning
  • collective working

GoldenGate has these features. It was created to be great scalable development tool.

Continue reading ‘GoldenGate Studio: Quick Start’ »

Oracle BigData Lite VM introduction

imageVirtual Machine Oracle BigData Lite is vm created by Oracle for development and demo purposes of BigData (Cloudera) and Oracle technologies working together.

You can download BigData Lite VM from Oracle site. This vm includes following software (for vm version 4.4):

Continue reading ‘Oracle BigData Lite VM introduction’ »

Configuring GoldenGate to replicate data from MSSQL Standard Edition (CDC, Change Data Capture)

Introduction

imageOne of GoldenGate 12.2 new features is replication support for MSSQL Standard Edition. In the previous GoldenGate 12.1 release, GoldenGate supported capture of DML only from SQL Server Enterprise Edition, due to the method of enabling Supplemental Logging, which is done via a feature only available to Enterprise Editions of SQL Server. Fortunately, beginning with Oracle GoldenGate 12.2, GoldenGate is able to enable Supplemental Logging for Standard Edition instances via a different mechanism. That mechanism to enable Supplemental Logging is done via certain SQL Server Replication components. These SQL Server Replication components are required to be installed and configured in order to enable Supplemental Logging,

Also it is very interesting that MSSQL doesn’t offer Change Data Capture for MSSQL Standard Edition. So if we have 100 offices with MSSQL and we need to collect only changed data to Data Warehouse then it is required to install MSSQL Enterprise Edition in every office because CDC is only available in MSSQL EE.

Continue reading ‘Configuring GoldenGate to replicate data from MSSQL Standard Edition (CDC, Change Data Capture)’ »

GoldenGate Monitor 12.2: how to change password for internal communications

When install GoldenGate Monitor we create special credentials used by GoldenGate Agent and GoldenGate Monitor to secure communications.

image

Also we set this credentials while configuring GoldenGate Agent using pw_agent_util.sh.

But is not well documented how these credentials are used and how change them. So let fill these gap.

Continue reading ‘GoldenGate Monitor 12.2: how to change password for internal communications’ »

Configuring GoldenGate Monitor 12.2

I already wrote about configuring GoldenGate agent for ODI and Enterprise Manager. Now I would like to talk about configuring it for GoldenGate Monitor. GoldenGate is interesting tool for GoldenGate monitoring. It is more powerful and flexible than GoldenGate plugin for Enterprise Manager but it is still standalone tool not integrated with OEM. So it has its own advantages and disadvantages.

From the technical standpoint configuring GoldenGate Agent for Monitor is a little bit different than configuring GoldenGate Agent for OEM and ODI. Also I see some inaccuracies in documentation and step-by-step configuration is not crystal clear. That’s why I writing this blog post — I would like to have short step-by-step instruction how to configure GoldenGate Agent for Monitor.

So let’s look at GoldenGate monitoring framework architecture

image

We can see

  1. GoldenGate Monitor has its own Server with Repository (which is Oracle Database)
  2. Each GoldenGate instance has Agent (Monitor Agent)
  3. Monitor Server and Monitor agents communicates through JMX (which can also be encrypted by SSL)
  4. Agent communicates with Manager to get information about GoldenGate state

Continue reading ‘Configuring GoldenGate Monitor 12.2’ »

GoldenGate 12.2.0.1 New Features: Metadata Encapsulation

Basics

imageGoldenGate pre-12.2.0.1 doesn’t store table structure in trail files and in some cases it was an issue: when we received trail file — we didn’t know source table structure from the file. We had only record and this record contained field values. But we didn’t know field names. So earlier we chose how to manage this. There are two ways:

— assume that source table is the same as target table (ASSUMETARGETDEFS). In this case first value from trail record was written to first field of target table, the second value to the second field, etc. But this method doesn’t work in general case where source and target tables are different

— extract table definitions from source and gave this definition to replicat (SOURCEDEFS). In this case trail files were parsed and applied according to the static source table definition (stored in file). This method gave us flexibility but increased number of manual work: when source table structure changes — we must regenerate DEF-file or we can lose data consistency.

Now we don’t need ASSUMETARGETDEFS and SOURCEDEFS anymore. GoldenGate automatically transfers definitions in trail files. Moreover it retransfer definition when table structure changes (but retransfer will be triggered only after DML on this table) and also retransfer will be done after switch to new trail file (also we should wait for first DML).

There are two special new record types for table definition transfer: DDR (Database Definition Record) and TDR (Table Definition Record). The first one describes database, the second one is for table structure.

Continue reading ‘GoldenGate 12.2.0.1 New Features: Metadata Encapsulation’ »