Currently we see that Hadoop is becoming part of Enterprise Data Warehouse family. But family should be connected to each other. Sometimes we need access to Hadoop from Oracle Database. Sometimes Hadoop users need enterprise data stored in Oracle database.
Hive has very interesting concept - External Tables which allow you to define Java classes to access external database and present it as a native hive table.
Oracle Datasource for Apache Hadoop (formerly Oracle Table Access for Apache Hadoop) turns Oracle Database tables into a Hadoop data source (i.e., external table) enabling direct and consistent Hive QL/Spark SQL queries, as well as direct Hadoop API access. Applications can join master data or dimension data in Oracle Database with data stored in Hadoop. Additionally data can be written back to Oracle Database after processing.
Oracle Datasource for Apache Hadoop optimizes a query’s execution plans using predicate and projection pushdown, and partition pruning. Database table access is performed in parallel based on the selected split patterns, using smart and secure connections (Kerberos, SSL, Oracle Wallet), regulated by both Hadoop (i.e., maximum concurrent tasks) and Oracle DBAs (i.e., max pool size).
Continue reading ‘Oracle DataSource for Apache Hadoop (OD4H): introduction’ »
GoldenGate Cloud Service is part of Oracle's PaaS portfolio. From technical perspective it is just standard GoldenGate deployed on VM in Oracle Cloud. So same already proven architecture works in Cloud.
GGCS can be used for different cases from zero downtime migration to real-time DWH feeding. More cases like BigData and data pipeline feeding are on the way.
So what do you need to use GoldenGate Cloud Service. You should have:
- database instance in cloud (DBaaS or ExadataCS)
- subscription for GoldenGate Cloud Service.
- storage cloud service (it used for backup)
GGCS is available as Non Metered service now. If you use GGCS Non-Metered Service then you should pay money even if your GoldenGate instance is down.
Soon GGCS will be available as a Metered Service. So it will possible to pay on per hour basis. This capability will open new cases like Dev/Test Cloud Environment Synchronization. Just imagine you have database in cloud for testing purposes. You should periodically (every week/month) synchronize it with production database. So you don't need GGCS running for all time but run it for 2 hours every Sunday to apply captured data. This approach can save a lot of money.
Continue reading ‘GoldenGate Cloud Service (GGCS): Configure GoldenGate to replicate data’ »
Oracle Cloud provides several Oracle Database offerings. You can choose from
- a single schema based service
- virtual machine with a fully configured and running Oracle Database Instance
- Exadata Service with all the database features.
You can look into details here: https://cloud.oracle.com/en_US/database
We will talk about Database as a Service and not about Schema or Exadata here. So my final goal is to create database for GoldenGate replication which is separate service. Ok let's start.
Continue reading ‘Oracle Database Cloud Service: Create database’ »
Let's define terms which Oracle uses in Oracle Cloud Services.
- Block Storage - optimizes storage for IOPS and block-based access and provides POSIX-compliant file systems for Oracle Compute Cloud Service instances. This is just standard disk device. Sometimes it is only one drive, sometimes it is RAID device. But anyway application access it using standard disk operations
- Object Storage - scalable storage which can store large binary objects with metadata and unique ID. Multiple storage nodes form a single, shared, horizontally scalable pool. Application can access data using REST API.
Oracle Storage Cloud Service provides a low cost, reliable, secure, and scalable object-storage solution for storing unstructured data and accessing it anytime from anywhere. It is ideal for data backup, archival, file sharing, and for storing large amounts of unstructured data like logs, sensor-generated data, and VM images.
Continue reading ‘Oracle Storage Cloud Service: Creating Containers Using the REST API’ »
You work with GoldenGate then you know that there was tool GoldenGate Director for configuration, deployment, monitoring and management. Really these responsibilities are distributed among development and support. So there should be two tools: for development and for support. Oracle is following this ideology and has created two tools:
- GoldenGate Monitor (or GG plugin for OEM if you are OEM fan) - for management and monitoring
- GoldenGate Studio - for design and deployment
What do we need from development tools? I have the following list in my head:
- WYSIWYG/drag'n'drop interface
- easy moving configuration through the path Dev/QA/Prod
- collective working
GoldenGate has these features. It was created to be great scalable development tool.
Continue reading ‘GoldenGate Studio: Quick Start’ »
Virtual Machine Oracle BigData Lite is vm created by Oracle for development and demo purposes of BigData (Cloudera) and Oracle technologies working together.
You can download BigData Lite VM from Oracle site. This vm includes following software (for vm version 4.4):
Continue reading ‘Oracle BigData Lite VM introduction’ »
Интеграция данных является неотъемлемой фазой в построении оптимальных бизнес-процессов компании любого уровня и сферы деятельности. Компания Oracle предлагает лучшую в своем классе интеграционную платформу для решения задач:< ?xml:namespace prefix = "o" />
- Быстрого обогащения данных – Big Data Preparation Cloud Service
- Трансформации данных – Oracle Data Integrator
- Управления и повышения качества данных при помощи – Oracle Enterprise Data Quality
- Репликации данных в режиме реального времени – Oracle GoldenGate
- Управления слоем метаданных различных приложений – Oracle Metadata Management
Continue reading ‘Семинар "Интеграция в эпоху облачных технологий и больших данных" 25 февраля 2016 в 11:00’ »
One of GoldenGate 12.2 new features is replication support for MSSQL Standard Edition. In the previous GoldenGate 12.1 release, GoldenGate supported capture of DML only from SQL Server Enterprise Edition, due to the method of enabling Supplemental Logging, which is done via a feature only available to Enterprise Editions of SQL Server. Fortunately, beginning with Oracle GoldenGate 12.2, GoldenGate is able to enable Supplemental Logging for Standard Edition instances via a different mechanism. That mechanism to enable Supplemental Logging is done via certain SQL Server Replication components. These SQL Server Replication components are required to be installed and configured in order to enable Supplemental Logging,
Also it is very interesting that MSSQL doesn't offer Change Data Capture for MSSQL Standard Edition. So if we have 100 offices with MSSQL and we need to collect only changed data to Data Warehouse then it is required to install MSSQL Enterprise Edition in every office because CDC is only available in MSSQL EE.
Continue reading ‘Configuring GoldenGate to replicate data from MSSQL Standard Edition (CDC, Change Data Capture)’ »
When install GoldenGate Monitor we create special credentials used by GoldenGate Agent and GoldenGate Monitor to secure communications.
Also we set this credentials while configuring GoldenGate Agent using pw_agent_util.sh.
But is not well documented how these credentials are used and how change them. So let fill these gap.
Continue reading ‘GoldenGate Monitor 12.2: how to change password for internal communications’ »
I already wrote about configuring GoldenGate agent for ODI and Enterprise Manager. Now I would like to talk about configuring it for GoldenGate Monitor. GoldenGate is interesting tool for GoldenGate monitoring. It is more powerful and flexible than GoldenGate plugin for Enterprise Manager but it is still standalone tool not integrated with OEM. So it has its own advantages and disadvantages.
From the technical standpoint configuring GoldenGate Agent for Monitor is a little bit different than configuring GoldenGate Agent for OEM and ODI. Also I see some inaccuracies in documentation and step-by-step configuration is not crystal clear. That's why I writing this blog post - I would like to have short step-by-step instruction how to configure GoldenGate Agent for Monitor.
So let's look at GoldenGate monitoring framework architecture
We can see
- GoldenGate Monitor has its own Server with Repository (which is Oracle Database)
- Each GoldenGate instance has Agent (Monitor Agent)
- Monitor Server and Monitor agents communicates through JMX (which can also be encrypted by SSL)
- Agent communicates with Manager to get information about GoldenGate state
Continue reading ‘Configuring GoldenGate Monitor 12.2’ »