What do you need help with?

Ingest Oracle Cloud Infrastructure Logs into Third-Party SIEM Platforms using Log Shippers

Juan Romero

23-Nov-2025

Ingest Oracle Cloud Infrastructure Logs into Third-Party SIEM Platforms using Fluent-Bit

Introduction

Oracle Cloud Infrastructure (OCI) is an Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS) platform trusted by large-scale enterprises. It offers a comprehensive array of managed services, including hosting, storage, networking, databases, and more.

Proactively presenting security-related event logs for triage to the appropriate resources is crucial for detecting and preventing cybersecurity incidents. Many organizations rely on Security Information and Event Management (SIEM) platforms to correlate, analyze logs and alerts from relevant assets. Proper configuration of log capture, retention for the appropriate duration, and near-real-time monitoring and alerting enables security operations teams to identify issues, focus on critical information based on system tuning, and take timely action.

A best practice for ingesting OCI logs involves sending them to OCI Streaming, which is Apache Kafka-compatible, allowing third-party SIEM platforms to consume the logs as Kafka consumers. This approach reduces delays, provides resilience, and ensures retention in case of temporary issues with data consumption on the SIEM side.

However, some third-party SIEM platforms lack default connectors for consuming logs directly from OCI streams and do not natively support data consumption from Kafka topics, the widely used open-source event streaming platform, complicating the integration process. In such cases, log shippers serve as a solution to bridge this gap.

A log shipper functions as a standalone tool that collects logs from various sources and then forwards them to one or more specified destinations. To ensure seamless communication with both OCI Streaming and third-party SIEM platforms, the log shipper software should run on a machine with internet access. In this tutorial, we’ll deploy the log shipper software on a Compute Instance within OCI.

The log shipper will:

Pull logs from OCI streams using Kafka source.
Route the logs to third-party SIEM platform using any supported destination type or plugin, such as TCP port, HTTP, or a file on third-party SIEM collector.

Now, let us look at the high-level representation of the solution architecture as shown in the following image.

Note: While this solution can bridge the gap, it is advisable to consider it only as a last option if other methods are not feasible. It is important to coordinate closely with your SIEM provider to explore any native or recommended approaches first. If you decide to proceed with a log shipper, working with your SIEM provider in selecting the most suitable one will ensure better support from your SIEM provider during and after implementation, helping to tailor the setup to meet your organization’s specific needs.

There are different log shippers available and some of them are:

Filebeat
Fluent Bit
Fluentd
Vector
Logstash

In our case, we’ll use Fluent-Bit as log shipper.

Objectives

This tutorial focuses on how logs can be consumed from OCI streams using the log shipper Fluent Bit and examples of how they can integrate with third-party SIEM platforms are provided.

Prerequisites

Users in OCI must have the required policies for OCI Streaming, OCI Connector Hub and OCI Logging services to manage the resources. For more information on policy reference of all the services, see Policy Reference.
A compute instance designated as the log shipper should be available with internet access to access OCI streams. For more information, see Creating an Instance.

Note: The following tasks (Task 1 to Task 4) should be performed on the OCI end, regardless of the chosen method or log shipper.

Task 1: Configure the Logs to Capture

OCI Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. OCI Logging provides access to logs from OCI resources. A log is a first-class OCI resource that stores and captures log events collected in a given context. A log group is a collection of logs stored in a compartment. Log groups are logical containers for logs. Use log groups to organize and streamline management of logs by applying Oracle Cloud Infrastructure Identity and Access Management (OCI IAM) policy or grouping logs for analysis.

To get started, enable a log for a resource. Services provide log categories for the different types of logs available for resources. For example, the OCI Object Storage service supports the following log categories for storage buckets: read and write access events. Read access events capture download events, while write access events capture write events. Each service can have different log categories for resources.

Log in to the OCI Console, navigate to Observability & Management, Logging and Log Groups.
Select your compartment, click Create Log Group and enter the following information.
- Name: Enter SIEM_log_group.
- Description (Optional): Enter the description.
- Tags (Optional): Enter the tags.
Click Create to create a new log group.
Under Resources, click Logs.
Click Create custom log or Enable service log as desired.

For example, to enable write logs for an OCI Object Storage bucket, follow the steps:
1. Click Enable Service Log.
2. Select your resource compartment and enter Object Storage in the Search services.
3. Click Enable Logs and select your OCI Object Storage bucket name in the Resource.
4. Select log group (SIEM_log_group) created in Task 1.2 and Write Access Events in the Log Category. Optionally, enter SIEM_bucket_write as Log name.
5. Click Enable to create your new OCI log.

Task 2: Create a Stream using OCI Streaming

OCI Streaming service is a real-time, serverless, Apache Kafka-compatible event streaming platform for developers and data scientists. It provides a fully managed, scalable, and durable solution for ingesting and consuming high-volume data streams in real-time such as logs. We can use OCI Streaming for any use case in which data is produced and processed continually and sequentially in a publish-subscribe messaging model.

Go to the OCI Console, navigate to Analytics & AI, Messaging and Streaming.
Click Create Stream to create stream.
Enter the following information and click Create.
- Name: Enter the stream name. For this tutorial, it is SIEM_Stream.
- Stream Pool: Select existing stream pool or create a new one with public endpoint.
- Retention (in hours): Enter the number of hours to retain messages in this stream.
- Number of Partitions: Enter the number of partitions for the stream.
- Total Write Rate and Total Read Rate: Enter based on the amount of data you need to process.

You can start with default values for testing. For more information, see Partitioning a Stream.

Task 3: Set up an OCI Connector Hub

OCI Connector Hub orchestrates data movement between services in OCI. OCI Connector Hub provides a central place for describing, executing and monitoring data movements between services, such as OCI Logging, OCI Object Storage, OCI Streaming, OCI Logging Analytics, and OCI Monitoring. It can also trigger OCI Functions for lightweight data processing and OCI Notifications to set up alerts.

Go to the OCI Console, navigate to Observability & Management, Logging and Connectors.
Click Create Connector to create the connector.
Enter the following information.
- Name: Enter SIEM_SC.
- Description (Optional): Enter the description.
- Compartment: Select your compartment.
- Source: Select Logging.
- Target: Select Streaming.
Under Configure Source Connection, select a Compartment name, Log Group, and Log (log group and log created in Task 1).
If you also want to send Audit Logs, click +Another Log and select the same compartment while replacing _Audit as your log group.
Under Configure target, select a Compartment, and Stream (stream created in Task 2).
To accept default policies, click the Create link provided for each default policy. Default policies are offered for any authorization required for this connector to access source, task, and target services.
Click Create.

Task 4: Set Up an Access Control for Log Shippers to Retrieve Logs

To allow log shippers to access data from an OCI stream, create a user and grant stream-pull permissions for retrieving logs.

Create an OCI user. For more information, see Managing Users.
Create an OCI group named SIEM_User_Group and add the OCI user to the group. For more information, see Managing groups.
Create the following OCI IAM policy.

Allow group to use stream-pull in compartment

Use Fluent Bit as a Log Shipper

Fluent Bit is a lightweight, high-performance log shipper, serving as an alternative to Fluentd. Fluent Bit emerged in response to the growing need for an optimal solution capable of collecting logs from numerous sources while efficiently processing and filtering them. Notably, Fluent Bit excels in resource-constrained environments such as containers or embedded systems.

To use Fluent Bit, SOCFortress will define inputs, filters, outputs, and global configurations in a configuration file.

Let us examine these components in detail:

Service: Contains global settings for the running service.
Input: Specifies sources of log records for Fluent Bit to collect.
Filter: Applies transformations to log records.
Output: Determines the destination where Fluent Bit sends the processed logs.

Fluent Bit inputs and outputs plugins:

Fluent Bit Inputs Plugins: Fluent Bit offers a diverse range of input plugins tailored to different log sources. For more information, see Inputs.
Fluent Bit Outputs Plugins: To efficiently dispatch logs to different destinations, Fluent Bit provides a versatile array of output plugins. For more information, see Outputs.

To collect logs from the OCI Streaming we’ll configure the Fluent Bit input as Kafka. Parameters required: