Difficulty: intermediate
Estimated Time: 30 minutes

In this scenario you will create work with the OCI [Event] Streaming Service - a managed Pub/Sub service, similar to Apache Kafka. Events are published to Streams (aka Topics) over HTTP calls and consumed from those Streams, again using HTTP calls. Consumption can be done from several starting point (at or from a specific offset, from a specific time, as much as is available, only new events) . Note that consumers have to actively come to the Streaming service and collect messages. There is no push mechanism that will trigger listeners whenever new messages have been published to the Stream. There is no concept of subscription: consumers just create a 'cursor' (somewhat similar to a session or a [long running] query) and start pulling messages in the context of that cursor.

Streaming can be used for messaging, ingesting high-volume data such as application logs, operational telemetry, web click-stream data, or other use cases in which data is produced and processed continually and sequentially in a publish-subscribe messaging model. Messages are retained for up to 7 days. There are limits on how much data a Stream can handle per second, but these limits are fairly high (1MB per partition per second, 1MB maximum message size, five consume calls per second).

The scenario uses an Ubuntu 18.04 environment with Docker, OCI CLI and Fn CLI. Before you can start the steps in the scenario, the two Command Line interfaces are downloaded and installed. This will take about one minute. You will need Postman as well, if you want to invoke the API Gateway directly from your computer.

The scenario expects a number of preparations:

  • you already have deployed a function called hello in an application called lab# where # is a number assigned to you (in OCI compartment lab-compartment)
  • an API Gateway already has been provided in compartment lab-compartment with permissions to access functions in this compartment; the API Gateway is associated with a public subnet
  • there is already an API Deployment on the API Gateway - called MY_API_DEPL#
  • there already is a Stream in compartment lab-compartment, called lab-stream

The scenario was prepared for the Meetup Workshop Cloud Native application development on Oracle Cloud Infrastructure on January 20th, hosted by AMIS|Conclusion in Nieuwegein in collaboration with REAL (the Red Expert Alliance) and Link from Portugal.


Blog article introducing OCI Streaming [https://technology.amis.nl/2020/01/07/oracle-cloud-streaming-service-scalable-reliable-kafka-like-event-service-on-oci/]

Documentation on OCI Streaming Service: https://docs.cloud.oracle.com/iaas/Content/Streaming/Concepts/streamingoverview.htm

Blog article Todd Sharp: Getting your feet wet with Oracle Streaming: https://blogs.oracle.com/developers/getting-your-feet-wet-with-oci-streams

OCI Streaming - Publisher => Stream => Consumer(s)


This completes your explorations with Functions on a local Fn Server. The next scenario you may want to explore looks at deploying Function to Oracle Cloud Infrastructure.

Background Resources

Tutorial Getting Started with Fn environment https://github.com/fnproject/tutorials/blob/master/install/README.md

Tutorial FN with Node https://github.com/fnproject/tutorials/blob/master/node/intro/README.md

Docs on Fn on OCI https://docs.cloud.oracle.com/iaas/Content/Functions/Tasks/functionscreatefncontext.htm

OCI (Event) Streaming, combined with Functions and API Gateway

Step 1 of 5

Step 1 - Check on Stream on Console and Prepare Environment Settings, Fn and OCI CLI

The environment is currently being prepared. While that is happening, we can take a look at the Stream lab-stream in the lab-compartment. Note: there is a fairly strict limit on the number of Stream (partitions) that we are allowed to create (out of the box); therefore we are all sharing the stream in this workshop.

Open the OCI Console as lab-user at: https://console.us-ashburn-1.oraclecloud.com/storage/streaming . Here you should see the stream lab-stream.

Click on the stream lab-stream to go to the details page. Click on Produce Test Message to... well, produce a test message of course.

Type a message and press Produce. The console will indicate that the message was produced successfully.

To check on the message, close the popup window and press Load Messages. All recently published messages on the stream are displayed – including the one that was just published.

This is the simplest example of pub/sub through OCI Streaming.

Wait for OCI CLI (and Fn CLI) to be installed

You need to provide details on the OCI tenancy you will work in and the OCI user you will work as. Please edit these two files:

  • ~/.oci/config
  • ~/.oci/oci_api_key.pem

Paste the contents provided by the workshop instructor into these two files.

Set the environment variable LAB_ID to the number provided to you by the workshop instructor.

export LAB_ID=1

Do not continue until you see the file /root/allSetInBackground appear. If it appears, then the OCI CLI has been installed and you can continue.

Try out the following command to get a list of all namespaces you currently have access to - based on the OCI Configuration defined above.

oci os ns get

If you get a proper response, the OCI is configured correctly and you can proceed. If you run into an error, ask for help from your instructor.

Prepare a number of environment variables. Note: the assumptions here are a compartment called lab-compartment and an API Gateway lab-apigw in that same compartment as well as an API Deployment called MY_API_DEPL# on the API Gateway. We need to get references to these resources in order to create new resources in the right place.

cs=$(oci iam compartment list)
export compartmentId=$(echo $cs | jq -r --arg display_name "lab-compartment" '.data | map(select(."name" == $display_name)) | .[0] | .id')

apigws=$(oci api-gateway gateway list -c $compartmentId)
export apiGatewayId=$(echo $apigws | jq -r --arg display_name "lab-apigw" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .id')
depls=$(oci api-gateway deployment list -c $compartmentId)
deploymentEndpoint=$(echo $depls | jq -r --arg display_name "MY_API_DEPL_$LAB_ID" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .endpoint')
apiDeploymentId=$(echo $depls | jq -r --arg display_name "MY_API_DEPL_$LAB_ID" '.data.items | map(select(."display-name" == $display_name)) | .[0] | .id')
nss=$(oci os ns get)
export ns=$(echo $nss | jq -r '.data')

echo "Compartment OCID: $compartmentId"
echo "Namespace: $ns"
echo "API Deployment Id and Endpoint: $deploymentEndpoint and $apiDeploymentId"

Fn Environment Preparation

Now Check the installed version of Fn CLI. Note: we do not need the Fn server at this stage.

fn version

To prepare the proper Fn context - associated with the OCI tenancy for this workshop, execute the next set of commands:

fn create context lab-fn-context --provider oracle

fn use context lab-fn-context

fn update context oracle.compartment-id $compartmentId

fn update context api-url https://functions.us-ashburn-1.oci.oraclecloud.com

fn update context registry iad.ocir.io/$ns/cloudlab-repo

fn update context oracle.profile FN

A remote Fn context based on Oracle as provider should now be set up. List the currently available and set Fn contexts to verify this.

fn list contexts

Next and finally, login to the private Docker Registry that is prepared for you on OCI.

docker login iad.ocir.io

The username you have to provide is composed of <tenancy-namespace>/<username>. The password is an Authentication Token generated for the specified user. Both these values are provided by your workshop instructor.

And now we are ready.