Create a stream on Pivotal Cloud Foundry (PCF) using Spring Cloud Data Flow (SCDF) server
[ Spring Cloud Stream
Spring Cloud Dataflow
Spring Cloud Skipper
Pivotal Cloud Foundry
] In this tutorial, let’s use Spring provided HTTP source and LOG sink applications to create a stream and deploy it to cloudfoundry. We shall use SCDF server running locally on our machine to deploy the stream to cloudfoundry.
Table of contents
- Prerequisites
- Start SCDF Shell CLI on your PC
- Register the out-of-the-box
http
andlog
spring boot apps - Create the stream
- Deploy the stream
- Validate the deployed stream
- Test
httpLogStream
stream using PCF CLI
Prerequisites
- An account on Pivotal Cloud Foundry (PCF). You can create one here.
- PCF Command Line Interface (CLI) installed on your computer. PCF CLI can be found in
tools
section of your PCF account. - An up and running SCDF server on PCF. You can follow the tutorial here to install SCDF server on PCF.
- Download SCDF CLI jar to your PC. You can follow the tutorial here to download.
Start SCDF Shell CLI on your PC
java -jar spring-cloud-dataflow-shell-2.3.0.RELEASE.jar --dataflow.uri=https://my-codeaches-scdf-server.cfapps.io --dataflow.username=user001 --dataflow.password=pass001 --skip-ssl-validation=true
Output
____ ____ _ __
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
____ |_| _ __|___/ __________
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
2.3.0.RELEASE
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
Successfully targeted https://my-codeaches-scdf-server.cfapps.io
dataflow:>
Register the out-of-the-box http
and log
spring boot apps
Cloudfoundry provides with few out-of-the-box source and sink spring boot applications which can be used for stream creation. Lets register the out-of-the-box http
and log
spring boot apps, specific to rabbit
messaging broker, in SCDF server.
Register HTTP Source Spring Boot App
dataflow:>app register --name http --type source --uri maven://org.springframework.cloud.stream.app:http-source-rabbit:2.1.1.RELEASE
Output
Successfully registered application 'source:http'
Register LOG Sink Spring Boot App
dataflow:>app register --name log --type sink --uri maven://org.springframework.cloud.stream.app:log-sink-rabbit:2.1.2.RELEASE
Output
Successfully registered application 'sink:log'
Create the stream
Let’s utilize the above registered apps http
and log
to create http|log
stream using stream create
command on SCDF shell terminal.
This stream will take HTTP POST request and prints the body in log file.
dataflow:>stream create --name httpLogStream --definition "http | log"
Output
Created new stream 'httpLogStream'
Deploy the stream
dataflow:>stream deploy httpLogStream --platformName dev
Output
Deployment request has been sent for stream 'httpLogStream'
Validate the deployed stream
dataflow:>stream validate 'httpLogStream'
Output
╔═════════════╤═════════════════╗
║ Stream Name │Stream Definition║
╠═════════════╪═════════════════╣
║httpLogStream│http | log ║
╚═════════════╧═════════════════╝
httpLogStream is a valid stream.
╔═══════════╤═════════════════╗
║ App Name │Validation Status║
╠═══════════╪═════════════════╣
║source:http│valid ║
║sink:log │valid ║
╚═══════════╧═════════════════╝
dataflow:>stream info 'httpLogStream'
Output
╔═════════════╤═════════════════╤═══════════╤════════╗
║ Stream Name │Stream Definition│Description│ Status ║
╠═════════════╪═════════════════╪═══════════╪════════╣
║httpLogStream│http | log │ │deployed║
╚═════════════╧═════════════════╧═══════════╧════════╝
Stream Deployment properties: {
"log" : {
"resource" : "maven://org.springframework.cloud.stream.app:log-sink-rabbit:jar",
"spring.cloud.deployer.group" : "httpLogStream",
"version" : "2.1.2.RELEASE"
},
"http" : {
"resource" : "maven://org.springframework.cloud.stream.app:http-source-rabbit:jar",
"spring.cloud.deployer.group" : "httpLogStream",
"version" : "2.1.1.RELEASE"
}
}
Test httpLogStream
stream using PCF CLI
Log into PCF using the cf login
command.
$ cf login -a api.run.pivotal.io -u "you@some-domain.com" -p "yourpassword" -o "your-org" -s "dev"
Output
API endpoint: api.run.pivotal.io
Authenticating...
OK
Targeted org your-org
Targeted space dev
API endpoint: https://api.run.pivotal.io (API version: 2.138.0)
User: you@some-domain.com
Org: your-org
Space: dev
Once the stream creation and deployment is successful, PCF creates random routes (urls) for both log and sink applications which can be validated using cf apps
command
$ cf apps
Output
Getting apps in org your-org / space dev as you@some-domain.com...
OK
name requested state instances memory disk urls
NEe1JMP-httpLogStream-http-v1 started 1/1 2G 1G NEe1JMP-httpLogStream-http-v1.cfapps.io
NEe1JMP-httpLogStream-log-v1 started 1/1 2G 1G NEe1JMP-httpLogStream-log-v1.cfapps.io
Post a sample message to the stream
Post a sample hello world
message to http
application using the route NEe1JMP-httpLogStream-http-v1.cfapps.io
as shown below. The message will be picked up by http
app and passed to log
application.
$ curl -i -H "Content-Type:application/text" -X POST -d 'hello world' https://NEe1JMP-httpLogStream-http-v1.cfapps.io
Output
HTTP/1.1 202 Accepted
Date: Sat, 04 Jan 2020 01:58:19 GMT
X-Vcap-Request-Id: c94b6a06-f3e2-4982-7f4b-34fbe3664713
Content-Length: 0
Connection: keep-alive
Once the message is posted successfully, hello world
will be printed in the logs of log
application.
Tail the log of log
application using cf logs
command
Tail the log of data-flow-server-hd6lIb0-httpLogStream-log
application using cf logs
command.
cf logs --recent NEe1JMP-httpLogStream-log-v1
Output
[httpLogStream-1] NEe1JMP-httpLogStream-log-v1 : hello world
Summary
Congratulations! You just deployed a stream to cloudfoundry using Spring Cloud Dataflow Server (SCDF).
Your feedback is always appreciated. Happy coding!
Related Posts: