Run Robotframework service in Google Cloud
I wrapped a robotframework task inside a webservice, deployed it in to Google Cloud and trigger it now on schedule.
It is no secret I use Robot Framework for all my automation projects. I had this robot task that ran for more 18 months smoothly in a scheduled GitLab CI pipeline. But last week, a job got stuck and consumed all my remaining processing minutes of this month. A good opportunity to rethink the setup and move to Google Cloud.
The task crawls data from a website that has Shadow DOMs, so BrowserLibrary with Playwright is the library of choice here. SeleniumLibrary is easier to setup, but unstable under these conditions. The crawled data is uploaded to a FTP server, therefore SSHLibrary is on board, too.
Goal
- Wrap Robot Task in container
- Trigger execution of task every 2 hours
Run task as webservice
Cloud depends on APIs. But a robot task is shell script. How could you trigger that through an API? By placing the task inside a webservice. There is the robotframework-webservice project that does exactly that. It is right now in prototype state, but it is sufficient for this project:
You simply define task-directory on startup with paramter t. The last command produces following output:
Afterwards you can access API documentation at endpoint docs, for example http://localhost:5003/docs
We now have several endpoints available to trigger any task in the directory we provided at parameter t during startup of the service.
Define image
We now define a Dockerfile as build recipe for our service. As I need BrowserLibrary, I do not go through the struggle installing node-js on top of some slim python image, but use directly the image provided by the maintainers of BrowserLibrary:
Be aware that you can also configure a custom port for your webservice, if you like. Now, what can cause you a lot of trouble using rfbrowser as base image are user privileges . The maintainers are experts and follow best practice not executing everything as root user. However, if you had been lazy like me, it will cost you a lot of trial and error to get it right. Your task will fail due to lack of some permission, because output directory cannot be created, if you do not follow best practice. For that you must check the Dockerfiles from BrowserLibrary GitHub repository.
Deploy on Cloud Run
Go to your Google Cloud Console , create a project for your service (if needed), and open Cloud Run on the menu top left.
If you do not find Cloud Run, use the search bar instead. If Cloud Run needs to be activated first, activate it.
There are several ways to deploy this container to Google Cloud. I choose Cloud Run using the gcloud cli. First you need to install gcloud cli, of course. Afterwards simply open a terminal in your main folder of the project and call
You need to authenticate your cli, which you do once and is guided on command line. You also need to create some cloud storage for docker artefacts and activate Cloud Build, but as with authentication, you are guided through that process. It takes some time, after you are finished, you find your service on in Cloud Run:
Access Webservice API in Browser
Click on the service. You see an url on top. If you granted public access, you can click on it and attach /docs to the end in order to access the Swagger-UI. You can trigger your task now from here using the endpoints from this documentation!
Setup robot logging
Go back the service and open the logs panel in order to validate, if your service throws any exceptions. You should see startup messages from the webservice. But you miss logging from Robot. Luckily, you can easily tell robot to create some system logs file by setting the environment variable ROBOT_SYSLOG_FILE . Click on Edit & Deploy New Revision :
And then choose Variables & Secrets
Here you can define environment variables for your container and thus determine that robot syslog shall be stored in /var/log . Why /var/log ? Because Google Logging exports all logs found in that directory. Click on deploy and wait a few seconds until the container is redeployed. Call the webservice again and execute a task. You should now see logs from Robot.
Hardware Resources
You might not get a lot of logging, if you kept the hardware resources to default. My task requires 4 GB of RAM. 1 cpu is enough. But sometimes, it even exceeds even 4 GB of RAM. When your service consumes more resources than reserved, the service is cancelled immediately.
You define the hardware resources as well as scaling options at the Container panel next to Variables & Secrets
Create Cloud Scheduler Job
So, you have now a service that is waiting to be triggered. Create a Cloud Scheduler job by opening Cloud Scheduler from the menu:
Creating a Cloud Scheduler job is straight forward. Click on Create Job on top of the table and configure it accordingly. Important the target type. It must be HTTP. Provide the URL to your desired endpoint of the webservice and the HTTP method (mostly GET).
Next steps
Google Cloud will show you some hints: how to use secrets in case you want to inject passwords as environment variables. Or to setup a custom service account for this explicit service with most limited amount of privileges.
As I am also maintainer of robotframework-webservice, I will make robotframework-webservice production ready. I will also combine the service task with Camunda 8, which offers outbound http connectors. That should replace the scheduled job and thus open a playground for a swarm of orchestrated robot tasks.
Conclusion
Now you have a robot task webservice that is triggered frequently from a job. In Cloud Run configuration you can even define how many instances of your task service shall run concurrently, how many requests per service are accepted and what the maximum amount of instances are. In this example, we only need 1 instance at most, because the scheduling job will only trigger once in while. No need to waste hardware resources.
Set the minimum amount of instances to 0. You will then see that your webservice is shutdown gracefully after 15 minutes of inactivity. That saves energy and budget. With a setup like that you can easily run your task for less than 1 cent per day. The webservice will start automatically as soon as a new request comes in.
