hello everyone, my name is George Vafeiadis,
I'm a senior software engineer working for a fintech company called Quantile Technologies.
In Quantile, we're using Guix for most of our production systems
Our systems are mainly written in Python and C++
Python for the business logic and C++ for highly optimized numerical algorithms.
We believe Guix gives us a competitive advantage, compared to the previous approaches,
we use it to handle packaging and deployment
and those are very transparent approach in terms of
having all the dependencies and knowing which package is used across all the company.
Today I'm going to discuss how we're using Guix to build AWS Lambda images.
AWS Lambda is a service from Amazon that is used to run code without provisioning or managing a service
Your code… You organize your code into Lambda functions
and it's a fast and easy way to build a scalable service.
The platform is taking care of all the required configuration for managing the service,
scaling the service, and logging, and security, and all of these things.
So you just provide your own function,
and the platform is doing the rest.
To deploy a lambda function you can use a zip file or you can use docker.
The zip files can seem limited in terms of the size and programming language you can use
So Docker is the only real alternative
especially if you need to handle native dependencies
so in our case, using Docker was the only approach we could do.
The only problem is that AWS provides some base images for Python in particular,
which are compatible with AWS Lambda in order to build your own Docker images.
The only problem is, we wanted to use Guix
because we use Guix for all our code, so
using Guix for AWS Lambda was also something we…
It was a requirement for our case.
The base images Amazon provides are Open Source.
So it was one approach that we could use
so we can actually use one of the base images
and then add any other dependencies.
Guix provides a way to have a relocatable binary so it was one approach we could use.
But we wanted to avoid the extra step,
of adding some other particular code that is not handled by Guix
so we want Guix to build the full image,
including all the
all our code and all the dependencies for our app.
The only way to do that is,
to use a custom AWS runtime.
Amazon provides some documentation on how to implement an AWS Lambda runtime API.
In our case for Python,
there is an AWS Lambda Python runtime interface client.
This client is a Python package that also bundles some C++ components internally,
that you can use if you want to create a custom image.
Our first approach,
in order to make Guix create an AWS Lambda compatible image is
to check if this package is actually provided by Guix.
When we were checking it was not available, so the next step was to actually build this package.
This awslambdaric is the runtime interface client,
that we had to bring in to Guix in order to build anything based on that.
So, this is what I'm going to show you today.
We started with this packaging,
here you can see we created our own package,
because it's not provided by the main Guix repository
So this python-awslambdaric
is an Open Source…
…package.
We had to create a Guix recipe in order to make it available in Guix.
Let me show you…
the code…
This is how we are def…
This is a typical Guix package.
We define a module,
(all the Guix packages are located in a module)
so in our case we just call it aws
and python-awslambdaric.
Here you can see some of the modules that we're using,
and then here we define our module.
So we're using the URL which is the actual code that Guix is going to fetch
it's available here, in Github.
We specify which version we're interested in,
and then this is the hash that uniquely identifies the package.
Any tampering or anything is going to break,
if for any reason the location of the package changes,
So this what is important for us
because in our domain we need… This is what actually Guix is giving us.
It's a very tranparent way and an easy way
to audit all the code,
and you know,
every single time,
all the dependencies and where they're coming from.
In order to build this Python package,
we use a build system wich is the python-build-system.
This is provided by Guix for building Python packages.
In this case,
the code is a little bit more than the typical Python package,
because internally, this AWS Lambda runtime
they have their own curl and their own C++ component
There is one called aws-lambda-cpp.
which has some C++ component but also a couple of patches
Here we just add and modify the standard phases,
the phases are all the steps Guix is doing in order to build your package.
When we actually build the package, it's going to fetch the source code,
and then here we add another hook,
which, we are saying, after the unpacking phase,
just extract the dependencies
so if we check the source code of the github of the AWS Lambda Python runtime,
We can see that internally, it downloads a specific version of curl, a specific version of the awslambda-cpp,
and it's applying a couple of patches.
So we do some patches on our side here,
in order to make it build,
and then we're actually invoking the configuration inside the Guix environment.
It's going to build the actual dependencies,
which are the specific version of curl and the specific version of aws-lambda-cpp
One approach could be to have them as dedicated packages,
this is the common approach
We could have curl specific package
and reference it in the inputs of the Guix package.
But in our case,
because it requires some specific patches that are not useful for any other package,
We decided just to… handle everything inside the main package description.
so the next step is just building the second one, the aws-lambda-cpp
and you can see here,
the native-inputs.
Those are the inputs that are used for building the actual Python package.
Thos packages are used typically for supporting the building process.
So this means for running tests,
linters,
and all of these things
So here we can see all those dependencies
Another interesting thing was,
because it's using a custom version of aws-lambda-cpp,
it required a specific version of binutils due to a bug
that the main code had.
This is why here we're just using a specific version of binutils.
that's required by the package.
The only propagated-input is simplejson.
Here you can see the full description of the recipe,
that we use for the AWS
Lambda runtime.
The code for…
The code is not yet in the upstream,
so we're planning also to…
to push it to the main guix, but
in the meantime you can actually use it freely from the guix-quantile
guix-quantile is our own channel,
that we try to…
We will try to push more of our packages there,
and then eventually also we're going to push to the main…
the main Guix repository.
If you need it, you can always find it here.
It's the same code we are releasing as an Open Source package.
Let's see now how we can…
We have defined our main dependencies
This is quite important
because the runtime is what makes AWS Docker image actually compatible with AWS Lambda.
So now we have it there, we can actually
build it
I have built it before, so that's why it is quite fast,
but let's see also what is inside.
So you can see that it is a proper Python package,
that has all the required runtime and that is under the /gnu/store,
So we have this ready.
Now, the next step
Let's say that we want to bundle our own Python application,
create a Docker image,
the Python application is going to provide the AWS Lambda
so in our case,
we are going to use this simple application.
which is just a simple handler
This is a lambda function
The lambda function just defines an event,
defines a function that has parameters which are the event and the context
so you can get something…
When someone is invoking a lambda function,
you can actually get all the parameters
so we have a very simple, here, lambda function
It's just going to return a string
This is the one application that we would like to create a Lambda-compatible docker image [for]
Another thing is
we need to define an entry
The entry.sh is the main entrypoint
that will be used
when we deploy the image
So this is the entrypoint that is going to run,
the first time it's going to deploy the image
Mainly, we use this in order to source the /etc/profile
Sourcing the /etc/profile is going to get everything that is defined in /gnu/store.
and then we just call the main Python module by passing the handler, pretty much
So those are the two files.
Now, let me show you,
what are the steps.
In order to build a Docker image,
in Guix
Guix provides this "guix pack" command.
The "guix pack" command is able to create a number of outputs.
One of them is a Docker image.
So, we can see here, in our case,
We want to…
In our example we're not going to bundle the app I showed you earlier,
using Guix, so we're going to keep it as a second stage,
But this is something we can actually add here
So if we have an actual recipe for the app.py application,
we can add it here as a package.
So here, I'm going to show you…
how to actually build this Docker image.
So first,
we just define an environment variable,
we use this dependency, which is the awslambdaric we defined earlier,
We would like to have bash,
we'd like to have a minimal Python,
and we need to have coreutils.
Here, you can actually put any Guix-provided package.
or any custom package for your application.
So it's very flexible, you can add all your dependencies here.
So we use "guix pack",
The -RR is… we'd like to have relocatable binaries.
We define here a local registry for packages
which is actually the one I showed you earlier for AWS,
If you're going to use our channel,
you can avoid this, but you need extra configuration for…
In our case we just pass it here locally.
So we have defined this package description
Guix is going to search for any package,
apart from the main channel,
it's going to search for packages in the ./packages here.
And then we create a symbolic link for easier…
just to make it a bit easier to…
to have them in the image.
So this means it's just going to mount this… Guix specific /bin in this specific location.
so let's do that
So we define the command we're going to run,
So the next step is: we're going to run the command,
this is a simple bash script, pretty much
and then we're going to load and create the actual image
Let's do that.
So now it's going to actually create the image.
You can see here,
it's creating a tar.gz,
bundle, that has all the dependencies,
which means it has a dependency for the Lambda runtime,
bash,
python-minimal,
and coreutils, and anything else we want to put here.
So this means now, we can see here,
we have a Docker image in tar.gz form,
and it is located in /gnu/store.
Now, we're going to do a couple of other things.
The first thing,
we're going to actually create the Docker image.
because now,
let's see what is the result,
the result is, you can see, "Loaded image"
Let's get the actual image.
OK, now we have created a local docker image.
We can actually see here, images
We can see that we have created the Docker image.
I'm going to tag the image.
The tag is just adding, and this is Docker specific stuff, we just put a Docker specific name.
And then, we're going to do a second step.
The second step,
This is something… in order to add this app.py application inside the docker file.
But this is something we could avoid if we had used
the actual packages here.
So we would have bundled our simple application as myapp here
We could have avoided it because Guix would have added it to our main archive.
So, let me show you what is this docker file.
This is a simple Docker file,
It's going to use the Guix base that we created earlier,
by using Guix,
This would be our built image,
Then we're just going to copy the directory I showed you earlier with the application,
I will add the entry point,
which is the entry script,
and then, the final thing is going to just specify what is the actual handler.
So we have the entry point and the handler.
So this is pretty much the only Docker description,
So let's do that.
It takes some time to pretty much create the…
to copy everything from the build image in order to generate the final Docker image.
This is the most time-consuming operation along with the process that is building the actual…
is packing from Guix, all the dependencies.
takes a bit of time…
And then I will show you how to actually,
what is going to be inside when we have the actual image.
Now you can see it's completed.
Let see, "docker images"
You can see it's there,
So now I'm going to run it,
and instead, I'm going to use as entrypoint the /bin/sh.
This will allow us
OK, now I'm inside, this will allow us to see what are the files inside the Docker image.
So we are now in the /home/app,
as we have specified,
we can see here our application
As I said, this is not required if we had bundled also this app as a Guix application.
We can go here. We can see here everything is Guix-specific.
We can see we have the /gnu/store,
with all the dependencies that are required in order for our application to run,
so they are mainly python specific, but also the dependencies of some of the dependencies,
This is something that can be optimized,
You can see some stuff that you might say you might not need it.
but they are required by some of the packages.
So this is typically quite small.
the actual Docker image,
because it has only the dependencies that are required in order to run it.
and then we can see here in the /bin,
that all of them are symbolic links to the actual /gnu/store.
Everything from the operating system files, up to the application,
are all managed by Guix.
So this is our Guix Docker image.
So, now,
that we have our image,
we need to use,
we need to use the AWS registry in order to push the image there.
So first,
We define our own registry,
then we create a repository,
the repository is going to be called "guix"
this is creating in AWS an actual repository.
So if I go to the… here,
we can see now,
in the container service, we can see that we have created a registry.
So this is just a place for pushing the image,
that is going to be used later on by the AWS Lambda.
Now we have created the image,
let's now
we're already authenticated
Now let's tag the image.
so we're going to just tag it
and then we're just going to push the image.
We take some time to upload the image to the registry.
One thing that we have seen,
the packages are quite small in general,
Definitely there are some packages that are smaller than that,
but this means that there are some dependencies that Guix is using,
which are using some stuff that are not required, in a way, in the runtime.
but this means we might need to do some cleanup work,
in terms of finding out which packages are using things that we don't care.
For example, we might have a Guix package that bundles a library,
and maybe we don't need this library in the runtime,
This is something that we have seen we can improve a lot,
but this is something for future work.
Now, this process has completed successfully.
Now if we go to…
Now let's create the actual Lambda function.
The actual Lambda function
Let's create from the UI
So we have the container image, let's say mylambda,
Now let's browse the images,
guix…
It's one minute ago
and then let's create the function here
alright…
ok, now it's creating our Lambda image,
Our Lambda function.
The Lambda function is…
is going to use our Guix…
Docker image
In order to…
to run.
Let's wait a few seconds just to create the function.
And as I was saying, this is very convenient for us,
because we know now that we have full control of what we actually have in our Docker image.
so this means that vulnerabilities,
or things that are…
before that we had to find out if there are any vulnerability in all the layers in a Docker image,
This is something that goes away because Guix is handling all layers, so pretty much we have full control of what we are bundling there.
Ok, now we have the Lambda image,
so we can actually test it.
The test is just invoking the function.
typically the first time it takes a few more seconds to load, then it's a little bit faster.
Alright, so we can see here,
that our…
brand new Guix Docker image is run successfully,
it's running our application,
and everything is handled by Guix.
So, you see,
It takes some… it took… the most…
In order to do the whole process,
it was mainly how we can have the main runtime available
So this goes to the main package I showed you earlier.
It goes the only major work.
Apart from that, it goes pretty much a few scripts
Defining the packages
Packaging the Guix Docker image,
and then pretty much push the image and make it available.
So this approach could be used for many serverless platforms.
Currently we are just building for AWS specific, but if you are using Kubernetes,
or maybe Mesos,
or Yarn,
or some other cluster management that requires to have dependencies which are native,
or a hybrid with native and non-native dependencies,
Guix is a good tool,
is a great solution actually in order to bundle all of this and have full control.
Because now, we know exactly what will… we can actually see all of this
where they're coming from,
all the dependencies of the dependencies,
so there is nothing that…
the transparency we're getting out of this is quite important.
So, this is pretty much my simple demo.
As I said, the code is available,
you can actually…
take it from there,
it's an actual channel, but we're planning also to…
push it in the main Guix repository.
For now, we just use this Github
and you can either clone it locally,
or you can configure your Guix to use it as an external channel,
so you can have this package available.
This is how it is.
OK, thank you very much for your time.
Hope you enjoyed the presentation. Bye bye.