Turning rusty tech into Rust ~ When you need to FTP but don’t want to

Hannes de Jager unFTP libunftp Rust

I believe that FTP is due for a makeover because it’s so ancient that few companies want to host it, yet so many customers still want to use it. That’s why fellow techies and I embarked on two open-source projects to develop a modernized FTP server for the cloud called unFTP.

In this writeup I would like to tell you about this server and library, how you can use, customize and extend it and finally ask you to help us make it even better by contributing to its Rust codebases.

unFTP you say…?

unFTP is an open-source FTP(S) (not SFTP) server aimed at the cloud that allows bespoke extension through its pluggable authenticator, storage back-end and user detail store architectures. It aims to bring features typically needed in cloud environments like integration with proxy servers, Prometheus monitoring and shipping of structured logs while capitalizing on the memory safety and speed provided by its implementation language, Rust.

unFTP is first an embeddable library (libunftp) and second an FTPS server application (unFTP). You can run it out of the box, embed it in your app, craft your own server or build extensions for it.

unFTP tries to untangle you from old-school environments so you can move all the things, even FTP, to the cloud while your users still get that familiar FTP feeling.

Did you stop to ask if you should?

“What year is it? Why are we talking about FTP?”

…might as well be a comment on this blog post similar to the one I’ve seen on Reddit the other day. And one has to understand this sentiment because this protocol comes from the ARPANET days in the early 1970s. I mean the Internet was not even a thing yet. And yet, in this day and age, it is interesting to see how widely it is still used as a medium of business system integration or file exchange with customers. Sometimes you can’t seem to escape FTP. Some of us are lucky or persistent and get our customers to move from FTP to FTPS and/or SFTP which were developed to address the growing security concerns of plaintext FTP. Most of us just make the best of it and try to touch those undiscussed servers as little as possible.

Here at bol.com - the largest online retailer in the Netherlands and Belgium and tech business employer of scores of IT people that build its seller platform - we are not able to escape FTP just yet either. We need to FTP even though we prefer not to. We also need to integrate with FTP from our microservices running on Kubernetes in the Cloud. Developers expressing their latest platform innovations in languages like Kotlin and Go are not particularly empowered by technologies like FTP. No, it’s rather a millstone around the keyboard and yet, the need for data remains king. Vendors like AWS offer great solutions but still, sometimes custom business integration needs makes the use of even this not viable. This is one of the reasons why at bol.com we decided to develop unFTP.

OK, fair enough, how do I run it?

If you’re on Linux or macOS then you can head over to the unFTP home page at github.com/bolcom/unFTP and download the binaries from there.

Chances are, though, that you would like to run this in Kubernetes. Here is an example of running unFTP in a docker container to nudge you in that direction:

docker run \
  -e ROOT_DIR=/ \
  -e UNFTP_LOG_LEVEL=info \
  -e UNFTP_FTPS_CERTS_FILE='/unftp.crt' \
  -e UNFTP_FTPS_KEY_FILE='/unftp.key' \
  -e UNFTP_PASSIVE_PORTS=50000-50005 \
  -e UNFTP_SBE_TYPE=gcs \
  -e UNFTP_SBE_GCS_BUCKET=the-bucket-name \
  -e UNFTP_SBE_GCS_KEY_FILE=/key.json \
  -e UNFTP_AUTH_TYPE=json \
  -e UNFTP_AUTH_JSON_PATH='/secrets/unftp_credentials.json' \
  -e UNFTP_LOG_REDIS_HOST='redislogging.internal.io' \
  -e UNFTP_LOG_REDIS_KEY='logs-list' \
  -e UNFTP_BIND_ADDRESS='0.0.0.0:2121' \
  -e UNFTP_PASSIVE_PORTS='50000-50005' \
  -p 2121:2121 \
  -p 50000:50000 \
  -p 50001:50001 \
  -p 50002:50002 \
  -p 50003:50003 \
  -p 50004:50004 \
  -p 50005:50005 \
  -p 8080:8080 \
  -ti \
  bolcom/unftp:v0.12.10-alpine

We have a couple of docker images availale on docker hub. For instance there is also an alpine image with scuttle installed if you need to run with the istio service mesh.

From the above example you can extrapolate how one can deploy to Google Cloud Platform and:

run unFTP in a Kubernetes pod
store files in a Google Cloud Storage bucket
Get TLS certificates from Let’s Encrypt via Google Cert-Manager
authenticate against encrypted credentials in a JSON payload sourced from a Kubernetes Secret
have a Prometheus instance in our environment scrape metrics from a the /metrics endpoint running on port 8080
ship logging to a redis instance from where its picked up by Logstash to find its home in an ELK stack.

Nice, I’d like to build my own

You can embed libunftp in your Rust app or create your own FTP server with libunftp in just a couple of lines of Rust code. If you’ve got the Rust compiler and Cargo installed then create your project with:

cargo new myftp

Next let’s add the libunftp and tokio crates to your project’s dependencies in Cargo.toml:

[dependencies]
libunftp = "0.17.4"
tokio = { version = "1", features = ["full"] }

We will also need a dependency on a storage back-end where the files will go. We’ll use the filesystem back-end (unftp-sbe-fs) here:

[dependencies]
...
unftp-sbe-fs = "0.1"

Finally, let’s code up the server! Add the following to src/main.rs:

use unftp_sbe_fs::ServerExt;

#[tokio::main]
pub async fn main() {
    let ftp_home = std::env::temp_dir();
    let server = libunftp::Server::with_fs(ftp_home)
        .greeting("Welcome to my FTP server")
        .passive_ports(50000..65535);
    
    server.listen("127.0.0.1:2121").await;
}

The above creates an FTP server that uses a filesystem back-end that will put and serve files from the operating system temporary directory. We set the greeting message an FTP client will see when it connects and we define the port range that the server can listen on for data connections originating from the FTP client. Lastly, we listen for control connections on port 2121 on the local host. Not that useful yet but you get the idea.

Let’s proceed to run your server with cargo run and connect to localhost:2121 with your favourite FTP client. For example:

lftp -p 2121 localhost

This should allow you to upload and download files to and from your temporary directory via FTP.

I have this other authentication system…

Of course you do. If you’re opting to create your own FTP server with libunftp then chances are that you will also be implementing your own storage back-end and/or authenticator. To illustrate how this is done we show you how to implement an authenticator that will always give access to the user. Start off by adding a dependency to the async-trait crate in your Cargo.toml file:

[dependencies]
...
async-trait = "0.1.50"

Then implement the Authenticator and optionally the UserDetail trait:

use libunftp::auth::{Authenticator, AuthenticationError, UserDetail};
use async_trait::async_trait;


#[derive(Debug)]
struct RandomAuthenticator;

#[async_trait]
impl Authenticator<RandomUser> for RandomAuthenticator {
    async fn authenticate(&self, _username: &str, _password: &str) -> Result<RandomUser, AuthenticationError> {
        Ok(RandomUser{})
    }
}


#[derive(Debug)]
struct RandomUser;

impl UserDetail for RandomUser {}

impl std::fmt::Display for RandomUser {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "RandomUser")
    }
}

Lastly, register this authenticator with libunftp:

let server = libunftp::Server::with_authenticator(
  Box::new(move || { unftp_sbe_fs::Filesystem::new("/srv/ftp") }),
  std::sync::Arc::new(RandomAuthenticator{})
);
.greeting("Welcome to my FTP server")
.passive_ports(50000..65535);

If you don’t want to allow a user access to the files then simply return a AuthenticationError from the authenticate method above.

Call for help

unFTP as it stands today provides a minimal viable product for certain use cases but is nowhere near the one stop solution for FTP that it can become. For this a village of contributors are needed. A secret hope with this blog post is that we inspire warriors that will pick a fight with the Rust borrow checker and bring a unFTP ecosystem to life that we can be proud of, and, dare I say, perhaps even to the point where people may want to FTP even though they don’t need to :-).

We would love to see many unftp-* results returned when a search is done on crates.io, the Rust package registry. Results ranging from the obviously needed to the super creative. We imagine a state where FTP integrators can bring their very custom needs to crates.io and have a shopping list to pick from:

Authenticator implementations like:

unftp-auth-ldap
unftp-auth0

User detail stores like:

unftp-usr-postgres
unftp-usr-mysql
unftp-usr-auth0

Storage back-end implementations like:

unftp-sbe-s3
unftp-sbe-azure-blobs
unftp-sbe-chrooted-fs
unftp-sbe-dropbox
unftp-sbe-gmail

The project also needs contributors to its core because what’s the use of a lot of extensions without a solid core product:

Experts in FTP or ardent RFC readers to extend on the provided FTP protocol command implementations.
Experts in distributed computing to help build a truly scalable and highly available FTP solution for the cloud.
People helping with testing and benchmarking.
Implementors for things like event notification to cloud pubsub, a possible new extension point.
Scriptability through Lua.
Security experts to help ensure libunftp is as secure as it can be.

unFTP also needs to be available. So the project also needs:

Developers experienced in packaging for deployment: .deb, .rpm, archlinux
A Homebrew formula.
Support on more platforms: Windows, Arm?

And last but not least, good documentation with pleasant pictures.

Summary

In this post we explored the libunftp crate for Rust and its companion server project unFTP that was birthed by the often found need to run FTP even though it would be much rather avoided. We touched on how unFTP is aimed at solving custom FTP integration challenges in today’s cloud environments. We gave an introduction of running the server and how to use the library and invited our readers to get their hands dirty with Rust by contributing to the unFTP project. We hope to meet some of you in the near future even though Covid-19 won’t allow us to shake hands.

Hannes de Jager

Software Builder