h1 surf
- 2019-08-14

Today we’re happy to announce Surf, an asynchronous cross-platform streaming HTTP client for Rust. This project was a collaboration between Kat Marchán (Entropic / Microsoft), Stjepan Glavina (Ferrous Systems), and myself (Yoshua Wuyts).

Ergonomics

Surf is a friendly HTTP client built for casual Rustaceans and veterans alike. It’s completely modular, works directly with async/await, and is easy to use. Whether it’s a quick script, or a full SDK, Surf will make it work.

Take for example the classic GET request. With Surf it’s one line to make a request and print the response.

dbg!(surf::get("https://example.com").await?);

Similarly sending and receiving JSON is real easy too thanks to integration with serde:

let res = surf::post("https://httpbin.org/post")
    .body_json(&json!({ "name": "rey" }))?
    .await?;

#[derive(Deserialize)]
struct Ip { ip: String }

let Ip { ip } = surf::get("https://api.ipify.org?format=json")
    .recv_json()
    .await?;

And even better: we fully support streaming inbound and outbound request bodies. Not only can we upload files directly and while figuring out the Media Types (MIME) for you. You can also pass requests as the bodies to other requests, making it really convenient to proxy requests.

let res = surf::post("https://httpbin.org/post")
    .body(surf::get("https://httpbin.org/get").await?)
    .await?;

Write once, run everywhere

One our main inspirations for Surf has been make-fetch-happen, which is a JavaScript client that provides a convenient user interface for Node.js, built on top of the cross-platform fetch API.

Similarly we wanted to make sure Surf works in lots of environments, and thanks to the help of the Rust WASM WG we were able to get really far on this! Out of the box Surf will work on Linux, MacOS, Windows and in browsers. Setting it up in the browser is a matter of using wasm-bindgen-futures, and you should be good to go:

#[wasm_bindgen(start)]
pub fn main() {
    async fn main() -> Result<(), surf::Exception> {
        femme::start(log::LevelFilter::Trace)?;
        let body = surf::get("https://httpbin.org/get").recv_string().await?;
        log::info!("{:?}", body);
        Ok(())
    }

    spawn_local(async {
        let res = main().await;
        res.unwrap_or(|e| panic!(format!("{}", e)))
    });
}

This example uses log and femme for logging, and must be built with wasm-pack. Also WASM doesn’t support returning Result from start yet, so that’s why we handle the error manually.

Out of the box Surf uses curl on servers (through isahc ✨🐶), and window.fetch in browsers. Additionally we also ship a runtime-enabled hyper backend, which can be used by enabling the hyper-client feature in Cargo.toml:

[dependencies.surf]
version = "1.0.0"
default-features = false
features = ["hyper-client", "middleware-logger"]

Middleware

One of the core insights we had early on in development is that whatever needs we anticipate people might have, it’s impossible to hit the mark for everyone out of the box. That’s why Surf can be extended with middleware to run both after a request is sent, and after a response is received.

This allows a lot of important functionality to be built as middleware. Examples include caching, retries, logging, header injection, and more. A basic logger can be written in a few lines:

use surf::middleware::{Middleware, Request, Response, Next, HttpClient};
use futures::future::BoxFuture;

struct Printer;

impl<C: HttpClient> Middleware<C> for Printer {
    fn handle<'a>(&'a self, req: Request, client: C, next: Next<'a, C>)
      -> BoxFuture<'a, Result<Response, surf::Exception>> {
        Box::pin(async move {
            println!("sending a request!");
            let res = next.run(req, client).await?;
            println!("request completed!");
            Ok(res)
        })
    }
}

Almost all of this is boilerplate that can be copied between projects, and only exists because async in traits doesn’t work quite yet.

async-trait exists, but we couldn’t get it to work quite yet. We should follow up on this though!

Client reuse

An important part of building “serious” HTTP clients is building out SDKs that act as a dedicated client for a particular endpoint. This is especially convenient because it means that instead of needing to remember urls and methods, we can simply cargo add a client and we’re good to go.

To this purpose Surf supports persistent clients using the Client interface. Instead of performing one-off requests, these are intended to stick around and handle many requests.

let client = surf::Client::new();

let req1 = client.get("https://httpbin.org/get").recv_string();
let req2 = client.get("https://httpbin.org/get").recv_string();

let (str1, str2) = try_join(req1, req2).await?;

A fun implication of Surf’s portability means that it’ll become possible to write SDKs that will work in any platform. Even better: with Surf it becomes possible to use the same serde payload definitions on both sides of the wire, making it possible to add a lot of robustness to (JSON) APIs. We’re very excited for the interactions that this will enable.

Conclusion

It’s been a fun few weeks since Kat first raised that Entropic was looking for a new HTTP client. This is just the start for Surf. We’re incredibly happy the direction, and excited to be opening it up to the rest of the Rust community! You can check out the project on github, crates.io, and docs.rs.

Thanks for reading, and have a great Thursday!