Introduction

Recently, I've been developing an experimental project called ccache, which is a Redis client-side caching that guarantees consistency. Since it operates on the client side, I need to ensure it supports different programming languages.

The common practice is to write similar logic in different languages, as seen with Redis client and OpenTelemetry instrument library. However, this approach involves tedious and repetitive work.

One potential solution is to write the core functionality in Rust and integrate it with different languages. To achieve this, we need to address the discrepancies between Rust and other languages, such as how to represent data and manage memory safely.

Rust2go is a practical FFI framework that enables calling Go from Rust. In this article, I will introduce how it works.

Benefits

Due to its low overhead and safety guarantees, Rust has been integrated into many other systems traditionally written in C, such as Ruby and Linux.

Integrating Rust with other high-level languages is beneficial as it can improve performance and reduce repetitive work.

For example, ByteDance reduced CPU usage by more than 30% after migrating a core service from Golang to Rust[2].

Additionally, OpenTelemetry supports 11 languages[3]. Using Rust for core functionality can significantly reduce development efforts and prevent inconsistencies between different language implementations.

How Rust2go Works

Calling Go Functions

After building the Go code into a library and linking it to Rust, the Go functions become accessible within the Rust project.

However, Rust and Go have different calling conventions, so Rust cannot directly call Go functions. One solution is to use a trampoline to handle this issue[4]. Due to the unstable Rust ABI and the desire to address goroutine stack expansion, the author of rust2go chose not to use this method.

rust2go uses the C ABI as a "bridge" between Rust and Go. The Go functions are exposed as C functions through cgo, and Rust calls these C functions.

Memory Representation

Rust and Go represent structs in different ways. In Rust2go, a struct is first converted to a C struct and then to a Go struct. For example, a Rust struct DemoUser is converted to DemoUserRef and then to a Go DemoUser.

pub struct DemoUser {
    pub name: String,
    pub age: u8,
}

typedef struct DemoUserRef {
  struct StringRef name;
  uint8_t age;
} DemoUserRef;

func newDemoUser(p C.DemoUserRef) DemoUser {
    return DemoUser{
        name: newString(p.name),
        age:  newC_uint8_t(p.age),
    }
}

And then it coverts the primate types, for example, StringRef is converted to Go string by newString

func newString(s_ref C.StringRef) string {
    return unsafeString((*byte)(unsafe.Pointer(s_ref.ptr)), int(s_ref.len))
}

func unsafeString(ptr *byte, length int) string {
    sliceHeader := &reflect.SliceHeader{
        Data: uintptr(unsafe.Pointer(ptr)),
        Len:  length,
        Cap:  length,
    }
    return *(*string)(unsafe.Pointer(sliceHeader))
}

I will explain why Rust2go uses XXXRef in the next section.

Passing Variables Between Rust and Go

In the previous section, I explained how rust2go understands structs in Rust and Go. Now, I will explain how it passes variables between the two languages.

Passing Arguments to Go

The most simple and straightforward method is to use serialization protocols like Thrift and Protocol Buffers. Rust2go does not choose this method as it wastes CPU time converting the data back and forth.

Instead, it passes arguments through pointers and converts the data to make it understandable for Rust and Go. This avoids deep copying, such as strings and binary data.

This method adheres to Rust's safety rules because the arguments are "borrowed" by Go, and the memory is "owned" by Rust. Once Go finishes using the data, it frees its allocated memory, but the variables' memory allocated by Rust is not freed by Go.

Receiving Return Variables from Go

The return variables are created by Go, so Go can free them when necessary. Rust calling Rust does not have this problem because the variable can own the return result, such as let x = some_func();.

Rust2go handles this by copying the variable in the C callback so that Rust and Go can manage the "same" variable independently.

Summary

This article provides an introduction to how rust2go works. For more details, please refer to the author's article[2].

References

1. https://github.com/ihciah/rust2go
2. https://en.ihcblog.com/rust2go/
3. https://opentelemetry.io/status/
4. https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-guide-to-the-go-laxy/