Packed and bitfield annotations

[fanpei91]

Currently solod does not have directives about using packed and bitfield on struct. I hope that there will be such two directive annotations, which can be used like this: .

//so:packed
type sdshdr16 struct {
    len uint16
    alloc uint16
    flags uint8
    buf [0]byte
}

type node struct {
    size uint64
    count uint32 //so:bitfield(2)
    encoding uint32 //so:bitfield(2)
    recompress uint32 //so:bitfield(1)
    extra uint32 //so:bitfield(27)
}

Thank your job! Solod is the system programming language I am looking for years.

[nalgeon]

Thank you! I can't promise these, but I'll look into it while working on v0.2. Do you have a specific use case for these structures (sdshdr16 and node), or are they just examples?

[fanpei91]

I am rewriting resis 8/valkey 9 using solod. These are all real data structures:

struct __attribute__((__packed__)) sdshdr5 {
    unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
    char buf[];
};
struct __attribute__((__packed__)) sdshdr8 {
    uint8_t len;         /* used */
    uint8_t alloc;       /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__((__packed__)) sdshdr16 {
    uint16_t len;        /* used */
    uint16_t alloc;      /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__((__packed__)) sdshdr32 {
    uint32_t len;        /* used */
    uint32_t alloc;      /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__((__packed__)) sdshdr64 {
    uint64_t len;        /* used */
    uint64_t alloc;      /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};

typedef struct quicklistNode {
    struct quicklistNode *prev;
    struct quicklistNode *next;
    unsigned char *entry;
    size_t sz;             /* entry size in bytes */
    unsigned int count : 16;     /* count of items in listpack */
    unsigned int encoding : 2;   /* RAW==1 or LZF==2 */
    unsigned int container : 2;  /* PLAIN==1 or PACKED==2 */
    unsigned int recompress : 1; /* was this node previous compressed? */
    unsigned int attempted_compress : 1; /* node can't compress; too small */
    unsigned int dont_compress : 1; /* prevent compression of entry that will be used later */
    unsigned int extra : 9; /* more bits to steal for future usage */
} quicklistNode;

Due to the lack of packed, I use []byte to achieve this. For the absence of bitfield, I use a single uint32 field to partition it as 16 + 2 + 2 + 1 + 1 + 1 + 9 bits. The code readability is not good as the C language version.

[nalgeon]

I see, thanks for explaining!

[fanpei91]

Regarding the bitfield feature, if future support is planned, an approach like the following may be a choice:

type node struct {
    size uint64
    count uint32 `so:"bits:1"`
    encoding uint32 `so:"bits:2"`
    ...
}

[nalgeon]

Starting with 600e881, you can use the so:attr directive to add GCC attributes:

//so:attr packed
type sdshdr16 struct {
	len   uint16
	alloc uint16
	flags uint8
	buf   [0]byte
}

To install the latest version:

go install solod.dev/cmd/so@600e881fe72c
go get solod.dev@600e881fe72c

But honestly, I doubt that porting something as big as Redis to So makes much sense. It's probably better to just write some bindings and call it, instead of porting the whole thing.

[fanpei91]

Nice! Smart design!

I am implementing Redis using Solod in order to deeply understand it, and the best way to deeply understand it is to re-implement it in another language. I have previously implemented the standalone version of Redis 5 in Zig, but the experience was not enjoyable—it was too verbose, function parameters can not be reassigned, and goto was removed from the language. A few days ago, I looked into Odin; it is largely good, but lacks goto. I have always believed that an excellent systems programming language should have simple syntax and come with a rich standard library. As a Go developer, I appreciate Go’s design philosophy. Solod is exactly that kind of language—a pragmatic language I have been looking for for years.

Very thank you for your hard working!

[nalgeon]

Wow, this is really impressive! I once thought about porting Redis to Go, but after looking through the source code, I realized it would be way too much work.

Thank you!

[fanpei91]

It is difficult to implement AOF rewriting and RDB generation in Go. To obtain a memory snapshot, AOF rewriting and RDB generation need to be run in a subprocess. Even if the subprocess is forcibly forked, the GC is modifying memory at every moment, COW is happening in the kernel at all times, memory usage spikes, and the point of reimplementation is completely lost. If these two tasks were done within a sub-goroutine, the data structure would need to implement MVCC, making it very difficult and requiring high memory consumption. So you can imagine how happy I was to see Solod!