There are numerous formats for configuration tables: XML, JSON, SQLite, ScriptableObject, Protobuf, and more. In Lua-based projects, many developers still export configuration tables directly as Lua tables. However, using a straightforward Key-Value (KV) approach is clearly not optimal in terms of memory usage or parsing time.

To address this, we have developed several optimization strategies:

1. Structural Optimizations

• Default Value Extraction: Fields that appear most frequently are extracted as default values to reduce redundancy.
• Key-Value Separation: By separating Keys and Values and storing data in arrays, we significantly save memory and improve parsing speed. Testing shows that parsing time is halved with this method.
• Field Deduplication: Identical fields can be further abstracted, providing an additional 30% performance boost.

I have provided a simple implementation for exporting Excel to optimized Lua tables here:

GitHub: https://github.com/506638093/ExcelToLua

2. The setconfigtable Approach

While using setmetatable with an __index function to retrieve values is viable, it inevitably incurs the overhead of a Lua function call. Using rawset is another option, but it increases memory consumption.

To solve this, I introduced a new method called setconfigtable to retrieve values by modifying the Lua source code. This involves adding a new flag, TM_INDIRECT_INDEX, and modifying luaV_finishget.

Core C Source Modification:

C

/* Added flag: TM_INDIRECT_INDEX */

LUA_API int lua_setconfigtable(lua_State *L, int objindex) {
    // ... (Logic to set metatable and flag the TM_INDIRECT_INDEX)
    if (mt) {
        mt->flags |= (1u << TM_INDIRECT_INDEX); 
    }
    // ...
}

void luaV_finishget (lua_State *L, const TValue *t, TValue *key, StkId val, const TValue *slot) {
    // ... inside the loop
    if (tm == NULL) {  /* no metamethod? */
        /* Custom Logic: Indirect Indexing */
        if (mt != NULL && (mt->flags & (1u << TM_INDIRECT_INDEX))){
            mk = luaH_get(mt, key); // Find the index/key in the metatable
            if (mk != NULL){
                slot = luaH_get(ot, mk); // Get the actual value from the original table
                setobj2s(L, val, slot);
                return;
            }
        }
        setnilvalue(val);
        return;
    }
    // ...
}

This approach is over 20 times faster than using a standard Lua-level __index metamethod.

3. Handling Large-Scale “Cold Data”

If a project’s configuration files are excessively large, Lua’s memory usage becomes enormous upon loading. In reality, most configuration data is “cold data” (rarely accessed).

• Text-based Databases: Replacing Lua tables with a database like SQLite is a solid alternative. Optimized SQLite performance is comparable to Lua tables because it avoids full-file loading.
• FastDB: For read-only requirements, FastDB is an excellent solution, performing 4–5 times faster than SQLite. Although the project hasn’t been maintained recently and contains some bugs, it is worth reviving for high-performance configuration needs.
  • Reference: gavioto/fastdb