Multi-rank send of a complex conduit node returns corrupted data (nan / garbage)

Bugs and Error Reporting
1

What I’m seeing

I’m using the C++ client API (melissa::send(node_name, conduit::Node&)) and sending one node per timestep. The node has three leaves:

Leaf Type How it’s set
time float64 scalar by value (node["time"] = t;)
p float64 array (~N elements) external pointer (set_external_float64_ptr)
U float64 array (~3·N elements) external pointer (set_external_float64_ptr)

On a single rank, everything arrives correctly. On more than one MPI rank, the payload reaching the server is corrupted: p comes through as nan, and even time shows up as nonsense floats (e.g. 7.27e-321, 4.67e-310, 1.78e+161).

In this case time is set by value, not via an external pointer, so I wouldn’t expect it to be affected. But since the whole gathered sub-node looks bad, I suspect something goes wrong during assembly.

I also checked v3.1.0 and current develop, and the relevant code seems unchanged, so I don’t think this is already fixed, though I could be mistaken.

Where I think the problem might be

In api/comm.cpp, function gather_and_process_payloads. The local gathered_payloads is declared at line 381:

conduit::Node gathered_payloads;                       // line 381

try
{
    // Use Conduit's MPI relay to pull everything into an
    // anonymous sequence on Rank 0
    conduit::relay::mpi::gather_using_schema(
        local_payload, gathered_payloads, 0, session.mpi.comm);   // lines 388-389
}
catch (const conduit::Error& e)
{
    ...
}

and then, for a complex schema, the nesting happens at lines 462–468:

// If layout is a complex schema, map using zero-copy set_external handles
for (int i = 0; i < session.mpi.comm_size; ++i)
{
    std::string path_key = "r" + std::to_string(i);
    payload_to_send[path_key].set_external(gathered_payloads.child(i));
}

return Result<void>::ok();

My reading is that gathered_payloads is a local variable, and set_external doesn’t copy, it just points into it. So when the function returns and gathered_payloads goes out of scope, payload_to_send would hold pointers to freed memory, which is then serialized later in send_user_node. That would explain both the garbage values and why even the by-value time inside each rank’s sub-node is affected.

For contrast, the single-rank branch (lines 375–379) uses set_external too, but there local_payload stays alive for the duration of the call which matches the fact that single-rank works:

// If single ranked, then simply send that
if (session.mpi.comm_size == 1)
{
    payload_to_send.set_external(local_payload);
    return Result<void>::ok();
}

I’m not fully confident this is the actual cause though. Maybe set_external behaves differently than I assume, or the issue is elsewhere. Happy to be corrected.

How to reproduce

  • Client sends a conduit node with multiple leaves per timestep (mix of by-value scalars and external-pointer arrays).
  • Run the client with >1 MPI rank.
  • On the server, read the nested node["r0"]/... → nan/garbage. Same code with a single rank → correct.
2

Thanks for cathing the bug!

It was indeed corrupted because of gathered_payload being a local variable.

The fix is merged in the develop.