Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relax Size Check #168

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,10 @@ The data communicated are presented as key-value
index and the value might be the according `float` gradient.
1. Basic synchronization functions: \ref ps::KVWorker::Push, \ref
ps::KVWorker::Pull, and \ref ps::KVWorker::Wait
2. Dynamic length value push and pull: \ref ps::KVWorker::VPush and \ref
ps::KVWorker::VPull
3. Zero-copy versions: \ref ps::KVWorker::ZPush, \ref
ps::KVWorker::ZPull, \ref ps::KVWorker::ZVPush and \ref
ps::KVWorker::ZVPull
2. Zero-copy versions: \ref ps::KVWorker::ZPush, \ref
ps::KVWorker::ZPull

To support dynamic length, pull operations(`Pull` and `ZPull`), do not require the buffer(`vals`) to be the same size as the total data size of pulling down. Larger buffer is allowed while `lens` records the actual size of each key. So the reliable way to read a valid message is to read `lens` bytes. If you ensure that the data size of a key does not change during push or pull, you can verify it by checking whether `lens` of the key is equal to the fixed size.

often server *i* handles the keys (feature indices) within the i-th
segment of <em>[0, uint64_max]</em>. The server node allows user-defined handles to
Expand Down
14 changes: 13 additions & 1 deletion include/ps/kv_app.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ struct KVPairs {
* \brief A worker node that can \ref Push (\ref Pull) key-value pairs to (from) server
* nodes
*
* 1. Basic synchronization functions: \ref ps::KVWorker::Push,
* \ref ps::KVWorker::Pull, and \ref ps::KVWorker::Wait
*
* 2. Zero-copy versions: \ref ps::KVWorker::ZPush,
* \ref ps::KVWorker::ZPull
*
* \tparam Val the type of value, which should be primitive types such as
* int32_t and float
*/
Expand Down Expand Up @@ -140,6 +146,12 @@ class KVWorker : public SimpleApp {
* It's a non-blocking call. The actual pulling is finished,
* namely \a vals (and \a lens) is filled with pulled values, only
* if \ref Wait returns or the callback is called.
*
* Note that \a vals can be larger than the length of the pulled values.
* \a lens is the actual length of the pulled values. The reliable way to
* read a valid message is to read \a lens bytes. If you ensure that the
* data size of a key does not change during push or pull, you can verify
* it by checking whether \a lens of the key is equal to the fixed size.
*
* @param keys a list of keys, must be unique and sorted in increasing order
* @param vals the buffer for the pulled values. It can be 0 size.
Expand Down Expand Up @@ -686,7 +698,7 @@ int KVWorker<Val>::AddPullCB(
if (vals->empty()) {
vals->resize(total_val);
} else {
CHECK_EQ(vals->size(), total_val);
CHECK_GE(vals->size(), total_val);
}
Val* p_vals = vals->data();
int *p_lens = nullptr;
Expand Down