Question on PacketArray expand behavior on a specific example

[tloubrieu-jpl]

Hello,

I wanted to start using the "expand" option on PacketArray instead of a fixed shape as I used to do for my dataset but I am getting a different behavior in both case where I am seeing 34 more bytes of data using the expand option and I don't understand where they are coming from.

The following bits after the PacketArray are used the same way in both case (expand or fixed size).

I am wondering if I am doing something wrong or missing a behavior here or if there is actually something weird happening with the expand option.

I am attaching the test data file and the code.

import ccsdspy

fgx = ccsdspy.VariableLength([
        ccsdspy.PacketField(
            name="Instrument SCLK Time second", bit_length=32, data_type="uint"
        ),
        ccsdspy.PacketField(
            name="Instrument SCLK Time subsec", bit_length=16, data_type="uint"
        ),
        ccsdspy.PacketField(name="Accountability ID", bit_length=32, data_type="uint"),
        # option 1: let ccsdspy guess the size, we get an array of size 1474
        # len(parsed_apids["FGX_CHANNELS"][0]) --> 1474
        #ccsdspy.PacketArray(
        #    name="FGX_CHANNELS", data_type="uint", bit_length=8, array_shape="expand",
        #),
        # option 2: explicitely read expected size 1440
        # len(parsed_apids["FGX_CHANNELS"][0]) --> 1440
        ccsdspy.PacketArray(
            name="FGX_CHANNELS", data_type="uint", bit_length=8, array_shape=1440,
        ),

        # the fields following, in both option 1 and 2 are properly filled
        # option 1
        #array([-163418, -163417, -163417, -163417, -163415, -163415, -163418,
        #       -163417, -163418, -163418, -163417, -163418, -163418, -163419,
        #       -163416, -163416, -163415, -163418, -163415, -163417, -163417,
        #       -163416], dtype='>i4')
        # option 2:
        #array([-163418, -163417, -163417, -163417, -163415, -163415, -163418,
        #       -163417, -163418, -163418, -163417, -163418, -163418, -163419,
        #       -163416, -163416, -163415, -163418, -163415, -163417, -163417,
        #       -163416], dtype='>i4')
        ccsdspy.PacketField(name="FGx_-4.7VHK", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_+4.7VHK", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_2VREF", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_1VREF", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_DRV_SNS", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_OP_PRTA", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_FBX", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_FBY", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_FBZ", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_BPFX", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_BPFY", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_BPFZ", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_+4.7_I", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_-4.7_I", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_HK_CH14", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="FGx_HK_CH15", bit_length=24, data_type="int"),
        ccsdspy.PacketField(name="Register 80", bit_length=16, data_type="uint"),
        ccsdspy.PacketField(name="PEC (CRC-16-CCITT)", bit_length=16, data_type="uint"),
    ]
)


pkt_defs = {
    1219: fgx,
    1223: fgx,
    1227: fgx
}

def parse_file():
    stream_by_apid = ccsdspy.utils.split_by_apid("./ecm_test.bin")
    for apid, streams in stream_by_apid.items():
        print(f"Parse APID {apid}")
        if apid in pkt_defs:
            pkt_def = pkt_defs[apid]
            parsed_apids = pkt_def.load(streams, include_primary_header=True)

if __name__ == '__main__':
    parse_file()

The use test dataset is there:
ecm_test.bin.zip

[tloubrieu-jpl]

A quick update, I tried to read the number of entries given by the expand option using the fixed length option, as follow:

ccsdspy.PacketArray(
            name="FGX_CHANNELS", data_type="uint", bit_length=8, array_shape=1474,
        ),

And it generates an error related to the size of the packet not matching what is expected.

[ddasilva]

Hi @tloubrieu-jpl,
Thanks for sending your question along. I'll try to look at it soon and get a response to you. What's your timeline for needing a response?

[tloubrieu-jpl]

Thanks @ddasilva , I am not blocked since I can work on something else before finding a solution for that.

I did not mention it previously but the expand option is really what we need here since for this type of packet there are 3 different sizes for the packet array and not an easy way to tell which size applies to which packet.

That would be great if you can test and provide me feedback within 2 weeks from now.

Thanks again,

Thomas

[ddasilva]

Hi @tloubrieu-jpl . Thanks for reporting this! There was a minor bug which arose from a typo in a list comprehension. This caused the expanding field to calculate the incorrect length, but only when there are fields following it. This isn't usually the case (most expanding fields come at the end), so we didn't notice.

Can you try the branch fix-expand-footer-bits and let me know if this fixed your issue? If it does, I'll open a merge request.

[tloubrieu-jpl]

Thanks @ddasilva , I will try that tomorrow morning.

[tloubrieu-jpl]

Thanks @ddasilva , I was able to make expand work as expected with your branch.

[ddasilva]

Great, it's been merged. Can you work off main for now, or do you need a new release?

[tloubrieu-jpl]

Thanks Daniel, I can work from main. Do you plan to have a release soon ? I can tell you later when we would need a stable version.

[ddasilva]

I have one other issue to look at from someone at Canadian Space Agency (#101)... if that requires a bug fix then I'll wait and bundle them together in the same revision release (1.2.1). If it doesn't I can publish this one sooner.

[tloubrieu-jpl]

Hi @ddasilva , we have a release next February, so we'll need a stable release working for us then. I am now integrating the code the Nischay wrote last summer into our official Science Data System. That should take me overall one of 2 more months. Let's say, that should be done by the end of the calendar year.

[ddasilva]

That's absolutely reasonable, @tloubrieu-jpl. I'm still working through this issue in #101. If you don't hear from me by mid December, feel free to ping me here or send me an email.

[ddasilva]

Hi @tloubrieu-jpl, the new update (v1.2.1) has been pushed to PyPI.

[tloubrieu-jpl]

Thanks @ddasilva !