Miscellaneous

Special

Renamed

Adds a name string to a field (which by default is None). This class is only used internally and you should use the / and * operators instead. Naming fields is needed when working with Struct and Union, but also sometimes with Sequence and FocusedSeq.

"num" / Byte       <-->  Renamed(Byte, newname="num")
Byte * "comment"   <-->  Renamed(Byte, newdocs="comment")
Byte * parsedhook  <-->  Renamed(byte, newparsed=parsedhook)

Miscellaneous

Const

A constant value that is required to exist in the data and match a given value. If the value is not matching, ConstError is raised. Useful for so called magic numbers, signatures, asserting correct protocol version, etc.

>>> d = Const(b"IHDR")
>>> d.build(None)
b'IHDR'
>>> d.parse(b"JPEG")
construct.core.ConstError: expected b'IHDR' but parsed b'JPEG'

By default, Const uses a Bytes field with size matching the value. However, other fields can also be used:

>>> d = Const(255, Int32ul)
>>> d.build(None)
b'\xff\x00\x00\x00'

The shortcoming is that it only works if the amount and exact bytes are known in advance. To check if a “variable” data meets some criterium (not mere equality), you would need the Check class. There is also OneOf and NoneOf class.

Computed

Represents a value dynamically computed from the context. Computed does not read or write anything to the stream. It only computes a value (usually by extracting a key from a context dictionary) and returns its computed value as the result. Usually Computed fields are used for computations on the context dict. Context is explained in a previous chapter. However, Computed can also produce values based on external environment, random module, or constants. For example:

>>> d = Struct(
...     "width" / Byte,
...     "height" / Byte,
...     "total" / Computed(this.width * this.height),
... )
>>> d.parse(b"12")
Container(width=49, height=50, total=2450)
>>> d.build(dict(width=4, height=5))
b'\x04\x05'
>>> d = Computed(lambda ctx: os.urandom(10))
>>> d.parse(b"")
b'[\x86\xcc\xf1b\xd9\x10\x0f?\x1a'

Index

Fields that are inside Array, GreedyRange or RepeatUntil can reference their index within the outer list. This is being effectuated by repeater class maintaining a context entry _index and updating it between each iteration. Note that some classes do context nesting (like Struct), but they do copy the key over. You can access the key using Index class, or refer to the context entry directly, using this._index expression. Some constructions are only possible with direct method, when you want to use the index as parameter of a construct, like in Bytes(this._index + 1).

>>> d = Array(3, Index)
>>> d.parse(b"")
ListContainer([0, 1, 2])
>>> d = Array(3, Struct("i" / Index))
>>> d.parse(b"")
ListContainer([Container(i=0), Container(i=1), Container(i=2)])
>>> d = Array(3, Computed(this._index + 1))
>>> d.parse(b"")
ListContainer([1, 2, 3])
>>> d = Array(3, Struct("i" / Computed(this._index + 1)))
>>> d.parse(b"")
ListContainer([Container(i=1), Container(i=2), Container(i=3)])

Rebuild

When there is an array separated from its length field, the Rebuild wrapper can be used to measure the length of the list when building. Note that both the len_ and this expressions are used as discussed in meta chapter. Only building is affected, parsing is simply deferred to subcon.

>>> d = Struct(
...     "count" / Rebuild(Byte, len_(this.items)),
...     "items" / Byte[this.count],
... )
>>> d.build(dict(items=[1,2,3]))
b'\x03\x01\x02\x03'

When the count field is directly before the items, PrefixedArray can be used instead:

>>> d = PrefixedArray(Byte, Byte)
>>> d.build([1,2,3])
b'\x03\x01\x02\x03'

Default

Allows to make a field have a default value, which comes handly when building a Struct from a dict with missing keys. Only building is affected, parsing is simply deferred to subcon.

>>> d = Struct(
...     "a" / Default(Byte, 0),
... )
>>> d.build(dict(a=1))
b'\x01'
>>> d.build(dict())
b'\x00'

Check

When fields are expected to be coherent in some way but integrity cannot be checked by merely comparing data with constant bytes using Const field, then a Check field can be put in place to get a key from context dict and check if the integrity is preserved. For example, maybe there is a count field (implied being non-negative but the field is signed type):

>>> d = Struct(
...     "num" / Int8sb,
...     "integrity1" / Check(this.num > 0),
... )
>>> d.parse(b"\xff")
CheckError: Error in path (parsing) -> integrity1
check failed during parsing

Or there is a collection and a count provided and the count is expected to match the collection length (which might go out of sync by mistake). Note that Rebuild is more appropriate but the check is also possible:

>>> d = Struct(
...     "count" / Byte,
...     "items" / Byte[this.count],
... )
>>> st.build(dict(count=9090, items=[]))
FormatFieldError: Error in path (building) -> count
struct '>B' error during building, given value 9090
>>> d = Struct(
...     "integrity" / Check(this.count == len_(this.items)),
...     "count" / Byte,
...     "items" / Byte[this.count],
... )
>>> d.build(dict(count=9090, items=[]))
CheckError: Error in path (building) -> integrity
check failed during building

Error

You can also explicitly raise an error, declaratively with a construct.

>>> Error.parse(b"")
ExplicitError: Error in path (parsing)
Error field was activated during parsing

FocusedSeq

When a sequence has some fields that could be ommited like Const, Padding or Terminated, the user can focus on one particular field that is useful. Only one field can be focused on, and can be referred by index or name. Other fields must be able to build without a value:

>>> d = FocusedSeq(1 or "num",
...     Const(b"MZ"),
...     "num" / Byte,
...     Terminated,
... )
>>> d.parse(b"MZ\xff")
255
>>> d.build(255)
b'MZ\xff'

Pickled

For convenience, arbitrary Python objects can be preserved using the famous pickle protocol. Almost any type can be pickled, but you have to understand that pickle uses its own (homebrew) protocol that is not a standard outside Python. Therefore, you can forget about parsing the binary blobs using other languages. Its useful, but it automates things beyond your understanding.

>>> obj = [1, 2.3, {}]
>>> Pickled.build(obj)
b'\x80\x03]q\x00(K\x01G@\x02ffffff}q\x01e.'
>>> Pickled.parse(_)
[1, 2.3, {}]

Numpy

Numpy arrays can be preserved and retrived along with their element type (dtype), dimensions (shape) and items. This is effectuated using the Numpy binary protocol, so parsing blobs produced by this class with other langagues (or other frameworks than Numpy for that matter) is not possible. Otherwise you could use PrefixedArray but this class is more convenient.

>>> import numpy
>>> obj = numpy.asarray([1,2,3])
>>> Numpy.build(obj)
b"\x93NUMPY\x01\x00F\x00{'descr': '<i8', 'fortran_order': False, 'shape': (3,), }            \n\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00"

NamedTuple

Both arrays, structs and sequences can be mapped to a namedtuple from collections module. To create a named tuple, you need to provide a name and a sequence of fields, either a string with space-separated names or a list of strings. Just like the stadard namedtuple does.

>>> d = NamedTuple("coord", "x y z", Byte[3])
>>> d = NamedTuple("coord", "x y z", Byte >> Byte >> Byte)
>>> d = NamedTuple("coord", "x y z", "x"/Byte + "y"/Byte + "z"/Byte)
>>> d.parse(b"123")
coord(x=49, y=50, z=51)

Timestamp

Datetimes can be parsed using Timestamp class. It supports modern formats and even MSDOS one. Note however that this class is not guaranteed to provide “exact” accurate values, due to several reasons explained in the docstring.

>>> d = Timestamp(Int64ub, 1., 1970)
>>> d.parse(b'\x00\x00\x00\x00ZIz\x00')
<Arrow [2018-01-01T00:00:00+00:00]>
>>> d = Timestamp(Int32ub, "msdos", "msdos")
>>> d.parse(b'H9\x8c"')
<Arrow [2016-01-25T17:33:04+00:00]>

Hex and HexDump

Integers and bytes can be displayed in hex form, for convenience. Note that parsing still results in int-alike and bytes-alike objects, and those results are unmodified, the hex form appears only when pretty-printing. If you want to obtain hexlified bytes, you need to use binascii.hexlify() on parsed results.

>>> d = Hex(Int32ub)
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
258
>>> print(obj)
0x00000102
>>> d = Hex(GreedyBytes)
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
b'\x00\x00\x01\x02'
>>> print(obj)
unhexlify('00000102')
>>> d = Hex(RawCopy(Int32ub))
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
{'data': b'\x00\x00\x01\x02',
 'length': 4,
 'offset1': 0,
 'offset2': 4,
 'value': 258}
>>> print(obj)
unhexlify('00000102')

Another variant is hexdumping, which shows both ascii representaion, hexadecimal representation, and offsets. Functionality is identical.

>>> d = HexDump(GreedyBytes)
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
b'\x00\x00\x01\x02'
>>> print(obj)
hexundump('''
0000   00 00 01 02                                       ....
''')
>>> d = HexDump(RawCopy(Int32ub))
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
{'data': b'\x00\x00\x01\x02',
 'length': 4,
 'offset1': 0,
 'offset2': 4,
 'value': 258}
>>> print(obj)
hexundump('''
0000   00 00 01 02                                       ....
''')

Warning

Note that Hex and possibly HexDump do not work correctly within a Bitwise context.

Conditional

Union

Treats the same data as multiple constructs (similar to C union statement) so you can “look” at the data in multiple views.

When parsing, all fields read the same data bytes, but stream remains at initial offset (or rather seeks back to original position after each subcon was parsed), unless parsefrom selects a subcon by index or name. When building, the first subcon that can find an entry in the dict (or builds from None, so it does not require an entry) is automatically selected.

Warning

If you skip parsefrom parameter then stream will be left back at starting offset, not seeked to any common denominator.

>>> d = Union(0,
...     "raw" / Bytes(8),
...     "ints" / Int32ub[2],
...     "shorts" / Int16ub[4],
...     "chars" / Byte[8],
... )
>>> d = Union(0, # alternative syntax
...     raw=Bytes(8),
...     ints=Int32ub[2],
...     shorts=Int16ub[4],
...     chars=Byte[8],
... )
>>> d.parse(b"12345678")
Container:
    raw = b'12345678' (total 8)
    ints = ListContainer:
        825373492
        892745528
    shorts = ListContainer:
        12594
        13108
        13622
        14136
    chars = ListContainer:
        49
        50
        51
        52
        53
        54
        55
        56
>>> d.build(dict(chars=range(8)))
b'\x00\x01\x02\x03\x04\x05\x06\x07'

Select

Attempts to parse or build each of the subcons, in order they were provided.

>>> d = Select(Int32ub, CString("utf8"))
>>> d = Select(num=Int32ub, text=CString("utf8")) # alternative syntax
>>> d.build(1)
b'\x00\x00\x00\x01'
>>> d.build("Афон")
b'\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd\x00'

Optional

Attempts to parse or build the subconstruct. If it fails during parsing, returns a None. If it fails during building, it puts nothing into the stream.

>>> d = Optional(Int64ul)
>>> d.parse(b"12345678")
4050765991979987505
>>> d.parse(b"")
None
>>> d.build(1)
b'\x01\x00\x00\x00\x00\x00\x00\x00'
>>> d.build(None)
b''

If

Parses or builds the subconstruct only if a certain condition is met. Otherwise, returns a None when parsing and puts nothing into the stream when building. The condition is a lambda that computes on the context just like in Computed examples.

>>> d = If(this.x > 0, Byte)
>>> d.build(255, x=1)
b'\xff'
>>> d.build(255, x=0)
b''

IfThenElse

Branches the construction path based on a given condition. If the condition is met, the thensubcon is used, otherwise the elsesubcon is used. Fields like Pass and Error can be used here. Just for your curiosity, If is just a macro around this class.

>>> d = IfThenElse(this.x > 0, VarInt, Byte)
>>> d.build(255, x=1)
b'\xff\x01'
>>> d.build(255, x=0)
b'\xff'

In particular, you can use different subcons for parsing and building. The _parsing, _building and _sizing context entries have boolean values that always exist, only one of them that corresponds to current action is set to True. For convenience, those two entries are duplicated in Struct, Sequence, FocusedSeq and Union nested contexts. You dont need to reach for the top-most entry. This comes handy when using hackish constructs to achieve some complex semantics that are not available in the core library.

d = Struct(
    If(this._parsing, ...),
    If(this._building, ...),
)

Switch

Branches the construction based on a return value from a context function. This is a more general implementation than IfThenElse. If no cases match the actual, it just passes successfully, although that behavior can be overriden.

>>> d = Switch(this.n, {1: Int8ub, 2: Int16ub, 4: Int32ub})
>>> d.build(5, n=1)
b'\x05'
>>> d.build(5, n=4)
b'\x00\x00\x00\x05'
>>> d = Switch(this.n, {}, default=Byte)
>>> d.parse(b"\x01", n=255)
1
>>> d.build(1, n=255)
b'\x01'

StopIf

Checks for a condition after each element, and stops a Struct, Sequence or GreedyRange from parsing or building further.

Struct('x'/Byte, StopIf(this.x == 0), 'y'/Byte)
Sequence('x'/Byte, StopIf(this.x == 0), 'y'/Byte)
GreedyRange(FocusedSeq(0, 'x'/Byte, StopIf(this.x == 0)))

Alignment and padding

Padding

Adds additional null bytes (a filler) analog to Padded but without a subcon. This field is usually anonymous inside a Struct. Internally this is just Padded(n, Pass) where n is an amount of null bytes.

>>> d = Padding(4)
>>> d.parse(b"****")
None
>>> d.build(None)
b'\x00\x00\x00\x00'

Padded

Appends additional null bytes after subcon to achieve a fixed length. Note that implementation of this class uses stream.tell() to find how many bytes were written by the subcon.

>>> d = Padded(4, Byte)
>>> d.build(255)
b'\xff\x00\x00\x00'

Similar effect can be obtained using FixedSized, but the implementation is rather different. FixedSized uses a separate BytesIO, which means that Greedy* fields should work properly with it (and fail with Padded) and also the stream does not need to be tellable (like pipes sockets etc).

Aligned

Appends additional null bytes after subcon to achieve a given modulus boundary. This implementation also uses stream.tell().

>>> d = Aligned(4, Int16ub)
>>> d.build(1)
b'\x00\x01\x00\x00'

AlignedStruct

Automatically aligns each member to modulus boundary. It does NOT align entire Struct, but each member separately.

>>> d = AlignedStruct(4, "a"/Int8ub, "b"/Int16ub)
>>> d.build(dict(a=0xFF, b=0xFFFF))
b'\xff\x00\x00\x00\xff\xff\x00\x00'