2012-08-22 14:37:32 +00:00
|
|
|
PACKING ALGORITHM
|
|
|
|
|
=================
|
|
|
|
|
|
|
|
|
|
Begin with a list of content set paths:
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
/content/dist/rhel/$releasever/$basearch/os
|
|
|
|
|
/content/dist/rhel/$releasever/$basearch/debug
|
|
|
|
|
/content/dist/rhel/$releasever/$basearch/source/SRPMS
|
|
|
|
|
/content/dist/jboss/source
|
|
|
|
|
/content/beta/rhel/$releasever/$basearch/os
|
|
|
|
|
/content/beta/rhel/$releasever/$basearch/debug
|
|
|
|
|
/content/beta/rhel/$releasever/$basearch/source/SRPMS
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
|
|
|
|
|
Create a prefix tree out of the paths, where each node in the tree contains the
|
|
|
|
|
names of its children, rather than the children containing its own name:
|
|
|
|
|
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
+-------+ +------+
|
|
|
|
|
| | | |
|
|
|
|
|
|-------+ |------|
|
|
|
|
|
|content+--+-------+ |source+-+---+
|
|
|
|
|
+-------+ | | +------+ | |
|
|
|
|
|
|-------| | |---|
|
|
|
|
|
|dist +--------+-------+ | +---+
|
|
|
|
|
+--+beta | | | |
|
|
|
|
|
| +-------+ |-------| |
|
|
|
|
|
| |jboss +--+
|
|
|
|
|
+-------+ |rhel +--+
|
|
|
|
|
| | +-------+ |
|
|
|
|
|
|-------| |
|
|
|
|
|
|rhel +--+-----------+ +-----------+
|
|
|
|
|
+-------+ | | | |
|
|
|
|
|
|-----------| |-----------|
|
|
|
|
|
+----------+$releasever| |$releasever+--+
|
|
|
|
|
| +-----------+ +-----------+ |
|
|
|
|
|
| |
|
|
|
|
|
+---------+ +---------+
|
|
|
|
|
| | | |
|
|
|
|
|
|---------| |---------|
|
|
|
|
|
|$basearch+--+ +--+$basearch| +---+
|
|
|
|
|
+---------+ | | +---------+ | |
|
|
|
|
|
| | |---|
|
|
|
|
|
+--------+ +-------+ +---+
|
|
|
|
|
| | | | |
|
|
|
|
|
|--------| |-------| |
|
|
|
|
|
+---+--------+os | |os +---------+
|
|
|
|
|
| | +--+debug | |debug +-----------+---+
|
|
|
|
|
|---| | |source +--+ |source +--+ | |
|
|
|
|
|
+---+ | +--------+ | +-------+ | |---|
|
|
|
|
|
| | | +---+
|
|
|
|
|
+---+ +------+ |
|
|
|
|
|
| | | | +-----+
|
|
|
|
|
|---| |------| | |
|
|
|
|
|
+---+ +--+SRPMS | |-----|
|
|
|
|
|
| +------+ |SRPMS+--+---+
|
|
|
|
|
| +-----+ | |
|
|
|
|
|
+---+ |---|
|
|
|
|
|
| | +---+
|
|
|
|
|
|---|
|
|
|
|
|
+---+
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
|
|
|
|
|
This eliminates the duplication in the prefixes of our paths. We now only have
|
|
|
|
|
one instance of the word "content", for example. We now find any duplicate
|
|
|
|
|
subtrees, remove the duplicate, and point all references from the duplicate to
|
|
|
|
|
the original:
|
|
|
|
|
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
+-------+ +------+
|
|
|
|
|
| | | |
|
|
|
|
|
|-------+ |------|
|
|
|
|
|
|content+--+-------+ |source+-+
|
|
|
|
|
+-------+ | | +------+ |
|
|
|
|
|
|-------| | |
|
|
|
|
|
|dist +--------+-------+ | |
|
|
|
|
|
+--+beta | | | | |
|
|
|
|
|
| +-------+ |-------| | |
|
|
|
|
|
| |jboss +--+ |
|
2012-08-22 14:50:53 +00:00
|
|
|
+-------+ +--+rhel | |
|
|
|
|
|
| | | +-------+ |
|
|
|
|
|
|-------| | |
|
|
|
|
|
|rhel +--+-----------+-+ |
|
2012-08-22 14:37:32 +00:00
|
|
|
+-------+ | | |
|
|
|
|
|
|-----------| |
|
|
|
|
|
+----------+$releasever| |
|
|
|
|
|
| +-----------+ |
|
|
|
|
|
| |
|
|
|
|
|
+---------+ |
|
|
|
|
|
| | |
|
|
|
|
|
|---------| |
|
|
|
|
|
|$basearch+--+ |
|
|
|
|
|
+---------+ | |
|
|
|
|
|
| |
|
|
|
|
|
+--------+ |
|
|
|
|
|
| | |
|
|
|
|
|
|--------| |
|
|
|
|
|
+-------+os | |
|
|
|
|
|
| +--+debug | |
|
|
|
|
|
| | |source +--+ |
|
|
|
|
|
| | +--------+ | |
|
|
|
|
|
| | | |
|
|
|
|
|
| | +------+ |
|
|
|
|
|
| +-------+ | | |
|
|
|
|
|
| | |------| |
|
|
|
|
|
| | +--+SRPMS | |
|
|
|
|
|
| | | +------+ |
|
|
|
|
|
| | | |
|
|
|
|
|
+------------+---+-------------------------+
|
|
|
|
|
| |
|
|
|
|
|
|---|
|
|
|
|
|
+---+
|
2012-08-22 14:38:33 +00:00
|
|
|
```
|
2012-08-22 14:37:32 +00:00
|
|
|
|
2012-08-22 14:50:53 +00:00
|
|
|
With this structure, we can begin creating the packed data. The first step is
|
|
|
|
|
to build a huffman coding for the string components of the path. In the
|
|
|
|
|
example, all strings are used only once, except for rhel and source. We create
|
|
|
|
|
a list of all the strings, ordered from least to most occurance. This list is
|
|
|
|
|
the one written out in the path dictionary section of the binary packing. The
|
|
|
|
|
ordering used in the list is what we then feed into a huffman tree for paths.
|
|
|
|
|
thus, even though both os and debug occur just once in the above DAG, depending
|
|
|
|
|
on the ordering in the list, one will be assigned a higher weight than the
|
|
|
|
|
other (which helps the other side decode the binary format). The string huffman
|
|
|
|
|
tree also includes a special sentinal value to indicate end of node in the
|
|
|
|
|
binary format. This value should not be written to the binary packing, should
|
|
|
|
|
be given the highest weight, and should be some string that is not used in the
|
|
|
|
|
path strings themselves (to avoid collision).
|
|
|
|
|
|
|
|
|
|
Next, we order the nodes from the above DAG by order of reference from other
|
|
|
|
|
nodes, from least to most references. This ensures the root of the DAG is
|
|
|
|
|
always first, as it has no references. Similar to the strings, we create a
|
2012-09-05 15:40:18 +00:00
|
|
|
huffman tree for the nodes. The root node should not be included in the huffman
|
|
|
|
|
tree, as it is never referenced by any other nodes, so we don't need to use up
|
|
|
|
|
any potential address space on it.
|
2012-08-22 14:50:53 +00:00
|
|
|
|
|
|
|
|
We can then iterate over the node list, writing each one out to the binary
|
|
|
|
|
packing. Within each node, for each string and node pair that it references, we
|
|
|
|
|
look up each in their respective huffman trees, and write the huffman coding to
|
|
|
|
|
the binary packing.
|
|
|
|
|
|
|
|
|
|
At the end of a node, we use the special sentinal value from the string huffman
|
|
|
|
|
tree to indicate end of node.
|
