![]() Success! So basically it was copy over deflate as a template for snappy, reduce block byte by 4, then make sure to resize the byte array to block byte size before getting Ironsnappy to decompress. I reduced it to read size and did the minus 4 size thing again. snappyunittests contains unit tests, verifying correctness on your machine in various scenarios. I found its because Apache.Avro always throws the deflate codec a 4096 byte array after the first block. snappybenchmark contains microbenchmarks used to tune compression and decompression performance. (which was basically whatever block Apache.Avro took minus 4 bytes which I assume is the Snappy CRC check which I ignored. I modified the codec portion of my Apache.Avro snappy codec code to make it work with 1 block. ![]() I found my Snappy block and was able to decompress it with IronSnappy. So I ran a loop on the byte start index and end index. While also tracking the index of bytes and such with the hex editor and the debugger in C#. I used that as my base and followed the flow of Apache.Avro in the debugger. Then I used that cat my avro into 1 row, uncompressed and snappy. To make it easier to use, I made it an executable with launch4j totally superfluous move but whatever. I found the avro-tools.jar tool inside Apache.Avro. ![]() I think Apache.Avro takes a long or a varint somehow, and the hex editor I used doesn't give me that info. ![]() Somehow the binary in the file gives that info somehow, but I still have no idea, I couldn't get it with a hex editor even. Well, I have no idea where a block starts and ends. I research Avro and see that it is seperated by an uncompressed Schema, followed by the name of the compression codec of the data, then the data itself, which are divided into blocks.Since Parquet.Net uses IronSnappy, I built/added out the snappy codec in Apache.Avro by basically cloning its deflate code and changing a few names.But I can get the schema as that is uncompressed apparently. I tried Apache.Avro and Confluent version as well, but their. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |