Dogfooding at Hedera: Trust Bucket for verifiable object storage with Amazon S3 and Hedera Consensus Service
Hedera just wrapped up its Hedera20 virtual hackathon — with over 800 competing developers across dozens of countries, it’s been the largest and most widespread Hedera developer event ever. You can read more about the event, criteria, prizes, and our winners in this blog posting recap.
As part of this event, employees were encouraged to participate by building a unique application, developer tool, or service utilizing the Hedera Consensus Service (HCS). It also happens that we had just spun up a #dogfooding Slack channel in our team chat — if you’re not familiar with the term, “dog fooding” or “eating your own dog food” is slang for using the products or services that you sell.
The Marketing team at Hedera regularly makes important (and sometimes sensitive) PDFs available on Hedera.com for transparency with the community — things like the minutes from Hedera Governing Council meetings, a detailed look at Hedera’s coin economics, the Governing Council LLC agreement, Hedera Whitepaper updates, and more. We thought about our core values in verifiability, transparency, and trust for the web and asked ourselves how we could apply those values in practice.
This is when we decided to bring verifiability and transparency to the PDFs on Hedera.com and we came up with Trust Bucket: a hackathon submission for Hedera20 that offers verifiable object storage for Amazon S3 using the Hedera Consensus Service.
PDFs on Hedera.com are now verifiable
Using Trust Bucket, we’ve hashed files listed on the Council, Council Meeting Minutes, and Papers pages of hedera.com and notarised them using the Hedera Consensus Service. Thus, file-hashes and timestamps are now recorded on a Hedera Hashgraph owned account and HCS Topic giving our readers the assurance that files have not been tampered after these have been put in the public domain.
What Trust Bucket does is rather simple: whenever a Hedera Hashgraph team member uploads a file into a tracked Amazon S3 bucket an automated process hashes the file and records it with HCS.
Each file-link listed on hedera.com is now published along with a button that will help you verify the file. We’ll use the Meeting Minutes page as an example of how to verify the authenticity of a PDF. From the Meeting Minutes page, click or tap on “Read” to display and download the meeting minutes PDF from the council meeting that took place on February 26, 2020.
Next, you’ll need to generate a SHA384 cryptographic hash of this meeting minutes file. There’s an easy-to-use SHA384 hash generator found here. Upload the PDF and click “Convert File”. You’ll receive a SHA384 hash that you can then compare with the hash that has been generated by Trust Bucket and notarized with HCS.
Cryptographic hashes take an input of data (in this case, a PDF file), and use an algorithm to produce an output of fixed length for that data. It’s nearly impossible to find two distinct inputs that result in the same output (otherwise known as “collision resistant”) — this is one of the most critical components of verification, because if anyone ever changed the contents of the file and tried to pass it off as authentic, the SHA384 hash that’s generated also be different.
Once the conversion completes, you’ll receive your output as displayed below. The output we’ll use is “hex”.
Go back to the Council Meeting Minutes page and click or tap HCS TXN INFO for the same February 26, 2020 file to see all of the information recorded by Trust Bucket (using the Hedera Consensus Service).
Compare the SHA384 hash on that screen to the SHA384 hash you received in the instructions above — they should match! You will also see the associated transaction ID, consensus timestamp, topic ID, sequence #, running hash, and link to network explorers.
Back on the Council Meeting Minutes page, you’ll see a button labeled “VERIFY ON EXPLORERS” (this link is the same one found in the HCS TXN INFO, in the very last row). This tool aggregates transactions across multiple network explorers (today this includes Kabuto, DragonGlass, and Hedera); this ensures a strong confidence level in the information and is more decentralized than only verifying on one network explorer.
Follow that link to verify that all of the transaction information matches, including the Hedera-owned account ID 0.0.44599 from where the transaction was submitted.
If everything matches, you’ve successfully verified the authenticity of a file.
How does Trust Bucket work and is it open source?
Trust Bucket is an open source tool that anyone can download, fork, and use for free with their website or application. You can check out the details and find implementation instructions here: https://github.com/hashgraph/hedera-hcs-s3-monitor-java
Trust Bucket uses AWS Lambda to execute actions and allocate resources in response to events that happen when using Trust Bucket, such as the upload of a PDF file. When a file is added to a S3 bucket, Trust Bucket takes the following actions:
- A file hash is generated.
- The hash is sent to HCS.
- HCS responds with a record containing information such as the timestamp and the sequence number.
- TrustBucket then creates log-files in the same bucket containing the file-name, file-hash, and timestamp among other file information.
After uploading a PDF (we’ll use abc.pdf as an example) the log folder will contain a text file named abc.pdf.hcs.txt. You can access that log file via the object url https://[bucket-name].s3.[region].amazonaws.com/tracked-docs-log/abc.pdf.hcs.txt. You’ll also find a file named abc.pdf.hcs.json and one named abc.pdf.hcs.html. The latter is a HTML formatted output of the same data.
Who built & implemented Trust Bucket?
The creators of Trust Bucket are Nik Siafakas, Software Engineer based in London, UK, and Greg Scullard, Lead Developer Advocate at Hedera Hashgraph, based in Saint-Amand-Montrond, France. The implementation on Hedera.com was completed by Mark Williamson, Senior Systems Engineer, and Paul Bugeja, Web Production, both at Hedera Hashgraph.
We're looking for curious and passionate individuals who are dedicated to learning and growing with us. We’re committed to building an inclusive team that values ideas over anything else. If you enjoyed reading about this project and are inspired to help us decentralized the web, please check out our open roles: https://www.hedera.com/future