Skip to content

ETL to QE, Update 34, Failed CGFS Experiments

Date: 2024-05-05 I have tried to develop the core of CGFS at least three times so far each time learning something new.

ddaemon/question-engine

The goal of the original question-engine was to enable anyone with a Ethereum browser wallet to be able to engage in a social media experience where content was permissioned using signature chain tokens.

I had implemented a signature chain mechanism which supported a wide range of token transactions. While writing the transactions types required to manage cryptographic social media I realized all social media is a graph therefore at the core we need a node and edge transaction type.

Stuff leftover from this era include,

The original Question Engine prototype used the sequelize ORM and used Ethereum signatures via ethersjs.

What did I learn from this proof of concept?

After I started doing the frontend work I became overwhelmed and gave up on the project. I had lost track of the core vision and what I could grasp seemed so far away I could not break down the task effectively. One thing I could have done at the time is called up a friend to talk through some of the problems or just describe what I was trying to build... Or do a blog post like I am doing now over a year later.

My React and SQL ORM skills really improved while trying to develop this POC so my time was not completely wasted.

dentropy/nostr-nip05-server-failed-POC

A couple months back I thought to myself that a NIP05 identity server.

For those that don't know NIP05 give a nostr identity, which is a public key, a human readable name attached to a domain name and looks exactly like an email address.

NIP05 is about as simple as it gets. NIP05 literally just requires a JSON file be hosted on a server under the right path with a domain name. The hard part is regulating who has access to edit the file.

So nearly two and a half months ago I started with this goal on 2024-02-25, which followed by scope creep getting the better of me in a way that I hope will eventually pay off.

You see when I started this proof of concept I had not obtained the CGFS - Requirements that I hold today. In fact CGFS was not even in my vocabulary yet. I originally tried using sqlite for the NIP05 server but after thinking through a couple problem I realized that I already needed a browser plus backend compatible database or datastructure.

While I was coding the logic to update NIP05 addresses I realized that the Nostr client I want to build would need the same data structure running in the forntend to be a proper self sovereign nostr client. This is when I decided to scrap sqlite and look for something else.

I had discovered RxDB's mention of JSONSchema and connected this JSONSchema with IPLD's DAG-JSON project. I realized RxDB would not operate on a KV store like S3 without heavy modification therefore I decided to try to create my own NOSQL database like RxDB inside IndexedDB.

Long story short, I fucked trying to build my own database and should have stood on the shoulders of RxDB.

What did I learn from this proof of concept?

Don't go on a side quest to develop your own NOSQL database using a KV store when you are trying to actually ship something on a reasonable time frame.

Test driven development can be great but write your code in more composable chunks that follow the unix philosophy of do one thing and do one thing well. Then test the shit out of those components.

During the development of this POC I realized that I can represent a ERD(Entity Relationship Diagram) using a Graph and easily visually represent it using Cytoscape.js

dentropy/nostr-nip05-server

After failing to develop my own NOSQL database I was now standing on the shoulders of giants via RxDB.

The development of this server went smoothly, I developed a schema system that can load in a series of collections assigning each of them a IPNS address, and setting up the logic to log specific indexes via Content Identifiers.

I am proud of how I wrote the backend code in a modular fashion so it would be extendable with new functions.

I am proud of the tests I wrote. Requiring myself to write tests. Having tests and no proper documentation meant that my tests were the documentation when I became lost.

What did I learn from this proof of concept?

I had not scoped out my ideas on Provenance deeply enough, therefore did not have the correct requirements, and therefore made architectural mistakes whiled developing the application.

The CGFS component of the project should really be developed, documented, and tested separately because I waste so much time trying to coordinate and develop two projects at the same time.

Functions were not thought out properly. When I got to the part where we were doing nostr token transfers to redeem NIP05 identities I realized that the function logic was not functional when it can be.

I also obtained a new value when developing this POC.

Also I should have used bignumber.js in the beginning for the token stuff because I knew javascript numbers maxed out easily. Don't openly make a mistake to get a POC out, if you know better, DO BETTER.